## Context
We receive Recall rate limit alerts on `/api/v1/bot`. Recall's List Bots
endpoint allows only 60 requests/min per Recall workspace (vs 300/min
for Retrieve and 120/min for Create), and that budget is shared by every
Twenty workspace on the instance since `RECALL_API_KEY` is a
server-level variable. The call-recorder recovery crons fanned out one
list call per stuck recording, fired at the same wall-clock minute for
every workspace, and never resolved dead rows, so the pending set only
grew.
This PR reworks the bot-scheduling recovery mechanism so that
crash-recovery work is rare, cheap, and mostly event-driven. One commit
per change:
## Changes
1. **Fail never-scheduled recordings once their meeting ends**
(`bot_never_scheduled` failure reason). Previously these rows stayed
`REQUESTED+SCHEDULED` forever and were re-fetched by every recovery run.
Rows with an unresolved creation attempt keep their recovery chance
until the 7-day convergence lookback passes (a bot may have recorded
before the id write-back was lost), then fail as
`bot_schedule_outcome_unknown`.
2. **Batch bot lookups into one list call per run.** The pending-bot
sweep and the failed-cancellation retry each issue at most one
workspace-wide `GET /api/v1/bot/` (filtered by `twentyWorkspaceId` +
active statuses) and match bots to recordings in memory via
`twentyCallRecordingId` metadata, instead of one list call per stuck
row. Truncated lists count as failed lookups so an incomplete map never
authorizes a duplicate creation.
3. **Record a `botScheduleAttemptedAt` marker before POSTing a bot.**
Recovery can now distinguish rows that never reached Recall (re-schedule
directly, zero Recall reads) from rows whose creation outcome is unknown
(only these join the lookup).
4. **Store the bot-creation `Idempotency-Key` on the row and recover by
re-sending.** When a stuck row's stored key still hashes from the
current scheduling inputs, recovery re-sends the creation: Recall either
returns the existing bot or creates the intended one, all on the Create
budget (120/min) without touching the List budget (60/min). Drifted
inputs still fall back to the lookup. Re-sends preserve the first
attempt's timestamp and are only trusted within a 12-hour window, so
repeated unknown outcomes age into the lookup path rather than risking a
twin bot after Recall's key retention expires.
5. **Resume pending rows on `callRecording.updated` events and slow the
cron.** A new database-event trigger resumes scheduling within seconds
when a row transitions back to pending (bot vanished at Recall, canceled
request re-requested, failed row reset by reconciliation), with queue
retries. It skips creations (the inserting run schedules inline), skips
its own progress writes, uses slim-payload diffs to skip cheaply, and
defers ambiguous rows to the cron so event bursts cannot fan out list
calls. The pending-requests cron becomes a backstop and drops from every
5 minutes to every 15.
Follow-up commits harden edge cases raised in review (status
revalidation before POST, per-row cancellation recovery window,
future-timestamp guard, attempt-state cleanup when a bot is confirmed
gone at Recall) and add a lifecycle integration test.
## Notes
- Two new app fields on `callRecording`: `botScheduleAttemptedAt`
(DATE_TIME) and `botScheduleIdempotencyKey` (TEXT), both nullable and
not UI-editable.
- A tight race between the event trigger and the cron converges on one
bot via the deterministic idempotency key.
- Not addressed here (needs a server-side change): per-workspace jitter
when dispatching logic-function cron triggers, so identical patterns
don't fire for every workspace on the same minute.
## Test
- New `call-recorder-lifecycle.integration-test.ts` on the app's
integration harness: the global setup installs the app on a live test
server, all reads and writes go through the real API into the test
database, and only externals are mocked — the Recall API (a fetch
interceptor that replays the same bot for a repeated `Idempotency-Key`,
like the real API) and the trigger transports (webhook payloads invoke
the webhook logic function handler; cron and database-event triggers run
their flows). Thirteen scenarios assert the resulting CallRecording rows
in the DB: scheduling from calendar reconciliation (events attached to a
seeded `SHARE_EVERYTHING` calendar channel, since unassociated events
are invisible), webhook status progression with artifact-import route
calls, transcript completion, out-of-order delivery protection, fatal
failure, unknown bots, cancellation with retried Recall delete, and
every crash recovery path. Verified locally against a live server: 15
integration tests pass (including the existing schema contract test).
- `yarn test:unit`: 488 tests pass. `yarn typecheck` and `yarn lint`
clean.
---------
Co-authored-by: martmull <martin@twenty.com>
martmull
89609c520c