martmull
89609c520c
## Context We receive Recall rate limit alerts on `/api/v1/bot`. Recall's List Bots endpoint allows only 60 requests/min per Recall workspace (vs 300/min for Retrieve and 120/min for Create), and that budget is shared by every Twenty workspace on the instance since `RECALL_API_KEY` is a server-level variable. The call-recorder recovery crons fanned out one list call per stuck recording, fired at the same wall-clock minute for every workspace, and never resolved dead rows, so the pending set only grew. This PR reworks the bot-scheduling recovery mechanism so that crash-recovery work is rare, cheap, and mostly event-driven. One commit per change: ## Changes 1. **Fail never-scheduled recordings once their meeting ends** (`bot_never_scheduled` failure reason). Previously these rows stayed `REQUESTED+SCHEDULED` forever and were re-fetched by every recovery run. Rows with an unresolved creation attempt keep their recovery chance until the 7-day convergence lookback passes (a bot may have recorded before the id write-back was lost), then fail as `bot_schedule_outcome_unknown`. 2. **Batch bot lookups into one list call per run.** The pending-bot sweep and the failed-cancellation retry each issue at most one workspace-wide `GET /api/v1/bot/` (filtered by `twentyWorkspaceId` + active statuses) and match bots to recordings in memory via `twentyCallRecordingId` metadata, instead of one list call per stuck row. Truncated lists count as failed lookups so an incomplete map never authorizes a duplicate creation. 3. **Record a `botScheduleAttemptedAt` marker before POSTing a bot.** Recovery can now distinguish rows that never reached Recall (re-schedule directly, zero Recall reads) from rows whose creation outcome is unknown (only these join the lookup). 4. **Store the bot-creation `Idempotency-Key` on the row and recover by re-sending.** When a stuck row's stored key still hashes from the current scheduling inputs, recovery re-sends the creation: Recall either returns the existing bot or creates the intended one, all on the Create budget (120/min) without touching the List budget (60/min). Drifted inputs still fall back to the lookup. Re-sends preserve the first attempt's timestamp and are only trusted within a 12-hour window, so repeated unknown outcomes age into the lookup path rather than risking a twin bot after Recall's key retention expires. 5. **Resume pending rows on `callRecording.updated` events and slow the cron.** A new database-event trigger resumes scheduling within seconds when a row transitions back to pending (bot vanished at Recall, canceled request re-requested, failed row reset by reconciliation), with queue retries. It skips creations (the inserting run schedules inline), skips its own progress writes, uses slim-payload diffs to skip cheaply, and defers ambiguous rows to the cron so event bursts cannot fan out list calls. The pending-requests cron becomes a backstop and drops from every 5 minutes to every 15. Follow-up commits harden edge cases raised in review (status revalidation before POST, per-row cancellation recovery window, future-timestamp guard, attempt-state cleanup when a bot is confirmed gone at Recall) and add a lifecycle integration test. ## Notes - Two new app fields on `callRecording`: `botScheduleAttemptedAt` (DATE_TIME) and `botScheduleIdempotencyKey` (TEXT), both nullable and not UI-editable. - A tight race between the event trigger and the cron converges on one bot via the deterministic idempotency key. - Not addressed here (needs a server-side change): per-workspace jitter when dispatching logic-function cron triggers, so identical patterns don't fire for every workspace on the same minute. ## Test - New `call-recorder-lifecycle.integration-test.ts` on the app's integration harness: the global setup installs the app on a live test server, all reads and writes go through the real API into the test database, and only externals are mocked — the Recall API (a fetch interceptor that replays the same bot for a repeated `Idempotency-Key`, like the real API) and the trigger transports (webhook payloads invoke the webhook logic function handler; cron and database-event triggers run their flows). Thirteen scenarios assert the resulting CallRecording rows in the DB: scheduling from calendar reconciliation (events attached to a seeded `SHARE_EVERYTHING` calendar channel, since unassociated events are invisible), webhook status progression with artifact-import route calls, transcript completion, out-of-order delivery protection, fatal failure, unknown bots, cancellation with retried Recall delete, and every crash recovery path. Verified locally against a live server: 15 integration tests pass (including the existing schema contract test). - `yarn test:unit`: 488 tests pass. `yarn typecheck` and `yarn lint` clean. --------- Co-authored-by: martmull <martin@twenty.com>
The #1 Open-Source CRM
Website · 
Documentation · 
Roadmap · 
Discord · 
Figma
Why Twenty
Twenty gives technical teams the building blocks for a custom CRM that meets complex business needs and quickly adapts as the business evolves. Twenty is the CRM you build, ship, and version like the rest of your stack.
Learn more about why we built Twenty
Installation
Cloud
The fastest way to get started. Sign up at twenty.com and spin up a workspace in under a minute, with no infrastructure to manage and always up to date.
Build an app
Scaffold a new app with the Twenty CLI:
npx create-twenty-app my-app
Define objects, fields, and views as code:
import { defineObject, FieldType } from 'twenty-sdk/define';
export default defineObject({
nameSingular: 'deal',
namePlural: 'deals',
labelSingular: 'Deal',
labelPlural: 'Deals',
fields: [
{ name: 'name', label: 'Name', type: FieldType.TEXT },
{ name: 'amount', label: 'Amount', type: FieldType.CURRENCY },
{ name: 'closeDate', label: 'Close Date', type: FieldType.DATE_TIME },
],
});
Then ship it to your workspace:
npx twenty app:publish --private
See the app development guide for objects, views, agents, and logic functions.
Self-hosting
Run Twenty on your own infrastructure with Docker Compose, or contribute locally via the local setup guide.
Everything you need
Twenty gives you the building blocks of a modern CRM (objects, views, workflows, and agents) and lets you extend them as code. Here's a tour of what's in the box.
Want to go deeper? Read the 
User Guide for product walkthroughs, or the Documentation for developer reference.
|
|
|
|
|
|
Learn more about version control in docStack
TypeScript
Nx
NestJS, with BullMQ, 
PostgreSQL, 
Redis
React, with Jotai, Linaria and Lingui
Thanks
Thanks to these amazing services that we use and recommend for code review (Greptile), catching bugs (Sentry) and translating (Crowdin).
Join the Community
Star the repo · Discord · 
Feature requests · Releases · 
X · 
LinkedIn · 
Crowdin · 
Contribute