Reduce call-recorder Recall API load and harden bot scheduling recovery (#23014)
## Context We receive Recall rate limit alerts on `/api/v1/bot`. Recall's List Bots endpoint allows only 60 requests/min per Recall workspace (vs 300/min for Retrieve and 120/min for Create), and that budget is shared by every Twenty workspace on the instance since `RECALL_API_KEY` is a server-level variable. The call-recorder recovery crons fanned out one list call per stuck recording, fired at the same wall-clock minute for every workspace, and never resolved dead rows, so the pending set only grew. This PR reworks the bot-scheduling recovery mechanism so that crash-recovery work is rare, cheap, and mostly event-driven. One commit per change: ## Changes 1. **Fail never-scheduled recordings once their meeting ends** (`bot_never_scheduled` failure reason). Previously these rows stayed `REQUESTED+SCHEDULED` forever and were re-fetched by every recovery run. Rows with an unresolved creation attempt keep their recovery chance until the 7-day convergence lookback passes (a bot may have recorded before the id write-back was lost), then fail as `bot_schedule_outcome_unknown`. 2. **Batch bot lookups into one list call per run.** The pending-bot sweep and the failed-cancellation retry each issue at most one workspace-wide `GET /api/v1/bot/` (filtered by `twentyWorkspaceId` + active statuses) and match bots to recordings in memory via `twentyCallRecordingId` metadata, instead of one list call per stuck row. Truncated lists count as failed lookups so an incomplete map never authorizes a duplicate creation. 3. **Record a `botScheduleAttemptedAt` marker before POSTing a bot.** Recovery can now distinguish rows that never reached Recall (re-schedule directly, zero Recall reads) from rows whose creation outcome is unknown (only these join the lookup). 4. **Store the bot-creation `Idempotency-Key` on the row and recover by re-sending.** When a stuck row's stored key still hashes from the current scheduling inputs, recovery re-sends the creation: Recall either returns the existing bot or creates the intended one, all on the Create budget (120/min) without touching the List budget (60/min). Drifted inputs still fall back to the lookup. Re-sends preserve the first attempt's timestamp and are only trusted within a 12-hour window, so repeated unknown outcomes age into the lookup path rather than risking a twin bot after Recall's key retention expires. 5. **Resume pending rows on `callRecording.updated` events and slow the cron.** A new database-event trigger resumes scheduling within seconds when a row transitions back to pending (bot vanished at Recall, canceled request re-requested, failed row reset by reconciliation), with queue retries. It skips creations (the inserting run schedules inline), skips its own progress writes, uses slim-payload diffs to skip cheaply, and defers ambiguous rows to the cron so event bursts cannot fan out list calls. The pending-requests cron becomes a backstop and drops from every 5 minutes to every 15. Follow-up commits harden edge cases raised in review (status revalidation before POST, per-row cancellation recovery window, future-timestamp guard, attempt-state cleanup when a bot is confirmed gone at Recall) and add a lifecycle integration test. ## Notes - Two new app fields on `callRecording`: `botScheduleAttemptedAt` (DATE_TIME) and `botScheduleIdempotencyKey` (TEXT), both nullable and not UI-editable. - A tight race between the event trigger and the cron converges on one bot via the deterministic idempotency key. - Not addressed here (needs a server-side change): per-workspace jitter when dispatching logic-function cron triggers, so identical patterns don't fire for every workspace on the same minute. ## Test - New `call-recorder-lifecycle.integration-test.ts` on the app's integration harness: the global setup installs the app on a live test server, all reads and writes go through the real API into the test database, and only externals are mocked — the Recall API (a fetch interceptor that replays the same bot for a repeated `Idempotency-Key`, like the real API) and the trigger transports (webhook payloads invoke the webhook logic function handler; cron and database-event triggers run their flows). Thirteen scenarios assert the resulting CallRecording rows in the DB: scheduling from calendar reconciliation (events attached to a seeded `SHARE_EVERYTHING` calendar channel, since unassociated events are invisible), webhook status progression with artifact-import route calls, transcript completion, out-of-order delivery protection, fatal failure, unknown bots, cancellation with retried Recall delete, and every crash recovery path. Verified locally against a live server: 15 integration tests pass (including the existing schema contract test). - `yarn test:unit`: 488 tests pass. `yarn typecheck` and `yarn lint` clean. --------- Co-authored-by: martmull <martin@twenty.com>
This commit is contained in:
+858
@@ -0,0 +1,858 @@
|
||||
import { randomUUID } from 'crypto';
|
||||
|
||||
import { CoreApiClient } from 'twenty-client-sdk/core';
|
||||
import { afterEach, beforeAll, beforeEach, describe, expect, it, vi } from 'vitest';
|
||||
|
||||
import { cancelCallRecordingRequest } from 'src/logic-functions/flows/cancel-call-recording-request.util';
|
||||
import { reconcileCallRecorderForCalendarEventIds } from 'src/logic-functions/flows/reconcile-call-recorder.util';
|
||||
import { retryFailedRecallCancellations } from 'src/logic-functions/flows/retry-failed-recall-cancellations.util';
|
||||
import { scheduleRecallBotsForPendingCallRecordings } from 'src/logic-functions/flows/schedule-recall-bots-for-pending-call-recordings.util';
|
||||
import { processRecallWebhookHandler } from 'src/logic-functions/process-recall-webhook';
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Call Recorder end-to-end behavior against a live Twenty server.
|
||||
//
|
||||
// The app is installed on the test server by the vitest global setup, and all
|
||||
// reads and writes go through the real API into the test database. Only the
|
||||
// externals are mocked:
|
||||
// - the Recall API (a fetch interceptor that replays the same bot for a
|
||||
// repeated Idempotency-Key, like the real API),
|
||||
// - the trigger transports: webhook deliveries invoke the webhook logic
|
||||
// function handler directly, and cron / database-event triggers invoke
|
||||
// the flows they dispatch.
|
||||
//
|
||||
// Every scenario then asserts the resulting CallRecording rows in the DB.
|
||||
//
|
||||
// The suite issues a few hundred API requests; if the test server runs with
|
||||
// the default API_RATE_LIMITING_LONG_LIMIT of 100 requests per minute,
|
||||
// raise it (e.g. to 100000) or the runs trip the limiter.
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
const WORKSPACE_API_KEY_ENV = 'TWENTY_API_KEY';
|
||||
const RECALL_BASE_URL = 'https://us-west-2.recall.ai/api/v1';
|
||||
const FUNCTIONS_URL = 'https://call-recorder-functions.test';
|
||||
const ARTIFACT_IMPORT_ROUTE = '/call-recorder/import-call-recording-artifacts';
|
||||
const RESTRICTED_TITLE_PLACEHOLDER =
|
||||
'FIELD_RESTRICTED_ADDITIONAL_PERMISSIONS_REQUIRED';
|
||||
|
||||
// The app's generated client only covers the objects the app uses, but test
|
||||
// seeding also needs calendarChannelEventAssociations (see below); this hits
|
||||
// the workspace GraphQL API directly with the test API key.
|
||||
const workspaceGraphql = async (
|
||||
query: string,
|
||||
variables: Record<string, unknown> = {},
|
||||
): Promise<any> => {
|
||||
const response = await fetch(`${process.env.TWENTY_API_URL}/graphql`, {
|
||||
method: 'POST',
|
||||
headers: {
|
||||
Authorization: `Bearer ${process.env[WORKSPACE_API_KEY_ENV]}`,
|
||||
'Content-Type': 'application/json',
|
||||
},
|
||||
body: JSON.stringify({ query, variables }),
|
||||
});
|
||||
const payload = await response.json();
|
||||
|
||||
if (payload.errors !== undefined) {
|
||||
throw new Error(
|
||||
`Workspace GraphQL request failed: ${JSON.stringify(payload.errors)}`,
|
||||
);
|
||||
}
|
||||
|
||||
return payload.data;
|
||||
};
|
||||
|
||||
// Calendar events are only readable when a calendar channel with
|
||||
// SHARE_EVERYTHING visibility claims them, exactly like events coming from a
|
||||
// real calendar sync. The dev seeds provide such a channel; it is found by
|
||||
// following a fully visible seeded event to its channel association.
|
||||
const discoverShareEverythingChannelId = async (): Promise<string> => {
|
||||
const eventsData = await workspaceGraphql(
|
||||
`query { calendarEvents(first: 30) { edges { node { id title } } } }`,
|
||||
);
|
||||
const visibleEvent = eventsData.calendarEvents.edges
|
||||
.map((edge: any) => edge.node)
|
||||
.find(
|
||||
(node: any) =>
|
||||
node.title !== null && node.title !== RESTRICTED_TITLE_PLACEHOLDER,
|
||||
);
|
||||
|
||||
if (visibleEvent === undefined) {
|
||||
throw new Error(
|
||||
'No fully visible seeded calendar event found; run the dev seeds before the integration tests',
|
||||
);
|
||||
}
|
||||
|
||||
const associationsData = await workspaceGraphql(
|
||||
`query ($calendarEventId: UUID) {
|
||||
calendarChannelEventAssociations(
|
||||
filter: { calendarEventId: { eq: $calendarEventId } }
|
||||
first: 1
|
||||
) { edges { node { calendarChannelId } } }
|
||||
}`,
|
||||
{ calendarEventId: visibleEvent.id },
|
||||
);
|
||||
const channelId =
|
||||
associationsData.calendarChannelEventAssociations.edges[0]?.node
|
||||
?.calendarChannelId;
|
||||
|
||||
if (channelId === undefined) {
|
||||
throw new Error('Seeded visible calendar event has no channel association');
|
||||
}
|
||||
|
||||
return channelId;
|
||||
};
|
||||
|
||||
const inOneHour = () => new Date(Date.now() + 60 * 60 * 1000).toISOString();
|
||||
const inTwoHours = () => new Date(Date.now() + 2 * 60 * 60 * 1000).toISOString();
|
||||
const hoursAgo = (hours: number) =>
|
||||
new Date(Date.now() - hours * 60 * 60 * 1000).toISOString();
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Recall API fake, installed as a fetch interceptor. Twenty API traffic is
|
||||
// never intercepted: the shared client is built before the interceptor and
|
||||
// unknown URLs fall through to the real fetch.
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
type FakeRecallBot = {
|
||||
id: string;
|
||||
metadata: Record<string, string>;
|
||||
statusCode: string;
|
||||
};
|
||||
|
||||
class FakeRecallApi {
|
||||
bots = new Map<string, FakeRecallBot>();
|
||||
botIdByIdempotencyKey = new Map<string, string>();
|
||||
deletedBotIds: string[] = [];
|
||||
listRequestCount = 0;
|
||||
artifactImportRequests: object[] = [];
|
||||
failNextDelete = false;
|
||||
|
||||
seedBot(bot: FakeRecallBot): void {
|
||||
this.bots.set(bot.id, bot);
|
||||
}
|
||||
|
||||
botForCallRecording(callRecordingId: string): FakeRecallBot | undefined {
|
||||
return [...this.bots.values()].find(
|
||||
(bot) => bot.metadata.twentyCallRecordingId === callRecordingId,
|
||||
);
|
||||
}
|
||||
|
||||
handle(requestUrl: string, requestInit?: any): Response | undefined {
|
||||
const method: string = requestInit?.method ?? 'GET';
|
||||
|
||||
if (requestUrl.startsWith(`${FUNCTIONS_URL}${ARTIFACT_IMPORT_ROUTE}`)) {
|
||||
this.artifactImportRequests.push(JSON.parse(requestInit?.body ?? '{}'));
|
||||
|
||||
return jsonResponse(200, {});
|
||||
}
|
||||
|
||||
if (!requestUrl.startsWith(RECALL_BASE_URL)) {
|
||||
return undefined;
|
||||
}
|
||||
|
||||
if (method === 'POST' && requestUrl === `${RECALL_BASE_URL}/bot/`) {
|
||||
return this.createBot(requestInit);
|
||||
}
|
||||
|
||||
if (method === 'GET' && requestUrl.startsWith(`${RECALL_BASE_URL}/bot/?`)) {
|
||||
this.listRequestCount += 1;
|
||||
|
||||
return jsonResponse(200, {
|
||||
next: null,
|
||||
results: [...this.bots.values()].map((bot) => ({
|
||||
id: bot.id,
|
||||
metadata: bot.metadata,
|
||||
status: { code: bot.statusCode },
|
||||
})),
|
||||
});
|
||||
}
|
||||
|
||||
const botIdMatch = requestUrl.match(/\/bot\/([^/]+)\/$/);
|
||||
|
||||
if (method === 'DELETE' && botIdMatch !== null) {
|
||||
if (this.failNextDelete) {
|
||||
this.failNextDelete = false;
|
||||
|
||||
return jsonResponse(400, {});
|
||||
}
|
||||
|
||||
this.bots.delete(botIdMatch[1]);
|
||||
this.deletedBotIds.push(botIdMatch[1]);
|
||||
|
||||
return new Response(null, { status: 204 });
|
||||
}
|
||||
|
||||
throw new Error(`Unhandled Recall API request: ${method} ${requestUrl}`);
|
||||
}
|
||||
|
||||
private createBot(requestInit: any): Response {
|
||||
const idempotencyKey: string | undefined =
|
||||
requestInit?.headers?.['Idempotency-Key'];
|
||||
const alreadyCreatedBotId =
|
||||
idempotencyKey === undefined
|
||||
? undefined
|
||||
: this.botIdByIdempotencyKey.get(idempotencyKey);
|
||||
|
||||
if (alreadyCreatedBotId !== undefined) {
|
||||
return jsonResponse(200, { id: alreadyCreatedBotId });
|
||||
}
|
||||
|
||||
const body = JSON.parse(requestInit?.body ?? '{}');
|
||||
const bot: FakeRecallBot = {
|
||||
id: `recall-bot-${randomUUID()}`,
|
||||
metadata: body.metadata ?? {},
|
||||
statusCode: 'ready',
|
||||
};
|
||||
|
||||
this.bots.set(bot.id, bot);
|
||||
|
||||
if (idempotencyKey !== undefined) {
|
||||
this.botIdByIdempotencyKey.set(idempotencyKey, bot.id);
|
||||
}
|
||||
|
||||
return jsonResponse(201, { id: bot.id });
|
||||
}
|
||||
}
|
||||
|
||||
const jsonResponse = (status: number, body: object): Response =>
|
||||
new Response(JSON.stringify(body), { status });
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Recall webhook payloads, mirroring the shapes Recall actually delivers.
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
const buildBotMetadata = (callRecordingId: string, workspaceId: string) => ({
|
||||
twentyWorkspaceId: workspaceId,
|
||||
twentyCallRecordingId: callRecordingId,
|
||||
});
|
||||
|
||||
const buildBotStatusChangeWebhook = ({
|
||||
botId,
|
||||
metadata,
|
||||
statusCode,
|
||||
statusTimestamp,
|
||||
}: {
|
||||
botId: string;
|
||||
metadata: Record<string, string>;
|
||||
statusCode: string;
|
||||
statusTimestamp?: string;
|
||||
}) => ({
|
||||
event: 'bot.status_change',
|
||||
data: {
|
||||
bot_id: botId,
|
||||
status: {
|
||||
code: statusCode,
|
||||
created_at: statusTimestamp ?? new Date().toISOString(),
|
||||
},
|
||||
bot: { id: botId, metadata },
|
||||
},
|
||||
});
|
||||
|
||||
const buildRecordingDoneWebhook = ({
|
||||
botId,
|
||||
metadata,
|
||||
startedAt,
|
||||
completedAt,
|
||||
}: {
|
||||
botId: string;
|
||||
metadata: Record<string, string>;
|
||||
startedAt: string;
|
||||
completedAt: string;
|
||||
}) => ({
|
||||
event: 'recording.done',
|
||||
data: {
|
||||
bot: { id: botId, metadata },
|
||||
recording: {
|
||||
id: 'recall-recording-1',
|
||||
started_at: startedAt,
|
||||
completed_at: completedAt,
|
||||
},
|
||||
},
|
||||
});
|
||||
|
||||
const buildTranscriptDoneWebhook = ({
|
||||
botId,
|
||||
metadata,
|
||||
}: {
|
||||
botId: string;
|
||||
metadata: Record<string, string>;
|
||||
}) => ({
|
||||
event: 'transcript.done',
|
||||
data: {
|
||||
bot: { id: botId, metadata },
|
||||
transcript: { id: 'recall-transcript-1' },
|
||||
},
|
||||
});
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Test workspace helpers: real rows in the test database, destroyed after
|
||||
// each scenario.
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
describe('call recorder app lifecycle (integration)', () => {
|
||||
// Built before the fetch interceptor is installed so the shared client's
|
||||
// Twenty API traffic always uses the real fetch.
|
||||
let client: CoreApiClient;
|
||||
let workspaceId: string;
|
||||
let shareEverythingChannelId: string;
|
||||
let recall: FakeRecallApi;
|
||||
const createdCalendarEventIds: string[] = [];
|
||||
const createdCallRecordingIds: string[] = [];
|
||||
const createdAssociationIds: string[] = [];
|
||||
|
||||
const readWorkspaceIdFromApiKey = (): string => {
|
||||
const apiKey = process.env[WORKSPACE_API_KEY_ENV] ?? '';
|
||||
const payload = JSON.parse(
|
||||
Buffer.from(apiKey.split('.')[1] ?? '', 'base64url').toString('utf8'),
|
||||
);
|
||||
|
||||
return payload.workspaceId;
|
||||
};
|
||||
|
||||
beforeAll(async () => {
|
||||
client = new CoreApiClient();
|
||||
workspaceId = readWorkspaceIdFromApiKey();
|
||||
shareEverythingChannelId = await discoverShareEverythingChannelId();
|
||||
});
|
||||
|
||||
beforeEach(() => {
|
||||
recall = new FakeRecallApi();
|
||||
|
||||
const realFetch = globalThis.fetch;
|
||||
|
||||
vi.stubGlobal(
|
||||
'fetch',
|
||||
(requestUrl: any, requestInit?: any): Promise<Response> => {
|
||||
const intercepted =
|
||||
typeof requestUrl === 'string'
|
||||
? recall.handle(requestUrl, requestInit)
|
||||
: undefined;
|
||||
|
||||
return intercepted !== undefined
|
||||
? Promise.resolve(intercepted)
|
||||
: realFetch(requestUrl, requestInit);
|
||||
},
|
||||
);
|
||||
vi.stubEnv('RECALL_API_KEY', 'recall-api-key');
|
||||
vi.stubEnv('RECALL_REGION', 'us-west-2');
|
||||
vi.stubEnv('CALL_RECORDER_USE_WORKSPACE_LOGO', 'false');
|
||||
vi.stubEnv('TWENTY_FUNCTIONS_URL', FUNCTIONS_URL);
|
||||
// Logic functions normally run with an app access token; the workspace
|
||||
// API key is a token with the same workspaceId claim.
|
||||
vi.stubEnv(
|
||||
'TWENTY_APP_ACCESS_TOKEN',
|
||||
process.env[WORKSPACE_API_KEY_ENV] ?? '',
|
||||
);
|
||||
});
|
||||
|
||||
afterEach(async () => {
|
||||
vi.unstubAllGlobals();
|
||||
vi.unstubAllEnvs();
|
||||
|
||||
await destroyCreatedRows();
|
||||
});
|
||||
|
||||
const destroyCreatedRows = async (): Promise<void> => {
|
||||
const callRecordingsForCreatedCalendarEvents =
|
||||
createdCalendarEventIds.length === 0
|
||||
? []
|
||||
: await findCallRecordings({
|
||||
calendarEventId: { in: createdCalendarEventIds },
|
||||
});
|
||||
const callRecordingIds = [
|
||||
...new Set([
|
||||
...createdCallRecordingIds,
|
||||
...callRecordingsForCreatedCalendarEvents.map(({ id }) => id),
|
||||
]),
|
||||
];
|
||||
|
||||
for (const callRecordingId of callRecordingIds) {
|
||||
await client
|
||||
.mutation({
|
||||
destroyCallRecording: { __args: { id: callRecordingId }, id: true },
|
||||
})
|
||||
.catch(() => {});
|
||||
}
|
||||
|
||||
for (const associationId of createdAssociationIds) {
|
||||
await workspaceGraphql(
|
||||
`mutation ($id: UUID!) {
|
||||
destroyCalendarChannelEventAssociation(id: $id) { id }
|
||||
}`,
|
||||
{ id: associationId },
|
||||
).catch(() => {});
|
||||
}
|
||||
|
||||
for (const calendarEventId of createdCalendarEventIds) {
|
||||
await client
|
||||
.mutation({
|
||||
destroyCalendarEvent: { __args: { id: calendarEventId }, id: true },
|
||||
})
|
||||
.catch(() => {});
|
||||
}
|
||||
|
||||
createdCalendarEventIds.length = 0;
|
||||
createdCallRecordingIds.length = 0;
|
||||
createdAssociationIds.length = 0;
|
||||
};
|
||||
|
||||
const createCalendarEvent = async (
|
||||
overrides: Record<string, unknown> = {},
|
||||
): Promise<string> => {
|
||||
const calendarEventId = randomUUID();
|
||||
|
||||
await client.mutation({
|
||||
createCalendarEvent: {
|
||||
__args: {
|
||||
data: {
|
||||
id: calendarEventId,
|
||||
title: 'Customer Sync (call recorder integration test)',
|
||||
startsAt: inOneHour(),
|
||||
endsAt: inTwoHours(),
|
||||
iCalUid: `call-recorder-test-${calendarEventId}`,
|
||||
conferenceLink: {
|
||||
primaryLinkUrl: `https://meet.example.com/${calendarEventId}`,
|
||||
},
|
||||
callRecorderPreference: 'ON',
|
||||
...overrides,
|
||||
},
|
||||
},
|
||||
id: true,
|
||||
},
|
||||
});
|
||||
createdCalendarEventIds.push(calendarEventId);
|
||||
|
||||
// Without a SHARE_EVERYTHING channel association the event would be
|
||||
// invisible to every query, like an event no calendar sync produced.
|
||||
const associationData = await workspaceGraphql(
|
||||
`mutation ($data: CalendarChannelEventAssociationCreateInput!) {
|
||||
createCalendarChannelEventAssociation(data: $data) { id }
|
||||
}`,
|
||||
{
|
||||
data: {
|
||||
calendarChannelId: shareEverythingChannelId,
|
||||
calendarEventId,
|
||||
eventExternalId: `call-recorder-test-${calendarEventId}`,
|
||||
},
|
||||
},
|
||||
);
|
||||
|
||||
createdAssociationIds.push(
|
||||
associationData.createCalendarChannelEventAssociation.id,
|
||||
);
|
||||
|
||||
return calendarEventId;
|
||||
};
|
||||
|
||||
const createPendingCallRecording = async ({
|
||||
calendarEventId,
|
||||
...overrides
|
||||
}: Record<string, unknown> & { calendarEventId: string }): Promise<string> => {
|
||||
const callRecordingId = randomUUID();
|
||||
|
||||
await client.mutation({
|
||||
createCallRecording: {
|
||||
__args: {
|
||||
data: {
|
||||
id: callRecordingId,
|
||||
title: 'Customer Sync (call recorder integration test)',
|
||||
status: 'SCHEDULED',
|
||||
recordingRequestStatus: 'REQUESTED',
|
||||
calendarEventId,
|
||||
...overrides,
|
||||
},
|
||||
},
|
||||
id: true,
|
||||
},
|
||||
});
|
||||
createdCallRecordingIds.push(callRecordingId);
|
||||
|
||||
return callRecordingId;
|
||||
};
|
||||
|
||||
const findCallRecordings = async (
|
||||
filter: Record<string, unknown>,
|
||||
): Promise<Array<Record<string, any>>> => {
|
||||
const result = await client.query({
|
||||
callRecordings: {
|
||||
__args: { filter, first: 50 },
|
||||
edges: {
|
||||
node: {
|
||||
id: true,
|
||||
status: true,
|
||||
recordingRequestStatus: true,
|
||||
calendarEventId: true,
|
||||
externalBotId: true,
|
||||
externalRecordingId: true,
|
||||
botScheduleAttemptedAt: true,
|
||||
botScheduleIdempotencyKey: true,
|
||||
callRecorderFailureReason: true,
|
||||
startedAt: true,
|
||||
endedAt: true,
|
||||
},
|
||||
},
|
||||
},
|
||||
});
|
||||
|
||||
return (result.callRecordings?.edges ?? []).map(
|
||||
(edge: any) => edge.node,
|
||||
);
|
||||
};
|
||||
|
||||
const fetchCallRecording = async (
|
||||
callRecordingId: string,
|
||||
): Promise<Record<string, any>> => {
|
||||
const callRecording = (
|
||||
await findCallRecordings({ id: { eq: callRecordingId } })
|
||||
)[0];
|
||||
|
||||
expect(callRecording).toBeDefined();
|
||||
|
||||
return callRecording;
|
||||
};
|
||||
|
||||
// Mocked database-event trigger: runs the reconciliation the
|
||||
// calendarEvent.* trigger dispatches, then returns the recording it wrote
|
||||
// to the DB together with its live Recall bot.
|
||||
const scheduleRecordingThroughCalendarReconciliation = async (): Promise<{
|
||||
calendarEventId: string;
|
||||
callRecordingId: string;
|
||||
botId: string;
|
||||
metadata: Record<string, string>;
|
||||
}> => {
|
||||
const calendarEventId = await createCalendarEvent();
|
||||
|
||||
await reconcileCallRecorderForCalendarEventIds({
|
||||
client,
|
||||
calendarEventIds: [calendarEventId],
|
||||
});
|
||||
|
||||
const callRecording = (
|
||||
await findCallRecordings({ calendarEventId: { in: [calendarEventId] } })
|
||||
)[0];
|
||||
|
||||
expect(callRecording).toBeDefined();
|
||||
expect(callRecording.externalBotId).toBeTruthy();
|
||||
|
||||
return {
|
||||
calendarEventId,
|
||||
callRecordingId: callRecording.id,
|
||||
botId: callRecording.externalBotId,
|
||||
metadata: buildBotMetadata(callRecording.id, workspaceId),
|
||||
};
|
||||
};
|
||||
|
||||
// Mocked webhook trigger: invokes the webhook logic function handler with
|
||||
// the payload Recall would have delivered.
|
||||
const deliverRecallWebhook = (body: object) =>
|
||||
processRecallWebhookHandler(body);
|
||||
|
||||
// Mocked cron trigger: runs the flows the recovery cron dispatches.
|
||||
const runPendingRecoveryCron = () =>
|
||||
scheduleRecallBotsForPendingCallRecordings({ client, now: new Date() });
|
||||
const runCancellationRetryCron = () =>
|
||||
retryFailedRecallCancellations({ client, now: new Date() });
|
||||
|
||||
describe('scheduling from calendar changes', () => {
|
||||
it('creates a recording and schedules a Recall bot for a meeting with recording enabled', async () => {
|
||||
const { callRecordingId, botId } =
|
||||
await scheduleRecordingThroughCalendarReconciliation();
|
||||
|
||||
const callRecording = await fetchCallRecording(callRecordingId);
|
||||
|
||||
expect(callRecording.status).toBe('SCHEDULED');
|
||||
expect(callRecording.recordingRequestStatus).toBe('REQUESTED');
|
||||
expect(callRecording.botScheduleAttemptedAt).toBeTruthy();
|
||||
expect(callRecording.botScheduleIdempotencyKey).toBeTruthy();
|
||||
expect(recall.bots.get(botId)?.metadata).toEqual(
|
||||
buildBotMetadata(callRecordingId, workspaceId),
|
||||
);
|
||||
});
|
||||
|
||||
it('creates nothing for a meeting without a conference link', async () => {
|
||||
const calendarEventId = await createCalendarEvent({
|
||||
conferenceLink: { primaryLinkUrl: '' },
|
||||
});
|
||||
|
||||
await reconcileCallRecorderForCalendarEventIds({
|
||||
client,
|
||||
calendarEventIds: [calendarEventId],
|
||||
});
|
||||
|
||||
expect(
|
||||
await findCallRecordings({ calendarEventId: { in: [calendarEventId] } }),
|
||||
).toEqual([]);
|
||||
});
|
||||
});
|
||||
|
||||
describe('Recall webhook lifecycle', () => {
|
||||
it('moves the recording through joining, recording, and processing', async () => {
|
||||
const { callRecordingId, botId, metadata } =
|
||||
await scheduleRecordingThroughCalendarReconciliation();
|
||||
|
||||
await deliverRecallWebhook(
|
||||
buildBotStatusChangeWebhook({
|
||||
botId,
|
||||
metadata,
|
||||
statusCode: 'joining_call',
|
||||
}),
|
||||
);
|
||||
expect((await fetchCallRecording(callRecordingId)).status).toBe(
|
||||
'JOINING',
|
||||
);
|
||||
|
||||
const recordingStartedAt = new Date().toISOString();
|
||||
|
||||
await deliverRecallWebhook(
|
||||
buildBotStatusChangeWebhook({
|
||||
botId,
|
||||
metadata,
|
||||
statusCode: 'in_call_recording',
|
||||
statusTimestamp: recordingStartedAt,
|
||||
}),
|
||||
);
|
||||
const recordingCallRecording = await fetchCallRecording(callRecordingId);
|
||||
|
||||
expect(recordingCallRecording.status).toBe('RECORDING');
|
||||
expect(recordingCallRecording.startedAt).toBeTruthy();
|
||||
|
||||
const recordingEndedAt = new Date().toISOString();
|
||||
|
||||
await deliverRecallWebhook(
|
||||
buildRecordingDoneWebhook({
|
||||
botId,
|
||||
metadata,
|
||||
startedAt: recordingStartedAt,
|
||||
completedAt: recordingEndedAt,
|
||||
}),
|
||||
);
|
||||
const processedCallRecording = await fetchCallRecording(callRecordingId);
|
||||
|
||||
expect(processedCallRecording.status).toBe('PROCESSING');
|
||||
expect(processedCallRecording.externalRecordingId).toBe(
|
||||
'recall-recording-1',
|
||||
);
|
||||
expect(processedCallRecording.endedAt).toBeTruthy();
|
||||
// recording.done hands media and transcript work to the artifact
|
||||
// import route.
|
||||
expect(recall.artifactImportRequests).toHaveLength(1);
|
||||
expect(recall.artifactImportRequests[0]).toMatchObject({
|
||||
callRecordingId,
|
||||
});
|
||||
});
|
||||
|
||||
it('queues another artifact import when the transcript finishes later', async () => {
|
||||
const { callRecordingId, botId, metadata } =
|
||||
await scheduleRecordingThroughCalendarReconciliation();
|
||||
|
||||
await deliverRecallWebhook(
|
||||
buildRecordingDoneWebhook({
|
||||
botId,
|
||||
metadata,
|
||||
startedAt: new Date().toISOString(),
|
||||
completedAt: new Date().toISOString(),
|
||||
}),
|
||||
);
|
||||
await deliverRecallWebhook(
|
||||
buildTranscriptDoneWebhook({ botId, metadata }),
|
||||
);
|
||||
|
||||
expect(recall.artifactImportRequests).toHaveLength(2);
|
||||
expect(recall.artifactImportRequests[1]).toMatchObject({
|
||||
callRecordingId,
|
||||
});
|
||||
});
|
||||
|
||||
it('never moves the status backwards on late webhook deliveries', async () => {
|
||||
const { callRecordingId, botId, metadata } =
|
||||
await scheduleRecordingThroughCalendarReconciliation();
|
||||
|
||||
await deliverRecallWebhook(
|
||||
buildRecordingDoneWebhook({
|
||||
botId,
|
||||
metadata,
|
||||
startedAt: new Date().toISOString(),
|
||||
completedAt: new Date().toISOString(),
|
||||
}),
|
||||
);
|
||||
|
||||
const lateJoiningResult = await deliverRecallWebhook(
|
||||
buildBotStatusChangeWebhook({
|
||||
botId,
|
||||
metadata,
|
||||
statusCode: 'joining_call',
|
||||
}),
|
||||
);
|
||||
|
||||
expect(lateJoiningResult.status).toBe('skipped');
|
||||
expect((await fetchCallRecording(callRecordingId)).status).toBe(
|
||||
'PROCESSING',
|
||||
);
|
||||
});
|
||||
|
||||
it('marks the recording failed when the bot dies fatally', async () => {
|
||||
const { callRecordingId, botId, metadata } =
|
||||
await scheduleRecordingThroughCalendarReconciliation();
|
||||
|
||||
await deliverRecallWebhook(
|
||||
buildBotStatusChangeWebhook({ botId, metadata, statusCode: 'fatal' }),
|
||||
);
|
||||
|
||||
const callRecording = await fetchCallRecording(callRecordingId);
|
||||
|
||||
expect(callRecording.status).toBe('FAILED');
|
||||
expect(callRecording.callRecorderFailureReason).toBe('fatal');
|
||||
});
|
||||
|
||||
it('ignores webhooks that match no known recording', async () => {
|
||||
const { callRecordingId } =
|
||||
await scheduleRecordingThroughCalendarReconciliation();
|
||||
|
||||
const result = await deliverRecallWebhook(
|
||||
buildBotStatusChangeWebhook({
|
||||
botId: 'recall-bot-from-another-app',
|
||||
metadata: buildBotMetadata(randomUUID(), workspaceId),
|
||||
statusCode: 'joining_call',
|
||||
}),
|
||||
);
|
||||
|
||||
expect(result.status).toBe('skipped');
|
||||
expect((await fetchCallRecording(callRecordingId)).status).toBe(
|
||||
'SCHEDULED',
|
||||
);
|
||||
});
|
||||
});
|
||||
|
||||
describe('cancellation', () => {
|
||||
it('cancels the request and deletes the Recall bot', async () => {
|
||||
const { callRecordingId, botId } =
|
||||
await scheduleRecordingThroughCalendarReconciliation();
|
||||
|
||||
await cancelCallRecordingRequest({
|
||||
client,
|
||||
callRecording: { id: callRecordingId, externalBotId: botId },
|
||||
});
|
||||
|
||||
const callRecording = await fetchCallRecording(callRecordingId);
|
||||
|
||||
expect(callRecording.recordingRequestStatus).toBe('CANCELED');
|
||||
// The API stores a cleared TEXT field as an empty string.
|
||||
expect(callRecording.externalBotId).toBeFalsy();
|
||||
expect(recall.deletedBotIds).toEqual([botId]);
|
||||
});
|
||||
|
||||
it('retries a failed Recall cancellation on the next cron run', async () => {
|
||||
const { callRecordingId, botId } =
|
||||
await scheduleRecordingThroughCalendarReconciliation();
|
||||
|
||||
recall.failNextDelete = true;
|
||||
await cancelCallRecordingRequest({
|
||||
client,
|
||||
callRecording: { id: callRecordingId, externalBotId: botId },
|
||||
});
|
||||
|
||||
// The Recall half failed, so the bot id must survive for the retry.
|
||||
expect((await fetchCallRecording(callRecordingId)).externalBotId).toBe(
|
||||
botId,
|
||||
);
|
||||
|
||||
await runCancellationRetryCron();
|
||||
|
||||
expect(
|
||||
(await fetchCallRecording(callRecordingId)).externalBotId,
|
||||
).toBeFalsy();
|
||||
expect(recall.deletedBotIds).toContain(botId);
|
||||
});
|
||||
});
|
||||
|
||||
describe('crash recovery cron', () => {
|
||||
it('schedules a bot for a recording created without one, with zero Recall list reads', async () => {
|
||||
const calendarEventId = await createCalendarEvent();
|
||||
const callRecordingId = await createPendingCallRecording({
|
||||
calendarEventId,
|
||||
});
|
||||
|
||||
await runPendingRecoveryCron();
|
||||
|
||||
const callRecording = await fetchCallRecording(callRecordingId);
|
||||
|
||||
expect(callRecording.externalBotId).toBeTruthy();
|
||||
expect(recall.botForCallRecording(callRecordingId)?.id).toBe(
|
||||
callRecording.externalBotId,
|
||||
);
|
||||
expect(recall.listRequestCount).toBe(0);
|
||||
});
|
||||
|
||||
it('re-sends the creation after a lost write-back and lands on the same bot', async () => {
|
||||
const calendarEventId = await createCalendarEvent();
|
||||
const callRecordingId = await createPendingCallRecording({
|
||||
calendarEventId,
|
||||
});
|
||||
|
||||
// First recovery run creates the bot and records the attempt.
|
||||
await runPendingRecoveryCron();
|
||||
const firstBotId = (await fetchCallRecording(callRecordingId))
|
||||
.externalBotId;
|
||||
|
||||
// Simulate the id write-back getting lost after the POST reached
|
||||
// Recall.
|
||||
await client.mutation({
|
||||
updateCallRecording: {
|
||||
__args: { id: callRecordingId, data: { externalBotId: null } },
|
||||
id: true,
|
||||
},
|
||||
});
|
||||
|
||||
await runPendingRecoveryCron();
|
||||
|
||||
// Recall dedupes the repeated idempotency key: the very same bot is
|
||||
// written back, without any list request.
|
||||
expect((await fetchCallRecording(callRecordingId)).externalBotId).toBe(
|
||||
firstBotId,
|
||||
);
|
||||
expect(recall.listRequestCount).toBe(0);
|
||||
});
|
||||
|
||||
it('attaches an existing bot found by lookup when the recorded attempt drifted', async () => {
|
||||
const calendarEventId = await createCalendarEvent();
|
||||
const callRecordingId = await createPendingCallRecording({
|
||||
calendarEventId,
|
||||
botScheduleAttemptedAt: hoursAgo(1),
|
||||
botScheduleIdempotencyKey: 'key-from-before-the-meeting-moved',
|
||||
});
|
||||
|
||||
recall.seedBot({
|
||||
id: 'recall-bot-from-crashed-run',
|
||||
metadata: buildBotMetadata(callRecordingId, workspaceId),
|
||||
statusCode: 'ready',
|
||||
});
|
||||
|
||||
await runPendingRecoveryCron();
|
||||
|
||||
expect((await fetchCallRecording(callRecordingId)).externalBotId).toBe(
|
||||
'recall-bot-from-crashed-run',
|
||||
);
|
||||
expect(recall.listRequestCount).toBe(1);
|
||||
});
|
||||
|
||||
it('fails a recording whose meeting ended before any bot creation was attempted', async () => {
|
||||
const calendarEventId = await createCalendarEvent({
|
||||
startsAt: hoursAgo(3),
|
||||
endsAt: hoursAgo(2),
|
||||
});
|
||||
const callRecordingId = await createPendingCallRecording({
|
||||
calendarEventId,
|
||||
});
|
||||
|
||||
await runPendingRecoveryCron();
|
||||
|
||||
const callRecording = await fetchCallRecording(callRecordingId);
|
||||
|
||||
expect(callRecording.status).toBe('FAILED');
|
||||
expect(callRecording.callRecorderFailureReason).toBe(
|
||||
'bot_never_scheduled',
|
||||
);
|
||||
expect(recall.botForCallRecording(callRecordingId)).toBeUndefined();
|
||||
});
|
||||
});
|
||||
});
|
||||
+2
@@ -0,0 +1,2 @@
|
||||
export const BOT_SCHEDULE_ATTEMPTED_AT_FIELD_UNIVERSAL_IDENTIFIER =
|
||||
'ca1d0179-f611-46bd-ba8a-b503ea9d024e';
|
||||
+2
@@ -0,0 +1,2 @@
|
||||
export const BOT_SCHEDULE_IDEMPOTENCY_KEY_FIELD_UNIVERSAL_IDENTIFIER =
|
||||
'6f00ea4d-1c5a-47ac-8c18-baacf5ee7c3f';
|
||||
+2
@@ -0,0 +1,2 @@
|
||||
export const SCHEDULE_RECALL_BOT_ON_CALL_RECORDING_UPDATE_LOGIC_FUNCTION_UNIVERSAL_IDENTIFIER =
|
||||
'fad4b1b5-8b6e-48c7-a8c7-3aecae2672c1';
|
||||
+21
@@ -0,0 +1,21 @@
|
||||
import {
|
||||
defineField,
|
||||
FieldType,
|
||||
STANDARD_OBJECT_UNIVERSAL_IDENTIFIERS,
|
||||
} from 'twenty-sdk/define';
|
||||
|
||||
import { BOT_SCHEDULE_ATTEMPTED_AT_FIELD_UNIVERSAL_IDENTIFIER } from 'src/constants/bot-schedule-attempted-at-field-universal-identifier';
|
||||
|
||||
export default defineField({
|
||||
universalIdentifier: BOT_SCHEDULE_ATTEMPTED_AT_FIELD_UNIVERSAL_IDENTIFIER,
|
||||
objectUniversalIdentifier:
|
||||
STANDARD_OBJECT_UNIVERSAL_IDENTIFIERS.callRecording.universalIdentifier,
|
||||
type: FieldType.DATE_TIME,
|
||||
name: 'botScheduleAttemptedAt',
|
||||
label: 'Bot Schedule Attempted At',
|
||||
description:
|
||||
'Set right before a Recall bot creation request is sent; recovery uses it to tell rows that never reached Recall from rows whose creation outcome is unknown.',
|
||||
icon: 'IconClockPlay',
|
||||
isNullable: true,
|
||||
isUIEditable: false,
|
||||
});
|
||||
+21
@@ -0,0 +1,21 @@
|
||||
import {
|
||||
defineField,
|
||||
FieldType,
|
||||
STANDARD_OBJECT_UNIVERSAL_IDENTIFIERS,
|
||||
} from 'twenty-sdk/define';
|
||||
|
||||
import { BOT_SCHEDULE_IDEMPOTENCY_KEY_FIELD_UNIVERSAL_IDENTIFIER } from 'src/constants/bot-schedule-idempotency-key-field-universal-identifier';
|
||||
|
||||
export default defineField({
|
||||
universalIdentifier: BOT_SCHEDULE_IDEMPOTENCY_KEY_FIELD_UNIVERSAL_IDENTIFIER,
|
||||
objectUniversalIdentifier:
|
||||
STANDARD_OBJECT_UNIVERSAL_IDENTIFIERS.callRecording.universalIdentifier,
|
||||
type: FieldType.TEXT,
|
||||
name: 'botScheduleIdempotencyKey',
|
||||
label: 'Bot Schedule Idempotency Key',
|
||||
description:
|
||||
'Idempotency key of the last Recall bot creation attempt; when the scheduling inputs still hash to this key, recovery can safely re-send the creation instead of listing bots.',
|
||||
icon: 'IconKey',
|
||||
isNullable: true,
|
||||
isUIEditable: false,
|
||||
});
|
||||
+309
@@ -0,0 +1,309 @@
|
||||
import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest';
|
||||
|
||||
import { scheduleRecallBotOnCallRecordingUpdateHandler } from 'src/logic-functions/schedule-recall-bot-on-call-recording-update';
|
||||
|
||||
const queryMock = vi.hoisted(() => vi.fn());
|
||||
const mutationMock = vi.hoisted(() => vi.fn());
|
||||
|
||||
vi.mock('twenty-client-sdk/core', () => ({
|
||||
CoreApiClient: class {
|
||||
query = queryMock;
|
||||
mutation = mutationMock;
|
||||
},
|
||||
}));
|
||||
|
||||
const fetchMock = vi.fn();
|
||||
|
||||
const NOW = new Date('2026-01-01T12:00:00.000Z');
|
||||
const WORKSPACE_ID = '123e4567-e89b-12d3-a456-426614174000';
|
||||
const RECALL_CREATE_BOT_URL = 'https://us-west-2.recall.ai/api/v1/bot/';
|
||||
|
||||
const buildAccessToken = (payload: Record<string, unknown>): string =>
|
||||
[
|
||||
Buffer.from(JSON.stringify({ alg: 'none' })).toString('base64url'),
|
||||
Buffer.from(JSON.stringify(payload)).toString('base64url'),
|
||||
'signature',
|
||||
].join('.');
|
||||
|
||||
const buildConnection = <Node>(nodes: Node[]) => ({
|
||||
pageInfo: { hasNextPage: false, endCursor: undefined },
|
||||
edges: nodes.map((node) => ({ node })),
|
||||
});
|
||||
|
||||
type HandlerEvent = Parameters<
|
||||
typeof scheduleRecallBotOnCallRecordingUpdateHandler
|
||||
>[0];
|
||||
|
||||
const buildUpdateEvent = (overrides: Partial<HandlerEvent> = {}): HandlerEvent =>
|
||||
({
|
||||
name: 'callRecording.updated',
|
||||
recordId: 'call-recording-1',
|
||||
properties: {
|
||||
updatedFields: ['externalBotId'],
|
||||
before: {
|
||||
id: 'call-recording-1',
|
||||
status: 'SCHEDULED',
|
||||
recordingRequestStatus: 'REQUESTED',
|
||||
externalBotId: 'recall-bot-vanished',
|
||||
},
|
||||
after: {
|
||||
id: 'call-recording-1',
|
||||
status: 'SCHEDULED',
|
||||
recordingRequestStatus: 'REQUESTED',
|
||||
externalBotId: null,
|
||||
},
|
||||
},
|
||||
...overrides,
|
||||
}) as HandlerEvent;
|
||||
|
||||
const stubPendingCallRecordingQueries = () => {
|
||||
queryMock.mockImplementation(async (query: any) => {
|
||||
if (query.callRecordings !== undefined) {
|
||||
return {
|
||||
callRecordings: buildConnection([
|
||||
{
|
||||
id: 'call-recording-1',
|
||||
status: 'SCHEDULED',
|
||||
recordingRequestStatus: 'REQUESTED',
|
||||
calendarEventId: 'calendar-event-1',
|
||||
externalBotId: null,
|
||||
botScheduleAttemptedAt: null,
|
||||
},
|
||||
]),
|
||||
};
|
||||
}
|
||||
|
||||
if (query.calendarEvents !== undefined) {
|
||||
return {
|
||||
calendarEvents: buildConnection([
|
||||
{
|
||||
id: 'calendar-event-1',
|
||||
startsAt: '2026-01-01T13:00:00.000Z',
|
||||
endsAt: '2026-01-01T14:00:00.000Z',
|
||||
iCalUid: 'calendar-event-uid',
|
||||
conferenceLink: {
|
||||
primaryLinkUrl: 'https://meet.example.com/customer-sync',
|
||||
},
|
||||
},
|
||||
]),
|
||||
};
|
||||
}
|
||||
|
||||
throw new Error(`Unhandled query: ${JSON.stringify(query)}`);
|
||||
});
|
||||
};
|
||||
|
||||
describe('scheduleRecallBotOnCallRecordingUpdateHandler', () => {
|
||||
beforeEach(() => {
|
||||
vi.useFakeTimers();
|
||||
vi.setSystemTime(NOW);
|
||||
vi.spyOn(console, 'warn').mockImplementation(() => {});
|
||||
vi.stubGlobal('fetch', fetchMock);
|
||||
vi.stubEnv('RECALL_API_KEY', 'recall-api-key');
|
||||
vi.stubEnv('RECALL_REGION', 'us-west-2');
|
||||
vi.stubEnv('CALL_RECORDER_USE_WORKSPACE_LOGO', 'false');
|
||||
vi.stubEnv(
|
||||
'TWENTY_APP_ACCESS_TOKEN',
|
||||
buildAccessToken({ workspaceId: WORKSPACE_ID }),
|
||||
);
|
||||
queryMock.mockReset();
|
||||
mutationMock.mockReset();
|
||||
mutationMock.mockImplementation(async (mutation: any) => ({
|
||||
updateCallRecording: { id: mutation.updateCallRecording.__args.id },
|
||||
}));
|
||||
fetchMock.mockReset();
|
||||
fetchMock.mockImplementation(async (requestUrl: string) => {
|
||||
if (requestUrl === RECALL_CREATE_BOT_URL) {
|
||||
return new Response(JSON.stringify({ id: 'recall-bot-new' }), {
|
||||
status: 201,
|
||||
});
|
||||
}
|
||||
|
||||
throw new Error(`Unhandled fetch in test: ${requestUrl}`);
|
||||
});
|
||||
});
|
||||
|
||||
afterEach(() => {
|
||||
vi.unstubAllGlobals();
|
||||
vi.unstubAllEnvs();
|
||||
vi.useRealTimers();
|
||||
vi.restoreAllMocks();
|
||||
});
|
||||
|
||||
it('schedules a bot when an update clears the bot id of a requested recording', async () => {
|
||||
stubPendingCallRecordingQueries();
|
||||
|
||||
const result = await scheduleRecallBotOnCallRecordingUpdateHandler(
|
||||
buildUpdateEvent(),
|
||||
);
|
||||
|
||||
expect(result).toEqual({
|
||||
callRecordingId: 'call-recording-1',
|
||||
result: { status: 'scheduled' },
|
||||
});
|
||||
expect(fetchMock).toHaveBeenCalledTimes(1);
|
||||
expect(fetchMock).toHaveBeenCalledWith(
|
||||
RECALL_CREATE_BOT_URL,
|
||||
expect.objectContaining({ method: 'POST' }),
|
||||
);
|
||||
});
|
||||
|
||||
it('skips events that are not call recording updates', async () => {
|
||||
const result = await scheduleRecallBotOnCallRecordingUpdateHandler(
|
||||
buildUpdateEvent({ name: 'callRecording.created' } as HandlerEvent),
|
||||
);
|
||||
|
||||
expect(result).toEqual({
|
||||
skipped: true,
|
||||
reason: 'not a call recording update',
|
||||
});
|
||||
expect(queryMock).not.toHaveBeenCalled();
|
||||
expect(fetchMock).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it('skips its own scheduling-progress writes so it cannot re-trigger itself', async () => {
|
||||
const result = await scheduleRecallBotOnCallRecordingUpdateHandler(
|
||||
buildUpdateEvent({
|
||||
properties: {
|
||||
updatedFields: [
|
||||
'botScheduleAttemptedAt',
|
||||
'botScheduleIdempotencyKey',
|
||||
],
|
||||
before: {},
|
||||
after: {
|
||||
id: 'call-recording-1',
|
||||
status: 'SCHEDULED',
|
||||
recordingRequestStatus: 'REQUESTED',
|
||||
externalBotId: null,
|
||||
},
|
||||
},
|
||||
} as unknown as HandlerEvent),
|
||||
);
|
||||
|
||||
expect(result).toEqual({
|
||||
skipped: true,
|
||||
reason: 'no pending-transition field changed',
|
||||
});
|
||||
expect(queryMock).not.toHaveBeenCalled();
|
||||
expect(fetchMock).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it('skips updates that leave the recording with a bot attached', async () => {
|
||||
const result = await scheduleRecallBotOnCallRecordingUpdateHandler(
|
||||
buildUpdateEvent({
|
||||
properties: {
|
||||
updatedFields: ['externalBotId'],
|
||||
before: {},
|
||||
after: {
|
||||
id: 'call-recording-1',
|
||||
status: 'SCHEDULED',
|
||||
recordingRequestStatus: 'REQUESTED',
|
||||
externalBotId: 'recall-bot-1',
|
||||
},
|
||||
},
|
||||
} as unknown as HandlerEvent),
|
||||
);
|
||||
|
||||
expect(result).toEqual({
|
||||
skipped: true,
|
||||
reason: 'call recording is not pending',
|
||||
});
|
||||
expect(queryMock).not.toHaveBeenCalled();
|
||||
expect(fetchMock).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it('defers rows with an ambiguous prior attempt to the recovery cron instead of listing bots', async () => {
|
||||
queryMock.mockImplementationOnce(async () => ({
|
||||
callRecordings: buildConnection([
|
||||
{
|
||||
id: 'call-recording-1',
|
||||
status: 'SCHEDULED',
|
||||
recordingRequestStatus: 'REQUESTED',
|
||||
calendarEventId: 'calendar-event-1',
|
||||
externalBotId: null,
|
||||
botScheduleAttemptedAt: '2026-01-01T11:55:00.000Z',
|
||||
botScheduleIdempotencyKey: 'stale-key-from-moved-meeting',
|
||||
},
|
||||
]),
|
||||
}));
|
||||
queryMock.mockImplementationOnce(async () => ({
|
||||
calendarEvents: buildConnection([
|
||||
{
|
||||
id: 'calendar-event-1',
|
||||
startsAt: '2026-01-01T13:00:00.000Z',
|
||||
endsAt: '2026-01-01T14:00:00.000Z',
|
||||
conferenceLink: {
|
||||
primaryLinkUrl: 'https://meet.example.com/customer-sync',
|
||||
},
|
||||
},
|
||||
]),
|
||||
}));
|
||||
|
||||
const result = await scheduleRecallBotOnCallRecordingUpdateHandler(
|
||||
buildUpdateEvent(),
|
||||
);
|
||||
|
||||
expect(result).toEqual({
|
||||
callRecordingId: 'call-recording-1',
|
||||
result: {
|
||||
status: 'deferred',
|
||||
reason: 'ambiguous prior attempt; the recovery cron will reconcile it',
|
||||
},
|
||||
});
|
||||
expect(fetchMock).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it('skips slim payloads whose diff shows the bot id was written back', async () => {
|
||||
const result = await scheduleRecallBotOnCallRecordingUpdateHandler(
|
||||
buildUpdateEvent({
|
||||
properties: {
|
||||
updatedFields: ['externalBotId'],
|
||||
diff: {
|
||||
externalBotId: { before: null, after: 'recall-bot-1' },
|
||||
},
|
||||
},
|
||||
} as unknown as HandlerEvent),
|
||||
);
|
||||
|
||||
expect(result).toEqual({
|
||||
skipped: true,
|
||||
reason: 'call recording is not pending',
|
||||
});
|
||||
expect(queryMock).not.toHaveBeenCalled();
|
||||
expect(fetchMock).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it('does not schedule anything when the meeting already ended', async () => {
|
||||
stubPendingCallRecordingQueries();
|
||||
queryMock.mockImplementationOnce(async () => ({
|
||||
callRecordings: buildConnection([
|
||||
{
|
||||
id: 'call-recording-1',
|
||||
status: 'SCHEDULED',
|
||||
recordingRequestStatus: 'REQUESTED',
|
||||
calendarEventId: 'calendar-event-1',
|
||||
externalBotId: null,
|
||||
},
|
||||
]),
|
||||
}));
|
||||
queryMock.mockImplementationOnce(async () => ({
|
||||
calendarEvents: buildConnection([
|
||||
{
|
||||
id: 'calendar-event-1',
|
||||
startsAt: '2026-01-01T10:00:00.000Z',
|
||||
endsAt: '2026-01-01T11:00:00.000Z',
|
||||
},
|
||||
]),
|
||||
}));
|
||||
|
||||
const result = await scheduleRecallBotOnCallRecordingUpdateHandler(
|
||||
buildUpdateEvent(),
|
||||
);
|
||||
|
||||
expect(result).toEqual({
|
||||
callRecordingId: 'call-recording-1',
|
||||
result: { status: 'skipped', reason: 'meeting already ended' },
|
||||
});
|
||||
expect(fetchMock).not.toHaveBeenCalled();
|
||||
});
|
||||
});
|
||||
+9
@@ -0,0 +1,9 @@
|
||||
export const ACTIVE_RECALL_BOT_STATUSES = [
|
||||
'ready',
|
||||
'joining_call',
|
||||
'in_waiting_room',
|
||||
'in_call_not_recording',
|
||||
'recording_permission_allowed',
|
||||
'recording_permission_denied',
|
||||
'in_call_recording',
|
||||
];
|
||||
+3
-1
@@ -1 +1,3 @@
|
||||
export const PENDING_CALL_RECORDING_REQUESTS_CRON_PATTERN = '*/5 * * * *';
|
||||
// The callRecording.updated trigger resumes most stuck rows within seconds;
|
||||
// this cron is the backstop for crashed creations and missed events.
|
||||
export const PENDING_CALL_RECORDING_REQUESTS_CRON_PATTERN = '*/15 * * * *';
|
||||
|
||||
+10
@@ -21,6 +21,8 @@ type CallRecordingNode = {
|
||||
endedAt?: string | null;
|
||||
calendarEventId?: string | null;
|
||||
externalBotId?: string | null;
|
||||
botScheduleAttemptedAt?: string | null;
|
||||
botScheduleIdempotencyKey?: string | null;
|
||||
externalRecordingId?: string | null;
|
||||
callRecorderFailureReason?: string | null;
|
||||
};
|
||||
@@ -54,6 +56,8 @@ export const findCallRecordingsByFilter = async (
|
||||
endedAt: true,
|
||||
calendarEventId: true,
|
||||
externalBotId: true,
|
||||
botScheduleAttemptedAt: true,
|
||||
botScheduleIdempotencyKey: true,
|
||||
externalRecordingId: true,
|
||||
callRecorderFailureReason: true,
|
||||
},
|
||||
@@ -80,6 +84,12 @@ export const findCallRecordingsByFilter = async (
|
||||
endedAt: callRecording.endedAt ?? undefined,
|
||||
calendarEventId: callRecording.calendarEventId ?? undefined,
|
||||
externalBotId: normalizeOptionalString(callRecording.externalBotId),
|
||||
botScheduleAttemptedAt: normalizeOptionalString(
|
||||
callRecording.botScheduleAttemptedAt,
|
||||
),
|
||||
botScheduleIdempotencyKey: normalizeOptionalString(
|
||||
callRecording.botScheduleIdempotencyKey,
|
||||
),
|
||||
externalRecordingId: normalizeOptionalString(
|
||||
callRecording.externalRecordingId,
|
||||
),
|
||||
|
||||
+54
@@ -0,0 +1,54 @@
|
||||
import { isUndefined } from '@sniptt/guards';
|
||||
|
||||
import { type CalendarEventRecord } from 'src/logic-functions/types/calendar-event-record.type';
|
||||
import { type CallRecordingRecord } from 'src/logic-functions/types/call-recording-record.type';
|
||||
import { hasUnchangedBotScheduleIdempotencyKey } from 'src/logic-functions/domain/has-unchanged-bot-schedule-idempotency-key.util';
|
||||
|
||||
// Recall does not document how long idempotency keys are retained (24h is
|
||||
// the industry minimum), so re-sends are only trusted while the recorded
|
||||
// attempt is clearly fresh; a stale key would create a twin bot instead of
|
||||
// deduping. Recovery normally runs within minutes of the attempt.
|
||||
const IDEMPOTENT_RESEND_WINDOW_HOURS = 12;
|
||||
|
||||
// Rows without a schedule-attempt marker never reached Recall, so no bot can
|
||||
// exist for them. Rows whose stored idempotency key still matches the current
|
||||
// scheduling inputs can re-send the creation and let Recall dedupe it.
|
||||
export const canRescheduleCallRecordingWithoutRecallLookup = ({
|
||||
callRecording,
|
||||
calendarEvent,
|
||||
workspaceId,
|
||||
now,
|
||||
}: {
|
||||
callRecording: CallRecordingRecord;
|
||||
calendarEvent: CalendarEventRecord;
|
||||
workspaceId: string | undefined;
|
||||
now: Date;
|
||||
}): boolean =>
|
||||
isUndefined(callRecording.botScheduleAttemptedAt) ||
|
||||
(isWithinIdempotentResendWindow(callRecording.botScheduleAttemptedAt, now) &&
|
||||
!isUndefined(workspaceId) &&
|
||||
hasUnchangedBotScheduleIdempotencyKey({
|
||||
callRecording,
|
||||
calendarEvent,
|
||||
workspaceId,
|
||||
}));
|
||||
|
||||
const isWithinIdempotentResendWindow = (
|
||||
botScheduleAttemptedAt: string,
|
||||
now: Date,
|
||||
): boolean => {
|
||||
const attemptedTime = new Date(botScheduleAttemptedAt).getTime();
|
||||
|
||||
if (Number.isNaN(attemptedTime)) {
|
||||
return false;
|
||||
}
|
||||
|
||||
const elapsedMilliseconds = now.getTime() - attemptedTime;
|
||||
|
||||
// A future timestamp means clock skew or corrupt data; treat it as
|
||||
// untrustworthy rather than fresh.
|
||||
return (
|
||||
elapsedMilliseconds >= 0 &&
|
||||
elapsedMilliseconds < IDEMPOTENT_RESEND_WINDOW_HOURS * 60 * 60 * 1000
|
||||
);
|
||||
};
|
||||
+45
@@ -0,0 +1,45 @@
|
||||
import { isUndefined } from '@sniptt/guards';
|
||||
|
||||
import { type CalendarEventRecord } from 'src/logic-functions/types/calendar-event-record.type';
|
||||
import { type CallRecordingRecord } from 'src/logic-functions/types/call-recording-record.type';
|
||||
import { buildRecallRoutingMetadata } from 'src/logic-functions/domain/build-recall-routing-metadata.util';
|
||||
import { computeRecallBotJoinAt } from 'src/logic-functions/domain/compute-recall-bot-join-at.util';
|
||||
import { computeRecallBotCreationIdempotencyKey } from 'src/logic-functions/recall-api/schedule-recall-bot.util';
|
||||
|
||||
// True when re-sending the bot creation would carry the same idempotency key
|
||||
// as the recorded attempt, so Recall dedupes it instead of creating a twin.
|
||||
// A moved meeting, changed conference link, or changed join-early setting
|
||||
// drifts the key and recovery must fall back to a bot lookup.
|
||||
export const hasUnchangedBotScheduleIdempotencyKey = ({
|
||||
callRecording,
|
||||
calendarEvent,
|
||||
workspaceId,
|
||||
}: {
|
||||
callRecording: CallRecordingRecord;
|
||||
calendarEvent: CalendarEventRecord;
|
||||
workspaceId: string;
|
||||
}): boolean => {
|
||||
const storedIdempotencyKey = callRecording.botScheduleIdempotencyKey;
|
||||
const meetingUrl = calendarEvent.conferenceLinkUrl;
|
||||
const meetingStartsAt = calendarEvent.startsAt;
|
||||
|
||||
if (
|
||||
isUndefined(storedIdempotencyKey) ||
|
||||
isUndefined(meetingUrl) ||
|
||||
isUndefined(meetingStartsAt)
|
||||
) {
|
||||
return false;
|
||||
}
|
||||
|
||||
return (
|
||||
storedIdempotencyKey ===
|
||||
computeRecallBotCreationIdempotencyKey({
|
||||
meetingUrl,
|
||||
joinAt: computeRecallBotJoinAt(meetingStartsAt),
|
||||
metadata: buildRecallRoutingMetadata({
|
||||
callRecordingId: callRecording.id,
|
||||
workspaceId,
|
||||
}),
|
||||
})
|
||||
);
|
||||
};
|
||||
+2
@@ -281,6 +281,8 @@ describe('reconcileCallRecorderForCalendarEventIds', () => {
|
||||
recordingRequestStatus: 'REQUESTED',
|
||||
calendarEventId: 'calendar-event-1',
|
||||
externalBotId: 'recall-bot-1',
|
||||
botScheduleAttemptedAt: NOW.toISOString(),
|
||||
botScheduleIdempotencyKey: expect.any(String),
|
||||
},
|
||||
]);
|
||||
expect(recallBotCreateCalls()).toHaveLength(1);
|
||||
|
||||
+97
-5
@@ -304,8 +304,8 @@ describe('retryFailedRecallCancellations', () => {
|
||||
expect(listRequestParameters.get('metadata__twentyWorkspaceId')).toBe(
|
||||
WORKSPACE_ID,
|
||||
);
|
||||
expect(listRequestParameters.get('metadata__twentyCallRecordingId')).toBe(
|
||||
'call-recording-1',
|
||||
expect(listRequestParameters.has('metadata__twentyCallRecordingId')).toBe(
|
||||
false,
|
||||
);
|
||||
expect(fetchMock).toHaveBeenCalledWith(
|
||||
`${BASE_URL}/bot/recall-bot-recovered/`,
|
||||
@@ -338,7 +338,15 @@ describe('retryFailedRecallCancellations', () => {
|
||||
...buildJsonResponse(200),
|
||||
json: async () => ({
|
||||
next: null,
|
||||
results: [{ id: 'recall-bot-recovered' }],
|
||||
results: [
|
||||
{
|
||||
id: 'recall-bot-recovered',
|
||||
metadata: {
|
||||
twentyWorkspaceId: WORKSPACE_ID,
|
||||
twentyCallRecordingId: 'call-recording-1',
|
||||
},
|
||||
},
|
||||
],
|
||||
}),
|
||||
})
|
||||
.mockResolvedValueOnce(buildJsonResponse(400))
|
||||
@@ -422,7 +430,15 @@ describe('retryFailedRecallCancellations', () => {
|
||||
...buildJsonResponse(200),
|
||||
json: async () => ({
|
||||
next: null,
|
||||
results: [{ id: 'recall-bot-recovered' }],
|
||||
results: [
|
||||
{
|
||||
id: 'recall-bot-recovered',
|
||||
metadata: {
|
||||
twentyWorkspaceId: WORKSPACE_ID,
|
||||
twentyCallRecordingId: 'call-recording-1',
|
||||
},
|
||||
},
|
||||
],
|
||||
}),
|
||||
};
|
||||
}
|
||||
@@ -467,7 +483,15 @@ describe('retryFailedRecallCancellations', () => {
|
||||
...buildJsonResponse(200),
|
||||
json: async () => ({
|
||||
next: null,
|
||||
results: [{ id: 'recall-bot-recovered' }],
|
||||
results: [
|
||||
{
|
||||
id: 'recall-bot-recovered',
|
||||
metadata: {
|
||||
twentyWorkspaceId: WORKSPACE_ID,
|
||||
twentyCallRecordingId: 'call-recording-1',
|
||||
},
|
||||
},
|
||||
],
|
||||
}),
|
||||
};
|
||||
}
|
||||
@@ -490,6 +514,74 @@ describe('retryFailedRecallCancellations', () => {
|
||||
]);
|
||||
});
|
||||
|
||||
it('does not claim a listed bot for a cancellation whose recovery window elapsed', async () => {
|
||||
const client = new FakeCoreApiClient([
|
||||
{
|
||||
id: 'call-recording-recent',
|
||||
recordingRequestStatus: 'CANCELED',
|
||||
status: 'SCHEDULED',
|
||||
createdAt: '2026-01-02T11:30:00.000Z',
|
||||
calendarEventId: null,
|
||||
externalBotId: null,
|
||||
},
|
||||
{
|
||||
id: 'call-recording-aged',
|
||||
recordingRequestStatus: 'CANCELED',
|
||||
status: 'SCHEDULED',
|
||||
createdAt: '2026-01-01T11:00:00.000Z',
|
||||
calendarEventId: null,
|
||||
externalBotId: null,
|
||||
},
|
||||
]);
|
||||
fetchMock.mockImplementation(
|
||||
async (requestUrl: string, requestInit?: { method?: string }) => {
|
||||
if (requestInit?.method === 'DELETE') {
|
||||
return buildJsonResponse(204);
|
||||
}
|
||||
|
||||
if (requestUrl.startsWith(`${BASE_URL}/bot/?`)) {
|
||||
return {
|
||||
...buildJsonResponse(200),
|
||||
json: async () => ({
|
||||
next: null,
|
||||
results: [
|
||||
{
|
||||
id: 'recall-bot-recent',
|
||||
metadata: {
|
||||
twentyWorkspaceId: WORKSPACE_ID,
|
||||
twentyCallRecordingId: 'call-recording-recent',
|
||||
},
|
||||
},
|
||||
{
|
||||
id: 'recall-bot-aged',
|
||||
metadata: {
|
||||
twentyWorkspaceId: WORKSPACE_ID,
|
||||
twentyCallRecordingId: 'call-recording-aged',
|
||||
},
|
||||
},
|
||||
],
|
||||
}),
|
||||
};
|
||||
}
|
||||
|
||||
throw new Error(`Unhandled fetch: ${requestUrl}`);
|
||||
},
|
||||
);
|
||||
|
||||
const result = await retryFailedRecallCancellations({
|
||||
client: client as unknown as CoreApiClient,
|
||||
now: new Date('2026-01-02T12:00:00.000Z'),
|
||||
});
|
||||
|
||||
expect(result.canceledExternalBotCallRecordingIds).toEqual([
|
||||
'call-recording-recent',
|
||||
]);
|
||||
expect(fetchMock).not.toHaveBeenCalledWith(
|
||||
`${BASE_URL}/bot/recall-bot-aged/`,
|
||||
expect.objectContaining({ method: 'DELETE' }),
|
||||
);
|
||||
});
|
||||
|
||||
it('does not repeatedly look up botless cancellations after their meeting ended', async () => {
|
||||
const client = new FakeCoreApiClient(
|
||||
[
|
||||
|
||||
+252
-5
@@ -1,7 +1,9 @@
|
||||
import { type CoreApiClient } from 'twenty-client-sdk/core';
|
||||
import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest';
|
||||
|
||||
import { computeRecallBotJoinAt } from 'src/logic-functions/domain/compute-recall-bot-join-at.util';
|
||||
import { scheduleRecallBotsForPendingCallRecordings } from 'src/logic-functions/flows/schedule-recall-bots-for-pending-call-recordings.util';
|
||||
import { computeRecallBotCreationIdempotencyKey } from 'src/logic-functions/recall-api/schedule-recall-bot.util';
|
||||
|
||||
const NOW = new Date('2026-01-01T12:00:00.000Z');
|
||||
const WORKSPACE_ID = '123e4567-e89b-12d3-a456-426614174000';
|
||||
@@ -28,6 +30,9 @@ type CallRecordingNode = {
|
||||
recordingRequestStatus?: string | null;
|
||||
calendarEventId?: string | null;
|
||||
externalBotId?: string | null;
|
||||
botScheduleAttemptedAt?: string | null;
|
||||
botScheduleIdempotencyKey?: string | null;
|
||||
callRecorderFailureReason?: string | null;
|
||||
};
|
||||
|
||||
type CalendarEventNode = {
|
||||
@@ -216,6 +221,11 @@ describe('scheduleRecallBotsForPendingCallRecordings', () => {
|
||||
|
||||
expect(result.scheduledCallRecordingIds).toEqual(['call-recording-1']);
|
||||
expect(result.attachedCallRecordingIds).toEqual([]);
|
||||
// A row that never attempted a bot creation needs no Recall lookup.
|
||||
expect(listBotRequestUrls()).toHaveLength(0);
|
||||
expect(client.callRecordings[0].botScheduleAttemptedAt).toBe(
|
||||
NOW.toISOString(),
|
||||
);
|
||||
expect(createBotCalls()).toHaveLength(1);
|
||||
const [requestUrl, requestInit] = createBotCalls()[0];
|
||||
expect(requestUrl).toBe(RECALL_CREATE_BOT_URL);
|
||||
@@ -246,7 +256,11 @@ describe('scheduleRecallBotsForPendingCallRecordings', () => {
|
||||
],
|
||||
});
|
||||
const client = new FakeCoreApiClient({
|
||||
callRecordings: [buildPendingCallRecording()],
|
||||
callRecordings: [
|
||||
buildPendingCallRecording({
|
||||
botScheduleAttemptedAt: '2026-01-01T11:55:00.000Z',
|
||||
}),
|
||||
],
|
||||
calendarEvents: [buildCalendarEvent()],
|
||||
});
|
||||
|
||||
@@ -262,8 +276,8 @@ describe('scheduleRecallBotsForPendingCallRecordings', () => {
|
||||
expect(lookupParameters.get('metadata__twentyWorkspaceId')).toBe(
|
||||
WORKSPACE_ID,
|
||||
);
|
||||
expect(lookupParameters.get('metadata__twentyCallRecordingId')).toBe(
|
||||
'call-recording-1',
|
||||
expect(lookupParameters.has('metadata__twentyCallRecordingId')).toBe(
|
||||
false,
|
||||
);
|
||||
expect(lookupParameters.has('join_at_after')).toBe(false);
|
||||
expect(lookupParameters.has('join_at_before')).toBe(false);
|
||||
@@ -279,10 +293,54 @@ describe('scheduleRecallBotsForPendingCallRecordings', () => {
|
||||
expect(client.callRecordings[0].externalBotId).toBe('recall-bot-existing');
|
||||
});
|
||||
|
||||
it('looks up existing bots once for the whole run instead of per recording', async () => {
|
||||
stubRecallApi({
|
||||
listedBots: [
|
||||
{
|
||||
id: 'recall-bot-existing',
|
||||
metadata: {
|
||||
twentyWorkspaceId: WORKSPACE_ID,
|
||||
twentyCallRecordingId: 'call-recording-1',
|
||||
},
|
||||
},
|
||||
],
|
||||
});
|
||||
const client = new FakeCoreApiClient({
|
||||
callRecordings: [
|
||||
buildPendingCallRecording({
|
||||
botScheduleAttemptedAt: '2026-01-01T11:55:00.000Z',
|
||||
}),
|
||||
buildPendingCallRecording({
|
||||
id: 'call-recording-2',
|
||||
calendarEventId: 'calendar-event-2',
|
||||
botScheduleAttemptedAt: '2026-01-01T11:55:00.000Z',
|
||||
}),
|
||||
],
|
||||
calendarEvents: [
|
||||
buildCalendarEvent(),
|
||||
buildCalendarEvent({ id: 'calendar-event-2' }),
|
||||
],
|
||||
});
|
||||
|
||||
const result = await scheduleRecallBotsForPendingCallRecordings({
|
||||
client: client as unknown as CoreApiClient,
|
||||
now: NOW,
|
||||
});
|
||||
|
||||
expect(listBotRequestUrls()).toHaveLength(1);
|
||||
expect(result.attachedCallRecordingIds).toEqual(['call-recording-1']);
|
||||
expect(result.scheduledCallRecordingIds).toEqual(['call-recording-2']);
|
||||
expect(createBotCalls()).toHaveLength(1);
|
||||
});
|
||||
|
||||
it('defers scheduling when the existing-bot lookup fails so no duplicate bot is created', async () => {
|
||||
stubRecallApi({ listStatus: 400 });
|
||||
const client = new FakeCoreApiClient({
|
||||
callRecordings: [buildPendingCallRecording()],
|
||||
callRecordings: [
|
||||
buildPendingCallRecording({
|
||||
botScheduleAttemptedAt: '2026-01-01T11:55:00.000Z',
|
||||
}),
|
||||
],
|
||||
calendarEvents: [buildCalendarEvent()],
|
||||
});
|
||||
|
||||
@@ -297,6 +355,121 @@ describe('scheduleRecallBotsForPendingCallRecordings', () => {
|
||||
expect(client.callRecordings[0].externalBotId).toBeNull();
|
||||
});
|
||||
|
||||
it('re-sends the creation without any Recall lookup when the stored idempotency key still matches', async () => {
|
||||
const unchangedIdempotencyKey = computeRecallBotCreationIdempotencyKey({
|
||||
meetingUrl: 'https://meet.example.com/customer-sync',
|
||||
joinAt: computeRecallBotJoinAt(UPCOMING_STARTS_AT),
|
||||
metadata: {
|
||||
twentyWorkspaceId: WORKSPACE_ID,
|
||||
twentyCallRecordingId: 'call-recording-1',
|
||||
},
|
||||
});
|
||||
const client = new FakeCoreApiClient({
|
||||
callRecordings: [
|
||||
buildPendingCallRecording({
|
||||
botScheduleAttemptedAt: '2026-01-01T11:55:00.000Z',
|
||||
botScheduleIdempotencyKey: unchangedIdempotencyKey,
|
||||
}),
|
||||
],
|
||||
calendarEvents: [buildCalendarEvent()],
|
||||
});
|
||||
|
||||
const result = await scheduleRecallBotsForPendingCallRecordings({
|
||||
client: client as unknown as CoreApiClient,
|
||||
now: NOW,
|
||||
});
|
||||
|
||||
expect(listBotRequestUrls()).toHaveLength(0);
|
||||
expect(result.scheduledCallRecordingIds).toEqual(['call-recording-1']);
|
||||
const [, requestInit] = createBotCalls()[0];
|
||||
expect(requestInit.headers).toMatchObject({
|
||||
'Idempotency-Key': unchangedIdempotencyKey,
|
||||
});
|
||||
expect(client.callRecordings[0].externalBotId).toBe('recall-bot-1');
|
||||
// The first attempt timestamp survives the re-send so repeated unknown
|
||||
// outcomes age out of the resend window.
|
||||
expect(client.callRecordings[0].botScheduleAttemptedAt).toBe(
|
||||
'2026-01-01T11:55:00.000Z',
|
||||
);
|
||||
});
|
||||
|
||||
it('falls back to the Recall lookup when the recorded attempt is too old to trust its idempotency key', async () => {
|
||||
const unchangedIdempotencyKey = computeRecallBotCreationIdempotencyKey({
|
||||
meetingUrl: 'https://meet.example.com/customer-sync',
|
||||
joinAt: computeRecallBotJoinAt(UPCOMING_STARTS_AT),
|
||||
metadata: {
|
||||
twentyWorkspaceId: WORKSPACE_ID,
|
||||
twentyCallRecordingId: 'call-recording-1',
|
||||
},
|
||||
});
|
||||
const client = new FakeCoreApiClient({
|
||||
callRecordings: [
|
||||
buildPendingCallRecording({
|
||||
botScheduleAttemptedAt: '2025-12-30T12:00:00.000Z',
|
||||
botScheduleIdempotencyKey: unchangedIdempotencyKey,
|
||||
}),
|
||||
],
|
||||
calendarEvents: [buildCalendarEvent()],
|
||||
});
|
||||
|
||||
const result = await scheduleRecallBotsForPendingCallRecordings({
|
||||
client: client as unknown as CoreApiClient,
|
||||
now: NOW,
|
||||
});
|
||||
|
||||
expect(listBotRequestUrls()).toHaveLength(1);
|
||||
expect(result.scheduledCallRecordingIds).toEqual(['call-recording-1']);
|
||||
});
|
||||
|
||||
it('falls back to the Recall lookup when the meeting moved since the recorded attempt', async () => {
|
||||
const staleIdempotencyKey = computeRecallBotCreationIdempotencyKey({
|
||||
meetingUrl: 'https://meet.example.com/customer-sync',
|
||||
joinAt: computeRecallBotJoinAt('2026-01-01T09:00:00.000Z'),
|
||||
metadata: {
|
||||
twentyWorkspaceId: WORKSPACE_ID,
|
||||
twentyCallRecordingId: 'call-recording-1',
|
||||
},
|
||||
});
|
||||
const client = new FakeCoreApiClient({
|
||||
callRecordings: [
|
||||
buildPendingCallRecording({
|
||||
botScheduleAttemptedAt: '2026-01-01T11:55:00.000Z',
|
||||
botScheduleIdempotencyKey: staleIdempotencyKey,
|
||||
}),
|
||||
],
|
||||
calendarEvents: [buildCalendarEvent()],
|
||||
});
|
||||
|
||||
const result = await scheduleRecallBotsForPendingCallRecordings({
|
||||
client: client as unknown as CoreApiClient,
|
||||
now: NOW,
|
||||
});
|
||||
|
||||
expect(listBotRequestUrls()).toHaveLength(1);
|
||||
expect(result.scheduledCallRecordingIds).toEqual(['call-recording-1']);
|
||||
});
|
||||
|
||||
it('re-schedules an ambiguous recording when the lookup confirms no bot exists', async () => {
|
||||
const client = new FakeCoreApiClient({
|
||||
callRecordings: [
|
||||
buildPendingCallRecording({
|
||||
botScheduleAttemptedAt: '2026-01-01T11:55:00.000Z',
|
||||
}),
|
||||
],
|
||||
calendarEvents: [buildCalendarEvent()],
|
||||
});
|
||||
|
||||
const result = await scheduleRecallBotsForPendingCallRecordings({
|
||||
client: client as unknown as CoreApiClient,
|
||||
now: NOW,
|
||||
});
|
||||
|
||||
expect(listBotRequestUrls()).toHaveLength(1);
|
||||
expect(result.scheduledCallRecordingIds).toEqual(['call-recording-1']);
|
||||
expect(createBotCalls()).toHaveLength(1);
|
||||
expect(client.callRecordings[0].externalBotId).toBe('recall-bot-1');
|
||||
});
|
||||
|
||||
it('does not report a recording as scheduled when Recall scheduling fails', async () => {
|
||||
stubRecallApi({ createBotStatus: 500 });
|
||||
const client = new FakeCoreApiClient({
|
||||
@@ -318,7 +491,7 @@ describe('scheduleRecallBotsForPendingCallRecordings', () => {
|
||||
expect(client.callRecordings[0].externalBotId).toBeNull();
|
||||
});
|
||||
|
||||
it('skips a recording whose meeting has already ended', async () => {
|
||||
it('marks a recording failed when its meeting ended before any bot was scheduled', async () => {
|
||||
const client = new FakeCoreApiClient({
|
||||
callRecordings: [buildPendingCallRecording()],
|
||||
calendarEvents: [
|
||||
@@ -335,7 +508,81 @@ describe('scheduleRecallBotsForPendingCallRecordings', () => {
|
||||
});
|
||||
|
||||
expect(result.scheduledCallRecordingIds).toEqual([]);
|
||||
expect(result.markedFailedCallRecordingIds).toEqual(['call-recording-1']);
|
||||
expect(fetchMock).not.toHaveBeenCalled();
|
||||
expect(client.callRecordings[0].status).toBe('FAILED');
|
||||
expect(client.callRecordings[0].callRecorderFailureReason).toBe(
|
||||
'bot_never_scheduled',
|
||||
);
|
||||
});
|
||||
|
||||
it('keeps an ended recording with an unresolved attempt pending while convergence may still resolve it', async () => {
|
||||
const client = new FakeCoreApiClient({
|
||||
callRecordings: [
|
||||
buildPendingCallRecording({
|
||||
botScheduleAttemptedAt: '2026-01-01T09:55:00.000Z',
|
||||
}),
|
||||
],
|
||||
calendarEvents: [
|
||||
buildCalendarEvent({
|
||||
startsAt: PAST_STARTS_AT,
|
||||
endsAt: PAST_ENDS_AT,
|
||||
}),
|
||||
],
|
||||
});
|
||||
|
||||
const result = await scheduleRecallBotsForPendingCallRecordings({
|
||||
client: client as unknown as CoreApiClient,
|
||||
now: NOW,
|
||||
});
|
||||
|
||||
expect(result.markedFailedCallRecordingIds).toEqual([]);
|
||||
expect(fetchMock).not.toHaveBeenCalled();
|
||||
expect(client.callRecordings[0].status).toBe('SCHEDULED');
|
||||
});
|
||||
|
||||
it('fails an ended recording with an unresolved attempt once the convergence lookback has passed', async () => {
|
||||
const client = new FakeCoreApiClient({
|
||||
callRecordings: [
|
||||
buildPendingCallRecording({
|
||||
botScheduleAttemptedAt: '2025-12-20T09:55:00.000Z',
|
||||
}),
|
||||
],
|
||||
calendarEvents: [
|
||||
buildCalendarEvent({
|
||||
startsAt: '2025-12-20T10:00:00.000Z',
|
||||
endsAt: '2025-12-20T11:00:00.000Z',
|
||||
}),
|
||||
],
|
||||
});
|
||||
|
||||
const result = await scheduleRecallBotsForPendingCallRecordings({
|
||||
client: client as unknown as CoreApiClient,
|
||||
now: NOW,
|
||||
});
|
||||
|
||||
expect(result.markedFailedCallRecordingIds).toEqual(['call-recording-1']);
|
||||
expect(fetchMock).not.toHaveBeenCalled();
|
||||
expect(client.callRecordings[0].status).toBe('FAILED');
|
||||
expect(client.callRecordings[0].callRecorderFailureReason).toBe(
|
||||
'bot_schedule_outcome_unknown',
|
||||
);
|
||||
});
|
||||
|
||||
it('leaves a recording untouched when its calendar event is missing', async () => {
|
||||
const client = new FakeCoreApiClient({
|
||||
callRecordings: [buildPendingCallRecording()],
|
||||
calendarEvents: [],
|
||||
});
|
||||
|
||||
const result = await scheduleRecallBotsForPendingCallRecordings({
|
||||
client: client as unknown as CoreApiClient,
|
||||
now: NOW,
|
||||
});
|
||||
|
||||
expect(result.markedFailedCallRecordingIds).toEqual([]);
|
||||
expect(fetchMock).not.toHaveBeenCalled();
|
||||
expect(client.callRecordings[0].status).toBe('SCHEDULED');
|
||||
});
|
||||
|
||||
it('does nothing when every scheduled recording already has a bot', async () => {
|
||||
|
||||
-44
@@ -1,44 +0,0 @@
|
||||
import { isUndefined } from '@sniptt/guards';
|
||||
import { type CoreApiClient } from 'twenty-client-sdk/core';
|
||||
|
||||
import { type CallRecordingRecord } from 'src/logic-functions/types/call-recording-record.type';
|
||||
import { findScheduledRecallBotIdForCallRecording } from 'src/logic-functions/recall-api/find-scheduled-recall-bot-id-for-call-recording.util';
|
||||
import { getCurrentWorkspaceId } from 'src/logic-functions/data/get-current-workspace-id.util';
|
||||
import { updateCallRecording } from 'src/logic-functions/data/update-call-recording.util';
|
||||
|
||||
export type AttachExistingRecallBotToCallRecordingResult =
|
||||
| { status: 'attached'; externalBotId: string }
|
||||
| { status: 'no-existing-bot' }
|
||||
| { status: 'lookup-failed' };
|
||||
|
||||
// A run that POSTed a bot but died before the id write-back leaves the bot claimable by metadata; attaching it instead of re-POSTing prevents duplicate bots.
|
||||
export const attachExistingRecallBotToCallRecording = async (
|
||||
client: CoreApiClient,
|
||||
{ callRecording }: { callRecording: CallRecordingRecord },
|
||||
): Promise<AttachExistingRecallBotToCallRecordingResult> => {
|
||||
const workspaceId = getCurrentWorkspaceId();
|
||||
|
||||
if (isUndefined(workspaceId)) {
|
||||
return { status: 'no-existing-bot' };
|
||||
}
|
||||
|
||||
const findResult = await findScheduledRecallBotIdForCallRecording({
|
||||
callRecordingId: callRecording.id,
|
||||
workspaceId,
|
||||
});
|
||||
|
||||
if (!findResult.ok) {
|
||||
return { status: 'lookup-failed' };
|
||||
}
|
||||
|
||||
if (isUndefined(findResult.externalBotId)) {
|
||||
return { status: 'no-existing-bot' };
|
||||
}
|
||||
|
||||
await updateCallRecording(client, {
|
||||
id: callRecording.id,
|
||||
data: { externalBotId: findResult.externalBotId },
|
||||
});
|
||||
|
||||
return { status: 'attached', externalBotId: findResult.externalBotId };
|
||||
};
|
||||
+9
-2
@@ -53,11 +53,18 @@ export const rescheduleCallRecordingBot = async (
|
||||
return;
|
||||
}
|
||||
|
||||
// The bot vanished externally; drop the id so the stale-state cron re-creates it as the single writer.
|
||||
// The bot vanished externally; drop the id so recovery re-creates it as the
|
||||
// single writer. The recorded attempt state is resolved (its bot is
|
||||
// confirmed gone), so clearing it lets recovery schedule directly instead
|
||||
// of treating the row as an ambiguous attempt.
|
||||
if (rescheduleResult.status === RECALL_BOT_NOT_FOUND_STATUS) {
|
||||
await updateCallRecording(client, {
|
||||
id: callRecording.id,
|
||||
data: { externalBotId: null },
|
||||
data: {
|
||||
externalBotId: null,
|
||||
botScheduleAttemptedAt: null,
|
||||
botScheduleIdempotencyKey: null,
|
||||
},
|
||||
});
|
||||
|
||||
return;
|
||||
|
||||
+112
@@ -0,0 +1,112 @@
|
||||
import { isUndefined } from '@sniptt/guards';
|
||||
import { type CoreApiClient } from 'twenty-client-sdk/core';
|
||||
|
||||
import { CallRecordingRequestStatus } from 'src/logic-functions/constants/call-recording-request-status';
|
||||
import { CallRecordingStatus } from 'src/logic-functions/constants/call-recording-status';
|
||||
import { type CalendarEventRecord } from 'src/logic-functions/types/calendar-event-record.type';
|
||||
import { type CallRecordingRecord } from 'src/logic-functions/types/call-recording-record.type';
|
||||
import { canRescheduleCallRecordingWithoutRecallLookup } from 'src/logic-functions/domain/can-reschedule-call-recording-without-recall-lookup.util';
|
||||
import { fetchCalendarEventsByIds } from 'src/logic-functions/data/fetch-calendar-events-by-ids.util';
|
||||
import { findCallRecordingsByIds } from 'src/logic-functions/data/find-call-recordings-by-ids.util';
|
||||
import { getCurrentWorkspaceId } from 'src/logic-functions/data/get-current-workspace-id.util';
|
||||
import { hasMeetingEnded } from 'src/logic-functions/domain/has-meeting-ended.util';
|
||||
import { scheduleRecallBotForCallRecording } from 'src/logic-functions/flows/schedule-recall-bot-for-call-recording.util';
|
||||
|
||||
export type ResumePendingCallRecordingResult =
|
||||
| { status: 'scheduled' }
|
||||
| { status: 'skipped'; reason: string }
|
||||
| { status: 'deferred'; reason: string };
|
||||
|
||||
// Single-recording variant of the recovery sweep: finishes bot scheduling for
|
||||
// one row that transitioned back to pending. Deferred outcomes are retried by
|
||||
// the queue and ultimately by the recovery cron.
|
||||
export const resumePendingCallRecording = async ({
|
||||
client,
|
||||
callRecordingId,
|
||||
now,
|
||||
}: {
|
||||
client: CoreApiClient;
|
||||
callRecordingId: string;
|
||||
now: Date;
|
||||
}): Promise<ResumePendingCallRecordingResult> => {
|
||||
const callRecording = (
|
||||
await findCallRecordingsByIds(client, [callRecordingId])
|
||||
)[0];
|
||||
|
||||
if (isUndefined(callRecording)) {
|
||||
return { status: 'skipped', reason: 'call recording not found' };
|
||||
}
|
||||
|
||||
if (
|
||||
callRecording.recordingRequestStatus !==
|
||||
CallRecordingRequestStatus.REQUESTED ||
|
||||
callRecording.status !== CallRecordingStatus.SCHEDULED ||
|
||||
!isUndefined(callRecording.externalBotId)
|
||||
) {
|
||||
return { status: 'skipped', reason: 'call recording is not pending' };
|
||||
}
|
||||
|
||||
if (isUndefined(callRecording.calendarEventId)) {
|
||||
return { status: 'skipped', reason: 'no calendar event attached' };
|
||||
}
|
||||
|
||||
const calendarEvent = (
|
||||
await fetchCalendarEventsByIds(client, [callRecording.calendarEventId])
|
||||
)[0];
|
||||
|
||||
if (isUndefined(calendarEvent)) {
|
||||
return { status: 'skipped', reason: 'calendar event not found' };
|
||||
}
|
||||
|
||||
if (
|
||||
hasMeetingEnded({
|
||||
startsAt: calendarEvent.startsAt,
|
||||
endsAt: calendarEvent.endsAt,
|
||||
now,
|
||||
})
|
||||
) {
|
||||
// The recovery cron owns failing rows whose meeting is over.
|
||||
return { status: 'skipped', reason: 'meeting already ended' };
|
||||
}
|
||||
|
||||
if (
|
||||
canRescheduleCallRecordingWithoutRecallLookup({
|
||||
callRecording,
|
||||
calendarEvent,
|
||||
workspaceId: getCurrentWorkspaceId(),
|
||||
now,
|
||||
})
|
||||
) {
|
||||
return scheduleBot({ client, callRecording, calendarEvent });
|
||||
}
|
||||
|
||||
// Ambiguous attempts need a Recall bot lookup; doing one per event could
|
||||
// burst past the shared list budget, so the recovery cron owns them and
|
||||
// amortizes a single lookup across all ambiguous rows.
|
||||
return {
|
||||
status: 'deferred',
|
||||
reason: 'ambiguous prior attempt; the recovery cron will reconcile it',
|
||||
};
|
||||
};
|
||||
|
||||
const scheduleBot = async ({
|
||||
client,
|
||||
callRecording,
|
||||
calendarEvent,
|
||||
}: {
|
||||
client: CoreApiClient;
|
||||
callRecording: CallRecordingRecord;
|
||||
calendarEvent: CalendarEventRecord;
|
||||
}): Promise<ResumePendingCallRecordingResult> => {
|
||||
const didScheduleRecallBot = await scheduleRecallBotForCallRecording(
|
||||
client,
|
||||
{
|
||||
callRecording,
|
||||
calendarEvent,
|
||||
},
|
||||
);
|
||||
|
||||
return didScheduleRecallBot
|
||||
? { status: 'scheduled' }
|
||||
: { status: 'deferred', reason: 'Recall bot scheduling failed' };
|
||||
};
|
||||
+73
-55
@@ -7,10 +7,9 @@ import { NON_TERMINAL_CALL_RECORDING_STATUSES } from 'src/logic-functions/consta
|
||||
import { fetchCalendarEventsByIds } from 'src/logic-functions/data/fetch-calendar-events-by-ids.util';
|
||||
import { findCallRecordingsByFilter } from 'src/logic-functions/data/find-call-recordings-by-filter.util';
|
||||
import { findCallRecordingsByIds } from 'src/logic-functions/data/find-call-recordings-by-ids.util';
|
||||
import { getCurrentWorkspaceId } from 'src/logic-functions/data/get-current-workspace-id.util';
|
||||
import { replaceCanceledCallRecordingExternalBotId } from 'src/logic-functions/data/replace-canceled-call-recording-external-bot-id.util';
|
||||
import { cancelOrEjectRecallBot } from 'src/logic-functions/recall-api/cancel-or-eject-recall-bot.util';
|
||||
import { findScheduledRecallBotIdForCallRecording } from 'src/logic-functions/recall-api/find-scheduled-recall-bot-id-for-call-recording.util';
|
||||
import { findScheduledRecallBotIdsByCallRecordingId } from 'src/logic-functions/recall-api/find-scheduled-recall-bot-ids-by-call-recording-id.util';
|
||||
import { type CalendarEventRecord } from 'src/logic-functions/types/calendar-event-record.type';
|
||||
import { type CallRecordingRecord } from 'src/logic-functions/types/call-recording-record.type';
|
||||
import { getUniqueSortedIds } from 'src/logic-functions/utils/get-unique-sorted-ids.util';
|
||||
@@ -49,17 +48,32 @@ export const retryFailedRecallCancellations = async ({
|
||||
)
|
||||
).map((calendarEvent) => [calendarEvent.id, calendarEvent]),
|
||||
);
|
||||
const recoverableCallRecordingIds = new Set(
|
||||
canceledCallRecordings
|
||||
.filter(
|
||||
(callRecording) =>
|
||||
isUndefined(callRecording.externalBotId) &&
|
||||
isWithinCanceledBotRecoveryWindow({
|
||||
callRecording,
|
||||
calendarEvent: isUndefined(callRecording.calendarEventId)
|
||||
? undefined
|
||||
: calendarEventsById.get(callRecording.calendarEventId),
|
||||
now,
|
||||
}),
|
||||
)
|
||||
.map((callRecording) => callRecording.id),
|
||||
);
|
||||
const externalBotIdByCallRecordingId =
|
||||
await lookupRecoverableExternalBotIds(recoverableCallRecordingIds);
|
||||
const canceledExternalBotCallRecordingIds: string[] = [];
|
||||
|
||||
for (const callRecording of canceledCallRecordings) {
|
||||
const calendarEvent = isUndefined(callRecording.calendarEventId)
|
||||
? undefined
|
||||
: calendarEventsById.get(callRecording.calendarEventId);
|
||||
const externalBotId = await recoverRecallBotIdForCanceledCallRecording({
|
||||
client,
|
||||
callRecording,
|
||||
calendarEvent,
|
||||
now,
|
||||
listedExternalBotId: recoverableCallRecordingIds.has(callRecording.id)
|
||||
? externalBotIdByCallRecordingId?.get(callRecording.id)
|
||||
: undefined,
|
||||
});
|
||||
|
||||
if (isUndefined(externalBotId)) {
|
||||
@@ -98,21 +112,31 @@ export const retryFailedRecallCancellations = async ({
|
||||
return { canceledExternalBotCallRecordingIds };
|
||||
};
|
||||
|
||||
const recoverRecallBotIdForCanceledCallRecording = async ({
|
||||
client,
|
||||
// One workspace-wide list request covers every recoverable row; undefined
|
||||
// means the lookup failed and recovery must wait for the next run.
|
||||
const lookupRecoverableExternalBotIds = async (
|
||||
recoverableCallRecordingIds: Set<string>,
|
||||
): Promise<Map<string, string> | undefined> => {
|
||||
if (recoverableCallRecordingIds.size === 0) {
|
||||
return new Map();
|
||||
}
|
||||
|
||||
const lookupResult = await findScheduledRecallBotIdsByCallRecordingId();
|
||||
|
||||
return lookupResult.ok
|
||||
? lookupResult.externalBotIdByCallRecordingId
|
||||
: undefined;
|
||||
};
|
||||
|
||||
const isWithinCanceledBotRecoveryWindow = ({
|
||||
callRecording,
|
||||
calendarEvent,
|
||||
now,
|
||||
}: {
|
||||
client: CoreApiClient;
|
||||
callRecording: CallRecordingRecord;
|
||||
calendarEvent: CalendarEventRecord | undefined;
|
||||
now: Date;
|
||||
}): Promise<string | undefined> => {
|
||||
if (!isUndefined(callRecording.externalBotId)) {
|
||||
return callRecording.externalBotId;
|
||||
}
|
||||
|
||||
}): boolean => {
|
||||
if (
|
||||
!isUndefined(calendarEvent) &&
|
||||
hasMeetingEnded({
|
||||
@@ -122,50 +146,15 @@ const recoverRecallBotIdForCanceledCallRecording = async ({
|
||||
startGraceHours: CANCELED_BOT_RECOVERY_AFTER_START_HOURS,
|
||||
})
|
||||
) {
|
||||
return undefined;
|
||||
return false;
|
||||
}
|
||||
|
||||
// Recovery only closes the crash window right after cancellation; once a row ages out the daily cleanup sweep owns it, so stop the per-run Recall lookup instead of listing forever (notably for rows whose calendar event was deleted and can no longer bound the retry).
|
||||
// updatedAt tracks the cancellation write, so a request scheduled far ahead but canceled recently still gets its window; createdAt would age it out from scheduling time.
|
||||
if (
|
||||
hasCanceledRecoveryWindowElapsed({
|
||||
canceledAt: callRecording.updatedAt ?? callRecording.createdAt,
|
||||
now,
|
||||
})
|
||||
) {
|
||||
return undefined;
|
||||
}
|
||||
|
||||
const currentWorkspaceId = getCurrentWorkspaceId();
|
||||
|
||||
if (isUndefined(currentWorkspaceId)) {
|
||||
return undefined;
|
||||
}
|
||||
|
||||
const scheduledRecallBotLookupResult =
|
||||
await findScheduledRecallBotIdForCallRecording({
|
||||
callRecordingId: callRecording.id,
|
||||
workspaceId: currentWorkspaceId,
|
||||
});
|
||||
|
||||
if (
|
||||
!scheduledRecallBotLookupResult.ok ||
|
||||
isUndefined(scheduledRecallBotLookupResult.externalBotId)
|
||||
) {
|
||||
return undefined;
|
||||
}
|
||||
|
||||
const externalBotId = scheduledRecallBotLookupResult.externalBotId;
|
||||
const didClaimRecoveredBot = await replaceCanceledCallRecordingExternalBotId(
|
||||
client,
|
||||
{
|
||||
id: callRecording.id,
|
||||
expectedExternalBotId: null,
|
||||
nextExternalBotId: externalBotId,
|
||||
},
|
||||
);
|
||||
|
||||
return didClaimRecoveredBot ? externalBotId : undefined;
|
||||
return !hasCanceledRecoveryWindowElapsed({
|
||||
canceledAt: callRecording.updatedAt ?? callRecording.createdAt,
|
||||
now,
|
||||
});
|
||||
};
|
||||
|
||||
const hasCanceledRecoveryWindowElapsed = ({
|
||||
@@ -187,3 +176,32 @@ const hasCanceledRecoveryWindowElapsed = ({
|
||||
now.getTime()
|
||||
);
|
||||
};
|
||||
|
||||
const recoverRecallBotIdForCanceledCallRecording = async ({
|
||||
client,
|
||||
callRecording,
|
||||
listedExternalBotId,
|
||||
}: {
|
||||
client: CoreApiClient;
|
||||
callRecording: CallRecordingRecord;
|
||||
listedExternalBotId: string | undefined;
|
||||
}): Promise<string | undefined> => {
|
||||
if (!isUndefined(callRecording.externalBotId)) {
|
||||
return callRecording.externalBotId;
|
||||
}
|
||||
|
||||
if (isUndefined(listedExternalBotId)) {
|
||||
return undefined;
|
||||
}
|
||||
|
||||
const didClaimRecoveredBot = await replaceCanceledCallRecordingExternalBotId(
|
||||
client,
|
||||
{
|
||||
id: callRecording.id,
|
||||
expectedExternalBotId: null,
|
||||
nextExternalBotId: listedExternalBotId,
|
||||
},
|
||||
);
|
||||
|
||||
return didClaimRecoveredBot ? listedExternalBotId : undefined;
|
||||
};
|
||||
|
||||
+37
-5
@@ -2,13 +2,17 @@ import { isUndefined } from '@sniptt/guards';
|
||||
import { type CoreApiClient } from 'twenty-client-sdk/core';
|
||||
|
||||
import { CallRecordingRequestStatus } from 'src/logic-functions/constants/call-recording-request-status';
|
||||
import { CallRecordingStatus } from 'src/logic-functions/constants/call-recording-status';
|
||||
import { type MeetingRecording } from 'src/logic-functions/types/meeting-recording.type';
|
||||
import { buildRecallBotAutomaticVideoOutput } from 'src/logic-functions/domain/build-recall-bot-automatic-video-output.util';
|
||||
import { buildRecallRoutingMetadata } from 'src/logic-functions/domain/build-recall-routing-metadata.util';
|
||||
import { computeRecallBotJoinAt } from 'src/logic-functions/domain/compute-recall-bot-join-at.util';
|
||||
import { findCallRecordingsByIds } from 'src/logic-functions/data/find-call-recordings-by-ids.util';
|
||||
import { getCurrentWorkspaceId } from 'src/logic-functions/data/get-current-workspace-id.util';
|
||||
import { scheduleRecallBot } from 'src/logic-functions/recall-api/schedule-recall-bot.util';
|
||||
import {
|
||||
computeRecallBotCreationIdempotencyKey,
|
||||
scheduleRecallBot,
|
||||
} from 'src/logic-functions/recall-api/schedule-recall-bot.util';
|
||||
import { updateCallRecording } from 'src/logic-functions/data/update-call-recording.util';
|
||||
|
||||
// The sole place a Recall bot is created. Only the deterministic-create winner and the stale-state cron call it, so one writer per meeting POSTs exactly one bot.
|
||||
@@ -33,6 +37,7 @@ export const scheduleRecallBotForCallRecording = async (
|
||||
isUndefined(freshCallRecording) ||
|
||||
freshCallRecording.recordingRequestStatus !==
|
||||
CallRecordingRequestStatus.REQUESTED ||
|
||||
freshCallRecording.status !== CallRecordingStatus.SCHEDULED ||
|
||||
!isUndefined(freshCallRecording.externalBotId)
|
||||
) {
|
||||
return false;
|
||||
@@ -49,15 +54,42 @@ export const scheduleRecallBotForCallRecording = async (
|
||||
}
|
||||
|
||||
const automaticVideoOutput = await buildRecallBotAutomaticVideoOutput();
|
||||
const metadata = buildRecallRoutingMetadata({
|
||||
callRecordingId: callRecording.id,
|
||||
workspaceId,
|
||||
});
|
||||
const idempotencyKey = computeRecallBotCreationIdempotencyKey({
|
||||
meetingUrl,
|
||||
joinAt,
|
||||
metadata,
|
||||
});
|
||||
|
||||
// Persisted before the POST so a crash leaves proof that a bot creation may
|
||||
// have reached Recall; while the stored key still matches the scheduling
|
||||
// inputs, recovery can re-send the creation idempotently instead of asking
|
||||
// Recall whether a bot already exists. Re-sends of the same key keep the
|
||||
// first attempt's timestamp so repeated unknown outcomes age out of the
|
||||
// resend window instead of staying trusted forever.
|
||||
const recordedAttemptTimestamp =
|
||||
freshCallRecording.botScheduleIdempotencyKey === idempotencyKey
|
||||
? freshCallRecording.botScheduleAttemptedAt
|
||||
: undefined;
|
||||
|
||||
await updateCallRecording(client, {
|
||||
id: callRecording.id,
|
||||
data: {
|
||||
botScheduleAttemptedAt:
|
||||
recordedAttemptTimestamp ?? new Date().toISOString(),
|
||||
botScheduleIdempotencyKey: idempotencyKey,
|
||||
},
|
||||
});
|
||||
|
||||
const scheduleResult = await scheduleRecallBot({
|
||||
meetingUrl,
|
||||
joinAt,
|
||||
metadata: buildRecallRoutingMetadata({
|
||||
callRecordingId: callRecording.id,
|
||||
workspaceId,
|
||||
}),
|
||||
metadata,
|
||||
automaticVideoOutput,
|
||||
idempotencyKey,
|
||||
});
|
||||
|
||||
if (!scheduleResult.ok) {
|
||||
|
||||
+227
-31
@@ -1,16 +1,36 @@
|
||||
import { isUndefined } from '@sniptt/guards';
|
||||
import { type CoreApiClient } from 'twenty-client-sdk/core';
|
||||
|
||||
import { CallRecordingStatus } from 'src/logic-functions/constants/call-recording-status';
|
||||
import { type CalendarEventRecord } from 'src/logic-functions/types/calendar-event-record.type';
|
||||
import { type CallRecordingRecord } from 'src/logic-functions/types/call-recording-record.type';
|
||||
import { canRescheduleCallRecordingWithoutRecallLookup } from 'src/logic-functions/domain/can-reschedule-call-recording-without-recall-lookup.util';
|
||||
import { getCurrentWorkspaceId } from 'src/logic-functions/data/get-current-workspace-id.util';
|
||||
import { hasMeetingEnded } from 'src/logic-functions/domain/has-meeting-ended.util';
|
||||
import { attachExistingRecallBotToCallRecording } from 'src/logic-functions/flows/attach-existing-recall-bot-to-call-recording.util';
|
||||
import { scheduleRecallBotForCallRecording } from 'src/logic-functions/flows/schedule-recall-bot-for-call-recording.util';
|
||||
import { fetchCalendarEventsByIds } from 'src/logic-functions/data/fetch-calendar-events-by-ids.util';
|
||||
import { findOpenScheduledCallRecordings } from 'src/logic-functions/data/find-open-scheduled-call-recordings.util';
|
||||
import { findScheduledRecallBotIdsByCallRecordingId } from 'src/logic-functions/recall-api/find-scheduled-recall-bot-ids-by-call-recording-id.util';
|
||||
import { getUniqueSortedIds } from 'src/logic-functions/utils/get-unique-sorted-ids.util';
|
||||
import { updateCallRecording } from 'src/logic-functions/data/update-call-recording.util';
|
||||
|
||||
export const BOT_NEVER_SCHEDULED_FAILURE_REASON = 'bot_never_scheduled';
|
||||
export const BOT_SCHEDULE_OUTCOME_UNKNOWN_FAILURE_REASON =
|
||||
'bot_schedule_outcome_unknown';
|
||||
|
||||
// Mirrors the stale-state convergence lookback: past it no automatic pull
|
||||
// pass will resolve the row anymore, so keeping it pending only wastes runs.
|
||||
const UNRESOLVED_ATTEMPT_MAX_AGE_DAYS = 7;
|
||||
|
||||
export type ScheduleRecallBotsForPendingCallRecordingsResult = {
|
||||
attachedCallRecordingIds: string[];
|
||||
scheduledCallRecordingIds: string[];
|
||||
markedFailedCallRecordingIds: string[];
|
||||
};
|
||||
|
||||
type ResumableCallRecording = {
|
||||
callRecording: CallRecordingRecord;
|
||||
calendarEvent: CalendarEventRecord;
|
||||
};
|
||||
|
||||
// Resumes a CallRecording inserted before its Recall bot was scheduled.
|
||||
@@ -21,12 +41,17 @@ export const scheduleRecallBotsForPendingCallRecordings = async ({
|
||||
client: CoreApiClient;
|
||||
now: Date;
|
||||
}): Promise<ScheduleRecallBotsForPendingCallRecordingsResult> => {
|
||||
const result: ScheduleRecallBotsForPendingCallRecordingsResult = {
|
||||
attachedCallRecordingIds: [],
|
||||
scheduledCallRecordingIds: [],
|
||||
markedFailedCallRecordingIds: [],
|
||||
};
|
||||
const pendingCallRecordings = (
|
||||
await findOpenScheduledCallRecordings(client)
|
||||
).filter((callRecording) => isUndefined(callRecording.externalBotId));
|
||||
|
||||
if (pendingCallRecordings.length === 0) {
|
||||
return { attachedCallRecordingIds: [], scheduledCallRecordingIds: [] };
|
||||
return result;
|
||||
}
|
||||
|
||||
const calendarEventsById = new Map(
|
||||
@@ -41,51 +66,222 @@ export const scheduleRecallBotsForPendingCallRecordings = async ({
|
||||
)
|
||||
).map((calendarEvent) => [calendarEvent.id, calendarEvent]),
|
||||
);
|
||||
const attachedCallRecordingIds: string[] = [];
|
||||
const scheduledCallRecordingIds: string[] = [];
|
||||
const resumableCallRecordings: ResumableCallRecording[] = [];
|
||||
|
||||
for (const callRecording of pendingCallRecordings) {
|
||||
const calendarEvent = isUndefined(callRecording.calendarEventId)
|
||||
? undefined
|
||||
: calendarEventsById.get(callRecording.calendarEventId);
|
||||
|
||||
if (isUndefined(calendarEvent)) {
|
||||
continue;
|
||||
}
|
||||
|
||||
if (
|
||||
isUndefined(calendarEvent) ||
|
||||
hasMeetingEnded({
|
||||
startsAt: calendarEvent.startsAt,
|
||||
endsAt: calendarEvent.endsAt,
|
||||
now,
|
||||
})
|
||||
) {
|
||||
continue;
|
||||
}
|
||||
|
||||
const attachResult = await attachExistingRecallBotToCallRecording(client, {
|
||||
callRecording,
|
||||
});
|
||||
|
||||
if (attachResult.status === 'attached') {
|
||||
attachedCallRecordingIds.push(callRecording.id);
|
||||
continue;
|
||||
}
|
||||
|
||||
// A failed lookup can hide an existing bot; creating one now could duplicate it, so defer to the next run.
|
||||
if (attachResult.status === 'lookup-failed') {
|
||||
continue;
|
||||
}
|
||||
|
||||
const didScheduleRecallBot = await scheduleRecallBotForCallRecording(
|
||||
client,
|
||||
{
|
||||
await resolveEndedPendingCallRecording({
|
||||
client,
|
||||
callRecording,
|
||||
calendarEvent,
|
||||
},
|
||||
);
|
||||
|
||||
if (didScheduleRecallBot) {
|
||||
scheduledCallRecordingIds.push(callRecording.id);
|
||||
now,
|
||||
result,
|
||||
});
|
||||
continue;
|
||||
}
|
||||
|
||||
resumableCallRecordings.push({ callRecording, calendarEvent });
|
||||
}
|
||||
|
||||
return { attachedCallRecordingIds, scheduledCallRecordingIds };
|
||||
if (resumableCallRecordings.length === 0) {
|
||||
return result;
|
||||
}
|
||||
|
||||
// Rows without a schedule-attempt marker never reached Recall, so no bot
|
||||
// can exist for them. Rows whose stored idempotency key still matches the
|
||||
// current scheduling inputs can re-send the creation and let Recall dedupe
|
||||
// it. Only attempts whose inputs drifted since the attempt pay for a
|
||||
// Recall lookup.
|
||||
const workspaceId = getCurrentWorkspaceId();
|
||||
const ambiguousCallRecordings = resumableCallRecordings.filter(
|
||||
({ callRecording, calendarEvent }) =>
|
||||
!canRescheduleCallRecordingWithoutRecallLookup({
|
||||
callRecording,
|
||||
calendarEvent,
|
||||
workspaceId,
|
||||
now,
|
||||
}),
|
||||
);
|
||||
const unambiguousCallRecordings = resumableCallRecordings.filter(
|
||||
({ callRecording, calendarEvent }) =>
|
||||
canRescheduleCallRecordingWithoutRecallLookup({
|
||||
callRecording,
|
||||
calendarEvent,
|
||||
workspaceId,
|
||||
now,
|
||||
}),
|
||||
);
|
||||
|
||||
for (const { callRecording, calendarEvent } of unambiguousCallRecordings) {
|
||||
await scheduleBotForResumableCallRecording({
|
||||
client,
|
||||
callRecording,
|
||||
calendarEvent,
|
||||
result,
|
||||
});
|
||||
}
|
||||
|
||||
if (ambiguousCallRecordings.length === 0) {
|
||||
return result;
|
||||
}
|
||||
|
||||
// A run that POSTed a bot but died before the id write-back leaves the bot
|
||||
// claimable by metadata; one workspace-wide lookup finds them all without a
|
||||
// per-recording list call.
|
||||
const lookupResult = await findScheduledRecallBotIdsByCallRecordingId();
|
||||
|
||||
// A failed lookup can hide existing bots; creating one now could duplicate
|
||||
// them, so defer to the next run.
|
||||
if (!lookupResult.ok) {
|
||||
return result;
|
||||
}
|
||||
|
||||
for (const { callRecording, calendarEvent } of ambiguousCallRecordings) {
|
||||
const existingExternalBotId =
|
||||
lookupResult.externalBotIdByCallRecordingId.get(callRecording.id);
|
||||
|
||||
if (!isUndefined(existingExternalBotId)) {
|
||||
await updateCallRecording(client, {
|
||||
id: callRecording.id,
|
||||
data: { externalBotId: existingExternalBotId },
|
||||
});
|
||||
result.attachedCallRecordingIds.push(callRecording.id);
|
||||
continue;
|
||||
}
|
||||
|
||||
await scheduleBotForResumableCallRecording({
|
||||
client,
|
||||
callRecording,
|
||||
calendarEvent,
|
||||
result,
|
||||
});
|
||||
}
|
||||
|
||||
return result;
|
||||
};
|
||||
|
||||
const scheduleBotForResumableCallRecording = async ({
|
||||
client,
|
||||
callRecording,
|
||||
calendarEvent,
|
||||
result,
|
||||
}: {
|
||||
client: CoreApiClient;
|
||||
callRecording: CallRecordingRecord;
|
||||
calendarEvent: CalendarEventRecord;
|
||||
result: ScheduleRecallBotsForPendingCallRecordingsResult;
|
||||
}): Promise<void> => {
|
||||
const didScheduleRecallBot = await scheduleRecallBotForCallRecording(
|
||||
client,
|
||||
{
|
||||
callRecording,
|
||||
calendarEvent,
|
||||
},
|
||||
);
|
||||
|
||||
if (didScheduleRecallBot) {
|
||||
result.scheduledCallRecordingIds.push(callRecording.id);
|
||||
}
|
||||
};
|
||||
|
||||
// Only an absent attempt marker proves no POST reached Recall; a marked row
|
||||
// may have a bot that joined and recorded before the id write-back was lost,
|
||||
// so it keeps its recovery chance until the convergence lookback has passed.
|
||||
const resolveEndedPendingCallRecording = async ({
|
||||
client,
|
||||
callRecording,
|
||||
calendarEvent,
|
||||
now,
|
||||
result,
|
||||
}: {
|
||||
client: CoreApiClient;
|
||||
callRecording: CallRecordingRecord;
|
||||
calendarEvent: CalendarEventRecord;
|
||||
now: Date;
|
||||
result: ScheduleRecallBotsForPendingCallRecordingsResult;
|
||||
}): Promise<void> => {
|
||||
if (isUndefined(callRecording.botScheduleAttemptedAt)) {
|
||||
await markCallRecordingFailed({
|
||||
client,
|
||||
callRecording,
|
||||
failureReason: BOT_NEVER_SCHEDULED_FAILURE_REASON,
|
||||
logMessage: `call recording ${callRecording.id} never got a Recall bot and its meeting has ended; marking it failed`,
|
||||
});
|
||||
result.markedFailedCallRecordingIds.push(callRecording.id);
|
||||
|
||||
return;
|
||||
}
|
||||
|
||||
if (!hasUnresolvedAttemptAgedOut({ calendarEvent, now })) {
|
||||
console.warn(
|
||||
`[call-recorder] call recording ${callRecording.id} has an unresolved Recall bot creation attempt and its meeting has ended; waiting for convergence`,
|
||||
);
|
||||
|
||||
return;
|
||||
}
|
||||
|
||||
await markCallRecordingFailed({
|
||||
client,
|
||||
callRecording,
|
||||
failureReason: BOT_SCHEDULE_OUTCOME_UNKNOWN_FAILURE_REASON,
|
||||
logMessage: `call recording ${callRecording.id} has an unresolved Recall bot creation attempt older than the convergence lookback; marking it failed`,
|
||||
});
|
||||
result.markedFailedCallRecordingIds.push(callRecording.id);
|
||||
};
|
||||
|
||||
const hasUnresolvedAttemptAgedOut = ({
|
||||
calendarEvent,
|
||||
now,
|
||||
}: {
|
||||
calendarEvent: CalendarEventRecord;
|
||||
now: Date;
|
||||
}): boolean => {
|
||||
// Mirrors hasMeetingEnded: an unparseable end time falls back to the start
|
||||
// time so these rows still age out of the pending sweep eventually.
|
||||
const meetingEndTime = [calendarEvent.endsAt, calendarEvent.startsAt]
|
||||
.filter((candidate) => !isUndefined(candidate))
|
||||
.map((candidate) => new Date(candidate).getTime())
|
||||
.find((candidateTime) => !Number.isNaN(candidateTime));
|
||||
|
||||
return (
|
||||
!isUndefined(meetingEndTime) &&
|
||||
meetingEndTime + UNRESOLVED_ATTEMPT_MAX_AGE_DAYS * 24 * 60 * 60 * 1000 <=
|
||||
now.getTime()
|
||||
);
|
||||
};
|
||||
|
||||
const markCallRecordingFailed = async ({
|
||||
client,
|
||||
callRecording,
|
||||
failureReason,
|
||||
logMessage,
|
||||
}: {
|
||||
client: CoreApiClient;
|
||||
callRecording: CallRecordingRecord;
|
||||
failureReason: string;
|
||||
logMessage: string;
|
||||
}): Promise<void> => {
|
||||
console.warn(`[call-recorder] ${logMessage}`);
|
||||
|
||||
await updateCallRecording(client, {
|
||||
id: callRecording.id,
|
||||
data: {
|
||||
status: CallRecordingStatus.FAILED,
|
||||
callRecorderFailureReason: failureReason,
|
||||
},
|
||||
});
|
||||
};
|
||||
|
||||
-41
@@ -1,41 +0,0 @@
|
||||
import { listScheduledRecallBots } from 'src/logic-functions/recall-api/list-scheduled-recall-bots.util';
|
||||
|
||||
const ACTIVE_RECALL_BOT_STATUSES = [
|
||||
'ready',
|
||||
'joining_call',
|
||||
'in_waiting_room',
|
||||
'in_call_not_recording',
|
||||
'recording_permission_allowed',
|
||||
'recording_permission_denied',
|
||||
'in_call_recording',
|
||||
];
|
||||
|
||||
export type FindScheduledRecallBotIdResult =
|
||||
| { ok: true; externalBotId: string | undefined }
|
||||
| { ok: false };
|
||||
|
||||
export const findScheduledRecallBotIdForCallRecording = async ({
|
||||
callRecordingId,
|
||||
workspaceId,
|
||||
}: {
|
||||
callRecordingId: string;
|
||||
workspaceId: string;
|
||||
}): Promise<FindScheduledRecallBotIdResult> => {
|
||||
const listResult = await listScheduledRecallBots({
|
||||
metadata: {
|
||||
twentyWorkspaceId: workspaceId,
|
||||
twentyCallRecordingId: callRecordingId,
|
||||
},
|
||||
statuses: ACTIVE_RECALL_BOT_STATUSES,
|
||||
});
|
||||
|
||||
if (!listResult.ok) {
|
||||
console.warn(
|
||||
`[call-recorder] failed to look up existing Recall bot for call recording ${callRecordingId}: ${listResult.errorMessage}`,
|
||||
);
|
||||
|
||||
return { ok: false };
|
||||
}
|
||||
|
||||
return { ok: true, externalBotId: listResult.bots[0]?.id };
|
||||
};
|
||||
+62
@@ -0,0 +1,62 @@
|
||||
import { isUndefined } from '@sniptt/guards';
|
||||
|
||||
import { ACTIVE_RECALL_BOT_STATUSES } from 'src/logic-functions/constants/active-recall-bot-statuses';
|
||||
import { getCurrentWorkspaceId } from 'src/logic-functions/data/get-current-workspace-id.util';
|
||||
import { listScheduledRecallBots } from 'src/logic-functions/recall-api/list-scheduled-recall-bots.util';
|
||||
import { isNonEmptyString } from 'src/logic-functions/utils/is-non-empty-string.util';
|
||||
|
||||
export type FindScheduledRecallBotIdsByCallRecordingIdResult =
|
||||
| { ok: true; externalBotIdByCallRecordingId: Map<string, string> }
|
||||
| { ok: false };
|
||||
|
||||
// One workspace-wide list request covers every pending recording; Recall's
|
||||
// list endpoint has the tightest rate budget, so per-recording lookups must
|
||||
// not fan out.
|
||||
export const findScheduledRecallBotIdsByCallRecordingId =
|
||||
async (): Promise<FindScheduledRecallBotIdsByCallRecordingIdResult> => {
|
||||
const workspaceId = getCurrentWorkspaceId();
|
||||
|
||||
if (isUndefined(workspaceId)) {
|
||||
return { ok: true, externalBotIdByCallRecordingId: new Map() };
|
||||
}
|
||||
|
||||
const listResult = await listScheduledRecallBots({
|
||||
metadata: { twentyWorkspaceId: workspaceId },
|
||||
statuses: ACTIVE_RECALL_BOT_STATUSES,
|
||||
});
|
||||
|
||||
if (!listResult.ok) {
|
||||
console.warn(
|
||||
`[call-recorder] failed to look up existing Recall bots for pending call recordings: ${listResult.errorMessage}`,
|
||||
);
|
||||
|
||||
return { ok: false };
|
||||
}
|
||||
|
||||
// A truncated list can hide existing bots; callers treat a map miss as
|
||||
// permission to create, so an incomplete map must read as a failed lookup.
|
||||
if (listResult.truncated) {
|
||||
console.warn(
|
||||
'[call-recorder] Recall bot list was truncated; deferring bot recovery to the next run',
|
||||
);
|
||||
|
||||
return { ok: false };
|
||||
}
|
||||
|
||||
const externalBotIdByCallRecordingId = new Map<string, string>();
|
||||
|
||||
for (const bot of listResult.bots) {
|
||||
const callRecordingId = bot.metadata.twentyCallRecordingId;
|
||||
|
||||
if (
|
||||
!isNonEmptyString(callRecordingId) ||
|
||||
externalBotIdByCallRecordingId.has(callRecordingId)
|
||||
) {
|
||||
continue;
|
||||
}
|
||||
|
||||
externalBotIdByCallRecordingId.set(callRecordingId, bot.id);
|
||||
}
|
||||
|
||||
return { ok: true, externalBotIdByCallRecordingId };
|
||||
};
|
||||
+7
-6
@@ -20,6 +20,7 @@ export type ScheduleRecallBotArgs = {
|
||||
joinAt: string;
|
||||
metadata: RecallRoutingMetadata;
|
||||
automaticVideoOutput?: RecallBotAutomaticVideoOutput;
|
||||
idempotencyKey?: string;
|
||||
};
|
||||
|
||||
export const scheduleRecallBot = async ({
|
||||
@@ -27,6 +28,11 @@ export const scheduleRecallBot = async ({
|
||||
joinAt,
|
||||
metadata,
|
||||
automaticVideoOutput,
|
||||
idempotencyKey = computeRecallBotCreationIdempotencyKey({
|
||||
meetingUrl,
|
||||
joinAt,
|
||||
metadata,
|
||||
}),
|
||||
}: ScheduleRecallBotArgs): Promise<RecallBotScheduleResult> => {
|
||||
const configResult = getRecallApiConfig();
|
||||
|
||||
@@ -35,11 +41,6 @@ export const scheduleRecallBot = async ({
|
||||
}
|
||||
|
||||
const automaticLeave = getRecallBotAutomaticLeave();
|
||||
const idempotencyKey = computeRecallBotCreationIdempotencyKey({
|
||||
meetingUrl,
|
||||
joinAt,
|
||||
metadata,
|
||||
});
|
||||
|
||||
const result = await recallBotApiRequest<RecallBotResponse>({
|
||||
config: configResult.config,
|
||||
@@ -82,7 +83,7 @@ export const scheduleRecallBot = async ({
|
||||
};
|
||||
};
|
||||
|
||||
const computeRecallBotCreationIdempotencyKey = ({
|
||||
export const computeRecallBotCreationIdempotencyKey = ({
|
||||
meetingUrl,
|
||||
joinAt,
|
||||
metadata,
|
||||
|
||||
+129
@@ -0,0 +1,129 @@
|
||||
import { isUndefined } from '@sniptt/guards';
|
||||
import { CoreApiClient } from 'twenty-client-sdk/core';
|
||||
import {
|
||||
defineLogicFunction,
|
||||
type DatabaseEventPayload,
|
||||
type ObjectRecordBaseEvent,
|
||||
} from 'twenty-sdk/define';
|
||||
|
||||
import { SCHEDULE_RECALL_BOT_ON_CALL_RECORDING_UPDATE_LOGIC_FUNCTION_UNIVERSAL_IDENTIFIER } from 'src/constants/schedule-recall-bot-on-call-recording-update-logic-function-universal-identifier';
|
||||
import { CallRecordingRequestStatus } from 'src/logic-functions/constants/call-recording-request-status';
|
||||
import { CallRecordingStatus } from 'src/logic-functions/constants/call-recording-status';
|
||||
import {
|
||||
resumePendingCallRecording,
|
||||
type ResumePendingCallRecordingResult,
|
||||
} from 'src/logic-functions/flows/resume-pending-call-recording.util';
|
||||
|
||||
const CALL_RECORDING_OBJECT_NAME = 'callRecording';
|
||||
|
||||
// Only these fields can transition a row into the pending shape; ignoring the
|
||||
// rest keeps this trigger from re-firing on its own scheduling-progress
|
||||
// writes (botScheduleAttemptedAt, botScheduleIdempotencyKey, externalBotId
|
||||
// write-back) and on webhook artifact updates.
|
||||
const PENDING_TRANSITION_FIELDS = [
|
||||
'recordingRequestStatus',
|
||||
'status',
|
||||
'externalBotId',
|
||||
'calendarEventId',
|
||||
];
|
||||
|
||||
type CallRecordingForDatabaseEvent = {
|
||||
id: string;
|
||||
status?: string | null;
|
||||
recordingRequestStatus?: string | null;
|
||||
externalBotId?: string | null;
|
||||
};
|
||||
|
||||
type CallRecordingDatabaseEvent = DatabaseEventPayload<
|
||||
ObjectRecordBaseEvent<CallRecordingForDatabaseEvent>
|
||||
>;
|
||||
|
||||
// Created rows are scheduled inline by the run that inserts them; reacting to
|
||||
// creations here would race that run into duplicate Recall creates. This
|
||||
// trigger only resumes rows that fall back to pending later (a bot cleared
|
||||
// after vanishing at Recall, a canceled request re-requested, a failed row
|
||||
// reset by calendar reconciliation), which the recovery cron would otherwise
|
||||
// pick up minutes later.
|
||||
export const scheduleRecallBotOnCallRecordingUpdateHandler = async (
|
||||
event: CallRecordingDatabaseEvent,
|
||||
): Promise<
|
||||
| { skipped: true; reason: string }
|
||||
| { callRecordingId: string; result: ResumePendingCallRecordingResult }
|
||||
> => {
|
||||
const [objectName, action] = event.name.split('.');
|
||||
|
||||
if (objectName !== CALL_RECORDING_OBJECT_NAME || action !== 'updated') {
|
||||
return { skipped: true, reason: 'not a call recording update' };
|
||||
}
|
||||
|
||||
const updatedFields = event.properties.updatedFields ?? [];
|
||||
|
||||
if (
|
||||
!updatedFields.some((updatedField) =>
|
||||
PENDING_TRANSITION_FIELDS.includes(updatedField),
|
||||
)
|
||||
) {
|
||||
return { skipped: true, reason: 'no pending-transition field changed' };
|
||||
}
|
||||
|
||||
if (contradictsPendingCallRecording(event.properties)) {
|
||||
return { skipped: true, reason: 'call recording is not pending' };
|
||||
}
|
||||
|
||||
const result = await resumePendingCallRecording({
|
||||
client: new CoreApiClient(),
|
||||
callRecordingId: event.recordId,
|
||||
now: new Date(),
|
||||
});
|
||||
|
||||
return { callRecordingId: event.recordId, result };
|
||||
};
|
||||
|
||||
// Values can come from a full `after` snapshot or, on slim payloads, from the
|
||||
// per-field diff; any known value that contradicts the pending shape lets the
|
||||
// trigger skip without the authoritative fetch. Unknown values stay ambiguous
|
||||
// and the resume flow re-fetches to decide.
|
||||
const contradictsPendingCallRecording = (
|
||||
properties: CallRecordingDatabaseEvent['properties'],
|
||||
): boolean => {
|
||||
const knownValues = properties.after ?? resolveDiffAfterValues(properties);
|
||||
|
||||
return (
|
||||
(!isUndefined(knownValues.recordingRequestStatus) &&
|
||||
knownValues.recordingRequestStatus !==
|
||||
CallRecordingRequestStatus.REQUESTED) ||
|
||||
(!isUndefined(knownValues.status) &&
|
||||
knownValues.status !== CallRecordingStatus.SCHEDULED) ||
|
||||
(!isUndefined(knownValues.externalBotId) &&
|
||||
knownValues.externalBotId !== null)
|
||||
);
|
||||
};
|
||||
|
||||
const resolveDiffAfterValues = (
|
||||
properties: CallRecordingDatabaseEvent['properties'],
|
||||
): Partial<CallRecordingForDatabaseEvent> => {
|
||||
const diff = properties.diff ?? {};
|
||||
|
||||
return {
|
||||
...(isUndefined(diff.status) ? {} : { status: diff.status.after }),
|
||||
...(isUndefined(diff.recordingRequestStatus)
|
||||
? {}
|
||||
: { recordingRequestStatus: diff.recordingRequestStatus.after }),
|
||||
...(isUndefined(diff.externalBotId)
|
||||
? {}
|
||||
: { externalBotId: diff.externalBotId.after }),
|
||||
};
|
||||
};
|
||||
|
||||
export default defineLogicFunction({
|
||||
universalIdentifier:
|
||||
SCHEDULE_RECALL_BOT_ON_CALL_RECORDING_UPDATE_LOGIC_FUNCTION_UNIVERSAL_IDENTIFIER,
|
||||
name: 'schedule-recall-bot-on-call-recording-update',
|
||||
description:
|
||||
'Resumes Recall bot scheduling as soon as a call recording transitions back to pending instead of waiting for the recovery cron.',
|
||||
timeoutSeconds: 60,
|
||||
handler: scheduleRecallBotOnCallRecordingUpdateHandler,
|
||||
databaseEventTriggerSettings: {
|
||||
eventName: `${CALL_RECORDING_OBJECT_NAME}.updated`,
|
||||
},
|
||||
});
|
||||
+2
@@ -12,6 +12,8 @@ export type CallRecordingRecord = {
|
||||
endedAt?: string;
|
||||
calendarEventId?: string;
|
||||
externalBotId?: string;
|
||||
botScheduleAttemptedAt?: string;
|
||||
botScheduleIdempotencyKey?: string;
|
||||
externalRecordingId?: string;
|
||||
callRecorderFailureReason?: string;
|
||||
};
|
||||
|
||||
+3
@@ -13,6 +13,9 @@ export type CallRecordingUpdateFields = Partial<{
|
||||
calendarEventId: string;
|
||||
// null clears stale app-owned state on cancel/eject or reschedule.
|
||||
externalBotId: string | null;
|
||||
// null clears attempt state once its outcome is resolved (bot confirmed gone).
|
||||
botScheduleAttemptedAt: string | null;
|
||||
botScheduleIdempotencyKey: string | null;
|
||||
externalRecordingId: string;
|
||||
callRecorderFailureReason: string | null;
|
||||
transcript: Record<string, unknown>;
|
||||
|
||||
Reference in New Issue
Block a user