fix(ai-chat): keep streams alive on silent SSE death + make the stream job idempotent (#22201)
## Problem In production, an AI-chat assistant response sometimes freezes mid-stream (partial text, looks hung), then "picks up again on its own" later without the user resending and without a known worker restart. Root cause: the **agent-chat SSE subscription has no keepalive and no silent-death detection**. - Delivery is fire-and-forget Redis pub/sub (`SubscriptionService.publishToAgentChat`) and the resolver returns the **raw** iterator — unlike `EventStreamResolver`, which heartbeats every 30s via `wrapAsyncIteratorWithLifecycle`. - During a quiet model/tool gap the connection sends no bytes, so a proxy/LB/NAT can silently drop it mid-stream. `graphql-sse` neither surfaces an error nor resumes with `Last-Event-ID`, and **nothing re-pulls the existing Redis chunk catch-up on reconnect** (it only runs on thread (re)mount / `message-persisted` refetch). - So the live view freezes; recovery only happens when the terminal `message-persisted` fires a full refetch from the DB — the observed "self-recovery". This is the **same silent-SSE-death class fixed for the DB event stream in #21061**, which was never applied to the agent-chat path. The symptom also matches #21096 (worker logs the job finishing, client never updates, reload shows the message). It is **not** queue prioritization, and it is **not** addressed by #22193 (which only stabilizes the assistant message id and removes end-of-stream flicker). A secondary, independent self-recovery path also existed: BullMQ stalled-job re-run (default 30s `lockDuration`, no idempotency guard) re-streaming the whole turn → duplicate assistant messages / double billing. ## Changes ### Commit 1 — keepalive + silent-death recovery (ports the #21061 pattern to agent chat) - **Shared:** new `keepalive` variant on `AgentChatSubscriptionEvent`. - **Server:** wrap the agent-chat subscription iterator with `wrapAsyncIteratorWithLifecycle` — emit a `keepalive` on connect and every `APPLICATION_KEEPALIVE_INTERVAL_MS` (30s) so the connection keeps flushing bytes and a dead connection becomes detectable. - **Client:** track the last received event timestamp (refreshed on every chunk/keepalive in the SSE `next` sink); new `AgentChatStreamKeepAliveEffect` forces a resubscribe + messages refetch after 90s of silence, so the durable Redis chunk list backfills the gap (`firstLiveSeq` is reset on resubscribe). ### Commit 2 — stream-job idempotency + lockDuration - Thread a `lockDuration` option through `MessageQueueWorkerOptions` + the BullMQ driver; set `aiStreamQueue` to 10 min so long streams aren't falsely stalled. - Guard `StreamAgentChatJob.handle` with a `streamId`-scoped Redis lock (`SET NX PX` + compare-and-delete release) so a stalled re-run is skipped instead of double-processing. ## Verification ⚠️ I could **not run typecheck/lint locally** — `yarn install` could not complete in this environment (transient registry network aborts before the link step, so `node_modules` never populated). **Please rely on CI for type/lint verification.** The changes are written to match existing conventions; the points most worth a reviewer's eye are the resolver's iterator typing and the ioredis `set(..., 'PX', ttl, 'NX')` overload. How to confirm the root cause in prod: a frozen client with the worker logging `StreamAgentChatJob processed in …ms` and no `[AI_CHAT_NO_TEXT]` is the silent-death signature (check reverse-proxy idle/buffering). For the secondary path, watch `aiStreamQueue` `stalled`/re-processed metrics and duplicate turns around worker restarts. ## Notes / trade-offs - The 10-min `lockDuration` means a genuinely crashed worker's job isn't reclaimed for up to 10 min; the client-side keepalive/catch-up recovers the view independently, and the idempotency lock prevents duplicates. Faster dead-worker recovery could be a follow-up. - Touches `useAgentChatSubscription.ts` / `AgentChatRuntimeEffects.tsx` / `stream-agent-chat.job.ts`, which #22193 also touches — trivial rebase expected. Opened as **draft** pending CI. https://claude.ai/code/session_018dF82A1VcsuWMxPLmdY3dm --- _Generated by [Claude Code](https://claude.ai/code/session_018dF82A1VcsuWMxPLmdY3dm)_ <!-- This is an auto-generated description by cubic. --> <a href="https://cubic.dev/pr/twentyhq/twenty/pull/22201?utm_source=github" target="_blank" rel="noopener noreferrer" data-no-image-dialog="true"><picture><source media="(prefers-color-scheme: dark)" srcset="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"><source media="(prefers-color-scheme: light)" srcset="https://www.cubic.dev/buttons/review-in-cubic-light.svg"><img alt="Review in cubic" src="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"></picture></a> <!-- End of auto-generated description by cubic. -->
This commit is contained in:
+1
@@ -0,0 +1 @@
|
||||
export const AI_STREAM_LOCK_DURATION_MS = 10 * 60 * 1000;
|
||||
@@ -108,6 +108,9 @@ export class BullMQDriver
|
||||
...(isDefined(options?.concurrency)
|
||||
? { concurrency: options.concurrency }
|
||||
: {}),
|
||||
...(isDefined(options?.lockDuration)
|
||||
? { lockDuration: options.lockDuration }
|
||||
: {}),
|
||||
metrics: {
|
||||
maxDataPoints: MetricsTime.ONE_WEEK,
|
||||
collectInterval: 60000,
|
||||
|
||||
+1
@@ -1,3 +1,4 @@
|
||||
export interface MessageQueueWorkerOptions {
|
||||
concurrency?: number;
|
||||
lockDuration?: number;
|
||||
}
|
||||
|
||||
+7
-2
@@ -1,8 +1,13 @@
|
||||
import { AI_STREAM_LOCK_DURATION_MS } from 'src/engine/core-modules/message-queue/constants/ai-stream-lock-duration.constant';
|
||||
import { type MessageQueueWorkerOptions } from 'src/engine/core-modules/message-queue/interfaces/message-queue-worker-options.interface';
|
||||
import { MessageQueue } from 'src/engine/core-modules/message-queue/message-queue.constants';
|
||||
|
||||
export const QUEUE_WORKER_OPTIONS: Partial<
|
||||
Record<MessageQueue, { concurrency: number }>
|
||||
Record<MessageQueue, MessageQueueWorkerOptions>
|
||||
> = {
|
||||
[MessageQueue.aiStreamQueue]: { concurrency: 20 },
|
||||
[MessageQueue.aiStreamQueue]: {
|
||||
concurrency: 20,
|
||||
lockDuration: AI_STREAM_LOCK_DURATION_MS,
|
||||
},
|
||||
[MessageQueue.logicFunctionQueue]: { concurrency: 10 },
|
||||
};
|
||||
|
||||
Reference in New Issue
Block a user