fix(ai-chat): keep streams alive on silent SSE death + make the stream job idempotent (#22201)

## Problem

In production, an AI-chat assistant response sometimes freezes
mid-stream (partial text, looks hung), then "picks up again on its own"
later without the user resending and without a known worker restart.

Root cause: the **agent-chat SSE subscription has no keepalive and no
silent-death detection**.

- Delivery is fire-and-forget Redis pub/sub
(`SubscriptionService.publishToAgentChat`) and the resolver returns the
**raw** iterator — unlike `EventStreamResolver`, which heartbeats every
30s via `wrapAsyncIteratorWithLifecycle`.
- During a quiet model/tool gap the connection sends no bytes, so a
proxy/LB/NAT can silently drop it mid-stream. `graphql-sse` neither
surfaces an error nor resumes with `Last-Event-ID`, and **nothing
re-pulls the existing Redis chunk catch-up on reconnect** (it only runs
on thread (re)mount / `message-persisted` refetch).
- So the live view freezes; recovery only happens when the terminal
`message-persisted` fires a full refetch from the DB — the observed
"self-recovery".

This is the **same silent-SSE-death class fixed for the DB event stream
in #21061**, which was never applied to the agent-chat path. The symptom
also matches #21096 (worker logs the job finishing, client never
updates, reload shows the message).

It is **not** queue prioritization, and it is **not** addressed by
#22193 (which only stabilizes the assistant message id and removes
end-of-stream flicker).

A secondary, independent self-recovery path also existed: BullMQ
stalled-job re-run (default 30s `lockDuration`, no idempotency guard)
re-streaming the whole turn → duplicate assistant messages / double
billing.

## Changes

### Commit 1 — keepalive + silent-death recovery (ports the #21061
pattern to agent chat)
- **Shared:** new `keepalive` variant on `AgentChatSubscriptionEvent`.
- **Server:** wrap the agent-chat subscription iterator with
`wrapAsyncIteratorWithLifecycle` — emit a `keepalive` on connect and
every `APPLICATION_KEEPALIVE_INTERVAL_MS` (30s) so the connection keeps
flushing bytes and a dead connection becomes detectable.
- **Client:** track the last received event timestamp (refreshed on
every chunk/keepalive in the SSE `next` sink); new
`AgentChatStreamKeepAliveEffect` forces a resubscribe + messages refetch
after 90s of silence, so the durable Redis chunk list backfills the gap
(`firstLiveSeq` is reset on resubscribe).

### Commit 2 — stream-job idempotency + lockDuration
- Thread a `lockDuration` option through `MessageQueueWorkerOptions` +
the BullMQ driver; set `aiStreamQueue` to 10 min so long streams aren't
falsely stalled.
- Guard `StreamAgentChatJob.handle` with a `streamId`-scoped Redis lock
(`SET NX PX` + compare-and-delete release) so a stalled re-run is
skipped instead of double-processing.

## Verification

⚠️ I could **not run typecheck/lint locally** — `yarn install` could not
complete in this environment (transient registry network aborts before
the link step, so `node_modules` never populated). **Please rely on CI
for type/lint verification.** The changes are written to match existing
conventions; the points most worth a reviewer's eye are the resolver's
iterator typing and the ioredis `set(..., 'PX', ttl, 'NX')` overload.

How to confirm the root cause in prod: a frozen client with the worker
logging `StreamAgentChatJob processed in …ms` and no `[AI_CHAT_NO_TEXT]`
is the silent-death signature (check reverse-proxy idle/buffering). For
the secondary path, watch `aiStreamQueue` `stalled`/re-processed metrics
and duplicate turns around worker restarts.

## Notes / trade-offs
- The 10-min `lockDuration` means a genuinely crashed worker's job isn't
reclaimed for up to 10 min; the client-side keepalive/catch-up recovers
the view independently, and the idempotency lock prevents duplicates.
Faster dead-worker recovery could be a follow-up.
- Touches `useAgentChatSubscription.ts` / `AgentChatRuntimeEffects.tsx`
/ `stream-agent-chat.job.ts`, which #22193 also touches — trivial rebase
expected.

Opened as **draft** pending CI.

https://claude.ai/code/session_018dF82A1VcsuWMxPLmdY3dm

---
_Generated by [Claude
Code](https://claude.ai/code/session_018dF82A1VcsuWMxPLmdY3dm)_

<!-- This is an auto-generated description by cubic. -->
<a
href="https://cubic.dev/pr/twentyhq/twenty/pull/22201?utm_source=github"
target="_blank" rel="noopener noreferrer"
data-no-image-dialog="true"><picture><source
media="(prefers-color-scheme: dark)"
srcset="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"><source
media="(prefers-color-scheme: light)"
srcset="https://www.cubic.dev/buttons/review-in-cubic-light.svg"><img
alt="Review in cubic"
src="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"></picture></a>
<!-- End of auto-generated description by cubic. -->
This commit is contained in:
Félix Malfait
2026-06-26 15:19:34 +02:00
committed by GitHub
parent 8f7d6c24dd
commit da6a2ee300
18 changed files with 190 additions and 12 deletions
@@ -0,0 +1 @@
export const AI_STREAM_LOCK_DURATION_MS = 10 * 60 * 1000;
@@ -108,6 +108,9 @@ export class BullMQDriver
...(isDefined(options?.concurrency)
? { concurrency: options.concurrency }
: {}),
...(isDefined(options?.lockDuration)
? { lockDuration: options.lockDuration }
: {}),
metrics: {
maxDataPoints: MetricsTime.ONE_WEEK,
collectInterval: 60000,
@@ -1,3 +1,4 @@
export interface MessageQueueWorkerOptions {
concurrency?: number;
lockDuration?: number;
}
@@ -1,8 +1,13 @@
import { AI_STREAM_LOCK_DURATION_MS } from 'src/engine/core-modules/message-queue/constants/ai-stream-lock-duration.constant';
import { type MessageQueueWorkerOptions } from 'src/engine/core-modules/message-queue/interfaces/message-queue-worker-options.interface';
import { MessageQueue } from 'src/engine/core-modules/message-queue/message-queue.constants';
export const QUEUE_WORKER_OPTIONS: Partial<
Record<MessageQueue, { concurrency: number }>
Record<MessageQueue, MessageQueueWorkerOptions>
> = {
[MessageQueue.aiStreamQueue]: { concurrency: 20 },
[MessageQueue.aiStreamQueue]: {
concurrency: 20,
lockDuration: AI_STREAM_LOCK_DURATION_MS,
},
[MessageQueue.logicFunctionQueue]: { concurrency: 10 },
};