Félix Malfait
da6a2ee300
## Problem In production, an AI-chat assistant response sometimes freezes mid-stream (partial text, looks hung), then "picks up again on its own" later without the user resending and without a known worker restart. Root cause: the **agent-chat SSE subscription has no keepalive and no silent-death detection**. - Delivery is fire-and-forget Redis pub/sub (`SubscriptionService.publishToAgentChat`) and the resolver returns the **raw** iterator — unlike `EventStreamResolver`, which heartbeats every 30s via `wrapAsyncIteratorWithLifecycle`. - During a quiet model/tool gap the connection sends no bytes, so a proxy/LB/NAT can silently drop it mid-stream. `graphql-sse` neither surfaces an error nor resumes with `Last-Event-ID`, and **nothing re-pulls the existing Redis chunk catch-up on reconnect** (it only runs on thread (re)mount / `message-persisted` refetch). - So the live view freezes; recovery only happens when the terminal `message-persisted` fires a full refetch from the DB — the observed "self-recovery". This is the **same silent-SSE-death class fixed for the DB event stream in #21061**, which was never applied to the agent-chat path. The symptom also matches #21096 (worker logs the job finishing, client never updates, reload shows the message). It is **not** queue prioritization, and it is **not** addressed by #22193 (which only stabilizes the assistant message id and removes end-of-stream flicker). A secondary, independent self-recovery path also existed: BullMQ stalled-job re-run (default 30s `lockDuration`, no idempotency guard) re-streaming the whole turn → duplicate assistant messages / double billing. ## Changes ### Commit 1 — keepalive + silent-death recovery (ports the #21061 pattern to agent chat) - **Shared:** new `keepalive` variant on `AgentChatSubscriptionEvent`. - **Server:** wrap the agent-chat subscription iterator with `wrapAsyncIteratorWithLifecycle` — emit a `keepalive` on connect and every `APPLICATION_KEEPALIVE_INTERVAL_MS` (30s) so the connection keeps flushing bytes and a dead connection becomes detectable. - **Client:** track the last received event timestamp (refreshed on every chunk/keepalive in the SSE `next` sink); new `AgentChatStreamKeepAliveEffect` forces a resubscribe + messages refetch after 90s of silence, so the durable Redis chunk list backfills the gap (`firstLiveSeq` is reset on resubscribe). ### Commit 2 — stream-job idempotency + lockDuration - Thread a `lockDuration` option through `MessageQueueWorkerOptions` + the BullMQ driver; set `aiStreamQueue` to 10 min so long streams aren't falsely stalled. - Guard `StreamAgentChatJob.handle` with a `streamId`-scoped Redis lock (`SET NX PX` + compare-and-delete release) so a stalled re-run is skipped instead of double-processing. ## Verification ⚠️ I could **not run typecheck/lint locally** — `yarn install` could not complete in this environment (transient registry network aborts before the link step, so `node_modules` never populated). **Please rely on CI for type/lint verification.** The changes are written to match existing conventions; the points most worth a reviewer's eye are the resolver's iterator typing and the ioredis `set(..., 'PX', ttl, 'NX')` overload. How to confirm the root cause in prod: a frozen client with the worker logging `StreamAgentChatJob processed in …ms` and no `[AI_CHAT_NO_TEXT]` is the silent-death signature (check reverse-proxy idle/buffering). For the secondary path, watch `aiStreamQueue` `stalled`/re-processed metrics and duplicate turns around worker restarts. ## Notes / trade-offs - The 10-min `lockDuration` means a genuinely crashed worker's job isn't reclaimed for up to 10 min; the client-side keepalive/catch-up recovers the view independently, and the idempotency lock prevents duplicates. Faster dead-worker recovery could be a follow-up. - Touches `useAgentChatSubscription.ts` / `AgentChatRuntimeEffects.tsx` / `stream-agent-chat.job.ts`, which #22193 also touches — trivial rebase expected. Opened as **draft** pending CI. https://claude.ai/code/session_018dF82A1VcsuWMxPLmdY3dm --- _Generated by [Claude Code](https://claude.ai/code/session_018dF82A1VcsuWMxPLmdY3dm)_ <!-- This is an auto-generated description by cubic. --> <a href="https://cubic.dev/pr/twentyhq/twenty/pull/22201?utm_source=github" target="_blank" rel="noopener noreferrer" data-no-image-dialog="true"><picture><source media="(prefers-color-scheme: dark)" srcset="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"><source media="(prefers-color-scheme: light)" srcset="https://www.cubic.dev/buttons/review-in-cubic-light.svg"><img alt="Review in cubic" src="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"></picture></a> <!-- End of auto-generated description by cubic. -->
The #1 Open-Source CRM
Website · 
Documentation · 
Roadmap · 
Discord · 
Figma
Why Twenty
Twenty gives technical teams the building blocks for a custom CRM that meets complex business needs and quickly adapts as the business evolves. Twenty is the CRM you build, ship, and version like the rest of your stack.
Learn more about why we built Twenty
Installation
Cloud
The fastest way to get started. Sign up at twenty.com and spin up a workspace in under a minute, with no infrastructure to manage and always up to date.
Build an app
Scaffold a new app with the Twenty CLI:
npx create-twenty-app my-app
Define objects, fields, and views as code:
import { defineObject, FieldType } from 'twenty-sdk/define';
export default defineObject({
nameSingular: 'deal',
namePlural: 'deals',
labelSingular: 'Deal',
labelPlural: 'Deals',
fields: [
{ name: 'name', label: 'Name', type: FieldType.TEXT },
{ name: 'amount', label: 'Amount', type: FieldType.CURRENCY },
{ name: 'closeDate', label: 'Close Date', type: FieldType.DATE_TIME },
],
});
Then ship it to your workspace:
npx twenty app:publish --private
See the app development guide for objects, views, agents, and logic functions.
Self-hosting
Run Twenty on your own infrastructure with Docker Compose, or contribute locally via the local setup guide.
Everything you need
Twenty gives you the building blocks of a modern CRM (objects, views, workflows, and agents) and lets you extend them as code. Here's a tour of what's in the box.
Want to go deeper? Read the 
User Guide for product walkthroughs, or the Documentation for developer reference.
|
|
|
|
|
|
Learn more about version control in doc
Stack
TypeScript
Nx
NestJS, with BullMQ, 
PostgreSQL, 
Redis
React, with Jotai, Linaria and Lingui
Thanks
Thanks to these amazing services that we use and recommend for code review (Greptile), catching bugs (Sentry) and translating (Crowdin).
Join the Community
Star the repo · Discord · 
Feature requests · Releases · 
X · 
LinkedIn · 
Crowdin · 
Contribute