## Rationale
If the worker process dies mid-stream (OOM, deploy, crash), nothing ever
clears `activeStreamId`: `aiStreamQueue` runs with `attempts: 1`, the
job's `finally` never executes, and the SSE keepalive comes from the API
server — so it actively masks worker death. The thread is bricked: every
send queues behind a dead claim until someone intervenes manually. This
is a CONFIRMED-high from the chat-stack audit, and worker death is not
hypothetical: Sentry shows an unhandled promise rejection inside the AI
SDK in the worker
([TWENTY-SERVER-H7Y](https://twenty-v7.sentry.io/issues/TWENTY-SERVER-H7Y))
— unhandled rejections terminate Node by default.
## Design
- **Claim-time mark**: every enqueue site marks
`agent-chat-stream-alive:<streamId>` with a TTL matching the job lock
horizon (600s) — covering the enqueue→pickup window where a waiting job
holds no lock.
- **Running refresh**: the job tightens it to **30s, refreshed every
5s**; if the process dies, the interval dies with it and the key
expires. The expiry *is* the death signal. (Was 60s/15s — tightened
after review: detection latency is bounded by the TTL, robustness by
TTL−interval and the missed-beat tolerance; 30s/5s halves detection
while tolerating *more* missed beats, 5 vs 3.)
- **Read-path reap**: the send gate and the catchup query convert a
heartbeat-less claim into a normal retryable `STREAM_INTERRUPTED`
failed-turn state (conditional UPDATE guarded on the observed streamId,
so a newer stream's claim is never touched), reset the Redis chunk
state, and publish the terminal error. `isAlive` fails open on Redis
errors — a liveness probe must not turn a Redis blip into a broken send
path.
## Why this is the root cause, not a symptom patch
The strongest alternative — BullMQ's own stalled-job detection — fails
on four concrete grounds: detection latency is bounded by the deliberate
10-minute `AI_STREAM_LOCK_DURATION_MS` (long silent tool runs must not
spuriously stall); the stalled checker needs a *surviving* worker in the
pool; the signal fires in the worker process while the thing needing
repair is a DB claim read by API-server resolvers; and a `waiting` job
holds no lock at all. Reaping at the read path means recovery happens
exactly when a user is looking — the moment it matters — with zero
background machinery.
**Relationship to the graceful-shutdown work (planned follow-ups)**:
shutdown hooks + drain-then-abort will make *deploys* (cooperative
SIGTERM) end streams cleanly, and disabling stalled re-runs will stop
hard-killed jobs from zombie re-executing tools. This PR remains the
only recovery layer for non-cooperative deaths — OOMKill is a straight
SIGKILL, crashes and unhandled rejections never run shutdown hooks — and
the backstop when the drain path itself fails. The two are complements,
not alternatives.
## User impact
Today a worker crash mid-answer bricks the thread until manual
intervention; users see sends silently queue forever. With this, the
next interaction (send, reload) converts it into a visible "response was
interrupted" error with a working Retry, within ~30s of actual death.
## Test plan
- [x] Claim spec: live stream untouched; heartbeat-less claim reaped
into retryable `STREAM_INTERRUPTED` + chunk-state reset + published
terminal event; no-op when the claim moved to a newer stream mid-check
- [x] CI green
https://claude.ai/code/session_01Lyi6zTema2FMVVh8MD6c38
Félix Malfait
5d3b6d05b3