## Why
#22514 makes SIGTERM drain workers, but `worker.close()` waits for
active jobs with **no upper bound** (BullMQ semantics). An AI stream job
can run for 10 minutes; a deploy would either hang the rollout or hit
the pod's termination grace deadline and get SIGKILLed anyway — back to
the frozen-stream + stalled-rerun failure this series eliminates.
## What
On shutdown, `aiStreamQueue` gets a bounded drain: active stream jobs
have `AI_STREAM_SHUTDOWN_DRAIN_MS` (60s) to finish naturally; stragglers
are then aborted and terminate exactly like a stream failure —
`lastStreamError` persisted with the new typed
`AiExceptionCode.STREAM_INTERRUPTED`, the pinned `stream-error` →
`queue-updated` terminal sequence published, claim released. The client
shows the interrupted state with Retry (#22434) within seconds instead
of a stream frozen mid-sentence. This is deliberately **not** the
user-cancel path, which resolves cleanly and persists no error.
Mechanism — evaluated BullMQ 5.78's native cancellation vs a parallel
in-process registry, and picked native:
- The driver's processor now declares the 3-arg signature, which makes
BullMQ create a per-job `AbortController` (`processor.length >= 3` is
the trigger), and the signal is handed to job handlers as an optional
`MessageQueueJobContext`.
- `worker.cancelAllJobs()` is purely cooperative: it aborts the signal
and nothing else, so the job's own persist/publish/cleanup still runs to
completion and `worker.close()` still waits for it — no force-fail race,
no second signaling channel to maintain, and the timer lives inside the
same `closeWorker()` call so there is no dependence on Nest
module-destroy ordering.
- The stream job maps the shutdown signal onto its **existing**
AbortController (the one already wired through the AI SDK for user
cancel), with an `AiException(STREAM_INTERRUPTED)` reason to tell the
two apart. One abort path end to end, no new infrastructure.
Error-type choice: the job throws a plain `AiException`, not BullMQ's
`UnrecoverableError`. Stream jobs are enqueued with `attempts: 1` (no
`retryLimit`), so there is no BullMQ retry to suppress — retryability
for this queue lives at the app layer (`lastStreamError` + client
Retry), and an `UnrecoverableError` would only obscure the typed
exception.
`STREAM_INTERRUPTED` also replaces the string constant introduced on the
base branch (#22482's reap now uses the same enum member) — one code,
two producers (reap for dead workers, abort for live shutdowns),
identical client behavior.
Notes:
- 60s is a static constant mirroring `AI_STREAM_LOCK_DURATION_MS` rather
than an env var — it has to move in lockstep with the worker's
`terminationGracePeriodSeconds` (120s, twenty-infra PR) anyway, and we
ship multiple releases a day. Happy to lift it into a config variable if
you want runtime tunability.
- The `onModuleDestroy` scaffolding (drain logs,
workers-close-before-queues) deliberately matches #22514; whichever
lands second rebases clean.
Stacked on #22482 (needs the heartbeat/reap base). Merge order: #22482 →
this. Depends on #22514 for SIGTERM to reach the driver at all.
## Validation
- `stream-agent-chat.job.spec.ts`: shutdown-abort persists
`STREAM_INTERRUPTED`, publishes `stream-error` before `queue-updated`,
releases the claim, skips the queued-message flush; user-cancel
semantics unchanged with a wired-but-idle shutdown signal (60/60 ai-chat
tests green).
- Local end-to-end: real AI stream mid-flight, SIGTERM the worker →
drain window → abort → interrupted state persisted, process exits on its
own. (Transcript in the PR conversation.)
<!-- This is an auto-generated description by cubic. -->
<a
href="https://cubic.dev/pr/twentyhq/twenty/pull/22517?utm_source=github"
target="_blank" rel="noopener noreferrer"
data-no-image-dialog="true"><picture><source
media="(prefers-color-scheme: dark)"
srcset="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"><source
media="(prefers-color-scheme: light)"
srcset="https://www.cubic.dev/buttons/review-in-cubic-light.svg"><img
alt="Review in cubic"
src="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"></picture></a>
<!-- End of auto-generated description by cubic. -->
Félix Malfait
3a21089e0a
