Charles Bochet
1c3ae92c04
## Context
In prod-eu, **all `twenty-server` API pods crash simultaneously**
several times per hour (then restart in lockstep) since ~Jun 2. Each
crash is an **unhandled promise rejection** in SSE event-stream teardown
— `exitCode=1`, identical stack on every one of the 7 pods:
```
Error: Failed to acquire lock for key: workspace:<id>:activeStreams
at CacheLockService.withLock (cache-lock.service.ts:53)
at async EventStreamService.destroyEventStream (event-stream.service.ts:75)
at async cleanup (wrap-async-iterator-with-lifecycle.ts)
at async Object.return (wrap-async-iterator-with-lifecycle.ts)
at async Object.cancel (graphql-yoga/.../result-processor/sse.js:68)
```
### Mechanism
1. A busy workspace's SSE clients reconnect (no client backoff), so
connect/disconnect contend on a **single per-workspace Redis lock**
`workspace:<id>:activeStreams`.
2. Under contention `CacheLockService.withLock` exhausts its retries and
**throws**.
3. In the `destroyEventStream` teardown path the throw escapes
`wrapAsyncIteratorWithLifecycle`'s `cleanup()` — `return()` does `try {
await cleanup() } finally { … }` and does **not** catch a cleanup throw.
4. graphql-yoga invokes this from `cancel()` as a **fire-and-forget**
`Promise.all` on connection abort. With **no global `unhandledRejection`
handler**, Node's default policy terminates the process with **exit code
1**.
5. The crash drops all that pod's SSE clients → they reconnect to
surviving pods → contention moves there → the whole fleet crashes
together → restarts → reconnect storm → repeats (~14 min period,
matching the metrics).
## Changes
Crash-stopping hotfix (defense in depth). Does **not** change the
locking design or client reconnect behavior — see follow-ups.
- **`wrapAsyncIteratorWithLifecycle`**: `onCleanup()` is now best-effort
— wrapped in try/catch so teardown can never reject out of
`next()`/`return()`/`throw()`. The original iterator error is still
rethrown unchanged. Adds an `onCleanupError` hook so the swallowed error
is still reported.
- **`EventStreamResolver`**: wires `onCleanupError` to
`ExceptionHandlerService.captureExceptions` (→ Sentry) with workspace +
channel context, so these failures stay visible.
- **`main.ts`**: registers a global `process.on('unhandledRejection')`
that reports via `ExceptionHandlerService` (Sentry) instead of letting
Node terminate. Registering the listener also suppresses Node's default
process-termination. Non-`Error` reasons are formatted with
`util.inspect` (per Copilot review) so Sentry gets a readable message
rather than `[object Object]`.
## Verification
The fix was checked against the exact crash path — a wrapped iterator
whose `onCleanup` rejects:
- `return()` (graphql-yoga's `cancel()` path) **resolves** instead of
rejecting → no unhandled rejection.
- `next()` on a completed stream **resolves** despite a rejecting
cleanup.
- A genuine iterator error still surfaces on `next()` (cleanup failure
doesn't mask it).
- `onCleanupError` receives the original `Error`.
All four pass. `nx lint twenty-server` + `oxfmt` clean; no new `tsc`
errors in the changed files.
## Follow-ups (not in this PR)
- Remove the unnecessary `withLock` around the already-atomic Redis
`SADD`/`SREM` in `event-stream.service.ts` (the contention source).
- Restore exponential backoff + jitter on the frontend SSE reconnect
(regressed in #21061) to stop the thundering herd.
- Infra: make `/healthz` a meaningful liveness signal and add
`maxUnavailable` + a PodDisruptionBudget so pods can't all die together.
The #1 Open-Source CRM
Website · 
Documentation · 
Roadmap · 
Discord · 
Figma
Why Twenty
Twenty gives technical teams the building blocks for a custom CRM that meets complex business needs and quickly adapts as the business evolves. Twenty is the CRM you build, ship, and version like the rest of your stack.
Learn more about why we built Twenty
Installation
Cloud
The fastest way to get started. Sign up at twenty.com and spin up a workspace in under a minute, with no infrastructure to manage and always up to date.
Build an app
Scaffold a new app with the Twenty CLI:
npx create-twenty-app my-app
Define objects, fields, and views as code:
import { defineObject, FieldType } from 'twenty-sdk/define';
export default defineObject({
nameSingular: 'deal',
namePlural: 'deals',
labelSingular: 'Deal',
labelPlural: 'Deals',
fields: [
{ name: 'name', label: 'Name', type: FieldType.TEXT },
{ name: 'amount', label: 'Amount', type: FieldType.CURRENCY },
{ name: 'closeDate', label: 'Close Date', type: FieldType.DATE_TIME },
],
});
Then ship it to your workspace:
npx twenty app:publish --private
See the app development guide for objects, views, agents, and logic functions.
Self-hosting
Run Twenty on your own infrastructure with Docker Compose, or contribute locally via the local setup guide.
Everything you need
Twenty gives you the building blocks of a modern CRM (objects, views, workflows, and agents) and lets you extend them as code. Here's a tour of what's in the box.
Want to go deeper? Read the 
User Guide for product walkthroughs, or the Documentation for developer reference.
|
|
|
|
|
|
Learn more about version control in docStack
TypeScript
Nx
NestJS, with BullMQ, 
PostgreSQL, 
Redis
React, with Jotai, Linaria and Lingui
Thanks
Thanks to these amazing services that we use and recommend for code review (Greptile), catching bugs (Sentry) and translating (Crowdin).
Join the Community
Star the repo · Discord · 
Feature requests · Releases · 
X · 
LinkedIn · 
Crowdin · 
Contribute