## Context
`twenty_event_streams_live_total` (an observable gauge) calls
`getTotalActiveStreamCount()` →
`scanAndCountSetMembers('workspace:*:activeStreams')`, which runs a
full-keyspace `SCAN MATCH` over the entire Redis DB **on every metrics
scrape, on every pod**. `SCAN MATCH` walks every key (filtering only the
output), and the subscriptions namespace shares the node with the
workspace metadata cache (~190k keys in our prod), so this was the
dominant Redis command (billions of `SCAN` calls) to count a handful of
sets.
## What this does
Cache the count and refresh it via the scan at most once per
`ACTIVE_STREAM_COUNT_REFRESH_MS` (5 min) per pod, instead of on every
scrape. Steady-state scrapes return the cached value; the authoritative
scan still runs periodically so the gauge stays fresh.
Single-method change; no new Redis keys, no data-model changes.
## Context
Follow-up to #21228, which deprecated `isCustom` on object/field
metadata but kept it exposed because the frontend still relied on it.
This removes it from the GraphQL API and the frontend entirely.
## Implementation
### Server
- Remove `isCustom` `@Field` from the `Object`, `Field`, and
`MinimalObjectMetadata` GraphQL types
- Remove the `isCustom` `@ResolveField` resolvers and the
`isCustomLoader` dataloader (+ payload/interface)
- Remove `isCustom` as an internal `@HideField()` on the Object/Field
DTOs used by the i18n standard-override gate > Use an explicit
isStandard instead (which is the correct gating)
### Frontend
- Add `getIsMetadataItemCustom` helper + `useGetIsMetadataItemCustom`
hook: an item is custom when `applicationId ===
currentWorkspace.workspaceCustomApplication.id`
- Migrate all consumers off `objectMetadataItem.isCustom` /
`fieldMetadataItem.isCustom`; `isRecordFieldReadOnly` now takes a
precomputed `isFieldCustom`
- Drop `isCustom` from the metadata fragment/mutations/minimal query, FE
types, zod schemas, and mock generators; regenerate GraphQL types
## Notes
- Breaking change on the (already-deprecated) `Object.isCustom` /
`Field.isCustom` GraphQL fields and the `isCustom` filter
- FE semantic is "belongs to the workspace custom app" (third-party-app
objects/fields are treated as non-custom)
- `isCustom` on IndexMetadata / View / Skill / Agent is a separate
column and is untouched
- Breaking changes on REST metadata API
## Context
#21228 removed the stored `isCustom` column and, with it, the `isCustom`
early-return in `resolveObjectMetadataStandardOverride` /
`resolveFieldMetadataStandardOverride`, on the assumption that falling
through the `standardOverrides` checks was equivalent.
It isn't: custom object/field labels now reach the Lingui lookup. A
custom label that collides with a standard catalog string (e.g. a custom
field labeled "Status") gets translated for non-English locales against
the user's intent, and every other custom label pays a hash + catalog
miss — and, in production, an "Uncompiled message detected" warning
(#21415) — on each metadata resolution.
## Fix
Restore the gate. `isCustom` is no longer stored, so call sites that
build the resolver input from flat entities (dataloader,
minimal-metadata, view controller, command-menu-item navigation context)
derive it via `belongsToTwentyStandardApp`; GraphQL resolvers keep
passing DTOs, which already carry the derived value.
## Testing
- Unit tests for both resolvers, including a new regression test: a
custom label matching a standard catalog entry is returned verbatim,
Lingui never called.
---------
Co-authored-by: Weiko <corentin@twenty.com>
## Context
In prod-eu, **all `twenty-server` API pods crash simultaneously**
several times per hour (then restart in lockstep) since ~Jun 2. Each
crash is an **unhandled promise rejection** in SSE event-stream teardown
— `exitCode=1`, identical stack on every one of the 7 pods:
```
Error: Failed to acquire lock for key: workspace:<id>:activeStreams
at CacheLockService.withLock (cache-lock.service.ts:53)
at async EventStreamService.destroyEventStream (event-stream.service.ts:75)
at async cleanup (wrap-async-iterator-with-lifecycle.ts)
at async Object.return (wrap-async-iterator-with-lifecycle.ts)
at async Object.cancel (graphql-yoga/.../result-processor/sse.js:68)
```
### Mechanism
1. A busy workspace's SSE clients reconnect (no client backoff), so
connect/disconnect contend on a **single per-workspace Redis lock**
`workspace:<id>:activeStreams`.
2. Under contention `CacheLockService.withLock` exhausts its retries and
**throws**.
3. In the `destroyEventStream` teardown path the throw escapes
`wrapAsyncIteratorWithLifecycle`'s `cleanup()` — `return()` does `try {
await cleanup() } finally { … }` and does **not** catch a cleanup throw.
4. graphql-yoga invokes this from `cancel()` as a **fire-and-forget**
`Promise.all` on connection abort. With **no global `unhandledRejection`
handler**, Node's default policy terminates the process with **exit code
1**.
5. The crash drops all that pod's SSE clients → they reconnect to
surviving pods → contention moves there → the whole fleet crashes
together → restarts → reconnect storm → repeats (~14 min period,
matching the metrics).
## Changes
Crash-stopping hotfix (defense in depth). Does **not** change the
locking design or client reconnect behavior — see follow-ups.
- **`wrapAsyncIteratorWithLifecycle`**: `onCleanup()` is now best-effort
— wrapped in try/catch so teardown can never reject out of
`next()`/`return()`/`throw()`. The original iterator error is still
rethrown unchanged. Adds an `onCleanupError` hook so the swallowed error
is still reported.
- **`EventStreamResolver`**: wires `onCleanupError` to
`ExceptionHandlerService.captureExceptions` (→ Sentry) with workspace +
channel context, so these failures stay visible.
- **`main.ts`**: registers a global `process.on('unhandledRejection')`
that reports via `ExceptionHandlerService` (Sentry) instead of letting
Node terminate. Registering the listener also suppresses Node's default
process-termination. Non-`Error` reasons are formatted with
`util.inspect` (per Copilot review) so Sentry gets a readable message
rather than `[object Object]`.
## Verification
The fix was checked against the exact crash path — a wrapped iterator
whose `onCleanup` rejects:
- `return()` (graphql-yoga's `cancel()` path) **resolves** instead of
rejecting → no unhandled rejection.
- `next()` on a completed stream **resolves** despite a rejecting
cleanup.
- A genuine iterator error still surfaces on `next()` (cleanup failure
doesn't mask it).
- `onCleanupError` receives the original `Error`.
All four pass. `nx lint twenty-server` + `oxfmt` clean; no new `tsc`
errors in the changed files.
## Follow-ups (not in this PR)
- Remove the unnecessary `withLock` around the already-atomic Redis
`SADD`/`SREM` in `event-stream.service.ts` (the contention source).
- Restore exponential backoff + jitter on the frontend SSE reconnect
(regressed in #21061) to stop the thundering herd.
- Infra: make `/healthz` a meaningful liveness signal and add
`maxUnavailable` + a PodDisruptionBudget so pods can't all die together.
## Context
`isCustom` was a legacy denormalized boolean on `ObjectMetadataEntity`
and `FieldMetadataEntity`.
Now that every metadata row carries `applicationId` (via
`SyncableEntity`), "is this custom" is fully derivable, and the stored
boolean was a redundant second source of truth that could drift.
The real meaning of `isCustom` is **"the owning application is not the
twenty-standard application"** — i.e. `!belongsToTwentyStandardApp`.
Note this is *not* "belongs to the workspace custom app" as I initially
thought: third-party-application
objects/fields are custom too.
The standard application has a globally stable `universalIdentifier`, so
the value derives with no per-workspace lookup.
## Changed
## `isCustom` checks — before → after
`isCustom` is no longer a stored column. The table below lists every
site that branched on it and how it resolves now. The unifying rule:
`isCustom ≡
!isTwentyStandardApplicationUniversalIdentifier(applicationUniversalIdentifier)`.
### Server — behavioural checks
| Location | Purpose | Before | Now |
|---|---|---|---|
| `utils/compute-object-target-table.util.ts` | Physical table name `_`
prefix | `computeTableName(nameSingular, objectMetadata.isCustom)` |
derives from `applicationUniversalIdentifier` (single source for all
table-name callers) |
| `twenty-orm/factories/entity-schema.factory.ts` +
`…/entity-schema-metadata.type.ts` | ORM table name (hot path) |
`object.isCustom` | `object.applicationId !== standardApplicationId`
(computed in `buildEntitySchemaMetadataMaps`) |
|
`twenty-orm/repository/workspace-{delete,soft-delete,update}-query-builder.ts`
| Table name for mutations | `computeTableName(nameSingular,
objectMetadata.isCustom)` | `computeObjectTargetTable(objectMetadata)` |
| `index-metadata/utils/generate-deterministic-index-name-v2.ts` | Index
name hash (must stay bit-identical) | `flatObjectMetadata.isCustom` |
derives from `applicationUniversalIdentifier` |
| `object-metadata/object-record-count.service.ts` | Table name for
record count | `computeTableName(nameSingular, isCustom)` |
`computeObjectTargetTable(flatObjectMetadata)` |
|
`workspace-manager/dev-seeder/data/services/dev-seeder-data.service.ts`
| Match seed config by table name | `computeTableName(item.nameSingular,
item.isCustom)` | `computeObjectTargetTable(item)` |
| `commands/workspace-export/workspace-export.service.ts` +
`…/utils/generate-workspace-schema-ddl.util.ts` | Export table name (raw
entity) | `objectMetadata.isCustom` |
`!isTwentyStandard…(objectMetadata.application?.universalIdentifier)` |
|
`flat-field-metadata/services/flat-field-metadata-type-validator.service.ts`
| Block users creating reserved field types |
`args.flatEntityToValidate.isCustom` |
`!args.flatEntityToValidate.isSystem` |
| `api/common/.../common-create-many-query-runner.service.ts` | Don't
let client overwrite system `createdBy` |
`createdByFieldMetadata.isCustom === false` |
`createdByFieldMetadata.isSystem === true` |
|
`field-metadata/utils/resolve-field-metadata-standard-override.util.ts`
| Skip i18n/overrides for custom fields | `if (fieldMetadata.isCustom)
return raw` | **removed** — falls through on
`isDefined(standardOverrides)` |
|
`object-metadata/utils/resolve-object-metadata-standard-override.util.ts`
| Skip i18n/overrides for custom objects | `if (objectMetadata.isCustom)
return raw` | **removed** — same fall-through |
|
`command-menu-item/utils/build-navigation-interpolation-context.util.ts`
| Override context for nav labels | passed `isCustom` into resolver |
dropped (resolver no longer needs it) |
| `api/common/.../data-arg-processor.service.ts` | `isCustom` for
record-position table name | `flatObjectMetadata.isCustom` | derives
from `applicationUniversalIdentifier` |
| `metadata-modules/minimal-metadata/minimal-metadata.service.ts` |
Minimal DTO + override context | `flatObjectMetadata.isCustom` | derives
from `applicationUniversalIdentifier` |
|
`commands/upgrade-version-command/1-23/…backfill-record-page-layouts.command.ts`
| Filter to custom objects | `objectMetadata.isCustom` |
`!isTwentyStandard…(applicationUniversalIdentifier)` |
### Server — DTO / API population
| Location | Before | Now |
|---|---|---|
|
`flat-object-metadata/utils/from-flat-object-metadata-to-object-metadata-dto.util.ts`
| passthrough `isCustom` | derives from `applicationUniversalIdentifier`
|
|
`flat-field-metadata/utils/from-flat-field-metadata-to-field-metadata-dto.util.ts`
| passthrough `isCustom` | derives from `applicationUniversalIdentifier`
|
|
`object-metadata/utils/from-object-metadata-entity-to-object-metadata-dto.util.ts`
(REST) | `entity.isCustom` | `entity.applicationId !==
standardApplicationId` |
|
`field-metadata/utils/from-field-metadata-entity-to-field-metadata-dto.util.ts`
(REST) | `entity.isCustom` | `entity.applicationId !==
standardApplicationId` |
| `dataloaders/dataloader.service.ts` | passed
`flatFieldMetadata.isCustom` into override resolver | dropped (resolver
no longer needs it) |
> REST controllers (`object-metadata.controller.ts`,
`field-metadata.controller.ts`) resolve `standardApplicationId` once per
request from the cached `flatApplicationMaps`.
### Frontend
| Location | Purpose | Before | Now |
|---|---|---|---|
| `settings/.../SettingsObjectFieldDisabledActionDropdown.tsx` | Whether
an inactive field is deletable | `isDeletable = isCustomField` |
`isDeletable = isCustomField && !isSystemField` |
### Unchanged (out of scope)
`isCustom` on `IndexMetadata` / `View` / `Skill` / `Agent` and their
guards still read the persisted column.
Breaking change is on the isCustom filter on field and object APIs, this
is never used in the FE and unlikely used by external consumers
## Why
The five event-log streams (`workspaceEvent`, `pageview`, `objectEvent`,
`usageEvent`, `applicationLog`) each wrote to ClickHouse through their
own fire-and-forget writer (`AuditService`, `UsageEventWriterService`,
and the `application-logs` driver), with the per-type knowledge (table
names, normalization, access rules) spread across several modules. Three
of them reimplemented the same ClickHouse insert, and the read side, the
live stream, and the producers lived in different modules under two
different names.
This consolidates them into one `core-modules/event-logs/` subsystem
(emit, write, live, read), with the per-type config in a single registry
so adding an event type is roughly one file.
The base Logs settings tab and free application logs shipped separately
in #21180 (merged). This PR adds the unified backend, the registry, and
the viewer's live mode and entitlement gating.
## Pipeline
```mermaid
flowchart TB
subgraph PROD["Producers"]
A["auth, billing, impersonation,<br/>webhook, custom-domain"]
U["usage listener"]
F["logic-function executor (app logs)"]
R["record CRUD (entity events)"]
end
EM["EventLogEmitterService<br/>createContext().insert* / dispatch()"]
EQ(["entityEventsToDbQueue<br/>(existing, shared with timeline)"])
CIE["CreateEventLogFromInternalEvent"]
SINK["WorkspaceEventSinkService.ingest()"]
C1["ClickHouseEventSink"]
C2["ConsoleEventSink"]
LIVE["EventLogLiveService.publishWatched()<br/>(presence-gated)"]
CH[("ClickHouse, 5 tables, async_insert")]
CHAN(["WORKSPACE_EVENTS_CHANNEL"])
RS["EventLogsService (registry-driven read)"]
LR["EventLogsLiveResolver"]
UI["Settings > Logs"]
A --> EM
U --> EM
F --> EM
EM -->|direct| SINK
R --> EQ --> CIE -->|ingest| SINK
SINK --> C1 --> CH
SINK --> C2
SINK --> LIVE -.->|if a viewer is watching| CHAN --> LR --> UI
CH --> RS --> UI
```
## What it does
- Producers call `EventLogEmitterService.createContext().insert*()`,
which builds a typed `WorkspaceEventEnvelope` and writes it through
`WorkspaceEventSinkService` to the configured sinks (ClickHouse,
Console) plus a presence-gated live fan-out. Record/CRUD events reach
the same sink through the existing `entityEventsToDbQueue`. There is no
dedicated queue; ClickHouse `async_insert` batches server-side. Writes
are best-effort, as on main today.
- `EVENT_LOG_TYPES[table]` is the per-type source of truth: the
ClickHouse table, the required entitlement, the free-text filter column,
and the row-to-GraphQL mapping. Read row shapes derive from the write
rows.
- Four modules along their dependency boundaries:
`EventLogEmitterModule` (producer API), `EventLogIngestionModule` (sink
layer), `EventLogLiveModule` (fan-out), and `EventLogsViewerModule` (the
entitlement-gated GraphQL read, which is where
billing/enterprise/permissions stay so producers stay light).
- Logs viewer: per-table columns, filters (text, date, record), live
mode, and an upgrade card that points to Billing on Cloud or the Admin
Panel on self-hosted. Application logs are free on every plan; the other
four require the `AUDIT_LOGS` entitlement (with a `NO_ENTITLEMENT`
fallback to the upgrade card).
- Renames `AuditService` to `EventLogEmitterService`, and the generic
`Monitoring` event to a typed `Impersonation` event (`level` +
`action`).
- Removes `UsageEventWriterService`, the `application-logs`
driver/module, and `AuditService`'s direct inserts.
## Durability
Writes are best-effort, the same as main today (the old writers were
fire-and-forget). A dedicated queue was tried mid-PR and removed:
`async_insert` already batches server-side, so the queue only added
durability, which isn't a requirement right now. The `EventSink` seam
keeps a durable transport (e.g. a Redis-Streams buffer) easy to add
later without touching producers.
## Out of scope
S3 peer sink (seam only), Postgres or any second read path,
`ReplicatedMergeTree`, ClickHouse table-schema changes, and the
record-data `EVENT_STREAM_CHANNEL` (unchanged, separate concern).
## Testing
Unit tests cover the registry definitions and row normalization, the
entitlement gating, the envelope builders, and the producers.
Integration tests cover the write paths (record create produces an
`objectEvent`; the track mutation produces a `workspaceEvent`) and the
read/query path across all five tables. Verified with typecheck, lint, a
server boot, and GraphQL/SDK codegen.
## Context
Timeline activities never updated in real time. They were explicitly
excluded from the database-event pipeline
(formatTwentyOrmEventToDatabaseBatchEvent early-returned for the
timelineActivity object), so no SSE event was ever broadcast, and the
frontend timeline only refreshed on mount/manual refetch.
## Implementation
Backend
- Feat: Stop dropping timeline-activity events in
formatTwentyOrmEventToDatabaseBatchEvent.
Instead route them through EntityEventsToDbListener, which publishes
them directly to live subscriptions (but still skipping webhook/audit
handling).
- Fix: Harden ObjectRecordEventPublisher: wrap nested-relation
enrichment in try/catch so a failure broadcasts the event without
relations instead of dropping it (logs a warning).
- Fix: Skip unreadable relation targets in CommonSelectFieldsHelper when
the role lacks canReadObjectRecords, preventing errors while computing
selected fields.
- Fix: Support MORPH_RELATION alongside RELATION in RLS row-level
permission predicate matching (timeline activities use morph targets).
Frontend
- Feat: useTimelineActivities now registers the timeline query with the
SSE system via useListenToEventsForQuery and refetches on incoming
timeline-activity record operations.
- Feat: Add a skip option to useListenToEventsForQuery so the listener
isn't registered when the object has no timeline field.
## Test
https://github.com/user-attachments/assets/ed1d1c66-d6ea-434d-ac9c-9b83d2b78338
Note: "UpdatedBy" seems to be listen to and visible in the timeline
activity summary, this is probably a bug that we want to fix
## Summary
Two intertwined streams of work:
### UI — discovery hero pattern, settings shell, AI/API redesign
- **Generalize `SettingsDiscoveryHeroCard`** and use it on Layout, Data
Model, Apps, AI, API/Webhooks, Members. Drops 4 per-page wrapper files
(`SettingsObjectCoverImage`, `SettingsLayoutCoverImage`,
`SettingsLayoutCustomizeVideoModal`,
`SettingsDataModelVisualizeVideoModal`). Each page now supplies cover
src, modal id, and tab list.
- **Modal**: swap `<video>` placeholder for the Vimeo iframe pattern
from `twenty-docs`, per-tab `vimeoId`. Drop the parallel border-bottom
on the header (TabList draws its own baseline) and the grey background
behind the video. Note: Vimeo's embed allowlist applies — the iframes
load with the correct URL on `localhost` but the player itself requires
the video owner to allow the dev/staging domains in Vimeo settings.
- **AI page** rebuilt into a Cockpit pattern (Overview / Models / Skills
/ Tools / Usage). New `SettingsAiOverviewTab` with default Smart/Fast
pickers, at-a-glance stats, and an MCP signpost that deep-links to
`/settings/api-webhooks#mcp`. System Prompt link moved under Models.
Advanced tab removed.
- **API & Webhooks** now has 4 tabs (Playground / MCP / API Keys /
Webhooks). Hero card above tabs. Playground tab inverted to "Core API" /
"Metadata API" sections, each containing REST + GraphQL cards — schema
is the meaningful axis, protocol is secondary. Hash deep-link sync
delegated to the shared `TabListFromUrlOptionalEffect`.
- **Settings shell**: unified drawer outer padding (kill `isSettings`
branch), extract `CollapsibleNavigationDrawerSection`, add `iconColor`
on settings nav items, fix Exit Settings button alignment, 880px content
cap.
### Backend — strategy C: ephemeral playground token
The legacy paste-your-API-key flow is replaced by an on-demand
short-lived token scoped to the calling user's permissions. No shared
"Playground" API key to manage or revoke.
- New `JwtTokenTypeEnum.PLAYGROUND`. `PlaygroundTokenJwtPayload =
Omit<AccessTokenJwtPayload, 'type' | impersonation fields>` so any
future ACCESS claim flows through automatically.
- `AccessTokenService.generatePlaygroundToken` signs an access-shaped
JWT with `type: PLAYGROUND` and a configurable short TTL. A shared
private `resolveTokenSubject` helper parallelizes the user / workspace /
userWorkspace lookups for both generators.
- `JwtAuthStrategy.validateAccessToken` widened to accept
`AccessTokenJwtPayload | PlaygroundTokenJwtPayload`; impersonation gated
on `payload.type === ACCESS` so the union narrows without `as unknown
as` casts. The two branches in `validate()` collapse into one.
- New `PLAYGROUND_TOKEN_EXPIRES_IN` config var (default `2h`).
- New `generatePlaygroundToken` mutation (`WorkspaceAuthGuard`, no args,
returns `AuthToken`).
- Frontend `useOpenPlayground` hook centralizes mint → atom write →
navigate, with Apollo `onError` snackbar and a "use cached PLAYGROUND
token if still fresh" short-circuit (decodes via `jwt-decode`, checks
both `type` AND `exp`). Old API_KEY tokens left in localStorage from the
prior paste-form flow are rejected on `type` alone and force a re-mint —
this is what was causing the "This API Key is revoked" symptom on stale
browsers.
### Drive-by cleanups
- `PlaygroundToken` DTO removed (identical shape to `AuthToken` already
in use).
- 5 `customize-sidebar.webm` imports and the dead placeholder pipeline
removed.
## Test plan
### Discovery hero
- [ ] `/settings/layout`, `/settings/data-model`,
`/settings/applications`, `/settings/ai`, `/settings/api-webhooks`,
`/settings/members` each render the discovery hero card with its
illustration + play button + tabbed modal
- [ ] Modal tabs show the correct Vimeo embed URL per tab; aspect ratio
stays at 1440/900; no parallel border-bottom jog at the tab baseline
- [ ] AI Overview tab shows Smart/Fast model pickers + stats grid + MCP
signpost card; the MCP card lands on `/settings/api-webhooks#mcp` with
the MCP tab active
### API playground (ephemeral token)
- [ ] With an empty `playgroundApiKeyState` in localStorage, clicking
REST or GraphQL playground card opens the playground and the cached
token has `type: "PLAYGROUND"` with ~2h exp
- [ ] Clicking the card again within the freshness window does **not**
re-mint (`iat` / fingerprint stable across visits)
- [ ] Planting a fake API_KEY-shaped JWT in localStorage and clicking
the card forces a fresh mint (old token rejected on `type`)
- [ ] `GET /rest/companies?limit=1` with the cached token returns 200 +
real data
- [ ] `POST /graphql { __typename }` returns 200
### Settings shell
- [ ] Settings nav matches main app drawer padding; sections collapse;
Exit Settings button aligns with the workspace links above
- [ ] Active nav items have a right-gap (cleaner active state)
- [ ] Content area capped at 880px
### Verify
- [ ] `npx nx typecheck twenty-front` passes
- [ ] `npx nx typecheck twenty-server` passes
- [ ] `npx nx lint:diff-with-main twenty-front` passes
- [ ] `npx nx lint:diff-with-main twenty-server` passes
The SSE event stream could silently die from network partitions, NAT
table flushes, browser tab throttling, or server restarts. When this
happened:
1. The `error` callback only called `captureException` — no reconnection
was triggered
2. The `complete` callback was `() => {}` — a cleanly terminated stream
left the client permanently broken
3. No mechanism existed to detect a silently dead connection where no
FIN/RST was received
## Summary
- **Fix `error`/`complete` callbacks**: The `graphql-sse` subscription's
`error` callback only reported to Sentry, and `complete` was a no-op.
Both now set `shouldDestroyEventStreamState = true` to trigger the
destroy-recreate lifecycle, ensuring detected transport failures and
clean stream terminations lead to automatic reconnection.
- **Add server-side keepalive**: The existing heartbeat timer now runs
every 30s (instead of 6min) and publishes empty events through the Redis
pub/sub channel in addition to refreshing the Redis TTL (throttled to
~6min). Unlike GraphQL Yoga's opaque SSE comment pings, these are real
subscription events that flow through the client's `next`/`message`
handlers.
- **Add client-side keepalive monitor (`SSEKeepAliveEffect`)**: Tracks
the timestamp of the last received event. If no event arrives within 90
seconds (3x the keepalive interval), it clears query listeners and
triggers a stream destroy-recreate cycle.
## Test plan
- [x] Start the app, verify SSE events flow normally (workflow runs
update in real-time)
- [x] Leave the app idle for >90 seconds, then trigger a workflow run —
verify the stream auto-reconnects and events are delivered
- [x] Kill the server, restart it, verify the frontend recovers its
event stream
- [x] Verify keepalive events (empty
`objectRecordEventsWithQueryIds`/`metadataEvents`) appear in browser
network tab every ~30s
- [x] Verify no regressions in SSE-dependent features (record updates,
metadata changes, workflow run visualization)
# Introduction
Adding `no-miused-promise` lint rule to the twenty-server
In order to flag such pattern
```ts
// ❌ Flagged — forEach doesn't await the callback
items.forEach(async (item) => {
await process(item);
});
```
## What happened
- Refactored the code-interpreter driver to have a async onResult (
which is also expected by e2b )
- Workspace manager still dirty solution including force cast
- Basic fixes
## Summary
- Resolve standard field `label`, `description`, and `icon` overrides
through dedicated GraphQL field resolvers.
- Fall back to the source locale safely when the request locale is
missing, and allow direct overrides to apply for non-source locales when
translations are absent.
- Enrich metadata subscription payloads for both `before` and `after`,
reusing the same override application path for field and object
metadata.
- Update and extend tests to cover the revised override behavior.
## Testing
- Updated unit coverage for standard override resolution, including the
non-source-locale fallback path.
- Not run (not requested).
---------
Co-authored-by: Charles Bochet <charles@twenty.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
## Summary
- When an event stream expires (TTL), `addQueryToEventStream` and
`removeQueryFromEventStream` now return `false` instead of throwing
`EVENT_STREAM_DOES_NOT_EXIST` as an `InternalServerError`
- Frontend checks the mutation return value and triggers the
destroy/recreate cycle, same recovery behavior without the error path
- Removes `EVENT_STREAM_DOES_NOT_EXIST` from exception code, exception
filter, and frontend graceful error check since it's no longer thrown
## Test plan
- [x] Verify that when an event stream TTL expires, the frontend
silently recreates the stream without error noise in logs/Sentry
- [x] Verify that `NOT_AUTHORIZED` errors still throw correctly on both
mutations
- [ ] Verify that the subscription `onEventSubscription` still works
end-to-end with stream creation, query registration, and heartbeat TTL
refresh
Made with [Cursor](https://cursor.com)
## Summary
Fixes cross-user cache contamination for AI chat threads in multi-user
workspaces.
`agentChatThread` is the only user-scoped (`userWorkspaceId`-filtered)
entry in `METADATA_NAME_TO_ENTITY_KEY`, but its create/update events
were going through `WorkspaceEventBroadcaster`, which fans out to every
active SSE stream in the workspace. Every client therefore received
other users' threads into their local `agentChatThreads` metadata store
(which is persisted to localStorage). On a subsequent session,
`AgentChatThreadInitializationEffect` would pick the
most-recently-updated thread — potentially another user's — and fire
`GetChatMessages` against it; the server's `userWorkspaceId` filter
didn't match, producing "Thread not found" errors (now 404 thanks to
twentyhq/twenty#19831).
### The fix mirrors the RLS pattern already used for object records
`ObjectRecordEventPublisher` already reads
`streamData.authContext.userWorkspaceId` to filter per subscriber.
Metadata events had no equivalent. This PR closes that gap with a
minimal, opt-in change:
- Add optional `recipientUserWorkspaceIds?: string[]` to
`WorkspaceBroadcastEvent`. Omit → workspace-wide (unchanged for views,
objects, fields, etc.). Set → delivered only to streams whose
`authContext.userWorkspaceId` is in the list.
- `WorkspaceEventBroadcaster.broadcast` builds the payload per stream
and skips events whose recipient doesn't match.
- Both `agentChatThread` broadcast call sites in `AgentChatService` now
pass `recipientUserWorkspaceIds: [userWorkspaceId]`.
### Scope
3 files, +40/-11 LOC:
-
`subscriptions/workspace-event-broadcaster/types/workspace-broadcast-event.type.ts`
-
`subscriptions/workspace-event-broadcaster/workspace-event-broadcaster.service.ts`
- `metadata-modules/ai/ai-chat/services/agent-chat.service.ts`
### Stale cache note
Existing clients still carry poisoned localStorage from before this
lands. They self-heal on sign-out/in because
`clearAllSessionLocalStorageKeys` already drops `agentChatThreads`. If
we want to actively flush on deploy, a frontend cache-version bump can
follow in a separate PR.
### Related
- twentyhq/twenty#19831 turns the resulting "Thread not found" from 500
to 404. This PR addresses the root cause; #19831 stops the Sentry noise.
## Test plan
- [x] `npx nx typecheck twenty-server`
- [x] `npx oxlint --type-aware` on all 3 files (0 warnings/errors)
- [x] `npx prettier --check` on all 3 files
- [ ] Manual: two users in the same workspace, user A creates/sends a
chat thread — confirm user B's session does not receive the event and
their `chatThreads` sidebar is unaffected
- [ ] CI
## Context
Moving isActive filtering to the frontend for page layout tabs and
widgets, hiding inactive entities from the UI while keeping them in
state for future reactivation
Next we will implement deactivated standard tab re-activation during tab
creation (cc @Devessier)
<img width="234" height="303" alt="📋 Menu (Slots)"
src="https://github.com/user-attachments/assets/17a25ac6-55e2-4778-b7f0-e7554ed69704"
/>
## Context
SSE metadata events for overridable entities (viewField, viewFieldGroup,
pageLayoutWidget, pageLayoutTab) were sending raw flat entity data
without resolving overrides into base properties or filtering isActive:
false entities. This caused the frontend to display stale/incorrect
values (e.g., a hidden viewField still appearing visible).
## Implementation
Add a sanitization step in
MetadataEventPublisher.enrichMetadataEventBatch that resolves overrides
and converts isActive transitions into the appropriate event types
(deactivated -> delete, reactivated -> create), matching what GraphQL
resolvers already do
## Summary
- **Queue messages while streaming**: Messages sent during active AI
streaming are queued server-side and auto-flushed when the current
stream completes. Frontend renders queued messages optimistically in a
dedicated queue UI.
- **Drop `@ai-sdk/react` + `resumable-stream`**: Replace the dual HTTP
SSE + AI SDK client architecture with a single GraphQL SSE subscription
per thread. All events (token chunks, message persistence, queue
updates, errors) flow through Redis PubSub → GraphQL subscription.
- **Server-driven architecture**: The server decides whether to queue or
stream (via `POST /:threadId/message`). The frontend mirrors this
decision for optimistic rendering but defers to the server response.
- **Reuse AI SDK accumulation logic**: `readUIMessageStream` from the
`ai` package handles chunk-to-message accumulation on the frontend,
avoiding a custom 780-line accumulator.
## Key files
**Backend:**
- `agent-chat-event-publisher.service.ts` — publishes events to Redis
PubSub
- `agent-chat-subscription.resolver.ts` — GraphQL subscription resolver
- `stream-agent-chat.job.ts` — publishes chunks via PubSub instead of
resumable-stream
- `agent-chat.controller.ts` — unified `POST /:threadId/message`
endpoint
**Frontend:**
- `useAgentChatSubscription.ts` — subscribes to `onAgentChatEvent`,
bridges to `readUIMessageStream`
- `useAgentChat.ts` — send/stop/optimistic rendering (no more AI SDK)
- `AgentChatStreamSubscriptionEffect.tsx` — replaces
`AgentChatAiSdkStreamEffect.tsx`
## Test plan
- [ ] Send message on new thread → optimistic render, streaming response
appears
- [ ] Send message while streaming → queued instantly (no flash in main
thread)
- [ ] Queued message auto-flushes after current stream completes
- [ ] Remove queued message via queue UI
- [ ] Stop streaming mid-response
- [ ] Leave chat idle for several minutes → streaming still works after
(SSE client recycling)
- [ ] Token refresh during session → requests succeed (authenticated
fetch)
- [ ] Switch threads while streaming → clean subscription handoff
Made with [Cursor](https://cursor.com)
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
- enrich SSE events with relations
- remove queries from sse metadata events
- on sse event, manage store
- on object/field metadata changes, manage store
TODO left:
- fix other metadata items
---------
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions <github-actions@twenty.com>
Co-authored-by: Charles Bochet <charles@twenty.com>
## Summary
Replace the single `metadataVersion` integer with per-entity-type
**collection hashes** for granular metadata staleness detection. The
backend already generates a UUID per flat entity map on each cache
recompute (`crypto.randomUUID()` in `WorkspaceCacheService`); we now
expose these via the minimal metadata endpoint and SSE events so the
frontend can compare and know exactly which entity types are stale.
### Key changes
**Backend:**
- `WorkspaceCacheService.getCacheHashes()` — new public method that
reads only `:hash` keys from Redis without fetching full data
- `MinimalMetadataDTO` — added `collectionHashes: Record<string,
string>` (JSON scalar mapping `AllMetadataName` → collection hash),
removed `metadataVersion`
- `MetadataEventDTO` — added optional `updatedCollectionHash` field to
SSE events
- `MetadataEventsToDbListener` — reads the collection hash for the
affected entity type after cache invalidation and attaches it to the SSE
event before publishing
- `MinimalMetadataService` — no longer queries the workspace table; uses
`getCacheHashes()` for all flat entity maps and maps cache keys to
`AllMetadataName` locally
**Frontend:**
- `metadataCollectionHashesState` — new Jotai atom with
`atomWithStorage` + `getOnInit: true` storing
`Partial<Record<MetadataEntityKey, string>>`
- `mapAllMetadataNameToEntityKey()` — explicit mapping from backend
`AllMetadataName` to frontend `MetadataEntityKey` (23 entries)
- `useLoadMinimalMetadata` — stores `collectionHashes` from server,
computes `staleEntityKeys` by comparing local vs server hashes
- `patchMetadataStoreFromSSEEvent()` — accepts optional
`updatedCollectionHash` and updates `metadataCollectionHashesState`
- All 11 SSE effect components — pass
`eventDetail.updatedCollectionHash` through to the patch function
- `useStaleMetadataEntities` — new hook returning entity keys missing
from collection hashes (not yet loaded/synced)
- `resetMetadataStore()` — also clears collection hashes
- Deleted `metadataVersionState` (superseded by collection hashes)
### Design decisions
- **No change to hash generation** — existing `crypto.randomUUID()` is
sufficient. Hashes are persisted in Redis, survive server restarts, and
change only on `invalidateAndRecompute`.
- **"Collection hash" naming** — used consistently to clarify the hash
represents an entire entity collection (e.g., all views), not a single
record.
- **Mapping localized** — backend `WorkspaceCacheKeyName` →
`AllMetadataName` mapping lives in the minimal metadata service.
Frontend `AllMetadataName` → `MetadataEntityKey` mapping lives in a
local utility. Nothing in `twenty-shared`.
- **Backward compatible** — `collectionHashes` is additive;
`updatedCollectionHash` is nullable.
Create the necessary tooling to listen to metadata events and plug it to
the front components. Now we have a hot reload like experience when we
edit a component in an app.
## Backend
- Split `EventWithQueryIds` into `ObjectRecordEventWithQueryIds` and
`MetadataEventWithQueryIds`
- Publish metadata event batches to active SSE streams in
`MetadataEventsToDbListener`
## Frontend
- Create a metadata event dispatching pipeline: SSE metadata events are
grouped by metadata name, transformed into
`MetadataOperationBrowserEventDetail` objects, and dispatched as browser
`CustomEvents`
- Add `useListenToMetadataOperationBrowserEvent` hook for consuming
metadata operation events filtered by metadata name and operation type
- Rename `useListenToObjectRecordEventsForQuery` to
`useListenToEventsForQuery`, now accepting both
`RecordGqlOperationSignature` and `MetadataGqlOperationSignature`
- Implement `useOnFrontComponentUpdated` which subscribes to front
component metadata events and updates the Apollo cache when the
component is modified
- Add `builtComponentChecksum` to the front component query and appends
it to the component URL for browser cache invalidation
This PR handles SSE event stream edge cases.
- When a pod restarts, the front clients have to reconnect to SSE
- When the dev server restarts or is hot reloaded, the front client has
to reconnect to SSE
- When redis server restarts or the redis key is cleared for any reason,
the server has to recreate the event stream in redis, this can happen
when navigating for example.
- Log in / log out flow
With this PR we avoid error messages in the front end due to TTL or pod
crash, we implement a resilient way of reconnecting silently.
To avoid DDoSing our servers if pods crash or a full restart of the
cluster is made, we evenly space retry attempts to reconnect from all
the clients, to avoid n clients reconnection at the same time, we use a
random wait time between 0 and a constant max wait time (set to 2 mins
for now). This is the cheapest and most effective solution, clients who
want to force reconnect have to refresh or navigate to another page.
Fixes https://github.com/twentyhq/core-team-issues/issues/2045
This PR update the redis subscription iterator wrapper with an heartbeat
system. Heartbeat interval are based on event stream TTL. Those refresh
the event stream TTL and active streams in redis.
I also cleaned a few functions that were not used anymore.
---------
Co-authored-by: cubic-dev-ai[bot] <191113872+cubic-dev-ai[bot]@users.noreply.github.com>
This PR implements SSE update events across the main components of the
application : tables, boards, calendars, show pages.
There is still work to do on other event type and on making sure
everything works fine, but this first implementation should be robust
enough to start with.
Some problems encountered along the way :
- Events are returning raw Postgres output, because they are not
normalized by the GraphQL layer, so we ended up with amountMicros as
string values, which the frontend does not like, so I implemented a
small util in our `formatResult` generic pipeline to turn amountMicros
to a number if it's a string value. We could implement other formatters
for composite fields if we see problems with events.
-`action` property was missing in SSE events, which is required by the
frontend to know what kind of event it is.
# QA
https://github.com/user-attachments/assets/393b45ac-59d2-48b0-855b-2ce9c4b8ae57https://github.com/user-attachments/assets/cb214e7a-1595-4b85-bdb6-87630993d2e2
- new channel EVENT_STREAM_CHANNEL based on event stream id
- on event, perform the matching and publish only to the right streams
- store a list of active streams per workspace
- store the user id along with the queries for each stream
Bonus:
- remove onSubscriptionMatch
This PR is a follow-up of https://github.com/twentyhq/twenty/pull/16966
and implements a new mechanism to handle SSE events.
It creates only one event stream per browser tab, then use mutations to
tell the backend which query to listen to, without re-mounting the event
stream connexion.
Then each event that comes from this unique subscription is then
dispatched in a new JavaScript CustomEvent, per queryId, on which
specific hooks add an event listener.
This PR introduces the generic tooling as well as the handling of update
events on table.
- moved a few gql types to twenty shared to re-use in server
- added a new endpoint, onEventSubscription that expect a streamId to
create a connection
- two new endpoints to store queries in Redis
- updated the existing batch channel to directly use object record type
We were using a global ValidationPipe in main.ts. This is an issue as
@Controllers should return HttpExecption and @Resolvers should return
GraphqlErrors
Removing the global pipe and creating a ResolverValidationPipe able to
generate GraphqlError. We also need to handle the exception in a filter
to avoid nest to think it's unhandled and make it flow to logs
Next step:
- it would be nice to have both @UsePipes(ResolverValidationPipe) +
@UseFilters(GraphqlValidationExceptionFilter) come together. This should
be possible if we create a @GraphQLResolver annotation
related to https://github.com/twentyhq/core-team-issues/issues/601
## Done
- add a `onDbEvent` `Subscription` graphql endpoint to listen to
database_event using what we have done with webhooks:
- you can subscribe to any `action` (created, updated, ...) for any
`objectNameSingular` or a specific `recordId`. Parameters are nullable
and treated as wildcards when null.
- returns events with following shape
```typescript
@Field(() => String)
eventId: string;
@Field()
emittedAt: string;
@Field(() => DatabaseEventAction)
action: DatabaseEventAction;
@Field(() => String)
objectNameSingular: string;
@Field(() => GraphQLJSON)
record: ObjectRecord;
@Field(() => [String], { nullable: true })
updatedFields?: string[];
```
- front provide a componentEffect `<ListenRecordUpdatesEffect />` that
listen for an `objectNameSingular`, a `recordId` and a list of
`listenedFields`. It subscribes to record updates and updates its apollo
cached value for specified `listenedFields`
- subscription is protected with credentials
## Result
Here is an application with `workflowRun`
https://github.com/user-attachments/assets/c964d857-3b54-495f-bf14-587ba26c5a8c
---------
Co-authored-by: prastoin <paul@twenty.com>
Charles Bochet