Files
twenty/packages/twenty-server
Weiko 24ccdb9b5d Replace Redis key scanning with sorted-set event stream tracking (#23326)
## Context

The `twenty_event_streams_live_total` gauge counted live streams by
SCANning every `workspace:*:activeStreams` key and summing set
cardinalities. A full metric refresh walks the entire Redis keyspace, on
every server instance, and its cost grows with unrelated cache data
rather than with the number of streams.

## What changed

Adds a metric-only sorted set, `activeStreamExpirations`. Members are
`workspaceId:eventStreamChannelId`, scores are expiration timestamps:

- Create and successful heartbeat refresh: `ZADD` with score `now +
EVENT_STREAM_TTL_MS`
- Destroy and stale cleanup: `ZREM`
- Gauge read: `ZREMRANGEBYSCORE` + `ZCARD` in one transaction

The scan-and-count cache helper is removed.

Scores are written at the same moments the stream key TTL is set, so a
member expires exactly when its stream key would. Any missed cleanup
(crashed pod, failed heartbeat) resolves itself at the next gauge read.
Metric writes are best effort: failures are logged and never affect
stream creation, refresh, or cleanup. Existing stream keys and
application behavior are unchanged, and no migration is needed.

## Tradeoffs

- One extra `ZADD` per 30-second heartbeat
- During a rolling deploy, streams owned by old pods appear in the gauge
after their next heartbeat (undercount bounded by one heartbeat
interval)
- A destroy racing a concurrent refresh can leave one orphaned member
until its score lapses (gauge over-counts by 1 for at most one TTL)

## Testing

- Unit coverage for the sorted-set cache helpers and the stream
lifecycle (create, refresh success/failure, destroy, stale cleanup)
- `npx nx typecheck twenty-server`, targeted Oxlint, 14 tests passing
2026-07-27 16:35:54 +00:00
..