perf(server): lower workspace local cache cap to 6,000 entries (#22946)

## Context

#21954 raised `MAX_LOCAL_CACHE_ENTRIES` from 1,000 to 7,500 to stop the
per-pod workspace metadata cache from thrashing against Redis. That
worked: the cache node's egress dropped from ~1.1-1.8 TB/day to ~0.5-0.7
TB/day and has stayed there.

The memory side of the trade turned out tighter than the sizing assumed.
Prod-eu metrics over the past month:

- Average server pod working set rose from ~1.9-2.4 GiB (before the
rollout on 06/26) to ~3.0-3.4 GiB, with individual pods peaking at
3.5-3.6 GiB, right at the `--max-old-space-size=3500` heap ceiling on 4
GiB pods.
- The `workspace-metadata-cache/local-eviction` counter added in #21954
stayed at zero for three weeks, then on 07/15 between 08:30 and 09:30
UTC all 7 server pods hit the 7,500-entry cap within an hour (no deploy
that morning; organic growth crossed the threshold during the European
morning peak). Since then the fleet evicts continuously (~167k
entries/day).
- Server pods were terminated with reason OOMKilled on 8 days of the
past month (1-2 pods each time).

The per-entry payload also grew since the cap was sized: search field
metadata backfills, two new per-workspace cache components
(`flatSearchFieldMetadataMaps`, `workflowAutomatedTriggerMaps`), and the
heap-only `ORMEntityMetadatas` entries. So the byte ceiling implied by
7,500 entries now sits exactly at the heap limit instead of comfortably
under it.

## What this does

Lower `MAX_LOCAL_CACHE_ENTRIES` 7,500 → 6,000. At the measured ~170-200
KB average entry size this frees roughly 300-500 MB of steady-state heap
per pod, restoring headroom under the 3.5 GB old-space limit.

Traffic cost should be small: eviction churn at the 7,500 cap only moved
cache egress from ~0.55 to ~0.65-0.68 TB/day, so a 20% smaller cap
should keep the bulk of the #21954 reduction. The eviction counter gives
us the feedback loop; if egress climbs materially we can split the
difference, and if pods still run hot the next lever is byte-aware
eviction for the few known-heavy keys.

This commit is contained in:

Charles Bochet

2026-07-16 12:34:02 +02:00

committed by

GitHub

parent 1b9152d4c5

commit 9860871eac

1 changed files with 2 additions and 1 deletions

									
										packages/twenty-server/src/engine/workspace-cache/services/workspace-cache.service.ts
									
		+2
		-1
	
												View File
												
				@@ -35,7 +35,8 @@ const LOCAL_ENTRY_TTL_MS = 30 * 60 * 1000; // 30 minutes

				const MEMOIZER_TTL_MS = 10_000; // 10 seconds

				const STALE_VERSION_TTL_MS = 5_000; // 5 seconds

				const MAX_LOCAL_STALE_VERSIONS = 5; // 5 stale versions

				const MAX_LOCAL_CACHE_ENTRIES = 7_500;

				// Sized against 4 GiB pods (--max-old-space-size=3500): 7,500 sat at the heap ceiling

				const MAX_LOCAL_CACHE_ENTRIES = 6_000;

				const MIN_EVICT_KEYS = 100;

				type CacheDataType = WorkspaceCacheDataMap[WorkspaceCacheKeyName];