From 9860871eaccdab8846d0f22cedd6091004085e54 Mon Sep 17 00:00:00 2001 From: Charles Bochet Date: Thu, 16 Jul 2026 12:34:02 +0200 Subject: [PATCH] perf(server): lower workspace local cache cap to 6,000 entries (#22946) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Context #21954 raised `MAX_LOCAL_CACHE_ENTRIES` from 1,000 to 7,500 to stop the per-pod workspace metadata cache from thrashing against Redis. That worked: the cache node's egress dropped from ~1.1-1.8 TB/day to ~0.5-0.7 TB/day and has stayed there. The memory side of the trade turned out tighter than the sizing assumed. Prod-eu metrics over the past month: - Average server pod working set rose from ~1.9-2.4 GiB (before the rollout on 06/26) to ~3.0-3.4 GiB, with individual pods peaking at 3.5-3.6 GiB, right at the `--max-old-space-size=3500` heap ceiling on 4 GiB pods. - The `workspace-metadata-cache/local-eviction` counter added in #21954 stayed at zero for three weeks, then on 07/15 between 08:30 and 09:30 UTC all 7 server pods hit the 7,500-entry cap within an hour (no deploy that morning; organic growth crossed the threshold during the European morning peak). Since then the fleet evicts continuously (~167k entries/day). - Server pods were terminated with reason OOMKilled on 8 days of the past month (1-2 pods each time). The per-entry payload also grew since the cap was sized: search field metadata backfills, two new per-workspace cache components (`flatSearchFieldMetadataMaps`, `workflowAutomatedTriggerMaps`), and the heap-only `ORMEntityMetadatas` entries. So the byte ceiling implied by 7,500 entries now sits exactly at the heap limit instead of comfortably under it. ## What this does Lower `MAX_LOCAL_CACHE_ENTRIES` 7,500 → 6,000. At the measured ~170-200 KB average entry size this frees roughly 300-500 MB of steady-state heap per pod, restoring headroom under the 3.5 GB old-space limit. Traffic cost should be small: eviction churn at the 7,500 cap only moved cache egress from ~0.55 to ~0.65-0.68 TB/day, so a 20% smaller cap should keep the bulk of the #21954 reduction. The eviction counter gives us the feedback loop; if egress climbs materially we can split the difference, and if pods still run hot the next lever is byte-aware eviction for the few known-heavy keys. --- .../engine/workspace-cache/services/workspace-cache.service.ts | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/packages/twenty-server/src/engine/workspace-cache/services/workspace-cache.service.ts b/packages/twenty-server/src/engine/workspace-cache/services/workspace-cache.service.ts index 5adc529a2f..e79b55030b 100644 --- a/packages/twenty-server/src/engine/workspace-cache/services/workspace-cache.service.ts +++ b/packages/twenty-server/src/engine/workspace-cache/services/workspace-cache.service.ts @@ -35,7 +35,8 @@ const LOCAL_ENTRY_TTL_MS = 30 * 60 * 1000; // 30 minutes const MEMOIZER_TTL_MS = 10_000; // 10 seconds const STALE_VERSION_TTL_MS = 5_000; // 5 seconds const MAX_LOCAL_STALE_VERSIONS = 5; // 5 stale versions -const MAX_LOCAL_CACHE_ENTRIES = 7_500; +// Sized against 4 GiB pods (--max-old-space-size=3500): 7,500 sat at the heap ceiling +const MAX_LOCAL_CACHE_ENTRIES = 6_000; const MIN_EVICT_KEYS = 100; type CacheDataType = WorkspaceCacheDataMap[WorkspaceCacheKeyName];