## Context
#21954 raised `MAX_LOCAL_CACHE_ENTRIES` from 1,000 to 7,500 to stop the
per-pod workspace metadata cache from thrashing against Redis. That
worked: the cache node's egress dropped from ~1.1-1.8 TB/day to ~0.5-0.7
TB/day and has stayed there.
The memory side of the trade turned out tighter than the sizing assumed.
Prod-eu metrics over the past month:
- Average server pod working set rose from ~1.9-2.4 GiB (before the
rollout on 06/26) to ~3.0-3.4 GiB, with individual pods peaking at
3.5-3.6 GiB, right at the `--max-old-space-size=3500` heap ceiling on 4
GiB pods.
- The `workspace-metadata-cache/local-eviction` counter added in #21954
stayed at zero for three weeks, then on 07/15 between 08:30 and 09:30
UTC all 7 server pods hit the 7,500-entry cap within an hour (no deploy
that morning; organic growth crossed the threshold during the European
morning peak). Since then the fleet evicts continuously (~167k
entries/day).
- Server pods were terminated with reason OOMKilled on 8 days of the
past month (1-2 pods each time).
The per-entry payload also grew since the cap was sized: search field
metadata backfills, two new per-workspace cache components
(`flatSearchFieldMetadataMaps`, `workflowAutomatedTriggerMaps`), and the
heap-only `ORMEntityMetadatas` entries. So the byte ceiling implied by
7,500 entries now sits exactly at the heap limit instead of comfortably
under it.
## What this does
Lower `MAX_LOCAL_CACHE_ENTRIES` 7,500 → 6,000. At the measured ~170-200
KB average entry size this frees roughly 300-500 MB of steady-state heap
per pod, restoring headroom under the 3.5 GB old-space limit.
Traffic cost should be small: eviction churn at the 7,500 cap only moved
cache egress from ~0.55 to ~0.65-0.68 TB/day, so a 20% smaller cap
should keep the bulk of the #21954 reduction. The eviction counter gives
us the feedback loop; if egress climbs materially we can split the
difference, and if pods still run hot the next lever is byte-aware
eviction for the few known-heavy keys.
Charles Bochet
9860871eac