## What
Two related changes to the per-pod local workspace-metadata cache
(`WorkspaceCacheService`):
1. **Occupancy metrics** — per-pod gauges so we can measure how the
cache is actually used from prod instead of guessing:
- `twenty_workspace_cache_local_entries` — Map size
- `twenty_workspace_cache_local_workspaces` — distinct workspaces held
- `twenty_workspace_cache_local_versions_total` — total versions across
entries
- `twenty_workspace_cache_local_bytes_estimate` /
`..._bytes_by_provider{provider}` — sampled deep-size (circular-safe,
includes `localDataOnly` providers)
-
`twenty_workspace_cache_local_entries_by_version_count{versions=1|2|3|4|5+}`
— stale-version distribution
2. **Per-provider eviction budget (behavior change)** — a heavy provider
can override the global entry cap, and eviction now drops the
least-recently-read entry. `ORMEntityMetadatas` is capped at 128
entries.
## Why
Measured on a prod pod (with these gauges plus a live heap walk): the
local cache is ~1.5-2 GB of the pod's ~2.4 GB live heap, and the pod
sits at 85% of its 4 GB limit. Two providers own 89% of it:
- `orm:entity-metadatas` — **50%**, ~5 MB/entry (the full TypeORM
`EntityMetadata[]` graph), `localDataOnly` so it is pure per-pod RAM.
- `flat-maps:field-metadata` — 39%.
The pod held 434 ORM entries but served under one distinct workspace per
second, with 62% of entries idle for more than 5 minutes — it hoards.
Rebuilding an ORM entry is cheap (6-16 ms of synchronous CPU; the DB
read dominates the rest of a recompute), so bounding how many we retain
is nearly free.
## The eviction change
The only size control before this was a single global 6000-entry LRU,
which is byte-blind: a 5 MB ORM entry and a 760 B webhook entry each
count as "1", so "6000 entries" is anywhere from 300 MB to 3 GB.
This adds a **per-provider entry cap** and evicts the
**least-recently-read** entry, keyed on `version.lastReadAt` (replacing
the coarser `lastHashCheckedAt`, which was zeroed on invalidation and
only 100 ms-granular). `ORMEntityMetadatas` → 128 entries, down from 434
observed in prod: ≈640 MB at the measured ~1.0 versions/entry (worst
case bounded by `128 × MAX_LOCAL_STALE_VERSIONS`), against ~2.1 GB
today. The global 6000 cap remains as a coarse total backstop; providers
without an override are unchanged.
A cold miss (a workspace served again after its entry was evicted)
recomputes transparently. At the measured activation rate that is under
one rebuild per second per pod — well below 1% of a core. Cache
correctness is unchanged: entries are hash-versioned and disposable.
## Cost
- **Metrics**: stats are one pass over the local Map, memoized 5 s so
concurrent gauge callbacks reuse them. The byte estimate is a background
sampler (first run ~30 s after startup, then every 5 min) that
deep-sizes a few entries per provider, node-capped and yielding between
walks — off the request and scrape paths.
- **Eviction**: the per-provider cap is enforced on write; a cold-miss
rebuild is the 6-16 ms recompute above.
Charles Bochet
6abeb7b5e5