Files
twenty/packages/twenty-server/test/integration/metadata/suites/graphql-operation-cache/metadata-graphql-operations-cache-dependencies.integration-spec.ts
T
Weiko 0108a34765 Rekey metadata caches when flat map hashes change (#23164)
## Context

The `/metadata` GraphQL response cache (`ObjectMetadataItems`,
`FindAllViews`) and the workspace SDL cache were keyed on
`workspace.metadataVersion`, an integer bumped on every object/field
migration. That mechanism is legacy (the migration runner literally
calls it `getLegacyCacheInvalidationPromises`): the data plane already
moved to `WorkspaceCacheService`, which versions each flat entity map
with its own hash minted on invalidation.

Version keying had two concrete costs: the version bump was the only
proactive invalidation for `ObjectMetadataItems`, and keeping
`FindAllViews` fresh required `flushGraphQLOperation`, a full Redis
keyspace SCAN on every relevant migration. It is also the main blocker
for deprecating `metadataVersion` entirely.

This PR re-keys both caches on the flat-map hashes instead.

## What changed

**Response cache** (`use-cached-metadata.ts`): the key is now
`{operation}:{workspaceId}:{combinedDependencyHash}[:{userWorkspaceId}]:{locale}:{queryHash}`.
Each cached operation declares which flat maps its resolvers read
(`metadata-graphql-operations-to-cache.constant.ts`) plus a scope:
`ObjectMetadataItems` stays workspace-shared, `FindAllViews` is per-user
because unlisted-view visibility depends on the caller. When any
declared map changes, its hash rotates and the key rotates with it; no
flush needed. The key is resolved once per request and reused in
`onResponse`, so a rotation mid-request can never cache a response under
a fresher key than the data it was built from. If hash resolution fails,
the request is served uncached (Sentry-captured).

This also fixes three pre-existing key soundness gaps: `FindAllViews`
ignored locale although view names are translated server-side, the query
hash ignored GraphQL variables (`$viewTypes`), and mid-request version
rotation could re-key between request and response.

**SDL cache** (`workspace-graphql-schema-sdl.service.ts`): keyed on the
combined hash of the four maps the schema is generated from, taken from
the same `getOrRecomputeWithHashes` call that returns the data, so key
and content cannot skew. The `metadataVersion` read/seed block there is
gone; the Redis seed moved to `middleware.service.ts` so the
`X-Schema-Version` "refresh the page" check keeps working after the
Redis key's TTL expires.

**`WorkspaceCacheService`**: the internal pipeline now threads `{data,
hashes}` through every stage (local hit, hash validation, Redis fetch,
recompute) and the memoizer stores the pair, so returned hashes are
always consistent with returned data. New public
`getOrRecomputeWithHashes` and `getOrRecomputeCombinedHash`
(hashes-first: one MGET of the small `:hash` keys, full pipeline only
for missing ones, so cold pods never pull map payloads just to build a
key).

**Atomic pair writes** (`cache-storage.service.ts`): `mset` on the Redis
driver now delegates to the store's own `mset` (a MULTI of `SET ... PX`,
or native `MSET`), grouped by TTL. Previously it was a `Promise.all` of
independent SETs, so two concurrent recomputes could interleave and
leave one recompute's `:data` next to the other's `:hash`; with
hash-keyed caches that torn pair could persist a stale response under a
live key. `CoreEntityCacheService` writes through the same method and is
fixed for free.

**Cleanup**: `flush()` lost its `metadataVersion` parameter (always
pattern-flush per key on workspace deletion),
`METADATA_VERSIONED_WORKSPACE_CACHE_KEY` became
`HASH_KEYED_WORKSPACE_CACHE_KEYS` with the `MetadataVersion` key
relocated to `WORKSPACE_CACHE_KEYS` and the dead `ORMEntitySchemas`
entry removed.

## Deliberately unchanged

- `incrementMetadataVersion` and all its callers stay: the version still
feeds the `X-Schema-Version` check and the pinned upgrade commands.
Deprecating the column is a later stage.
- The runner's `FindAllViews` pattern-flush is kept for exactly one
release: view-only migrations never bump `metadataVersion`, so old pods
in a rolling deploy have no other invalidation signal for their
version-keyed entries. It gets deleted next release, which removes the
SCAN entirely.
- Old-shape cache entries are not migrated; they expire via the 7-day
TTL.

## Known limitations (follow-ups, not regressions)

- The plugin reads dependency hashes Redis-fresh while resolvers can
serve up to 10s-old memoized data, so a request landing right after a
migration can cache a pre-rotation response under the new key. Same
shape existed under `metadataVersion`; closing it needs request-scoped
snapshot plumbing.
- Concurrent recomputes are last-writer-wins (lost update). Fencing with
a conditional write is a follow-up.

## Validation

- Unit: response-cache plugin behavior (scope, key stash, serve-uncached
on failure, prototype-name guard), atomic `mset` batching, existing
`WorkspaceCacheService` spec passing unchanged.
- Integration: a new drift-guard spec runs the real
`ObjectMetadataItems`/`FindAllViews` operations with full frontend
selection sets against the in-process app, spies on
`WorkspaceCacheService`, and fails if resolvers read a flat map missing
from the declared dependency lists, so the constant cannot silently
drift.
- Manual against a live server: creating a field rotates the field-map
hash and the very next `ObjectMetadataItems` response contains it (hash
rotation is now its only invalidation path); warm hits are ~5ms; SDL
entries appear under hash-shaped keys via introspection.

## Suggested reading order

1. `workspace-cache.service.ts`, `workspace-cache-key.type.ts`,
`combine-cache-hashes.util.ts` (the `{data, hashes}` pipeline)
2. `use-cached-metadata.ts`,
`metadata-graphql-operations-to-cache.constant.ts`,
`metadata.module-factory.ts` (response cache)
3. `workspace-graphql-schema-sdl.service.ts`,
`workspace-cache-storage.service.ts` (SDL cache and renames)
4. `middleware.service.ts` (metadata version seed relocation)
5. `cache-storage.service.ts` (atomic writes)
6. Tests
2026-07-22 16:50:09 +00:00

296 lines
7.4 KiB
TypeScript

import { gql } from 'graphql-tag';
import { makeMetadataAPIRequest } from 'test/integration/metadata/suites/utils/make-metadata-api-request.util';
import { getAppProviderByClassName } from 'test/integration/utils/get-app-provider-by-class-name.util';
import { type ASTNode } from 'graphql';
import { METADATA_GRAPHQL_OPERATIONS_TO_CACHE } from 'src/engine/api/graphql/graphql-config/constants/metadata-graphql-operations-to-cache.constant';
import { FIND_ALL_VIEWS_GRAPHQL_OPERATION } from 'src/engine/metadata-modules/view/constants/find-all-views-graphql-operation.constant';
import { type WorkspaceCacheService } from 'src/engine/workspace-cache/services/workspace-cache.service';
import { type WorkspaceCacheKeyName } from 'src/engine/workspace-cache/types/workspace-cache-key.type';
// Mirrors OBJECT_METADATA_FRAGMENT from twenty-front
// (packages/twenty-front/src/modules/object-metadata/graphql/fragment.ts):
// the recorded resolver reads are only as complete as this selection set.
const OBJECT_METADATA_ITEMS_QUERY = gql`
query ObjectMetadataItems {
objects(paging: { first: 1000 }) {
edges {
node {
id
universalIdentifier
nameSingular
namePlural
labelSingular
labelPlural
color
description
icon
isRemote
isActive
isSystem
isUIEditable
isUICreatable
createdAt
updatedAt
labelIdentifierFieldMetadataId
imageIdentifierFieldMetadataId
applicationId
shortcut
isLabelSyncedWithName
isSearchable
duplicateCriteria
searchFieldMetadataList {
id
fieldMetadataId
tsVectorFieldMetadataId
position
}
indexMetadataList {
id
name
indexWhereClause
indexType
isUnique
isCustom
indexFieldMetadataList {
id
fieldMetadataId
subFieldName
order
}
}
fieldsList {
id
universalIdentifier
type
name
label
description
icon
isActive
isSystem
isUIEditable
isNullable
isUnique
defaultValue
options
settings
isLabelSyncedWithName
morphId
applicationId
relation {
type
sourceObjectMetadata {
id
nameSingular
namePlural
}
targetObjectMetadata {
id
nameSingular
namePlural
}
sourceFieldMetadata {
id
name
}
targetFieldMetadata {
id
name
}
}
morphRelations {
type
sourceObjectMetadata {
id
nameSingular
namePlural
}
targetObjectMetadata {
id
nameSingular
namePlural
}
sourceFieldMetadata {
id
name
}
targetFieldMetadata {
id
name
}
}
}
}
}
}
}
`;
const OBJECT_METADATA_ITEMS_BASELINE_QUERY = gql`
query ObjectMetadataItems {
objects(paging: { first: 1 }) {
edges {
node {
id
}
}
}
}
`;
// Mirrors VIEW_FRAGMENT from twenty-front
// (packages/twenty-front/src/modules/views/graphql/fragments/viewFragment.ts).
const FIND_ALL_VIEWS_QUERY = gql`
query FindAllViews {
getViews {
id
name
objectMetadataId
type
key
icon
position
isCompact
openRecordIn
kanbanAggregateOperation
kanbanAggregateOperationFieldMetadataId
mainGroupByFieldMetadataId
shouldHideEmptyGroups
kanbanColumnWidth
anyFieldFilterValue
calendarFieldMetadataId
calendarEndFieldMetadataId
calendarLayout
visibility
createdByUserWorkspaceId
isActive
viewFields {
id
fieldMetadataId
isVisible
size
position
aggregateOperation
}
viewFieldGroups {
id
name
position
isVisible
viewId
isActive
viewFields {
id
fieldMetadataId
isVisible
size
position
aggregateOperation
}
}
viewFilters {
id
fieldMetadataId
operand
value
viewFilterGroupId
positionInViewFilterGroup
subFieldName
}
viewFilterGroups {
id
parentViewFilterGroupId
logicalOperator
positionInViewFilterGroup
}
viewSorts {
id
fieldMetadataId
direction
}
viewGroups {
id
isVisible
fieldValue
position
viewId
}
}
}
`;
const FIND_ALL_VIEWS_BASELINE_QUERY = gql`
query FindAllViews {
getViews {
id
}
}
`;
describe('metadata GraphQL operations cache dependencies', () => {
const recordAccessedCacheKeys = async (
query: ASTNode,
): Promise<Set<WorkspaceCacheKeyName>> => {
const workspaceCacheService =
getAppProviderByClassName<WorkspaceCacheService>('WorkspaceCacheService');
const spy = jest.spyOn(workspaceCacheService, 'getOrRecomputeWithHashes');
try {
const response = await makeMetadataAPIRequest({ query });
expect(response.body.errors).toBeUndefined();
return new Set<WorkspaceCacheKeyName>(
spy.mock.calls.flatMap(([, cacheKeyNames]) => cacheKeyNames),
);
} finally {
spy.mockRestore();
}
};
const findUndeclaredDependencies = async ({
operationName,
fullQuery,
baselineQuery,
}: {
operationName: string;
fullQuery: ASTNode;
baselineQuery: ASTNode;
}): Promise<WorkspaceCacheKeyName[]> => {
const declaredDependencies = new Set<WorkspaceCacheKeyName>(
METADATA_GRAPHQL_OPERATIONS_TO_CACHE[operationName].dependencies,
);
const requestInfrastructureKeys =
await recordAccessedCacheKeys(baselineQuery);
const accessedCacheKeys = await recordAccessedCacheKeys(fullQuery);
return [...accessedCacheKeys].filter(
(cacheKeyName) =>
!declaredDependencies.has(cacheKeyName) &&
!requestInfrastructureKeys.has(cacheKeyName),
);
};
it('declares every flat map the ObjectMetadataItems resolvers read', async () => {
const undeclaredDependencies = await findUndeclaredDependencies({
operationName: 'ObjectMetadataItems',
fullQuery: OBJECT_METADATA_ITEMS_QUERY,
baselineQuery: OBJECT_METADATA_ITEMS_BASELINE_QUERY,
});
expect(undeclaredDependencies).toEqual([]);
});
it('declares every flat map the FindAllViews resolvers read', async () => {
const undeclaredDependencies = await findUndeclaredDependencies({
operationName: FIND_ALL_VIEWS_GRAPHQL_OPERATION,
fullQuery: FIND_ALL_VIEWS_QUERY,
baselineQuery: FIND_ALL_VIEWS_BASELINE_QUERY,
});
expect(undeclaredDependencies).toEqual([]);
});
});