0108a34765
## Context
The `/metadata` GraphQL response cache (`ObjectMetadataItems`,
`FindAllViews`) and the workspace SDL cache were keyed on
`workspace.metadataVersion`, an integer bumped on every object/field
migration. That mechanism is legacy (the migration runner literally
calls it `getLegacyCacheInvalidationPromises`): the data plane already
moved to `WorkspaceCacheService`, which versions each flat entity map
with its own hash minted on invalidation.
Version keying had two concrete costs: the version bump was the only
proactive invalidation for `ObjectMetadataItems`, and keeping
`FindAllViews` fresh required `flushGraphQLOperation`, a full Redis
keyspace SCAN on every relevant migration. It is also the main blocker
for deprecating `metadataVersion` entirely.
This PR re-keys both caches on the flat-map hashes instead.
## What changed
**Response cache** (`use-cached-metadata.ts`): the key is now
`{operation}:{workspaceId}:{combinedDependencyHash}[:{userWorkspaceId}]:{locale}:{queryHash}`.
Each cached operation declares which flat maps its resolvers read
(`metadata-graphql-operations-to-cache.constant.ts`) plus a scope:
`ObjectMetadataItems` stays workspace-shared, `FindAllViews` is per-user
because unlisted-view visibility depends on the caller. When any
declared map changes, its hash rotates and the key rotates with it; no
flush needed. The key is resolved once per request and reused in
`onResponse`, so a rotation mid-request can never cache a response under
a fresher key than the data it was built from. If hash resolution fails,
the request is served uncached (Sentry-captured).
This also fixes three pre-existing key soundness gaps: `FindAllViews`
ignored locale although view names are translated server-side, the query
hash ignored GraphQL variables (`$viewTypes`), and mid-request version
rotation could re-key between request and response.
**SDL cache** (`workspace-graphql-schema-sdl.service.ts`): keyed on the
combined hash of the four maps the schema is generated from, taken from
the same `getOrRecomputeWithHashes` call that returns the data, so key
and content cannot skew. The `metadataVersion` read/seed block there is
gone; the Redis seed moved to `middleware.service.ts` so the
`X-Schema-Version` "refresh the page" check keeps working after the
Redis key's TTL expires.
**`WorkspaceCacheService`**: the internal pipeline now threads `{data,
hashes}` through every stage (local hit, hash validation, Redis fetch,
recompute) and the memoizer stores the pair, so returned hashes are
always consistent with returned data. New public
`getOrRecomputeWithHashes` and `getOrRecomputeCombinedHash`
(hashes-first: one MGET of the small `:hash` keys, full pipeline only
for missing ones, so cold pods never pull map payloads just to build a
key).
**Atomic pair writes** (`cache-storage.service.ts`): `mset` on the Redis
driver now delegates to the store's own `mset` (a MULTI of `SET ... PX`,
or native `MSET`), grouped by TTL. Previously it was a `Promise.all` of
independent SETs, so two concurrent recomputes could interleave and
leave one recompute's `:data` next to the other's `:hash`; with
hash-keyed caches that torn pair could persist a stale response under a
live key. `CoreEntityCacheService` writes through the same method and is
fixed for free.
**Cleanup**: `flush()` lost its `metadataVersion` parameter (always
pattern-flush per key on workspace deletion),
`METADATA_VERSIONED_WORKSPACE_CACHE_KEY` became
`HASH_KEYED_WORKSPACE_CACHE_KEYS` with the `MetadataVersion` key
relocated to `WORKSPACE_CACHE_KEYS` and the dead `ORMEntitySchemas`
entry removed.
## Deliberately unchanged
- `incrementMetadataVersion` and all its callers stay: the version still
feeds the `X-Schema-Version` check and the pinned upgrade commands.
Deprecating the column is a later stage.
- The runner's `FindAllViews` pattern-flush is kept for exactly one
release: view-only migrations never bump `metadataVersion`, so old pods
in a rolling deploy have no other invalidation signal for their
version-keyed entries. It gets deleted next release, which removes the
SCAN entirely.
- Old-shape cache entries are not migrated; they expire via the 7-day
TTL.
## Known limitations (follow-ups, not regressions)
- The plugin reads dependency hashes Redis-fresh while resolvers can
serve up to 10s-old memoized data, so a request landing right after a
migration can cache a pre-rotation response under the new key. Same
shape existed under `metadataVersion`; closing it needs request-scoped
snapshot plumbing.
- Concurrent recomputes are last-writer-wins (lost update). Fencing with
a conditional write is a follow-up.
## Validation
- Unit: response-cache plugin behavior (scope, key stash, serve-uncached
on failure, prototype-name guard), atomic `mset` batching, existing
`WorkspaceCacheService` spec passing unchanged.
- Integration: a new drift-guard spec runs the real
`ObjectMetadataItems`/`FindAllViews` operations with full frontend
selection sets against the in-process app, spies on
`WorkspaceCacheService`, and fails if resolvers read a flat map missing
from the declared dependency lists, so the constant cannot silently
drift.
- Manual against a live server: creating a field rotates the field-map
hash and the very next `ObjectMetadataItems` response contains it (hash
rotation is now its only invalidation path); warm hits are ~5ms; SDL
entries appear under hash-shaped keys via introspection.
## Suggested reading order
1. `workspace-cache.service.ts`, `workspace-cache-key.type.ts`,
`combine-cache-hashes.util.ts` (the `{data, hashes}` pipeline)
2. `use-cached-metadata.ts`,
`metadata-graphql-operations-to-cache.constant.ts`,
`metadata.module-factory.ts` (response cache)
3. `workspace-graphql-schema-sdl.service.ts`,
`workspace-cache-storage.service.ts` (SDL cache and renames)
4. `middleware.service.ts` (metadata version seed relocation)
5. `cache-storage.service.ts` (atomic writes)
6. Tests
296 lines
7.4 KiB
TypeScript
296 lines
7.4 KiB
TypeScript
import { gql } from 'graphql-tag';
|
|
import { makeMetadataAPIRequest } from 'test/integration/metadata/suites/utils/make-metadata-api-request.util';
|
|
import { getAppProviderByClassName } from 'test/integration/utils/get-app-provider-by-class-name.util';
|
|
|
|
import { type ASTNode } from 'graphql';
|
|
|
|
import { METADATA_GRAPHQL_OPERATIONS_TO_CACHE } from 'src/engine/api/graphql/graphql-config/constants/metadata-graphql-operations-to-cache.constant';
|
|
import { FIND_ALL_VIEWS_GRAPHQL_OPERATION } from 'src/engine/metadata-modules/view/constants/find-all-views-graphql-operation.constant';
|
|
import { type WorkspaceCacheService } from 'src/engine/workspace-cache/services/workspace-cache.service';
|
|
import { type WorkspaceCacheKeyName } from 'src/engine/workspace-cache/types/workspace-cache-key.type';
|
|
|
|
// Mirrors OBJECT_METADATA_FRAGMENT from twenty-front
|
|
// (packages/twenty-front/src/modules/object-metadata/graphql/fragment.ts):
|
|
// the recorded resolver reads are only as complete as this selection set.
|
|
const OBJECT_METADATA_ITEMS_QUERY = gql`
|
|
query ObjectMetadataItems {
|
|
objects(paging: { first: 1000 }) {
|
|
edges {
|
|
node {
|
|
id
|
|
universalIdentifier
|
|
nameSingular
|
|
namePlural
|
|
labelSingular
|
|
labelPlural
|
|
color
|
|
description
|
|
icon
|
|
isRemote
|
|
isActive
|
|
isSystem
|
|
isUIEditable
|
|
isUICreatable
|
|
createdAt
|
|
updatedAt
|
|
labelIdentifierFieldMetadataId
|
|
imageIdentifierFieldMetadataId
|
|
applicationId
|
|
shortcut
|
|
isLabelSyncedWithName
|
|
isSearchable
|
|
duplicateCriteria
|
|
searchFieldMetadataList {
|
|
id
|
|
fieldMetadataId
|
|
tsVectorFieldMetadataId
|
|
position
|
|
}
|
|
indexMetadataList {
|
|
id
|
|
name
|
|
indexWhereClause
|
|
indexType
|
|
isUnique
|
|
isCustom
|
|
indexFieldMetadataList {
|
|
id
|
|
fieldMetadataId
|
|
subFieldName
|
|
order
|
|
}
|
|
}
|
|
fieldsList {
|
|
id
|
|
universalIdentifier
|
|
type
|
|
name
|
|
label
|
|
description
|
|
icon
|
|
isActive
|
|
isSystem
|
|
isUIEditable
|
|
isNullable
|
|
isUnique
|
|
defaultValue
|
|
options
|
|
settings
|
|
isLabelSyncedWithName
|
|
morphId
|
|
applicationId
|
|
relation {
|
|
type
|
|
sourceObjectMetadata {
|
|
id
|
|
nameSingular
|
|
namePlural
|
|
}
|
|
targetObjectMetadata {
|
|
id
|
|
nameSingular
|
|
namePlural
|
|
}
|
|
sourceFieldMetadata {
|
|
id
|
|
name
|
|
}
|
|
targetFieldMetadata {
|
|
id
|
|
name
|
|
}
|
|
}
|
|
morphRelations {
|
|
type
|
|
sourceObjectMetadata {
|
|
id
|
|
nameSingular
|
|
namePlural
|
|
}
|
|
targetObjectMetadata {
|
|
id
|
|
nameSingular
|
|
namePlural
|
|
}
|
|
sourceFieldMetadata {
|
|
id
|
|
name
|
|
}
|
|
targetFieldMetadata {
|
|
id
|
|
name
|
|
}
|
|
}
|
|
}
|
|
}
|
|
}
|
|
}
|
|
}
|
|
`;
|
|
|
|
const OBJECT_METADATA_ITEMS_BASELINE_QUERY = gql`
|
|
query ObjectMetadataItems {
|
|
objects(paging: { first: 1 }) {
|
|
edges {
|
|
node {
|
|
id
|
|
}
|
|
}
|
|
}
|
|
}
|
|
`;
|
|
|
|
// Mirrors VIEW_FRAGMENT from twenty-front
|
|
// (packages/twenty-front/src/modules/views/graphql/fragments/viewFragment.ts).
|
|
const FIND_ALL_VIEWS_QUERY = gql`
|
|
query FindAllViews {
|
|
getViews {
|
|
id
|
|
name
|
|
objectMetadataId
|
|
type
|
|
key
|
|
icon
|
|
position
|
|
isCompact
|
|
openRecordIn
|
|
kanbanAggregateOperation
|
|
kanbanAggregateOperationFieldMetadataId
|
|
mainGroupByFieldMetadataId
|
|
shouldHideEmptyGroups
|
|
kanbanColumnWidth
|
|
anyFieldFilterValue
|
|
calendarFieldMetadataId
|
|
calendarEndFieldMetadataId
|
|
calendarLayout
|
|
visibility
|
|
createdByUserWorkspaceId
|
|
isActive
|
|
viewFields {
|
|
id
|
|
fieldMetadataId
|
|
isVisible
|
|
size
|
|
position
|
|
aggregateOperation
|
|
}
|
|
viewFieldGroups {
|
|
id
|
|
name
|
|
position
|
|
isVisible
|
|
viewId
|
|
isActive
|
|
viewFields {
|
|
id
|
|
fieldMetadataId
|
|
isVisible
|
|
size
|
|
position
|
|
aggregateOperation
|
|
}
|
|
}
|
|
viewFilters {
|
|
id
|
|
fieldMetadataId
|
|
operand
|
|
value
|
|
viewFilterGroupId
|
|
positionInViewFilterGroup
|
|
subFieldName
|
|
}
|
|
viewFilterGroups {
|
|
id
|
|
parentViewFilterGroupId
|
|
logicalOperator
|
|
positionInViewFilterGroup
|
|
}
|
|
viewSorts {
|
|
id
|
|
fieldMetadataId
|
|
direction
|
|
}
|
|
viewGroups {
|
|
id
|
|
isVisible
|
|
fieldValue
|
|
position
|
|
viewId
|
|
}
|
|
}
|
|
}
|
|
`;
|
|
|
|
const FIND_ALL_VIEWS_BASELINE_QUERY = gql`
|
|
query FindAllViews {
|
|
getViews {
|
|
id
|
|
}
|
|
}
|
|
`;
|
|
|
|
describe('metadata GraphQL operations cache dependencies', () => {
|
|
const recordAccessedCacheKeys = async (
|
|
query: ASTNode,
|
|
): Promise<Set<WorkspaceCacheKeyName>> => {
|
|
const workspaceCacheService =
|
|
getAppProviderByClassName<WorkspaceCacheService>('WorkspaceCacheService');
|
|
const spy = jest.spyOn(workspaceCacheService, 'getOrRecomputeWithHashes');
|
|
|
|
try {
|
|
const response = await makeMetadataAPIRequest({ query });
|
|
|
|
expect(response.body.errors).toBeUndefined();
|
|
|
|
return new Set<WorkspaceCacheKeyName>(
|
|
spy.mock.calls.flatMap(([, cacheKeyNames]) => cacheKeyNames),
|
|
);
|
|
} finally {
|
|
spy.mockRestore();
|
|
}
|
|
};
|
|
|
|
const findUndeclaredDependencies = async ({
|
|
operationName,
|
|
fullQuery,
|
|
baselineQuery,
|
|
}: {
|
|
operationName: string;
|
|
fullQuery: ASTNode;
|
|
baselineQuery: ASTNode;
|
|
}): Promise<WorkspaceCacheKeyName[]> => {
|
|
const declaredDependencies = new Set<WorkspaceCacheKeyName>(
|
|
METADATA_GRAPHQL_OPERATIONS_TO_CACHE[operationName].dependencies,
|
|
);
|
|
const requestInfrastructureKeys =
|
|
await recordAccessedCacheKeys(baselineQuery);
|
|
const accessedCacheKeys = await recordAccessedCacheKeys(fullQuery);
|
|
|
|
return [...accessedCacheKeys].filter(
|
|
(cacheKeyName) =>
|
|
!declaredDependencies.has(cacheKeyName) &&
|
|
!requestInfrastructureKeys.has(cacheKeyName),
|
|
);
|
|
};
|
|
|
|
it('declares every flat map the ObjectMetadataItems resolvers read', async () => {
|
|
const undeclaredDependencies = await findUndeclaredDependencies({
|
|
operationName: 'ObjectMetadataItems',
|
|
fullQuery: OBJECT_METADATA_ITEMS_QUERY,
|
|
baselineQuery: OBJECT_METADATA_ITEMS_BASELINE_QUERY,
|
|
});
|
|
|
|
expect(undeclaredDependencies).toEqual([]);
|
|
});
|
|
|
|
it('declares every flat map the FindAllViews resolvers read', async () => {
|
|
const undeclaredDependencies = await findUndeclaredDependencies({
|
|
operationName: FIND_ALL_VIEWS_GRAPHQL_OPERATION,
|
|
fullQuery: FIND_ALL_VIEWS_QUERY,
|
|
baselineQuery: FIND_ALL_VIEWS_BASELINE_QUERY,
|
|
});
|
|
|
|
expect(undeclaredDependencies).toEqual([]);
|
|
});
|
|
});
|