Rekey metadata caches when flat map hashes change (#23164)

## Context

The `/metadata` GraphQL response cache (`ObjectMetadataItems`,
`FindAllViews`) and the workspace SDL cache were keyed on
`workspace.metadataVersion`, an integer bumped on every object/field
migration. That mechanism is legacy (the migration runner literally
calls it `getLegacyCacheInvalidationPromises`): the data plane already
moved to `WorkspaceCacheService`, which versions each flat entity map
with its own hash minted on invalidation.

Version keying had two concrete costs: the version bump was the only
proactive invalidation for `ObjectMetadataItems`, and keeping
`FindAllViews` fresh required `flushGraphQLOperation`, a full Redis
keyspace SCAN on every relevant migration. It is also the main blocker
for deprecating `metadataVersion` entirely.

This PR re-keys both caches on the flat-map hashes instead.

## What changed

**Response cache** (`use-cached-metadata.ts`): the key is now
`{operation}:{workspaceId}:{combinedDependencyHash}[:{userWorkspaceId}]:{locale}:{queryHash}`.
Each cached operation declares which flat maps its resolvers read
(`metadata-graphql-operations-to-cache.constant.ts`) plus a scope:
`ObjectMetadataItems` stays workspace-shared, `FindAllViews` is per-user
because unlisted-view visibility depends on the caller. When any
declared map changes, its hash rotates and the key rotates with it; no
flush needed. The key is resolved once per request and reused in
`onResponse`, so a rotation mid-request can never cache a response under
a fresher key than the data it was built from. If hash resolution fails,
the request is served uncached (Sentry-captured).

This also fixes three pre-existing key soundness gaps: `FindAllViews`
ignored locale although view names are translated server-side, the query
hash ignored GraphQL variables (`$viewTypes`), and mid-request version
rotation could re-key between request and response.

**SDL cache** (`workspace-graphql-schema-sdl.service.ts`): keyed on the
combined hash of the four maps the schema is generated from, taken from
the same `getOrRecomputeWithHashes` call that returns the data, so key
and content cannot skew. The `metadataVersion` read/seed block there is
gone; the Redis seed moved to `middleware.service.ts` so the
`X-Schema-Version` "refresh the page" check keeps working after the
Redis key's TTL expires.

**`WorkspaceCacheService`**: the internal pipeline now threads `{data,
hashes}` through every stage (local hit, hash validation, Redis fetch,
recompute) and the memoizer stores the pair, so returned hashes are
always consistent with returned data. New public
`getOrRecomputeWithHashes` and `getOrRecomputeCombinedHash`
(hashes-first: one MGET of the small `:hash` keys, full pipeline only
for missing ones, so cold pods never pull map payloads just to build a
key).

**Atomic pair writes** (`cache-storage.service.ts`): `mset` on the Redis
driver now delegates to the store's own `mset` (a MULTI of `SET ... PX`,
or native `MSET`), grouped by TTL. Previously it was a `Promise.all` of
independent SETs, so two concurrent recomputes could interleave and
leave one recompute's `:data` next to the other's `:hash`; with
hash-keyed caches that torn pair could persist a stale response under a
live key. `CoreEntityCacheService` writes through the same method and is
fixed for free.

**Cleanup**: `flush()` lost its `metadataVersion` parameter (always
pattern-flush per key on workspace deletion),
`METADATA_VERSIONED_WORKSPACE_CACHE_KEY` became
`HASH_KEYED_WORKSPACE_CACHE_KEYS` with the `MetadataVersion` key
relocated to `WORKSPACE_CACHE_KEYS` and the dead `ORMEntitySchemas`
entry removed.

## Deliberately unchanged

- `incrementMetadataVersion` and all its callers stay: the version still
feeds the `X-Schema-Version` check and the pinned upgrade commands.
Deprecating the column is a later stage.
- The runner's `FindAllViews` pattern-flush is kept for exactly one
release: view-only migrations never bump `metadataVersion`, so old pods
in a rolling deploy have no other invalidation signal for their
version-keyed entries. It gets deleted next release, which removes the
SCAN entirely.
- Old-shape cache entries are not migrated; they expire via the 7-day
TTL.

## Known limitations (follow-ups, not regressions)

- The plugin reads dependency hashes Redis-fresh while resolvers can
serve up to 10s-old memoized data, so a request landing right after a
migration can cache a pre-rotation response under the new key. Same
shape existed under `metadataVersion`; closing it needs request-scoped
snapshot plumbing.
- Concurrent recomputes are last-writer-wins (lost update). Fencing with
a conditional write is a follow-up.

## Validation

- Unit: response-cache plugin behavior (scope, key stash, serve-uncached
on failure, prototype-name guard), atomic `mset` batching, existing
`WorkspaceCacheService` spec passing unchanged.
- Integration: a new drift-guard spec runs the real
`ObjectMetadataItems`/`FindAllViews` operations with full frontend
selection sets against the in-process app, spies on
`WorkspaceCacheService`, and fails if resolvers read a flat map missing
from the declared dependency lists, so the constant cannot silently
drift.
- Manual against a live server: creating a field rotates the field-map
hash and the very next `ObjectMetadataItems` response contains it (hash
rotation is now its only invalidation path); warm hits are ~5ms; SDL
entries appear under hash-shaped keys via introspection.

## Suggested reading order

1. `workspace-cache.service.ts`, `workspace-cache-key.type.ts`,
`combine-cache-hashes.util.ts` (the `{data, hashes}` pipeline)
2. `use-cached-metadata.ts`,
`metadata-graphql-operations-to-cache.constant.ts`,
`metadata.module-factory.ts` (response cache)
3. `workspace-graphql-schema-sdl.service.ts`,
`workspace-cache-storage.service.ts` (SDL cache and renames)
4. `middleware.service.ts` (metadata version seed relocation)
5. `cache-storage.service.ts` (atomic writes)
6. Tests
This commit is contained in:
Weiko
2026-07-22 18:50:09 +02:00
committed by GitHub
parent f67e9c6b05
commit 0108a34765
20 changed files with 974 additions and 215 deletions
@@ -0,0 +1,32 @@
import { type CachedOperationConfig } from 'src/engine/api/graphql/graphql-config/hooks/use-cached-metadata';
import { FIND_ALL_VIEWS_GRAPHQL_OPERATION } from 'src/engine/metadata-modules/view/constants/find-all-views-graphql-operation.constant';
export const METADATA_GRAPHQL_OPERATIONS_TO_CACHE: Record<
string,
CachedOperationConfig
> = {
ObjectMetadataItems: {
scope: 'workspace',
dependencies: [
'flatObjectMetadataMaps',
'flatFieldMetadataMaps',
'flatIndexMaps',
'flatSearchFieldMetadataMaps',
'flatApplicationMaps',
],
},
[FIND_ALL_VIEWS_GRAPHQL_OPERATION]: {
scope: 'userWorkspace',
dependencies: [
'flatViewMaps',
'flatViewFieldMaps',
'flatViewFieldGroupMaps',
'flatViewFilterMaps',
'flatViewFilterGroupMaps',
'flatViewSortMaps',
'flatViewGroupMaps',
'flatObjectMetadataMaps',
'flatApplicationMaps',
],
},
};
@@ -4,6 +4,7 @@ import { type Request } from 'express';
import { useCachedMetadata } from 'src/engine/api/graphql/graphql-config/hooks/use-cached-metadata';
jest.mock('@sentry/node', () => ({
captureException: jest.fn(),
getCurrentScope: jest.fn(),
setTags: jest.fn(),
startSpan: jest.fn(),
@@ -40,18 +41,25 @@ describe('useCachedMetadata', () => {
},
locale: 'en',
userWorkspaceId: 'user-workspace-id',
workspace: { id: 'workspace-id', metadataVersion: 3 },
workspace: { id: 'workspace-id' },
...overrides,
}) as Request;
const createPlugin = ({
cacheGetter = jest.fn().mockResolvedValue(undefined),
cacheSetter = jest.fn(),
dependencyHashGetter = jest.fn().mockResolvedValue('dependency-hash'),
} = {}) =>
useCachedMetadata({
cacheGetter,
cacheSetter,
operationsToCache: ['FindAllViews'],
operationsToCache: {
FindAllViews: {
scope: 'userWorkspace',
dependencies: ['flatViewMaps'],
},
},
dependencyHashGetter,
});
beforeEach(() => {
@@ -133,6 +141,92 @@ describe('useCachedMetadata', () => {
expect(mockSpan.setAttribute).toHaveBeenNthCalledWith(2, 'cache.hit', true);
});
it('resolves dependency hashes once per request and caches the response under that key', async () => {
const cacheGetter = jest.fn().mockResolvedValue(undefined);
const cacheSetter = jest.fn();
const dependencyHashGetter = jest.fn().mockResolvedValue('dependency-hash');
const plugin = createPlugin({
cacheGetter,
cacheSetter,
dependencyHashGetter,
});
const request = createRequest();
const serverContext = { req: request };
const responseBody = { data: { views: [] } };
await plugin.onRequest?.({
endResponse: jest.fn(),
serverContext,
} as never);
await plugin.onResponse?.({
response: Response.json(responseBody),
serverContext,
} as never);
expect(dependencyHashGetter).toHaveBeenCalledTimes(1);
expect(dependencyHashGetter).toHaveBeenCalledWith('workspace-id', [
'flatViewMaps',
]);
const cacheKey = cacheGetter.mock.calls[0][0];
expect(cacheKey).toContain('dependency-hash');
expect(cacheKey).toContain('user-workspace-id');
expect(cacheSetter).toHaveBeenCalledWith(cacheKey, responseBody);
});
it('shares cache entries across users for workspace-scoped operations', async () => {
const cacheGetter = jest.fn().mockResolvedValue(undefined);
const plugin = useCachedMetadata({
cacheGetter,
cacheSetter: jest.fn(),
operationsToCache: {
FindAllViews: { scope: 'workspace', dependencies: ['flatViewMaps'] },
},
dependencyHashGetter: jest.fn().mockResolvedValue('dependency-hash'),
});
const request = createRequest();
const serverContext = { req: request };
await plugin.onRequest?.({
endResponse: jest.fn(),
serverContext,
} as never);
const cacheKey = cacheGetter.mock.calls[0][0];
expect(cacheKey).not.toContain('user-workspace-id');
expect(cacheKey).toContain(':en:');
});
it('serves the request uncached when dependency hashes cannot be resolved', async () => {
const cacheGetter = jest.fn();
const cacheSetter = jest.fn();
const dependencyHashGetter = jest
.fn()
.mockRejectedValue(new Error('cache storage unavailable'));
const plugin = createPlugin({
cacheGetter,
cacheSetter,
dependencyHashGetter,
});
const request = createRequest();
const serverContext = { req: request };
await plugin.onRequest?.({
endResponse: jest.fn(),
serverContext,
} as never);
await plugin.onResponse?.({
response: Response.json({ data: { views: [] } }),
serverContext,
} as never);
expect(cacheGetter).not.toHaveBeenCalled();
expect(cacheSetter).not.toHaveBeenCalled();
expect(Sentry.captureException).toHaveBeenCalledTimes(1);
});
it('does not trace client-controlled operations outside the cache allowlist', async () => {
const cacheGetter = jest.fn();
const plugin = createPlugin({ cacheGetter });
@@ -157,4 +251,25 @@ describe('useCachedMetadata', () => {
expect(Sentry.setTags).not.toHaveBeenCalled();
expect(Sentry.startSpan).not.toHaveBeenCalled();
});
it('ignores operation names inherited from Object.prototype', async () => {
const cacheGetter = jest.fn();
const dependencyHashGetter = jest.fn();
const plugin = createPlugin({ cacheGetter, dependencyHashGetter });
const request = createRequest({
body: {
operationName: 'constructor',
query: 'query { views { id } }',
},
});
const serverContext = { req: request };
await plugin.onRequest?.({
endResponse: jest.fn(),
serverContext,
} as never);
expect(cacheGetter).not.toHaveBeenCalled();
expect(dependencyHashGetter).not.toHaveBeenCalled();
});
});
@@ -5,48 +5,100 @@ import { type Request } from 'express';
import { type Plugin } from 'graphql-yoga';
import { isDefined } from 'twenty-shared/utils';
import { InternalServerError } from 'src/engine/core-modules/graphql/utils/graphql-errors.util';
import { type WorkspaceCacheKeyName } from 'src/engine/workspace-cache/types/workspace-cache-key.type';
export type CachedOperationConfig = {
dependencies: WorkspaceCacheKeyName[];
scope: 'workspace' | 'userWorkspace';
};
export type CacheMetadataPluginConfig = {
// oxlint-disable-next-line typescript/no-explicit-any
cacheGetter: (key: string) => any;
// oxlint-disable-next-line typescript/no-explicit-any
cacheSetter: (key: string, value: any) => void;
operationsToCache: string[];
operationsToCache: Record<string, CachedOperationConfig>;
dependencyHashGetter: (
workspaceId: string,
cacheKeyNames: WorkspaceCacheKeyName[],
) => Promise<string>;
};
export function useCachedMetadata(config: CacheMetadataPluginConfig): Plugin {
const computeCacheKey = ({
const computeCacheKey = async ({
operationName,
operationConfig,
workspaceId,
request,
}: {
operationName: string;
request: Pick<Request, 'workspace' | 'locale' | 'body' | 'userWorkspaceId'>;
}) => {
const workspace = request.workspace;
if (!isDefined(workspace)) {
throw new InternalServerError('Workspace is not defined');
}
const workspaceMetadataVersion = workspace.metadataVersion ?? '0';
const locale = request.locale;
operationConfig: CachedOperationConfig;
workspaceId: string;
request: Pick<Request, 'locale' | 'body' | 'userWorkspaceId'>;
}): Promise<string> => {
const dependencyHash = await config.dependencyHashGetter(
workspaceId,
operationConfig.dependencies,
);
const queryHash = createHash('sha256')
.update(request.body.query)
.update(JSON.stringify(request.body.variables ?? null))
.digest('hex');
const userScopeSegment =
operationConfig.scope === 'userWorkspace'
? `:${request.userWorkspaceId}`
: '';
if (operationName === 'FindAllViews') {
return `graphql:operations:${operationName}:${workspace.id}:${workspaceMetadataVersion}:${request.userWorkspaceId}:${queryHash}`;
}
return `graphql:operations:${operationName}:${workspace.id}:${workspaceMetadataVersion}:${locale}:${queryHash}`;
return `graphql:operations:${operationName}:${workspaceId}:${dependencyHash}${userScopeSegment}:${request.locale}:${queryHash}`;
};
// oxlint-disable-next-line typescript/no-explicit-any
const getOperationName = (serverContext: any) =>
serverContext?.req?.body?.operationName;
const getOperationCacheConfig = (
operationName: unknown,
): CachedOperationConfig | undefined =>
typeof operationName === 'string' &&
Object.prototype.hasOwnProperty.call(
config.operationsToCache,
operationName,
)
? config.operationsToCache[operationName]
: undefined;
const cacheHitRequests = new WeakSet<Request>();
const requestCacheKeys = new WeakMap<Request, string | null>();
const resolveCacheKey = async ({
operationName,
operationConfig,
workspaceId,
request,
}: {
operationName: string;
operationConfig: CachedOperationConfig;
workspaceId: string;
request: Request;
}): Promise<string | null> => {
let cacheKey: string | null;
try {
cacheKey = await computeCacheKey({
operationName,
operationConfig,
workspaceId,
request,
});
} catch (error) {
Sentry.captureException(error);
cacheKey = null;
}
requestCacheKeys.set(request, cacheKey);
return cacheKey;
};
const getCachedResponse = ({
cacheKey,
@@ -81,24 +133,33 @@ export function useCachedMetadata(config: CacheMetadataPluginConfig): Plugin {
onRequest: async ({ endResponse, serverContext }) => {
// TODO: we should probably override the graphql-yoga request type to include the workspace and locale
const request = (serverContext as unknown as { req: Request }).req;
const workspaceId = request.workspace?.id;
if (!request.workspace?.id) {
if (!workspaceId) {
return;
}
const operationName = getOperationName(serverContext);
const operationConfig = getOperationCacheConfig(operationName);
if (!config.operationsToCache.includes(operationName)) {
if (!isDefined(operationConfig)) {
return;
}
Sentry.setTags({ operationName, operation: 'query' });
Sentry.getCurrentScope().setTransactionName(operationName);
const cacheKey = computeCacheKey({
const cacheKey = await resolveCacheKey({
operationName,
operationConfig,
workspaceId,
request,
});
if (!isDefined(cacheKey)) {
return;
}
const cachedResponse = await getCachedResponse({
cacheKey,
operationName,
@@ -122,7 +183,7 @@ export function useCachedMetadata(config: CacheMetadataPluginConfig): Plugin {
const operationName = getOperationName(serverContext);
if (!config.operationsToCache.includes(operationName)) {
if (!isDefined(getOperationCacheConfig(operationName))) {
return;
}
@@ -130,10 +191,11 @@ export function useCachedMetadata(config: CacheMetadataPluginConfig): Plugin {
return;
}
const cacheKey = computeCacheKey({
operationName,
request,
});
const cacheKey = requestCacheKeys.get(request);
if (!isDefined(cacheKey)) {
return;
}
const cachedResponse = await getCachedResponse({
cacheKey,
@@ -16,6 +16,8 @@ import { TwentyConfigService } from 'src/engine/core-modules/twenty-config/twent
import { DataloaderModule } from 'src/engine/dataloaders/dataloader.module';
import { DataloaderService } from 'src/engine/dataloaders/dataloader.service';
import { MetadataEngineModule } from 'src/engine/metadata-modules/metadata-engine.module';
import { WorkspaceCacheService } from 'src/engine/workspace-cache/services/workspace-cache.service';
import { WorkspaceCacheModule } from 'src/engine/workspace-cache/workspace-cache.module';
@Module({
imports: [
@@ -27,6 +29,7 @@ import { MetadataEngineModule } from 'src/engine/metadata-modules/metadata-engin
DataloaderModule,
MetricsModule,
I18nModule,
WorkspaceCacheModule,
],
inject: [
TwentyConfigService,
@@ -36,6 +39,7 @@ import { MetadataEngineModule } from 'src/engine/metadata-modules/metadata-engin
MetricsService,
I18nService,
FeatureFlagService,
WorkspaceCacheService,
],
}),
MetadataEngineModule,
@@ -4,6 +4,7 @@ import GraphQLJSON from 'graphql-type-json';
import { NodeEnvironment } from 'src/engine/core-modules/twenty-config/interfaces/node-environment.interface';
import { METADATA_GRAPHQL_OPERATIONS_TO_CACHE } from 'src/engine/api/graphql/graphql-config/constants/metadata-graphql-operations-to-cache.constant';
import { useCachedMetadata } from 'src/engine/api/graphql/graphql-config/hooks/use-cached-metadata';
import { MetadataGraphQLApiModule } from 'src/engine/api/graphql/metadata-graphql-api.module';
import { type CacheStorageService } from 'src/engine/core-modules/cache-storage/services/cache-storage.service';
@@ -19,6 +20,7 @@ import { type MetricsService } from 'src/engine/core-modules/metrics/metrics.ser
import { type TwentyConfigService } from 'src/engine/core-modules/twenty-config/twenty-config.service';
import { type DataloaderService } from 'src/engine/dataloaders/dataloader.service';
import { renderApolloPlayground } from 'src/engine/utils/render-apollo-playground.util';
import { type WorkspaceCacheService } from 'src/engine/workspace-cache/services/workspace-cache.service';
export const metadataModuleFactory = async (
twentyConfigService: TwentyConfigService,
@@ -28,6 +30,7 @@ export const metadataModuleFactory = async (
metricsService: MetricsService,
i18nService: I18nService,
_featureFlagService: FeatureFlagService,
workspaceCacheService: WorkspaceCacheService,
): Promise<YogaDriverConfig> => {
const config: YogaDriverConfig = {
autoSchemaFile: true,
@@ -51,7 +54,11 @@ export const metadataModuleFactory = async (
useCachedMetadata({
cacheGetter: cacheStorageService.get.bind(cacheStorageService),
cacheSetter: cacheStorageService.set.bind(cacheStorageService),
operationsToCache: ['ObjectMetadataItems', 'FindAllViews'],
operationsToCache: METADATA_GRAPHQL_OPERATIONS_TO_CACHE,
dependencyHashGetter:
workspaceCacheService.getOrRecomputeCombinedHash.bind(
workspaceCacheService,
),
}),
useDisableIntrospectionAndSuggestionsForUnauthenticatedUsers(
twentyConfigService.get('NODE_ENV') === NodeEnvironment.PRODUCTION,
@@ -0,0 +1,6 @@
export const SCHEMA_SDL_CACHE_DEPENDENCIES = [
'flatObjectMetadataMaps',
'flatFieldMetadataMaps',
'flatIndexMaps',
'flatApplicationMaps',
] as const;
@@ -17,7 +17,9 @@ import { getSubFlatEntityMapsByApplicationIdsOrThrow } from 'src/engine/metadata
import { type FlatFieldMetadata } from 'src/engine/metadata-modules/flat-field-metadata/types/flat-field-metadata.type';
import { type FlatIndexMetadata } from 'src/engine/metadata-modules/flat-index-metadata/types/flat-index-metadata.type';
import { type FlatObjectMetadata } from 'src/engine/metadata-modules/flat-object-metadata/types/flat-object-metadata.type';
import { SCHEMA_SDL_CACHE_DEPENDENCIES } from 'src/engine/api/graphql/workspace-graphql-schema-sdl/constants/schema-sdl-cache-dependencies.constant';
import { WorkspaceCacheStorageService } from 'src/engine/workspace-cache-storage/workspace-cache-storage.service';
import { combineCacheHashes } from 'src/engine/workspace-cache/utils/combine-cache-hashes.util';
import { TWENTY_STANDARD_APPLICATION } from 'src/engine/workspace-manager/twenty-standard-application/constants/twenty-standard-applications';
export type WorkspaceGraphqlSchemaSDLResult = {
@@ -46,21 +48,20 @@ export class WorkspaceGraphqlSchemaSDLService {
}
const {
flatObjectMetadataMaps: allFlatObjectMetadataMaps,
flatFieldMetadataMaps: allFlatFieldMetadataMaps,
flatIndexMaps: allFlatIndexMaps,
flatApplicationMaps,
} = await this.workspaceManyOrAllFlatEntityMapsCacheService.getOrRecomputeManyOrAllFlatEntityMaps(
{
workspaceId: workspace.id,
flatMapsKeys: [
'flatObjectMetadataMaps',
'flatFieldMetadataMaps',
'flatIndexMaps',
'flatApplicationMaps',
],
data: {
flatObjectMetadataMaps: allFlatObjectMetadataMaps,
flatFieldMetadataMaps: allFlatFieldMetadataMaps,
flatIndexMaps: allFlatIndexMaps,
flatApplicationMaps,
},
);
hashes,
} =
await this.workspaceManyOrAllFlatEntityMapsCacheService.getOrRecomputeManyOrAllFlatEntityMapsWithHashes(
{
workspaceId: workspace.id,
flatMapsKeys: [...SCHEMA_SDL_CACHE_DEPENDENCIES],
},
);
if (!isDefined(allFlatObjectMetadataMaps)) {
throw new FlatEntityMapsException(
@@ -113,28 +114,20 @@ export class WorkspaceGraphqlSchemaSDLService {
}
}
let metadataVersion =
await this.workspaceCacheStorageService.getMetadataVersion(workspace.id);
if (!isDefined(metadataVersion)) {
metadataVersion = isDefined(workspace.metadataVersion)
? workspace.metadataVersion
: 0;
await this.workspaceCacheStorageService.setMetadataVersion(
workspace.id,
metadataVersion,
);
}
const metadataCacheHash = combineCacheHashes(
hashes,
SCHEMA_SDL_CACHE_DEPENDENCIES,
);
let sdl = await this.workspaceCacheStorageService.getGraphQLTypeDefs(
workspace.id,
metadataVersion,
metadataCacheHash,
applicationId,
);
let usedScalarNames =
await this.workspaceCacheStorageService.getGraphQLUsedScalarNames(
workspace.id,
metadataVersion,
metadataCacheHash,
applicationId,
);
@@ -152,13 +145,13 @@ export class WorkspaceGraphqlSchemaSDLService {
await this.workspaceCacheStorageService.setGraphQLTypeDefs(
workspace.id,
metadataVersion,
metadataCacheHash,
sdl,
applicationId,
);
await this.workspaceCacheStorageService.setGraphQLUsedScalarNames(
workspace.id,
metadataVersion,
metadataCacheHash,
usedScalarNames,
applicationId,
);