SearchVector derivation via migration-scoped index (alt to #2622 __warmedUpCache) (#22389)

## What this is close https://github.com/twentyhq/core-team-issues/issues/2622 A **POC / discussion branch** implementing the runner-scoped alternative to the `__warmedUpCache` design in [core-team-issues#2622](https://github.com/twentyhq/core-team-issues/issues/2622). Not for merge as-is — meant to diff against that plan. ## Problem `deriveSearchVectorAsExpressionForTsVectorField` scans the entire `flatSearchFieldMetadataMaps` (`Object.values(...).filter(...)`) once per object created in a migration. On install that's `O(objectsCreated × totalSearchFields)` — the quadratic #2622 targets. Only **one** of the three call sites is actually hot: - `create-object` (runner) — global maps, called per created object → the quadratic - export DDL — maps already built **per object** (O(k)) - `update-field` rebuild — one field, gated on `rebuildSearchVector` ## Approach Instead of a private `__warmedUpCache` side-channel on `FlatEntityMaps<T>` + drain-on-hydration, this keeps the index in the **consumer**: 1. `derive` now takes `targetSearchFieldMetadatas` (already scoped to the tsVector field) instead of scanning the map itself. 2. The runner builds a `Map<tsVectorFieldMetadataId, searchFields[]>` **once per migration**, lazily, and threads it through the action context. Safe because `searchFieldMetadata` creates are ordered before `objectMetadata` creates (`computeOrderedMigrationActions`), so the map is complete on first use. → `O(totalSearchFields)`. 3. `getTargetSearchFieldMetadatasForTsVectorField` (O(total) filter) stays as the fallback for the one-off callers (export, field-update) and when the accessor isn't provided. ## Why this over `__warmedUpCache` - **No `FlatEntityMaps<T>` type widening**, no convention-only privacy, no id/universalIdentifier drain to keep in sync. - **No referential-integrity obligation.** The index only ever contains entities present in the map, so the "search field created-then-deleted before its object hydrates" case (deferred as an edge in #2622) can't put a stale id into an aggregator and crash `derive` via the `-orThrow` lookup. - **One `derive` path**, not "aggregator + direct-filter fallback for export". - Blast radius: ~220 lines, mostly a new util + test. ## Benchmark (micro, isolated function) Median of 7 trials, 10 search fields per object, running the real shipped utils — old = `getTargetSearchFieldMetadatasForTsVectorField` once per object (identical to the old inline scan), new = `buildSearchFieldMetadatasByTsVectorFieldId` once + N lookups (both assert they resolve the same fields): | objects | total search fields | old (scan/obj) | new (index once) | speedup | |--------:|--------------------:|---------------:|-----------------:|--------:| | 50 | 500 | 2.08 ms | 0.06 ms | 33× | | 100 | 1,000 | 9.26 ms | 0.12 ms | 77× | | 200 | 2,000 | 36.2 ms | 0.23 ms | 160× | | 400 | 4,000 | 151 ms | 0.40 ms | 379× | | 800 | 8,000 | 701 ms | 0.92 ms | 766× | Confirms the old path is quadratic (~4× per doubling of object count) and the new path is linear (sub-ms throughout). **Caveats — read these before trusting the speedup:** - This is the **isolated derivation function**, no DB / DDL / inserts. In a real `create-object` action the derive is a small fraction of per-action cost, so the end-to-end win is far smaller than the ratios above. - A default workspace has ~20–30 objects, where the **old** code already costs only ~1–2 ms total across the whole install. The quadratic only becomes material (>50 ms, the runner's slow-action threshold) around **200–400 objects**. - The measurement that should actually gate this — `[install-perf] create:objectMetadata` on a real install against a real DB with a few hundred objects — has **not** been run yet. The micro-benchmark bounds the upside and locates the knee of the curve; it does not prove end-to-end payoff. ## Not done on purpose - **No end-to-end benchmark yet** — step 0 should still be measuring `[install-perf] create:objectMetadata` on a real large install to confirm the quadratic is worth removing at all. - Relies on the ordering invariant (commented at the build site). The fully self-contained variant is to put the object's search fields on `FlatCreateObjectAction` (builder change) — deliberately left out to keep this runner-scoped. ## Checks `nx typecheck twenty-server`, `nx lint:diff-with-main twenty-server`, new util spec + existing `generate-workspace-schema-ddl` spec all green.
2026-07-01 14:29:19 +02:00
parent 1a475d0edd
commit 2b1417770e
11 changed files with 323 additions and 24 deletions
@@ -0,0 +1,92 @@
+import { type FlatEntityMaps } from 'src/engine/metadata-modules/flat-entity/types/flat-entity-maps.type';
+import { type FlatSearchFieldMetadata } from 'src/engine/metadata-modules/flat-search-field-metadata/types/flat-search-field-metadata.type';
+import { buildSearchFieldMetadatasByTsVectorFieldId } from 'src/engine/metadata-modules/flat-search-field-metadata/utils/build-search-field-metadatas-by-ts-vector-field-id.util';
+import { getTargetSearchFieldMetadatasForTsVectorField } from 'src/engine/metadata-modules/flat-search-field-metadata/utils/get-target-search-field-metadatas-for-ts-vector-field.util';
+
+const buildFlatSearchFieldMetadata = (
+  overrides: Partial<FlatSearchFieldMetadata> &
+    Pick<FlatSearchFieldMetadata, 'universalIdentifier'>,
+): FlatSearchFieldMetadata =>
+  ({
+    id: overrides.universalIdentifier,
+    tsVectorFieldMetadataId: 'ts-vector-1',
+    fieldMetadataId: 'field-1',
+    position: 0,
+    ...overrides,
+  }) as unknown as FlatSearchFieldMetadata;
+
+const buildMaps = (
+  flatSearchFieldMetadatas: FlatSearchFieldMetadata[],
+): FlatEntityMaps<FlatSearchFieldMetadata> => ({
+  byUniversalIdentifier: Object.fromEntries(
+    flatSearchFieldMetadatas.map((flatSearchFieldMetadata) => [
+      flatSearchFieldMetadata.universalIdentifier,
+      flatSearchFieldMetadata,
+    ]),
+  ),
+  universalIdentifierById: {},
+  universalIdentifiersByApplicationId: {},
+});
+
+describe('buildSearchFieldMetadatasByTsVectorFieldId', () => {
+  it('groups search fields by their tsVector field id', () => {
+    const a1 = buildFlatSearchFieldMetadata({
+      universalIdentifier: 'a1',
+      tsVectorFieldMetadataId: 'ts-vector-a',
+    });
+    const a2 = buildFlatSearchFieldMetadata({
+      universalIdentifier: 'a2',
+      tsVectorFieldMetadataId: 'ts-vector-a',
+    });
+    const b1 = buildFlatSearchFieldMetadata({
+      universalIdentifier: 'b1',
+      tsVectorFieldMetadataId: 'ts-vector-b',
+    });
+
+    const grouped = buildSearchFieldMetadatasByTsVectorFieldId(
+      buildMaps([a1, a2, b1]),
+    );
+
+    expect(grouped.get('ts-vector-a')).toEqual([a1, a2]);
+    expect(grouped.get('ts-vector-b')).toEqual([b1]);
+    expect(grouped.get('ts-vector-unknown')).toBeUndefined();
+  });
+
+  it('matches the one-off filter helper for a given tsVector field', () => {
+    const maps = buildMaps([
+      buildFlatSearchFieldMetadata({
+        universalIdentifier: 'a1',
+        tsVectorFieldMetadataId: 'ts-vector-a',
+      }),
+      buildFlatSearchFieldMetadata({
+        universalIdentifier: 'b1',
+        tsVectorFieldMetadataId: 'ts-vector-b',
+      }),
+    ]);
+
+    const grouped = buildSearchFieldMetadatasByTsVectorFieldId(maps);
+
+    expect(grouped.get('ts-vector-a')).toEqual(
+      getTargetSearchFieldMetadatasForTsVectorField({
+        tsVectorFieldMetadataId: 'ts-vector-a',
+        flatSearchFieldMetadataMaps: maps,
+      }),
+    );
+  });
+
+  it('ignores undefined map entries', () => {
+    const maps = buildMaps([
+      buildFlatSearchFieldMetadata({
+        universalIdentifier: 'a1',
+        tsVectorFieldMetadataId: 'ts-vector-a',
+      }),
+    ]);
+
+    maps.byUniversalIdentifier['ghost'] = undefined;
+
+    const grouped = buildSearchFieldMetadatasByTsVectorFieldId(maps);
+
+    expect(grouped.get('ts-vector-a')).toHaveLength(1);
+    expect(grouped.size).toBe(1);
+  });
+});
@@ -0,0 +1,86 @@
+import { type FlatEntityMaps } from 'src/engine/metadata-modules/flat-entity/types/flat-entity-maps.type';
+import { type FlatSearchFieldMetadata } from 'src/engine/metadata-modules/flat-search-field-metadata/types/flat-search-field-metadata.type';
+import { createSearchFieldMetadatasByTsVectorFieldIdAccessor } from 'src/engine/metadata-modules/flat-search-field-metadata/utils/create-search-field-metadatas-by-ts-vector-field-id-accessor.util';
+
+const buildFlatSearchFieldMetadata = (
+  universalIdentifier: string,
+  tsVectorFieldMetadataId: string,
+): FlatSearchFieldMetadata =>
+  ({
+    id: universalIdentifier,
+    universalIdentifier,
+    tsVectorFieldMetadataId,
+    fieldMetadataId: `field-${universalIdentifier}`,
+    position: 0,
+  }) as unknown as FlatSearchFieldMetadata;
+
+const buildMaps = (
+  flatSearchFieldMetadatas: FlatSearchFieldMetadata[],
+): FlatEntityMaps<FlatSearchFieldMetadata> => ({
+  byUniversalIdentifier: Object.fromEntries(
+    flatSearchFieldMetadatas.map((flatSearchFieldMetadata) => [
+      flatSearchFieldMetadata.universalIdentifier,
+      flatSearchFieldMetadata,
+    ]),
+  ),
+  universalIdentifierById: {},
+  universalIdentifiersByApplicationId: {},
+});
+
+describe('createSearchFieldMetadatasByTsVectorFieldIdAccessor', () => {
+  it('resolves the search fields for a tsVector field', () => {
+    const maps = buildMaps([
+      buildFlatSearchFieldMetadata('a1', 'ts-vector-a'),
+      buildFlatSearchFieldMetadata('a2', 'ts-vector-a'),
+      buildFlatSearchFieldMetadata('b1', 'ts-vector-b'),
+    ]);
+
+    const accessor = createSearchFieldMetadatasByTsVectorFieldIdAccessor(
+      () => maps,
+    );
+
+    expect(accessor.get('ts-vector-a')).toHaveLength(2);
+    expect(accessor.get('ts-vector-b')).toHaveLength(1);
+    expect(accessor.get('ts-vector-unknown')).toEqual([]);
+  });
+
+  it('serves a stale result after the map changes until invalidate() is called', () => {
+    const maps = buildMaps([buildFlatSearchFieldMetadata('a1', 'ts-vector-a')]);
+
+    const accessor = createSearchFieldMetadatasByTsVectorFieldIdAccessor(
+      () => maps,
+    );
+
+    expect(accessor.get('ts-vector-a')).toHaveLength(1);
+
+    maps.byUniversalIdentifier['a2'] = buildFlatSearchFieldMetadata(
+      'a2',
+      'ts-vector-a',
+    );
+
+    expect(accessor.get('ts-vector-a')).toHaveLength(1);
+
+    accessor.invalidate();
+
+    expect(accessor.get('ts-vector-a')).toHaveLength(2);
+  });
+
+  it('rebuilds against the latest maps reference returned by the getter', () => {
+    let maps = buildMaps([buildFlatSearchFieldMetadata('a1', 'ts-vector-a')]);
+
+    const accessor = createSearchFieldMetadatasByTsVectorFieldIdAccessor(
+      () => maps,
+    );
+
+    expect(accessor.get('ts-vector-a')).toHaveLength(1);
+
+    maps = buildMaps([
+      buildFlatSearchFieldMetadata('a1', 'ts-vector-a'),
+      buildFlatSearchFieldMetadata('a2', 'ts-vector-a'),
+    ]);
+
+    accessor.invalidate();
+
+    expect(accessor.get('ts-vector-a')).toHaveLength(2);
+  });
+});
@@ -0,0 +1,35 @@
+import { isDefined } from 'twenty-shared/utils';
+
+import { type FlatEntityMaps } from 'src/engine/metadata-modules/flat-entity/types/flat-entity-maps.type';
+import { type FlatSearchFieldMetadata } from 'src/engine/metadata-modules/flat-search-field-metadata/types/flat-search-field-metadata.type';
+
+export const buildSearchFieldMetadatasByTsVectorFieldId = (
+  flatSearchFieldMetadataMaps: FlatEntityMaps<FlatSearchFieldMetadata>,
+): Map<string, FlatSearchFieldMetadata[]> => {
+  const searchFieldMetadatasByTsVectorFieldId = new Map<
+    string,
+    FlatSearchFieldMetadata[]
+  >();
+
+  for (const flatSearchFieldMetadata of Object.values(
+    flatSearchFieldMetadataMaps.byUniversalIdentifier,
+  )) {
+    if (!isDefined(flatSearchFieldMetadata)) {
+      continue;
+    }
+
+    const { tsVectorFieldMetadataId } = flatSearchFieldMetadata;
+
+    const existingSearchFieldMetadatas =
+      searchFieldMetadatasByTsVectorFieldId.get(tsVectorFieldMetadataId) ?? [];
+
+    existingSearchFieldMetadatas.push(flatSearchFieldMetadata);
+
+    searchFieldMetadatasByTsVectorFieldId.set(
+      tsVectorFieldMetadataId,
+      existingSearchFieldMetadatas,
+    );
+  }
+
+  return searchFieldMetadatasByTsVectorFieldId;
+};
@@ -0,0 +1,36 @@
+import { type FlatEntityMaps } from 'src/engine/metadata-modules/flat-entity/types/flat-entity-maps.type';
+import { type FlatSearchFieldMetadata } from 'src/engine/metadata-modules/flat-search-field-metadata/types/flat-search-field-metadata.type';
+import { buildSearchFieldMetadatasByTsVectorFieldId } from 'src/engine/metadata-modules/flat-search-field-metadata/utils/build-search-field-metadatas-by-ts-vector-field-id.util';
+
+// Scoped alternative to the warmed-up optimistic aggregator design in
+// https://github.com/twentyhq/core-team-issues/issues/2622. Prefer implementing
+// that centralized cache if a second create-before-parent consumer re-occurs.
+
+export type SearchFieldMetadatasByTsVectorFieldIdAccessor = {
+  get: (tsVectorFieldMetadataId: string) => FlatSearchFieldMetadata[];
+  invalidate: () => void;
+};
+
+export const createSearchFieldMetadatasByTsVectorFieldIdAccessor = (
+  getFlatSearchFieldMetadataMaps: () => FlatEntityMaps<FlatSearchFieldMetadata>,
+): SearchFieldMetadatasByTsVectorFieldIdAccessor => {
+  let searchFieldMetadatasByTsVectorFieldId:
+    | Map<string, FlatSearchFieldMetadata[]>
+    | undefined;
+
+  return {
+    get: (tsVectorFieldMetadataId) => {
+      searchFieldMetadatasByTsVectorFieldId ??=
+        buildSearchFieldMetadatasByTsVectorFieldId(
+          getFlatSearchFieldMetadataMaps(),
+        );
+
+      return (
+        searchFieldMetadatasByTsVectorFieldId.get(tsVectorFieldMetadataId) ?? []
+      );
+    },
+    invalidate: () => {
+      searchFieldMetadatasByTsVectorFieldId = undefined;
+    },
+  };
+};
@@ -1,7 +1,6 @@
 import { type FieldMetadataType } from 'twenty-shared/types';
 import { isDefined } from 'twenty-shared/utils';

-import { type FlatEntityMaps } from 'src/engine/metadata-modules/flat-entity/types/flat-entity-maps.type';
 import {
  buildSearchVectorTargetField,
  computeSearchVectorAsExpressionFromSearchFieldMetadatas,
@@ -9,27 +8,17 @@ import {
 import { type FlatSearchFieldMetadata } from 'src/engine/metadata-modules/flat-search-field-metadata/types/flat-search-field-metadata.type';

 export const deriveSearchVectorAsExpressionForTsVectorField = ({
-  tsVectorFieldMetadataId,
-  flatSearchFieldMetadataMaps,
+  targetSearchFieldMetadatas,
  indexedFieldById,
 }: {
-  tsVectorFieldMetadataId: string;
-  flatSearchFieldMetadataMaps: FlatEntityMaps<FlatSearchFieldMetadata>;
+  targetSearchFieldMetadatas: FlatSearchFieldMetadata[];
  indexedFieldById: ReadonlyMap<
    string,
    { name: string; type: FieldMetadataType }
  >;
 }): string => {
-  const targetSearchableFields = Object.values(
-    flatSearchFieldMetadataMaps.byUniversalIdentifier,
-  )
-    .filter(isDefined)
-    .filter(
-      (flatSearchFieldMetadata) =>
-        flatSearchFieldMetadata.tsVectorFieldMetadataId ===
-        tsVectorFieldMetadataId,
-    )
-    .flatMap((flatSearchFieldMetadata) => {
+  const targetSearchableFields = targetSearchFieldMetadatas.flatMap(
+    (flatSearchFieldMetadata) => {
      const indexedField = indexedFieldById.get(
        flatSearchFieldMetadata.fieldMetadataId,
      );
@@ -45,7 +34,8 @@ export const deriveSearchVectorAsExpressionForTsVectorField = ({
          sortKey: flatSearchFieldMetadata.universalIdentifier,
        }),
      ];
-    });
+    },
+  );

  return computeSearchVectorAsExpressionFromSearchFieldMetadatas(
    targetSearchableFields,
@@ -0,0 +1,19 @@
+import { isDefined } from 'twenty-shared/utils';
+
+import { type FlatEntityMaps } from 'src/engine/metadata-modules/flat-entity/types/flat-entity-maps.type';
+import { type FlatSearchFieldMetadata } from 'src/engine/metadata-modules/flat-search-field-metadata/types/flat-search-field-metadata.type';
+
+export const getTargetSearchFieldMetadatasForTsVectorField = ({
+  tsVectorFieldMetadataId,
+  flatSearchFieldMetadataMaps,
+}: {
+  tsVectorFieldMetadataId: string;
+  flatSearchFieldMetadataMaps: FlatEntityMaps<FlatSearchFieldMetadata>;
+}): FlatSearchFieldMetadata[] =>
+  Object.values(flatSearchFieldMetadataMaps.byUniversalIdentifier)
+    .filter(isDefined)
+    .filter(
+      (flatSearchFieldMetadata) =>
+        flatSearchFieldMetadata.tsVectorFieldMetadataId ===
+        tsVectorFieldMetadataId,
+    );