SearchVector derivation via migration-scoped index (alt to #2622 __warmedUpCache) (#22389)

## What this is

close https://github.com/twentyhq/core-team-issues/issues/2622

A **POC / discussion branch** implementing the runner-scoped alternative
to the `__warmedUpCache` design in
[core-team-issues#2622](https://github.com/twentyhq/core-team-issues/issues/2622).
Not for merge as-is — meant to diff against that plan.

## Problem

`deriveSearchVectorAsExpressionForTsVectorField` scans the entire
`flatSearchFieldMetadataMaps` (`Object.values(...).filter(...)`) once
per object created in a migration. On install that's `O(objectsCreated ×
totalSearchFields)` — the quadratic #2622 targets.

Only **one** of the three call sites is actually hot:
- `create-object` (runner) — global maps, called per created object →
the quadratic
- export DDL — maps already built **per object** (O(k))
- `update-field` rebuild — one field, gated on `rebuildSearchVector`

## Approach

Instead of a private `__warmedUpCache` side-channel on
`FlatEntityMaps<T>` + drain-on-hydration, this keeps the index in the
**consumer**:

1. `derive` now takes `targetSearchFieldMetadatas` (already scoped to
the tsVector field) instead of scanning the map itself.
2. The runner builds a `Map<tsVectorFieldMetadataId, searchFields[]>`
**once per migration**, lazily, and threads it through the action
context. Safe because `searchFieldMetadata` creates are ordered before
`objectMetadata` creates (`computeOrderedMigrationActions`), so the map
is complete on first use. → `O(totalSearchFields)`.
3. `getTargetSearchFieldMetadatasForTsVectorField` (O(total) filter)
stays as the fallback for the one-off callers (export, field-update) and
when the accessor isn't provided.

## Why this over `__warmedUpCache`

- **No `FlatEntityMaps<T>` type widening**, no convention-only privacy,
no id/universalIdentifier drain to keep in sync.
- **No referential-integrity obligation.** The index only ever contains
entities present in the map, so the "search field created-then-deleted
before its object hydrates" case (deferred as an edge in #2622) can't
put a stale id into an aggregator and crash `derive` via the `-orThrow`
lookup.
- **One `derive` path**, not "aggregator + direct-filter fallback for
export".
- Blast radius: ~220 lines, mostly a new util + test.

## Benchmark (micro, isolated function)

Median of 7 trials, 10 search fields per object, running the real
shipped utils — old = `getTargetSearchFieldMetadatasForTsVectorField`
once per object (identical to the old inline scan), new =
`buildSearchFieldMetadatasByTsVectorFieldId` once + N lookups (both
assert they resolve the same fields):

| objects | total search fields | old (scan/obj) | new (index once) |
speedup |

|--------:|--------------------:|---------------:|-----------------:|--------:|
| 50 | 500 | 2.08 ms | 0.06 ms | 33× |
| 100 | 1,000 | 9.26 ms | 0.12 ms | 77× |
| 200 | 2,000 | 36.2 ms | 0.23 ms | 160× |
| 400 | 4,000 | 151 ms | 0.40 ms | 379× |
| 800 | 8,000 | 701 ms | 0.92 ms | 766× |

Confirms the old path is quadratic (~4× per doubling of object count)
and the new path is linear (sub-ms throughout).

**Caveats — read these before trusting the speedup:**
- This is the **isolated derivation function**, no DB / DDL / inserts.
In a real `create-object` action the derive is a small fraction of
per-action cost, so the end-to-end win is far smaller than the ratios
above.
- A default workspace has ~20–30 objects, where the **old** code already
costs only ~1–2 ms total across the whole install. The quadratic only
becomes material (>50 ms, the runner's slow-action threshold) around
**200–400 objects**.
- The measurement that should actually gate this — `[install-perf]
create:objectMetadata` on a real install against a real DB with a few
hundred objects — has **not** been run yet. The micro-benchmark bounds
the upside and locates the knee of the curve; it does not prove
end-to-end payoff.

## Not done on purpose

- **No end-to-end benchmark yet** — step 0 should still be measuring
`[install-perf] create:objectMetadata` on a real large install to
confirm the quadratic is worth removing at all.
- Relies on the ordering invariant (commented at the build site). The
fully self-contained variant is to put the object's search fields on
`FlatCreateObjectAction` (builder change) — deliberately left out to
keep this runner-scoped.

## Checks

`nx typecheck twenty-server`, `nx lint:diff-with-main twenty-server`,
new util spec + existing `generate-workspace-schema-ddl` spec all green.

packages/twenty-server/src/engine/workspace-manager/workspace-migration/workspace-migration-runner/action-handlers/field/services/update-field-action-handler.service.ts

@@ -25,6 +25,7 @@ import { isEnumFlatFieldMetadata } from 'src/engine/metadata-modules/flat-field-
 import { isFlatFieldMetadataOfType } from 'src/engine/metadata-modules/flat-field-metadata/utils/is-flat-field-metadata-of-type.util';
 import { isMorphOrRelationFlatFieldMetadata } from 'src/engine/metadata-modules/flat-field-metadata/utils/is-morph-or-relation-flat-field-metadata.util';
 import { deriveSearchVectorAsExpressionForTsVectorField } from 'src/engine/metadata-modules/flat-search-field-metadata/utils/derive-search-vector-as-expression-for-ts-vector-field.util';
+import { getTargetSearchFieldMetadatasForTsVectorField } from 'src/engine/metadata-modules/flat-search-field-metadata/utils/get-target-search-field-metadatas-for-ts-vector-field.util';
 import { FlatObjectMetadata } from 'src/engine/metadata-modules/flat-object-metadata/types/flat-object-metadata.type';
 import { WorkspaceSchemaManagerService } from 'src/engine/twenty-orm/workspace-schema-manager/workspace-schema-manager.service';
 import { computeObjectTargetTable } from 'src/engine/utils/compute-object-target-table.util';
@@ -180,6 +181,7 @@ export class UpdateFieldActionHandlerService extends WorkspaceMigrationRunnerAct
         flatIndexMaps,
       },
       workspaceId,
+      getSearchFieldMetadatasByTsVectorFieldId,
     } = context;
     const { entityId, update } = flatAction;
 
@@ -367,8 +369,14 @@ export class UpdateFieldActionHandlerService extends WorkspaceMigrationRunnerAct
 
       const searchVectorAsExpression =
         deriveSearchVectorAsExpressionForTsVectorField({
-          tsVectorFieldMetadataId: optimisticFlatFieldMetadata.id,
-          flatSearchFieldMetadataMaps,
+          targetSearchFieldMetadatas:
+            getSearchFieldMetadatasByTsVectorFieldId?.(
+              optimisticFlatFieldMetadata.id,
+            ) ??
+            getTargetSearchFieldMetadatasForTsVectorField({
+              tsVectorFieldMetadataId: optimisticFlatFieldMetadata.id,
+              flatSearchFieldMetadataMaps,
+            }),
           indexedFieldById,
         });
 

packages/twenty-server/src/engine/workspace-manager/workspace-migration/workspace-migration-runner/action-handlers/object/services/create-object-action-handler.service.ts

@@ -10,6 +10,7 @@ import { isCompositeFlatFieldMetadata } from 'src/engine/metadata-modules/flat-f
 import { isEnumFlatFieldMetadata } from 'src/engine/metadata-modules/flat-field-metadata/utils/is-enum-flat-field-metadata.util';
 import { isFlatFieldMetadataOfType } from 'src/engine/metadata-modules/flat-field-metadata/utils/is-flat-field-metadata-of-type.util';
 import { deriveSearchVectorAsExpressionForTsVectorField } from 'src/engine/metadata-modules/flat-search-field-metadata/utils/derive-search-vector-as-expression-for-ts-vector-field.util';
+import { getTargetSearchFieldMetadatasForTsVectorField } from 'src/engine/metadata-modules/flat-search-field-metadata/utils/get-target-search-field-metadatas-for-ts-vector-field.util';
 import { WorkspaceSchemaManagerService } from 'src/engine/twenty-orm/workspace-schema-manager/workspace-schema-manager.service';
 import {
   FlatCreateObjectAction,
@@ -120,7 +121,13 @@ export class CreateObjectActionHandlerService extends WorkspaceMigrationRunnerAc
   async executeForWorkspaceSchema(
     context: WorkspaceMigrationActionRunnerContext<FlatCreateObjectAction>,
   ): Promise<void> {
-    const { flatAction, queryRunner, workspaceId, allFlatEntityMaps } = context;
+    const {
+      flatAction,
+      queryRunner,
+      workspaceId,
+      allFlatEntityMaps,
+      getSearchFieldMetadatasByTsVectorFieldId,
+    } = context;
     const { flatEntity: flatObjectMetadata, flatFieldMetadatas } = flatAction;
 
     const { schemaName, tableName } = getWorkspaceSchemaContextForMigration({
@@ -145,9 +152,15 @@ export class CreateObjectActionHandlerService extends WorkspaceMigrationRunnerAc
           FieldMetadataType.TS_VECTOR,
         )
           ? deriveSearchVectorAsExpressionForTsVectorField({
-              tsVectorFieldMetadataId: flatFieldMetadata.id,
-              flatSearchFieldMetadataMaps:
-                allFlatEntityMaps.flatSearchFieldMetadataMaps,
+              targetSearchFieldMetadatas:
+                getSearchFieldMetadatasByTsVectorFieldId?.(
+                  flatFieldMetadata.id,
+                ) ??
+                getTargetSearchFieldMetadatasForTsVectorField({
+                  tsVectorFieldMetadataId: flatFieldMetadata.id,
+                  flatSearchFieldMetadataMaps:
+                    allFlatEntityMaps.flatSearchFieldMetadataMaps,
+                }),
               indexedFieldById,
             })
           : undefined,

packages/twenty-server/src/engine/workspace-manager/workspace-migration/workspace-migration-runner/services/workspace-migration-runner.service.ts

@@ -13,6 +13,7 @@ import { getMetadataFlatEntityMapsKey } from 'src/engine/metadata-modules/flat-e
 import { getMetadataRelatedMetadataNamesForValidation } from 'src/engine/metadata-modules/flat-entity/utils/get-metadata-related-metadata-names-for-validation.util';
 import { getMetadataRelatedMetadataNames } from 'src/engine/metadata-modules/flat-entity/utils/get-metadata-related-metadata-names.util';
 import { getMetadataSerializedRelationNames } from 'src/engine/metadata-modules/flat-entity/utils/get-metadata-serialized-relation-names.util';
+import { createSearchFieldMetadatasByTsVectorFieldIdAccessor } from 'src/engine/metadata-modules/flat-search-field-metadata/utils/create-search-field-metadatas-by-ts-vector-field-id-accessor.util';
 import { FIND_ALL_VIEWS_GRAPHQL_OPERATION } from 'src/engine/metadata-modules/view/constants/find-all-views-graphql-operation.constant';
 import { WorkspaceMetadataVersionService } from 'src/engine/metadata-modules/workspace-metadata-version/services/workspace-metadata-version.service';
 import { WorkspaceCacheStorageService } from 'src/engine/workspace-cache-storage/workspace-cache-storage.service';
@@ -310,6 +311,11 @@ export class WorkspaceMigrationRunnerService {
     let slowestActionLabel = 'n/a';
     let actionCount = 0;
 
+    const searchFieldMetadatasByTsVectorFieldIdAccessor =
+      createSearchFieldMetadatasByTsVectorFieldIdAccessor(
+        () => allFlatEntityMaps.flatSearchFieldMetadataMaps,
+      );
+
     try {
       await queryRunner.query(`SET LOCAL lock_timeout = '8s'`);
 
@@ -330,6 +336,8 @@ export class WorkspaceMigrationRunnerService {
                 queryRunner,
                 workspaceId,
                 preallocatedIdByUniversalIdentifierByMetadataName,
+                getSearchFieldMetadatasByTsVectorFieldId:
+                  searchFieldMetadatasByTsVectorFieldIdAccessor.get,
               },
             },
           );
@@ -355,6 +363,10 @@ export class WorkspaceMigrationRunnerService {
           ...partialOptimisticCache,
         } as typeof allFlatEntityMaps;
 
+        if (action.metadataName === 'searchFieldMetadata') {
+          searchFieldMetadatasByTsVectorFieldIdAccessor.invalidate();
+        }
+
         allMetadataEvents.push(...metadataEvents);
         allAfterCommitSideEffects.push(...afterCommitSideEffects);
       }

packages/twenty-server/src/engine/workspace-manager/workspace-migration/workspace-migration-runner/types/workspace-migration-action-runner-args.type.ts

@@ -2,6 +2,7 @@ import { type QueryRunner } from 'typeorm';
 
 import { type FlatApplication } from 'src/engine/core-modules/application/types/flat-application.type';
 import { type AllFlatEntityMaps } from 'src/engine/metadata-modules/flat-entity/types/all-flat-entity-maps.type';
+import { type FlatSearchFieldMetadata } from 'src/engine/metadata-modules/flat-search-field-metadata/types/flat-search-field-metadata.type';
 import { type PreallocatedIdByUniversalIdentifierByMetadataName } from 'src/engine/workspace-manager/workspace-migration/universal-flat-entity/utils/resolve-universal-relation-identifiers-to-ids.util';
 import {
   type AllFlatWorkspaceMigrationAction,
@@ -17,6 +18,9 @@ export type WorkspaceMigrationActionRunnerArgs<
   workspaceId: string;
   flatApplication: FlatApplication;
   preallocatedIdByUniversalIdentifierByMetadataName?: PreallocatedIdByUniversalIdentifierByMetadataName;
+  getSearchFieldMetadatasByTsVectorFieldId?: (
+    tsVectorFieldMetadataId: string,
+  ) => FlatSearchFieldMetadata[];
 };
 
 export type WorkspaceMigrationActionRunnerContext<