From 2b1417770e214a9fad0b5b9aabb79bdbc849822e Mon Sep 17 00:00:00 2001 From: Paul Rastoin <45004772+prastoin@users.noreply.github.com> Date: Wed, 1 Jul 2026 14:29:19 +0200 Subject: [PATCH] SearchVector derivation via migration-scoped index (alt to #2622 __warmedUpCache) (#22389) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## What this is close https://github.com/twentyhq/core-team-issues/issues/2622 A **POC / discussion branch** implementing the runner-scoped alternative to the `__warmedUpCache` design in [core-team-issues#2622](https://github.com/twentyhq/core-team-issues/issues/2622). Not for merge as-is — meant to diff against that plan. ## Problem `deriveSearchVectorAsExpressionForTsVectorField` scans the entire `flatSearchFieldMetadataMaps` (`Object.values(...).filter(...)`) once per object created in a migration. On install that's `O(objectsCreated × totalSearchFields)` — the quadratic #2622 targets. Only **one** of the three call sites is actually hot: - `create-object` (runner) — global maps, called per created object → the quadratic - export DDL — maps already built **per object** (O(k)) - `update-field` rebuild — one field, gated on `rebuildSearchVector` ## Approach Instead of a private `__warmedUpCache` side-channel on `FlatEntityMaps` + drain-on-hydration, this keeps the index in the **consumer**: 1. `derive` now takes `targetSearchFieldMetadatas` (already scoped to the tsVector field) instead of scanning the map itself. 2. The runner builds a `Map` **once per migration**, lazily, and threads it through the action context. Safe because `searchFieldMetadata` creates are ordered before `objectMetadata` creates (`computeOrderedMigrationActions`), so the map is complete on first use. → `O(totalSearchFields)`. 3. `getTargetSearchFieldMetadatasForTsVectorField` (O(total) filter) stays as the fallback for the one-off callers (export, field-update) and when the accessor isn't provided. ## Why this over `__warmedUpCache` - **No `FlatEntityMaps` type widening**, no convention-only privacy, no id/universalIdentifier drain to keep in sync. - **No referential-integrity obligation.** The index only ever contains entities present in the map, so the "search field created-then-deleted before its object hydrates" case (deferred as an edge in #2622) can't put a stale id into an aggregator and crash `derive` via the `-orThrow` lookup. - **One `derive` path**, not "aggregator + direct-filter fallback for export". - Blast radius: ~220 lines, mostly a new util + test. ## Benchmark (micro, isolated function) Median of 7 trials, 10 search fields per object, running the real shipped utils — old = `getTargetSearchFieldMetadatasForTsVectorField` once per object (identical to the old inline scan), new = `buildSearchFieldMetadatasByTsVectorFieldId` once + N lookups (both assert they resolve the same fields): | objects | total search fields | old (scan/obj) | new (index once) | speedup | |--------:|--------------------:|---------------:|-----------------:|--------:| | 50 | 500 | 2.08 ms | 0.06 ms | 33× | | 100 | 1,000 | 9.26 ms | 0.12 ms | 77× | | 200 | 2,000 | 36.2 ms | 0.23 ms | 160× | | 400 | 4,000 | 151 ms | 0.40 ms | 379× | | 800 | 8,000 | 701 ms | 0.92 ms | 766× | Confirms the old path is quadratic (~4× per doubling of object count) and the new path is linear (sub-ms throughout). **Caveats — read these before trusting the speedup:** - This is the **isolated derivation function**, no DB / DDL / inserts. In a real `create-object` action the derive is a small fraction of per-action cost, so the end-to-end win is far smaller than the ratios above. - A default workspace has ~20–30 objects, where the **old** code already costs only ~1–2 ms total across the whole install. The quadratic only becomes material (>50 ms, the runner's slow-action threshold) around **200–400 objects**. - The measurement that should actually gate this — `[install-perf] create:objectMetadata` on a real install against a real DB with a few hundred objects — has **not** been run yet. The micro-benchmark bounds the upside and locates the knee of the curve; it does not prove end-to-end payoff. ## Not done on purpose - **No end-to-end benchmark yet** — step 0 should still be measuring `[install-perf] create:objectMetadata` on a real large install to confirm the quadratic is worth removing at all. - Relies on the ordering invariant (commented at the build site). The fully self-contained variant is to put the object's search fields on `FlatCreateObjectAction` (builder change) — deliberately left out to keep this runner-scoped. ## Checks `nx typecheck twenty-server`, `nx lint:diff-with-main twenty-server`, new util spec + existing `generate-workspace-schema-ddl` spec all green. --- .../generate-workspace-schema-ddl.util.ts | 8 +- ...tadatas-by-ts-vector-field-id.util.spec.ts | 92 +++++++++++++++++++ ...y-ts-vector-field-id-accessor.util.spec.ts | 86 +++++++++++++++++ ...ld-metadatas-by-ts-vector-field-id.util.ts | 35 +++++++ ...tas-by-ts-vector-field-id-accessor.util.ts | 36 ++++++++ ...-as-expression-for-ts-vector-field.util.ts | 22 ++--- ...ield-metadatas-for-ts-vector-field.util.ts | 19 ++++ .../update-field-action-handler.service.ts | 12 ++- .../create-object-action-handler.service.ts | 21 ++++- .../workspace-migration-runner.service.ts | 12 +++ ...space-migration-action-runner-args.type.ts | 4 + 11 files changed, 323 insertions(+), 24 deletions(-) create mode 100644 packages/twenty-server/src/engine/metadata-modules/flat-search-field-metadata/utils/__tests__/build-search-field-metadatas-by-ts-vector-field-id.util.spec.ts create mode 100644 packages/twenty-server/src/engine/metadata-modules/flat-search-field-metadata/utils/__tests__/create-search-field-metadatas-by-ts-vector-field-id-accessor.util.spec.ts create mode 100644 packages/twenty-server/src/engine/metadata-modules/flat-search-field-metadata/utils/build-search-field-metadatas-by-ts-vector-field-id.util.ts create mode 100644 packages/twenty-server/src/engine/metadata-modules/flat-search-field-metadata/utils/create-search-field-metadatas-by-ts-vector-field-id-accessor.util.ts create mode 100644 packages/twenty-server/src/engine/metadata-modules/flat-search-field-metadata/utils/get-target-search-field-metadatas-for-ts-vector-field.util.ts diff --git a/packages/twenty-server/src/database/commands/workspace-export/utils/generate-workspace-schema-ddl.util.ts b/packages/twenty-server/src/database/commands/workspace-export/utils/generate-workspace-schema-ddl.util.ts index 5d6eb18b3d..9e137463fd 100644 --- a/packages/twenty-server/src/database/commands/workspace-export/utils/generate-workspace-schema-ddl.util.ts +++ b/packages/twenty-server/src/database/commands/workspace-export/utils/generate-workspace-schema-ddl.util.ts @@ -7,6 +7,7 @@ import { isFlatFieldMetadataOfType } from 'src/engine/metadata-modules/flat-fiel import { type FlatObjectMetadata } from 'src/engine/metadata-modules/flat-object-metadata/types/flat-object-metadata.type'; import { type FlatSearchFieldMetadata } from 'src/engine/metadata-modules/flat-search-field-metadata/types/flat-search-field-metadata.type'; import { deriveSearchVectorAsExpressionForTsVectorField } from 'src/engine/metadata-modules/flat-search-field-metadata/utils/derive-search-vector-as-expression-for-ts-vector-field.util'; +import { getTargetSearchFieldMetadatasForTsVectorField } from 'src/engine/metadata-modules/flat-search-field-metadata/utils/get-target-search-field-metadatas-for-ts-vector-field.util'; import { type ObjectMetadataEntity } from 'src/engine/metadata-modules/object-metadata/object-metadata.entity'; import { SEARCH_VECTOR_FIELD } from 'src/engine/metadata-modules/search-field-metadata/constants/search-vector-field.constants'; import { type SearchFieldMetadataEntity } from 'src/engine/metadata-modules/search-field-metadata/search-field-metadata.entity'; @@ -125,8 +126,11 @@ export const generateWorkspaceSchemaDdl = ( FieldMetadataType.TS_VECTOR, ) ? deriveSearchVectorAsExpressionForTsVectorField({ - tsVectorFieldMetadataId: flatFieldMetadata.id, - flatSearchFieldMetadataMaps, + targetSearchFieldMetadatas: + getTargetSearchFieldMetadatasForTsVectorField({ + tsVectorFieldMetadataId: flatFieldMetadata.id, + flatSearchFieldMetadataMaps, + }), indexedFieldById, }) : undefined, diff --git a/packages/twenty-server/src/engine/metadata-modules/flat-search-field-metadata/utils/__tests__/build-search-field-metadatas-by-ts-vector-field-id.util.spec.ts b/packages/twenty-server/src/engine/metadata-modules/flat-search-field-metadata/utils/__tests__/build-search-field-metadatas-by-ts-vector-field-id.util.spec.ts new file mode 100644 index 0000000000..34604d2291 --- /dev/null +++ b/packages/twenty-server/src/engine/metadata-modules/flat-search-field-metadata/utils/__tests__/build-search-field-metadatas-by-ts-vector-field-id.util.spec.ts @@ -0,0 +1,92 @@ +import { type FlatEntityMaps } from 'src/engine/metadata-modules/flat-entity/types/flat-entity-maps.type'; +import { type FlatSearchFieldMetadata } from 'src/engine/metadata-modules/flat-search-field-metadata/types/flat-search-field-metadata.type'; +import { buildSearchFieldMetadatasByTsVectorFieldId } from 'src/engine/metadata-modules/flat-search-field-metadata/utils/build-search-field-metadatas-by-ts-vector-field-id.util'; +import { getTargetSearchFieldMetadatasForTsVectorField } from 'src/engine/metadata-modules/flat-search-field-metadata/utils/get-target-search-field-metadatas-for-ts-vector-field.util'; + +const buildFlatSearchFieldMetadata = ( + overrides: Partial & + Pick, +): FlatSearchFieldMetadata => + ({ + id: overrides.universalIdentifier, + tsVectorFieldMetadataId: 'ts-vector-1', + fieldMetadataId: 'field-1', + position: 0, + ...overrides, + }) as unknown as FlatSearchFieldMetadata; + +const buildMaps = ( + flatSearchFieldMetadatas: FlatSearchFieldMetadata[], +): FlatEntityMaps => ({ + byUniversalIdentifier: Object.fromEntries( + flatSearchFieldMetadatas.map((flatSearchFieldMetadata) => [ + flatSearchFieldMetadata.universalIdentifier, + flatSearchFieldMetadata, + ]), + ), + universalIdentifierById: {}, + universalIdentifiersByApplicationId: {}, +}); + +describe('buildSearchFieldMetadatasByTsVectorFieldId', () => { + it('groups search fields by their tsVector field id', () => { + const a1 = buildFlatSearchFieldMetadata({ + universalIdentifier: 'a1', + tsVectorFieldMetadataId: 'ts-vector-a', + }); + const a2 = buildFlatSearchFieldMetadata({ + universalIdentifier: 'a2', + tsVectorFieldMetadataId: 'ts-vector-a', + }); + const b1 = buildFlatSearchFieldMetadata({ + universalIdentifier: 'b1', + tsVectorFieldMetadataId: 'ts-vector-b', + }); + + const grouped = buildSearchFieldMetadatasByTsVectorFieldId( + buildMaps([a1, a2, b1]), + ); + + expect(grouped.get('ts-vector-a')).toEqual([a1, a2]); + expect(grouped.get('ts-vector-b')).toEqual([b1]); + expect(grouped.get('ts-vector-unknown')).toBeUndefined(); + }); + + it('matches the one-off filter helper for a given tsVector field', () => { + const maps = buildMaps([ + buildFlatSearchFieldMetadata({ + universalIdentifier: 'a1', + tsVectorFieldMetadataId: 'ts-vector-a', + }), + buildFlatSearchFieldMetadata({ + universalIdentifier: 'b1', + tsVectorFieldMetadataId: 'ts-vector-b', + }), + ]); + + const grouped = buildSearchFieldMetadatasByTsVectorFieldId(maps); + + expect(grouped.get('ts-vector-a')).toEqual( + getTargetSearchFieldMetadatasForTsVectorField({ + tsVectorFieldMetadataId: 'ts-vector-a', + flatSearchFieldMetadataMaps: maps, + }), + ); + }); + + it('ignores undefined map entries', () => { + const maps = buildMaps([ + buildFlatSearchFieldMetadata({ + universalIdentifier: 'a1', + tsVectorFieldMetadataId: 'ts-vector-a', + }), + ]); + + maps.byUniversalIdentifier['ghost'] = undefined; + + const grouped = buildSearchFieldMetadatasByTsVectorFieldId(maps); + + expect(grouped.get('ts-vector-a')).toHaveLength(1); + expect(grouped.size).toBe(1); + }); +}); diff --git a/packages/twenty-server/src/engine/metadata-modules/flat-search-field-metadata/utils/__tests__/create-search-field-metadatas-by-ts-vector-field-id-accessor.util.spec.ts b/packages/twenty-server/src/engine/metadata-modules/flat-search-field-metadata/utils/__tests__/create-search-field-metadatas-by-ts-vector-field-id-accessor.util.spec.ts new file mode 100644 index 0000000000..b60bcb78db --- /dev/null +++ b/packages/twenty-server/src/engine/metadata-modules/flat-search-field-metadata/utils/__tests__/create-search-field-metadatas-by-ts-vector-field-id-accessor.util.spec.ts @@ -0,0 +1,86 @@ +import { type FlatEntityMaps } from 'src/engine/metadata-modules/flat-entity/types/flat-entity-maps.type'; +import { type FlatSearchFieldMetadata } from 'src/engine/metadata-modules/flat-search-field-metadata/types/flat-search-field-metadata.type'; +import { createSearchFieldMetadatasByTsVectorFieldIdAccessor } from 'src/engine/metadata-modules/flat-search-field-metadata/utils/create-search-field-metadatas-by-ts-vector-field-id-accessor.util'; + +const buildFlatSearchFieldMetadata = ( + universalIdentifier: string, + tsVectorFieldMetadataId: string, +): FlatSearchFieldMetadata => + ({ + id: universalIdentifier, + universalIdentifier, + tsVectorFieldMetadataId, + fieldMetadataId: `field-${universalIdentifier}`, + position: 0, + }) as unknown as FlatSearchFieldMetadata; + +const buildMaps = ( + flatSearchFieldMetadatas: FlatSearchFieldMetadata[], +): FlatEntityMaps => ({ + byUniversalIdentifier: Object.fromEntries( + flatSearchFieldMetadatas.map((flatSearchFieldMetadata) => [ + flatSearchFieldMetadata.universalIdentifier, + flatSearchFieldMetadata, + ]), + ), + universalIdentifierById: {}, + universalIdentifiersByApplicationId: {}, +}); + +describe('createSearchFieldMetadatasByTsVectorFieldIdAccessor', () => { + it('resolves the search fields for a tsVector field', () => { + const maps = buildMaps([ + buildFlatSearchFieldMetadata('a1', 'ts-vector-a'), + buildFlatSearchFieldMetadata('a2', 'ts-vector-a'), + buildFlatSearchFieldMetadata('b1', 'ts-vector-b'), + ]); + + const accessor = createSearchFieldMetadatasByTsVectorFieldIdAccessor( + () => maps, + ); + + expect(accessor.get('ts-vector-a')).toHaveLength(2); + expect(accessor.get('ts-vector-b')).toHaveLength(1); + expect(accessor.get('ts-vector-unknown')).toEqual([]); + }); + + it('serves a stale result after the map changes until invalidate() is called', () => { + const maps = buildMaps([buildFlatSearchFieldMetadata('a1', 'ts-vector-a')]); + + const accessor = createSearchFieldMetadatasByTsVectorFieldIdAccessor( + () => maps, + ); + + expect(accessor.get('ts-vector-a')).toHaveLength(1); + + maps.byUniversalIdentifier['a2'] = buildFlatSearchFieldMetadata( + 'a2', + 'ts-vector-a', + ); + + expect(accessor.get('ts-vector-a')).toHaveLength(1); + + accessor.invalidate(); + + expect(accessor.get('ts-vector-a')).toHaveLength(2); + }); + + it('rebuilds against the latest maps reference returned by the getter', () => { + let maps = buildMaps([buildFlatSearchFieldMetadata('a1', 'ts-vector-a')]); + + const accessor = createSearchFieldMetadatasByTsVectorFieldIdAccessor( + () => maps, + ); + + expect(accessor.get('ts-vector-a')).toHaveLength(1); + + maps = buildMaps([ + buildFlatSearchFieldMetadata('a1', 'ts-vector-a'), + buildFlatSearchFieldMetadata('a2', 'ts-vector-a'), + ]); + + accessor.invalidate(); + + expect(accessor.get('ts-vector-a')).toHaveLength(2); + }); +}); diff --git a/packages/twenty-server/src/engine/metadata-modules/flat-search-field-metadata/utils/build-search-field-metadatas-by-ts-vector-field-id.util.ts b/packages/twenty-server/src/engine/metadata-modules/flat-search-field-metadata/utils/build-search-field-metadatas-by-ts-vector-field-id.util.ts new file mode 100644 index 0000000000..b59927efe2 --- /dev/null +++ b/packages/twenty-server/src/engine/metadata-modules/flat-search-field-metadata/utils/build-search-field-metadatas-by-ts-vector-field-id.util.ts @@ -0,0 +1,35 @@ +import { isDefined } from 'twenty-shared/utils'; + +import { type FlatEntityMaps } from 'src/engine/metadata-modules/flat-entity/types/flat-entity-maps.type'; +import { type FlatSearchFieldMetadata } from 'src/engine/metadata-modules/flat-search-field-metadata/types/flat-search-field-metadata.type'; + +export const buildSearchFieldMetadatasByTsVectorFieldId = ( + flatSearchFieldMetadataMaps: FlatEntityMaps, +): Map => { + const searchFieldMetadatasByTsVectorFieldId = new Map< + string, + FlatSearchFieldMetadata[] + >(); + + for (const flatSearchFieldMetadata of Object.values( + flatSearchFieldMetadataMaps.byUniversalIdentifier, + )) { + if (!isDefined(flatSearchFieldMetadata)) { + continue; + } + + const { tsVectorFieldMetadataId } = flatSearchFieldMetadata; + + const existingSearchFieldMetadatas = + searchFieldMetadatasByTsVectorFieldId.get(tsVectorFieldMetadataId) ?? []; + + existingSearchFieldMetadatas.push(flatSearchFieldMetadata); + + searchFieldMetadatasByTsVectorFieldId.set( + tsVectorFieldMetadataId, + existingSearchFieldMetadatas, + ); + } + + return searchFieldMetadatasByTsVectorFieldId; +}; diff --git a/packages/twenty-server/src/engine/metadata-modules/flat-search-field-metadata/utils/create-search-field-metadatas-by-ts-vector-field-id-accessor.util.ts b/packages/twenty-server/src/engine/metadata-modules/flat-search-field-metadata/utils/create-search-field-metadatas-by-ts-vector-field-id-accessor.util.ts new file mode 100644 index 0000000000..9ee732d550 --- /dev/null +++ b/packages/twenty-server/src/engine/metadata-modules/flat-search-field-metadata/utils/create-search-field-metadatas-by-ts-vector-field-id-accessor.util.ts @@ -0,0 +1,36 @@ +import { type FlatEntityMaps } from 'src/engine/metadata-modules/flat-entity/types/flat-entity-maps.type'; +import { type FlatSearchFieldMetadata } from 'src/engine/metadata-modules/flat-search-field-metadata/types/flat-search-field-metadata.type'; +import { buildSearchFieldMetadatasByTsVectorFieldId } from 'src/engine/metadata-modules/flat-search-field-metadata/utils/build-search-field-metadatas-by-ts-vector-field-id.util'; + +// Scoped alternative to the warmed-up optimistic aggregator design in +// https://github.com/twentyhq/core-team-issues/issues/2622. Prefer implementing +// that centralized cache if a second create-before-parent consumer re-occurs. + +export type SearchFieldMetadatasByTsVectorFieldIdAccessor = { + get: (tsVectorFieldMetadataId: string) => FlatSearchFieldMetadata[]; + invalidate: () => void; +}; + +export const createSearchFieldMetadatasByTsVectorFieldIdAccessor = ( + getFlatSearchFieldMetadataMaps: () => FlatEntityMaps, +): SearchFieldMetadatasByTsVectorFieldIdAccessor => { + let searchFieldMetadatasByTsVectorFieldId: + | Map + | undefined; + + return { + get: (tsVectorFieldMetadataId) => { + searchFieldMetadatasByTsVectorFieldId ??= + buildSearchFieldMetadatasByTsVectorFieldId( + getFlatSearchFieldMetadataMaps(), + ); + + return ( + searchFieldMetadatasByTsVectorFieldId.get(tsVectorFieldMetadataId) ?? [] + ); + }, + invalidate: () => { + searchFieldMetadatasByTsVectorFieldId = undefined; + }, + }; +}; diff --git a/packages/twenty-server/src/engine/metadata-modules/flat-search-field-metadata/utils/derive-search-vector-as-expression-for-ts-vector-field.util.ts b/packages/twenty-server/src/engine/metadata-modules/flat-search-field-metadata/utils/derive-search-vector-as-expression-for-ts-vector-field.util.ts index 0208cb40d5..56258f95db 100644 --- a/packages/twenty-server/src/engine/metadata-modules/flat-search-field-metadata/utils/derive-search-vector-as-expression-for-ts-vector-field.util.ts +++ b/packages/twenty-server/src/engine/metadata-modules/flat-search-field-metadata/utils/derive-search-vector-as-expression-for-ts-vector-field.util.ts @@ -1,7 +1,6 @@ import { type FieldMetadataType } from 'twenty-shared/types'; import { isDefined } from 'twenty-shared/utils'; -import { type FlatEntityMaps } from 'src/engine/metadata-modules/flat-entity/types/flat-entity-maps.type'; import { buildSearchVectorTargetField, computeSearchVectorAsExpressionFromSearchFieldMetadatas, @@ -9,27 +8,17 @@ import { import { type FlatSearchFieldMetadata } from 'src/engine/metadata-modules/flat-search-field-metadata/types/flat-search-field-metadata.type'; export const deriveSearchVectorAsExpressionForTsVectorField = ({ - tsVectorFieldMetadataId, - flatSearchFieldMetadataMaps, + targetSearchFieldMetadatas, indexedFieldById, }: { - tsVectorFieldMetadataId: string; - flatSearchFieldMetadataMaps: FlatEntityMaps; + targetSearchFieldMetadatas: FlatSearchFieldMetadata[]; indexedFieldById: ReadonlyMap< string, { name: string; type: FieldMetadataType } >; }): string => { - const targetSearchableFields = Object.values( - flatSearchFieldMetadataMaps.byUniversalIdentifier, - ) - .filter(isDefined) - .filter( - (flatSearchFieldMetadata) => - flatSearchFieldMetadata.tsVectorFieldMetadataId === - tsVectorFieldMetadataId, - ) - .flatMap((flatSearchFieldMetadata) => { + const targetSearchableFields = targetSearchFieldMetadatas.flatMap( + (flatSearchFieldMetadata) => { const indexedField = indexedFieldById.get( flatSearchFieldMetadata.fieldMetadataId, ); @@ -45,7 +34,8 @@ export const deriveSearchVectorAsExpressionForTsVectorField = ({ sortKey: flatSearchFieldMetadata.universalIdentifier, }), ]; - }); + }, + ); return computeSearchVectorAsExpressionFromSearchFieldMetadatas( targetSearchableFields, diff --git a/packages/twenty-server/src/engine/metadata-modules/flat-search-field-metadata/utils/get-target-search-field-metadatas-for-ts-vector-field.util.ts b/packages/twenty-server/src/engine/metadata-modules/flat-search-field-metadata/utils/get-target-search-field-metadatas-for-ts-vector-field.util.ts new file mode 100644 index 0000000000..2e171cc632 --- /dev/null +++ b/packages/twenty-server/src/engine/metadata-modules/flat-search-field-metadata/utils/get-target-search-field-metadatas-for-ts-vector-field.util.ts @@ -0,0 +1,19 @@ +import { isDefined } from 'twenty-shared/utils'; + +import { type FlatEntityMaps } from 'src/engine/metadata-modules/flat-entity/types/flat-entity-maps.type'; +import { type FlatSearchFieldMetadata } from 'src/engine/metadata-modules/flat-search-field-metadata/types/flat-search-field-metadata.type'; + +export const getTargetSearchFieldMetadatasForTsVectorField = ({ + tsVectorFieldMetadataId, + flatSearchFieldMetadataMaps, +}: { + tsVectorFieldMetadataId: string; + flatSearchFieldMetadataMaps: FlatEntityMaps; +}): FlatSearchFieldMetadata[] => + Object.values(flatSearchFieldMetadataMaps.byUniversalIdentifier) + .filter(isDefined) + .filter( + (flatSearchFieldMetadata) => + flatSearchFieldMetadata.tsVectorFieldMetadataId === + tsVectorFieldMetadataId, + ); diff --git a/packages/twenty-server/src/engine/workspace-manager/workspace-migration/workspace-migration-runner/action-handlers/field/services/update-field-action-handler.service.ts b/packages/twenty-server/src/engine/workspace-manager/workspace-migration/workspace-migration-runner/action-handlers/field/services/update-field-action-handler.service.ts index 48c42e4206..f9ff3c6657 100644 --- a/packages/twenty-server/src/engine/workspace-manager/workspace-migration/workspace-migration-runner/action-handlers/field/services/update-field-action-handler.service.ts +++ b/packages/twenty-server/src/engine/workspace-manager/workspace-migration/workspace-migration-runner/action-handlers/field/services/update-field-action-handler.service.ts @@ -25,6 +25,7 @@ import { isEnumFlatFieldMetadata } from 'src/engine/metadata-modules/flat-field- import { isFlatFieldMetadataOfType } from 'src/engine/metadata-modules/flat-field-metadata/utils/is-flat-field-metadata-of-type.util'; import { isMorphOrRelationFlatFieldMetadata } from 'src/engine/metadata-modules/flat-field-metadata/utils/is-morph-or-relation-flat-field-metadata.util'; import { deriveSearchVectorAsExpressionForTsVectorField } from 'src/engine/metadata-modules/flat-search-field-metadata/utils/derive-search-vector-as-expression-for-ts-vector-field.util'; +import { getTargetSearchFieldMetadatasForTsVectorField } from 'src/engine/metadata-modules/flat-search-field-metadata/utils/get-target-search-field-metadatas-for-ts-vector-field.util'; import { FlatObjectMetadata } from 'src/engine/metadata-modules/flat-object-metadata/types/flat-object-metadata.type'; import { WorkspaceSchemaManagerService } from 'src/engine/twenty-orm/workspace-schema-manager/workspace-schema-manager.service'; import { computeObjectTargetTable } from 'src/engine/utils/compute-object-target-table.util'; @@ -180,6 +181,7 @@ export class UpdateFieldActionHandlerService extends WorkspaceMigrationRunnerAct flatIndexMaps, }, workspaceId, + getSearchFieldMetadatasByTsVectorFieldId, } = context; const { entityId, update } = flatAction; @@ -367,8 +369,14 @@ export class UpdateFieldActionHandlerService extends WorkspaceMigrationRunnerAct const searchVectorAsExpression = deriveSearchVectorAsExpressionForTsVectorField({ - tsVectorFieldMetadataId: optimisticFlatFieldMetadata.id, - flatSearchFieldMetadataMaps, + targetSearchFieldMetadatas: + getSearchFieldMetadatasByTsVectorFieldId?.( + optimisticFlatFieldMetadata.id, + ) ?? + getTargetSearchFieldMetadatasForTsVectorField({ + tsVectorFieldMetadataId: optimisticFlatFieldMetadata.id, + flatSearchFieldMetadataMaps, + }), indexedFieldById, }); diff --git a/packages/twenty-server/src/engine/workspace-manager/workspace-migration/workspace-migration-runner/action-handlers/object/services/create-object-action-handler.service.ts b/packages/twenty-server/src/engine/workspace-manager/workspace-migration/workspace-migration-runner/action-handlers/object/services/create-object-action-handler.service.ts index 7531193b76..d0d85fddcb 100644 --- a/packages/twenty-server/src/engine/workspace-manager/workspace-migration/workspace-migration-runner/action-handlers/object/services/create-object-action-handler.service.ts +++ b/packages/twenty-server/src/engine/workspace-manager/workspace-migration/workspace-migration-runner/action-handlers/object/services/create-object-action-handler.service.ts @@ -10,6 +10,7 @@ import { isCompositeFlatFieldMetadata } from 'src/engine/metadata-modules/flat-f import { isEnumFlatFieldMetadata } from 'src/engine/metadata-modules/flat-field-metadata/utils/is-enum-flat-field-metadata.util'; import { isFlatFieldMetadataOfType } from 'src/engine/metadata-modules/flat-field-metadata/utils/is-flat-field-metadata-of-type.util'; import { deriveSearchVectorAsExpressionForTsVectorField } from 'src/engine/metadata-modules/flat-search-field-metadata/utils/derive-search-vector-as-expression-for-ts-vector-field.util'; +import { getTargetSearchFieldMetadatasForTsVectorField } from 'src/engine/metadata-modules/flat-search-field-metadata/utils/get-target-search-field-metadatas-for-ts-vector-field.util'; import { WorkspaceSchemaManagerService } from 'src/engine/twenty-orm/workspace-schema-manager/workspace-schema-manager.service'; import { FlatCreateObjectAction, @@ -120,7 +121,13 @@ export class CreateObjectActionHandlerService extends WorkspaceMigrationRunnerAc async executeForWorkspaceSchema( context: WorkspaceMigrationActionRunnerContext, ): Promise { - const { flatAction, queryRunner, workspaceId, allFlatEntityMaps } = context; + const { + flatAction, + queryRunner, + workspaceId, + allFlatEntityMaps, + getSearchFieldMetadatasByTsVectorFieldId, + } = context; const { flatEntity: flatObjectMetadata, flatFieldMetadatas } = flatAction; const { schemaName, tableName } = getWorkspaceSchemaContextForMigration({ @@ -145,9 +152,15 @@ export class CreateObjectActionHandlerService extends WorkspaceMigrationRunnerAc FieldMetadataType.TS_VECTOR, ) ? deriveSearchVectorAsExpressionForTsVectorField({ - tsVectorFieldMetadataId: flatFieldMetadata.id, - flatSearchFieldMetadataMaps: - allFlatEntityMaps.flatSearchFieldMetadataMaps, + targetSearchFieldMetadatas: + getSearchFieldMetadatasByTsVectorFieldId?.( + flatFieldMetadata.id, + ) ?? + getTargetSearchFieldMetadatasForTsVectorField({ + tsVectorFieldMetadataId: flatFieldMetadata.id, + flatSearchFieldMetadataMaps: + allFlatEntityMaps.flatSearchFieldMetadataMaps, + }), indexedFieldById, }) : undefined, diff --git a/packages/twenty-server/src/engine/workspace-manager/workspace-migration/workspace-migration-runner/services/workspace-migration-runner.service.ts b/packages/twenty-server/src/engine/workspace-manager/workspace-migration/workspace-migration-runner/services/workspace-migration-runner.service.ts index c61caac76d..58fefd7c2f 100644 --- a/packages/twenty-server/src/engine/workspace-manager/workspace-migration/workspace-migration-runner/services/workspace-migration-runner.service.ts +++ b/packages/twenty-server/src/engine/workspace-manager/workspace-migration/workspace-migration-runner/services/workspace-migration-runner.service.ts @@ -13,6 +13,7 @@ import { getMetadataFlatEntityMapsKey } from 'src/engine/metadata-modules/flat-e import { getMetadataRelatedMetadataNamesForValidation } from 'src/engine/metadata-modules/flat-entity/utils/get-metadata-related-metadata-names-for-validation.util'; import { getMetadataRelatedMetadataNames } from 'src/engine/metadata-modules/flat-entity/utils/get-metadata-related-metadata-names.util'; import { getMetadataSerializedRelationNames } from 'src/engine/metadata-modules/flat-entity/utils/get-metadata-serialized-relation-names.util'; +import { createSearchFieldMetadatasByTsVectorFieldIdAccessor } from 'src/engine/metadata-modules/flat-search-field-metadata/utils/create-search-field-metadatas-by-ts-vector-field-id-accessor.util'; import { FIND_ALL_VIEWS_GRAPHQL_OPERATION } from 'src/engine/metadata-modules/view/constants/find-all-views-graphql-operation.constant'; import { WorkspaceMetadataVersionService } from 'src/engine/metadata-modules/workspace-metadata-version/services/workspace-metadata-version.service'; import { WorkspaceCacheStorageService } from 'src/engine/workspace-cache-storage/workspace-cache-storage.service'; @@ -310,6 +311,11 @@ export class WorkspaceMigrationRunnerService { let slowestActionLabel = 'n/a'; let actionCount = 0; + const searchFieldMetadatasByTsVectorFieldIdAccessor = + createSearchFieldMetadatasByTsVectorFieldIdAccessor( + () => allFlatEntityMaps.flatSearchFieldMetadataMaps, + ); + try { await queryRunner.query(`SET LOCAL lock_timeout = '8s'`); @@ -330,6 +336,8 @@ export class WorkspaceMigrationRunnerService { queryRunner, workspaceId, preallocatedIdByUniversalIdentifierByMetadataName, + getSearchFieldMetadatasByTsVectorFieldId: + searchFieldMetadatasByTsVectorFieldIdAccessor.get, }, }, ); @@ -355,6 +363,10 @@ export class WorkspaceMigrationRunnerService { ...partialOptimisticCache, } as typeof allFlatEntityMaps; + if (action.metadataName === 'searchFieldMetadata') { + searchFieldMetadatasByTsVectorFieldIdAccessor.invalidate(); + } + allMetadataEvents.push(...metadataEvents); allAfterCommitSideEffects.push(...afterCommitSideEffects); } diff --git a/packages/twenty-server/src/engine/workspace-manager/workspace-migration/workspace-migration-runner/types/workspace-migration-action-runner-args.type.ts b/packages/twenty-server/src/engine/workspace-manager/workspace-migration/workspace-migration-runner/types/workspace-migration-action-runner-args.type.ts index 80c12593d8..524a3c1da8 100644 --- a/packages/twenty-server/src/engine/workspace-manager/workspace-migration/workspace-migration-runner/types/workspace-migration-action-runner-args.type.ts +++ b/packages/twenty-server/src/engine/workspace-manager/workspace-migration/workspace-migration-runner/types/workspace-migration-action-runner-args.type.ts @@ -2,6 +2,7 @@ import { type QueryRunner } from 'typeorm'; import { type FlatApplication } from 'src/engine/core-modules/application/types/flat-application.type'; import { type AllFlatEntityMaps } from 'src/engine/metadata-modules/flat-entity/types/all-flat-entity-maps.type'; +import { type FlatSearchFieldMetadata } from 'src/engine/metadata-modules/flat-search-field-metadata/types/flat-search-field-metadata.type'; import { type PreallocatedIdByUniversalIdentifierByMetadataName } from 'src/engine/workspace-manager/workspace-migration/universal-flat-entity/utils/resolve-universal-relation-identifiers-to-ids.util'; import { type AllFlatWorkspaceMigrationAction, @@ -17,6 +18,9 @@ export type WorkspaceMigrationActionRunnerArgs< workspaceId: string; flatApplication: FlatApplication; preallocatedIdByUniversalIdentifierByMetadataName?: PreallocatedIdByUniversalIdentifierByMetadataName; + getSearchFieldMetadatasByTsVectorFieldId?: ( + tsVectorFieldMetadataId: string, + ) => FlatSearchFieldMetadata[]; }; export type WorkspaceMigrationActionRunnerContext<