Deprecate asExpression from field metadata search_vector (#22287)

## Summary Fully deprecates the cached `asExpression` / `generatedType` settings on `TS_VECTOR` (searchVector) fields. Previously the generated-column expression was stored in `FieldMetadataSettings` and kept in sync via imperative recompute side-effects. It is now **derived at DDL time** from the `searchFieldMetadata` rows that describe which fields feed the search vector, making `searchFieldMetadata` the single source of truth and removing a whole class of cache-drift bugs. This is delivered across the milestones tracked in #2587 and coordinates with the frontend migration (#1428). ## Why - The searchVector expression lived in two places (stored `settings.asExpression` + the actual generated column), kept consistent by bespoke side-effects (`recompute-search-vector-on-field-rename`, label-identifier recompute, etc.). - The frontend reconstructed the searchable-fields list by **regex-parsing** the stored `asExpression`. - Both are brittle. Deriving the expression from `searchFieldMetadata` rows at build/run time removes the cache and the parsing. ## What changed ### Server - data model & derivation - Introduce the `tsVectorFieldMetadata` relation on `searchFieldMetadata` (`tsVectorFieldMetadataId` / universal identifier) linking each searchable-field row to its target `TS_VECTOR` field. - New runtime derivation `deriveSearchVectorAsExpressionForTsVectorField` (`flat-search-field-metadata/utils/...`) used by the create-object and update-field handlers to generate the column expression from `searchFieldMetadata` rows. - Remove `asExpression` / `generatedType` from stored settings: `FieldMetadataSettings.TS_VECTOR` is now `null`; the column builder (`generate-column-definitions.util.ts`) hardcodes `generatedType: 'STORED'` and requires the derived expression. - Delete the imperative recompute side-effects and the `compute-search-vector-universal-settings-from-object-manifest` path; drop the `settings` block from all 28 standard `compute-*-standard-flat-field-metadata` utils. ### Server - migration runner - New `rebuildSearchVector` marker on `update-field` actions: the orchestrator synthesizes targeted column rebuilds (`compute-search-vector-rebuild-target-universal-identifiers.util.ts` + the deprioritize aggregator) only when a searchFieldMetadata change or indexed-field rename actually requires it - instead of rebuilding on every settings touch. - Deferrable FKs + in-flight ID resolution so a `searchFieldMetadata` row and its `TS_VECTOR` field can be created in the same transaction (deterministic UUIDs). ### Frontend (contract change, #1428) - New `SearchFieldMetadataDTO` + dataloader exposing `searchFieldMetadataList` on object metadata. - `SettingsObjectSearchSection` now reads `objectMetadataItem.searchFieldMetadatas` instead of parsing `asExpression`; new `SearchFieldMetadataItem` type, fragment, and mapping updates. ### Upgrade commands (2.18) - `2-18-instance-command-fast-...-add-ts-vector-field-metadata-id-to-search-field-metadata` - `2-18-instance-command-fast-...-make-search-field-metadata-fks-deferrable` - `2-18-instance-command-slow-...-backfill-ts-vector-field-metadata-id-on-search-field-metadata` (These were relocated from 2.16 to 2.18 and re-timestamped into an ordered block - add column -> make FK deferrable -> backfill data - since 2.16/2.17 are released.) ### Tests - Updated search-vector side-effect integration specs to assert behavior (search works) rather than the now-removed `asExpression`; removed the obsolete expression-validation specs; refreshed the application-sync snapshot (`universalSettings: null`). ## Upgrade / compatibility notes - Existing workspaces keep their stored `settings` until a later cleanup; nothing reads it anymore. The new derivation drives all DDL going forward. - Schema changes are gated behind the 2.18 instance commands above. ## Known follow-up (separate PR) https://github.com/twentyhq/core-team-issues/issues/2620 - The column rebuild (`DROP`/`ADD` of the `searchVector` STORED column) cascade-drops its GIN index and does not recreate it - a pre-existing regression on `main` inherited here. A follow-up PR will fix the rebuild handler to recreate the GIN index and add a 2.18 workspace command to recompute every search vector + strip the deprecated settings. (Planned.) ## Test plan - [ ] `npx nx typecheck twenty-server` / `twenty-front` - [ ] `npx nx lint:diff-with-main twenty-server` / `twenty-front` - [ ] Server integration: create/update/delete field, rename indexed field, update object - search returns expected records - [ ] Run the 2.18 instance commands on a seeded DB; verify `tsVectorFieldMetadataId` backfilled and FKs deferrable - [ ] Frontend: object Search settings tab lists the correct searchable fields (no `asExpression` parsing) close https://github.com/twentyhq/core-team-issues/issues/2587  <a href="https://cubic.dev/pr/twentyhq/twenty/pull/22287?utm_source=github" target="_blank" rel="noopener noreferrer" data-no-image-dialog="true"><picture><source media="(prefers-color-scheme: dark)" srcset="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"><source media="(prefers-color-scheme: light)" srcset="https://www.cubic.dev/buttons/review-in-cubic-light.svg"><img alt="Review in cubic" src="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"></picture></a>
2026-06-30 11:16:01 +02:00
parent 68c33a37ad
commit b0d7516951
144 changed files with 2030 additions and 1759 deletions
@@ -14,6 +14,7 @@ exports[`ALL_UNIVERSAL_FLAT_ENTITY_FOREIGN_KEY_AGGREGATOR_PROPERTIES should matc
    "kanbanAggregateOperationViewUniversalIdentifiers",
    "calendarViewUniversalIdentifiers",
    "mainGroupByFieldMetadataViewUniversalIdentifiers",
+    "searchFieldMetadataUniversalIdentifiers",
  ],
  "fieldPermission": [],
  "frontComponent": [],
@@ -37,6 +37,8 @@ exports[`flatEntityDeletedCreatedUpdatedMatrixDispatcher It should detect a crea
        "relationTargetFieldMetadataUniversalIdentifier": null,
        "relationTargetObjectMetadataId": null,
        "relationTargetObjectMetadataUniversalIdentifier": null,
+        "searchFieldMetadataIds": [],
+        "searchFieldMetadataUniversalIdentifiers": [],
        "settings": null,
        "standardOverrides": null,
        "type": "TEXT",
@@ -102,6 +104,8 @@ exports[`flatEntityDeletedCreatedUpdatedMatrixDispatcher It should detect a dele
        "relationTargetFieldMetadataUniversalIdentifier": null,
        "relationTargetObjectMetadataId": null,
        "relationTargetObjectMetadataUniversalIdentifier": null,
+        "searchFieldMetadataIds": [],
+        "searchFieldMetadataUniversalIdentifiers": [],
        "settings": null,
        "standardOverrides": null,
        "type": "TEXT",
@@ -181,6 +185,8 @@ exports[`flatEntityDeletedCreatedUpdatedMatrixDispatcher It should detect create
        "relationTargetFieldMetadataUniversalIdentifier": null,
        "relationTargetObjectMetadataId": null,
        "relationTargetObjectMetadataUniversalIdentifier": null,
+        "searchFieldMetadataIds": [],
+        "searchFieldMetadataUniversalIdentifiers": [],
        "settings": null,
        "standardOverrides": null,
        "type": "TEXT",
@@ -232,6 +238,8 @@ exports[`flatEntityDeletedCreatedUpdatedMatrixDispatcher It should detect create
        "relationTargetFieldMetadataUniversalIdentifier": null,
        "relationTargetObjectMetadataId": null,
        "relationTargetObjectMetadataUniversalIdentifier": null,
+        "searchFieldMetadataIds": [],
+        "searchFieldMetadataUniversalIdentifiers": [],
        "settings": null,
        "standardOverrides": null,
        "type": "TEXT",
@@ -11,6 +11,7 @@ import { type MetadataRelatedFlatEntityMapsKeys } from 'src/engine/metadata-modu
 import { type MetadataUniversalFlatEntityAndRelatedUniversalFlatEntityMaps } from 'src/engine/metadata-modules/flat-entity/types/metadata-related-types.type';
 import { type MetadataUniversalFlatEntity } from 'src/engine/metadata-modules/flat-entity/types/metadata-universal-flat-entity.type';
 import { findFlatEntityByUniversalIdentifierOrThrow } from 'src/engine/metadata-modules/flat-entity/utils/find-flat-entity-by-universal-identifier-or-throw.util';
+import { findFlatEntityByUniversalIdentifier } from 'src/engine/metadata-modules/flat-entity/utils/find-flat-entity-by-universal-identifier.util';
 import { getMetadataFlatEntityMapsKey } from 'src/engine/metadata-modules/flat-entity/utils/get-metadata-flat-entity-maps-key.util';
 import { type UniversalFlatEntityMaps } from 'src/engine/workspace-manager/workspace-migration/universal-flat-entity/types/universal-flat-entity-maps.type';
 import { addUniversalFlatEntityToUniversalFlatEntityMapsThroughMutationOrThrow } from 'src/engine/workspace-manager/workspace-migration/universal-flat-entity/utils/add-universal-flat-entity-to-universal-flat-entity-maps-through-mutation-or-throw.util';
@@ -22,6 +23,7 @@ type AddUniversalFlatEntityToUniversalFlatEntityAndRelatedEntityMapsThroughMutat
  metadataName: T;
  universalFlatEntity: MetadataUniversalFlatEntity<T>;
  universalFlatEntityAndRelatedMapsToMutate: MetadataUniversalFlatEntityAndRelatedUniversalFlatEntityMaps<T>;
+  skipMissingRelatedEntities?: boolean;
 };

 export const addUniversalFlatEntityToUniversalFlatEntityAndRelatedEntityMapsThroughMutationOrThrow =
@@ -29,6 +31,7 @@ export const addUniversalFlatEntityToUniversalFlatEntityAndRelatedEntityMapsThro
    metadataName,
    universalFlatEntity,
    universalFlatEntityAndRelatedMapsToMutate,
+    skipMissingRelatedEntities = false,
  }: AddUniversalFlatEntityToUniversalFlatEntityAndRelatedEntityMapsThroughMutationOrThrowArgs<T>) => {
    const flatEntityMapsKey = getMetadataFlatEntityMapsKey(metadataName);

@@ -96,6 +99,18 @@ export const addUniversalFlatEntityToUniversalFlatEntityAndRelatedEntityMapsThro
        continue;
      }

+      if (skipMissingRelatedEntities) {
+        const maybeRelatedUniversalFlatEntity =
+          findFlatEntityByUniversalIdentifier({
+            universalIdentifier: universalForeignKeyValue,
+            flatEntityMaps: relatedUniversalFlatEntityMaps,
+          });
+
+        if (!isDefined(maybeRelatedUniversalFlatEntity)) {
+          continue;
+        }
+      }
+
      const relatedUniversalFlatEntity =
        findFlatEntityByUniversalIdentifierOrThrow({
          universalIdentifier: universalForeignKeyValue,
@@ -13,6 +13,10 @@ import { type MetadataToFlatEntityMapsKey } from 'src/engine/metadata-modules/fl
 import { findFlatEntityByUniversalIdentifier } from 'src/engine/metadata-modules/flat-entity/utils/find-flat-entity-by-universal-identifier.util';
 import { getMetadataFlatEntityMapsKey } from 'src/engine/metadata-modules/flat-entity/utils/get-metadata-flat-entity-maps-key.util';

+export type PreallocatedIdByUniversalIdentifierByMetadataName = Partial<
+  Record<AllMetadataName, Record<string, string>>
+>;
+
 type ManyToOneRelationsConfig<T extends AllMetadataName> =
  (typeof ALL_MANY_TO_ONE_METADATA_RELATIONS)[T];

@@ -74,6 +78,7 @@ export const resolveUniversalRelationIdentifiersToIds = <
  metadataName,
  universalForeignKeyValues,
  flatEntityMaps,
+  preallocatedIdByUniversalIdentifierByMetadataName,
 }: {
  metadataName: T;
  universalForeignKeyValues: Record<TProvidedKeys, string | null | undefined>;
@@ -81,6 +86,7 @@ export const resolveUniversalRelationIdentifiersToIds = <
    T,
    TProvidedKeys
  >;
+  preallocatedIdByUniversalIdentifierByMetadataName?: PreallocatedIdByUniversalIdentifierByMetadataName;
 }): ResolvedForeignKeyIds<T, TProvidedKeys> => {
  const relationEntries = ALL_MANY_TO_ONE_METADATA_RELATIONS[metadataName];
  const result: Record<string, string | null> = {};
@@ -131,6 +137,18 @@ export const resolveUniversalRelationIdentifiersToIds = <
      continue;
    }

+    const preallocatedId = isDefined(universalIdentifierValue)
+      ? preallocatedIdByUniversalIdentifierByMetadataName?.[
+          targetMetadataName
+        ]?.[universalIdentifierValue]
+      : undefined;
+
+    if (isDefined(preallocatedId)) {
+      result[foreignKey] = preallocatedId;
+
+      continue;
+    }
+
    const targetEntity = findFlatEntityByUniversalIdentifier<
      MetadataFlatEntity<typeof targetMetadataName>
    >({