Deprecate asExpression from field metadata search_vector (#22287)

## Summary Fully deprecates the cached `asExpression` / `generatedType` settings on `TS_VECTOR` (searchVector) fields. Previously the generated-column expression was stored in `FieldMetadataSettings` and kept in sync via imperative recompute side-effects. It is now **derived at DDL time** from the `searchFieldMetadata` rows that describe which fields feed the search vector, making `searchFieldMetadata` the single source of truth and removing a whole class of cache-drift bugs. This is delivered across the milestones tracked in #2587 and coordinates with the frontend migration (#1428). ## Why - The searchVector expression lived in two places (stored `settings.asExpression` + the actual generated column), kept consistent by bespoke side-effects (`recompute-search-vector-on-field-rename`, label-identifier recompute, etc.). - The frontend reconstructed the searchable-fields list by **regex-parsing** the stored `asExpression`. - Both are brittle. Deriving the expression from `searchFieldMetadata` rows at build/run time removes the cache and the parsing. ## What changed ### Server - data model & derivation - Introduce the `tsVectorFieldMetadata` relation on `searchFieldMetadata` (`tsVectorFieldMetadataId` / universal identifier) linking each searchable-field row to its target `TS_VECTOR` field. - New runtime derivation `deriveSearchVectorAsExpressionForTsVectorField` (`flat-search-field-metadata/utils/...`) used by the create-object and update-field handlers to generate the column expression from `searchFieldMetadata` rows. - Remove `asExpression` / `generatedType` from stored settings: `FieldMetadataSettings.TS_VECTOR` is now `null`; the column builder (`generate-column-definitions.util.ts`) hardcodes `generatedType: 'STORED'` and requires the derived expression. - Delete the imperative recompute side-effects and the `compute-search-vector-universal-settings-from-object-manifest` path; drop the `settings` block from all 28 standard `compute-*-standard-flat-field-metadata` utils. ### Server - migration runner - New `rebuildSearchVector` marker on `update-field` actions: the orchestrator synthesizes targeted column rebuilds (`compute-search-vector-rebuild-target-universal-identifiers.util.ts` + the deprioritize aggregator) only when a searchFieldMetadata change or indexed-field rename actually requires it - instead of rebuilding on every settings touch. - Deferrable FKs + in-flight ID resolution so a `searchFieldMetadata` row and its `TS_VECTOR` field can be created in the same transaction (deterministic UUIDs). ### Frontend (contract change, #1428) - New `SearchFieldMetadataDTO` + dataloader exposing `searchFieldMetadataList` on object metadata. - `SettingsObjectSearchSection` now reads `objectMetadataItem.searchFieldMetadatas` instead of parsing `asExpression`; new `SearchFieldMetadataItem` type, fragment, and mapping updates. ### Upgrade commands (2.18) - `2-18-instance-command-fast-...-add-ts-vector-field-metadata-id-to-search-field-metadata` - `2-18-instance-command-fast-...-make-search-field-metadata-fks-deferrable` - `2-18-instance-command-slow-...-backfill-ts-vector-field-metadata-id-on-search-field-metadata` (These were relocated from 2.16 to 2.18 and re-timestamped into an ordered block - add column -> make FK deferrable -> backfill data - since 2.16/2.17 are released.) ### Tests - Updated search-vector side-effect integration specs to assert behavior (search works) rather than the now-removed `asExpression`; removed the obsolete expression-validation specs; refreshed the application-sync snapshot (`universalSettings: null`). ## Upgrade / compatibility notes - Existing workspaces keep their stored `settings` until a later cleanup; nothing reads it anymore. The new derivation drives all DDL going forward. - Schema changes are gated behind the 2.18 instance commands above. ## Known follow-up (separate PR) https://github.com/twentyhq/core-team-issues/issues/2620 - The column rebuild (`DROP`/`ADD` of the `searchVector` STORED column) cascade-drops its GIN index and does not recreate it - a pre-existing regression on `main` inherited here. A follow-up PR will fix the rebuild handler to recreate the GIN index and add a 2.18 workspace command to recompute every search vector + strip the deprecated settings. (Planned.) ## Test plan - [ ] `npx nx typecheck twenty-server` / `twenty-front` - [ ] `npx nx lint:diff-with-main twenty-server` / `twenty-front` - [ ] Server integration: create/update/delete field, rename indexed field, update object - search returns expected records - [ ] Run the 2.18 instance commands on a seeded DB; verify `tsVectorFieldMetadataId` backfilled and FKs deferrable - [ ] Frontend: object Search settings tab lists the correct searchable fields (no `asExpression` parsing) close https://github.com/twentyhq/core-team-issues/issues/2587  <a href="https://cubic.dev/pr/twentyhq/twenty/pull/22287?utm_source=github" target="_blank" rel="noopener noreferrer" data-no-image-dialog="true"><picture><source media="(prefers-color-scheme: dark)" srcset="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"><source media="(prefers-color-scheme: light)" srcset="https://www.cubic.dev/buttons/review-in-cubic-light.svg"><img alt="Review in cubic" src="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"></picture></a>
2026-06-30 11:16:01 +02:00
parent 68c33a37ad
commit b0d7516951
144 changed files with 2030 additions and 1759 deletions
@@ -1,29 +1,23 @@
-import { type FieldMetadataType } from 'twenty-shared/types';
 import { isDefined } from 'twenty-shared/utils';

 import { type AllFlatEntityMaps } from 'src/engine/metadata-modules/flat-entity/types/all-flat-entity-maps.type';
 import { findFlatEntityByIdInFlatEntityMaps } from 'src/engine/metadata-modules/flat-entity/utils/find-flat-entity-by-id-in-flat-entity-maps.util';
 import { findManyFlatEntityByIdInFlatEntityMapsOrThrow } from 'src/engine/metadata-modules/flat-entity/utils/find-many-flat-entity-by-id-in-flat-entity-maps-or-throw.util';
 import { type FlatFieldMetadata } from 'src/engine/metadata-modules/flat-field-metadata/types/flat-field-metadata.type';
-import { recomputeSearchVectorFieldFromSearchFieldMetadatas } from 'src/engine/metadata-modules/flat-object-metadata/utils/recompute-search-vector-field-from-search-field-metadatas.util';
 import { type FlatSearchFieldMetadata } from 'src/engine/metadata-modules/flat-search-field-metadata/types/flat-search-field-metadata.type';
 import { type UniversalFlatSearchFieldMetadata } from 'src/engine/workspace-manager/workspace-migration/universal-flat-entity/types/universal-flat-search-field-metadata.type';

 export const computeSearchFieldMetadataDeletionForDeletedFields = ({
  flatFieldMetadatasToDelete,
  flatObjectMetadataMaps,
-  flatFieldMetadataMaps,
  flatSearchFieldMetadataMaps,
 }: {
  flatFieldMetadatasToDelete: FlatFieldMetadata[];
 } & Pick<
  AllFlatEntityMaps,
-  | 'flatObjectMetadataMaps'
-  | 'flatFieldMetadataMaps'
-  | 'flatSearchFieldMetadataMaps'
+  'flatObjectMetadataMaps' | 'flatSearchFieldMetadataMaps'
 >): {
  searchFieldMetadatasToDelete: UniversalFlatSearchFieldMetadata[];
-  flatSearchVectorFieldsToUpdate: FlatFieldMetadata<FieldMetadataType.TS_VECTOR>[];
 } => {
  const deletedFieldIds = new Set(
    flatFieldMetadatasToDelete.map((flatFieldMetadata) => flatFieldMetadata.id),
@@ -35,8 +29,6 @@ export const computeSearchFieldMetadataDeletionForDeletedFields = ({
  );

  const searchFieldMetadatasToDelete: UniversalFlatSearchFieldMetadata[] = [];
-  const flatSearchVectorFieldsToUpdate: FlatFieldMetadata<FieldMetadataType.TS_VECTOR>[] =
-    [];

  for (const objectMetadataId of affectedObjectMetadataIds) {
    const flatObjectMetadata = findFlatEntityByIdInFlatEntityMaps({
@@ -59,31 +51,10 @@ export const computeSearchFieldMetadataDeletionForDeletedFields = ({
        deletedFieldIds.has(searchFieldMetadata.fieldMetadataId),
    );

-    if (rowsToDelete.length === 0) {
-      continue;
-    }
-
    searchFieldMetadatasToDelete.push(...rowsToDelete);
-
-    const remainingSearchFieldMetadatas = objectSearchFieldMetadatas.filter(
-      (searchFieldMetadata) =>
-        !deletedFieldIds.has(searchFieldMetadata.fieldMetadataId),
-    );
-
-    const flatSearchVectorFieldToUpdate =
-      recomputeSearchVectorFieldFromSearchFieldMetadatas({
-        flatObjectMetadata,
-        flatFieldMetadataMaps,
-        searchFieldMetadatas: remainingSearchFieldMetadatas,
-      });
-
-    if (isDefined(flatSearchVectorFieldToUpdate)) {
-      flatSearchVectorFieldsToUpdate.push(flatSearchVectorFieldToUpdate);
-    }
  }

  return {
    searchFieldMetadatasToDelete,
-    flatSearchVectorFieldsToUpdate,
  };
 };
@@ -1,4 +1,3 @@
-import { type FieldMetadataType } from 'twenty-shared/types';
 import {
  isDefined,
  trimAndRemoveDuplicatedWhitespacesFromObjectStringProperties,
@@ -13,7 +12,6 @@ import { type AllFlatEntityMaps } from 'src/engine/metadata-modules/flat-entity/
 import { findFlatEntityByIdInFlatEntityMaps } from 'src/engine/metadata-modules/flat-entity/utils/find-flat-entity-by-id-in-flat-entity-maps.util';
 import { findFlatEntityByUniversalIdentifierOrThrow } from 'src/engine/metadata-modules/flat-entity/utils/find-flat-entity-by-universal-identifier-or-throw.util';
 import { findManyFlatEntityByUniversalIdentifierInUniversalFlatEntityMapsOrThrow } from 'src/engine/metadata-modules/flat-entity/utils/find-many-flat-entity-by-universal-identifier-in-universal-flat-entity-maps-or-throw.util';
-import { type FlatFieldMetadata } from 'src/engine/metadata-modules/flat-field-metadata/types/flat-field-metadata.type';
 import { computeFlatFieldMetadataRelatedFlatFieldMetadata } from 'src/engine/metadata-modules/flat-field-metadata/utils/compute-flat-field-metadata-related-flat-field-metadata.util';
 import { computeSearchFieldMetadataDeletionForDeletedFields } from 'src/engine/metadata-modules/flat-field-metadata/utils/compute-search-field-metadata-deletion-for-deleted-fields.util';
 import { type FlatIndexMetadata } from 'src/engine/metadata-modules/flat-index-metadata/types/flat-index-metadata.type';
@@ -44,7 +42,6 @@ export const fromDeleteFieldInputToFlatFieldMetadatasToDelete = ({
  flatIndexesToUpdate: UniversalFlatIndexMetadata[];
  flatIndexesToDelete: UniversalFlatIndexMetadata[];
  searchFieldMetadatasToDelete: UniversalFlatSearchFieldMetadata[];
-  flatSearchVectorFieldsToUpdate: FlatFieldMetadata<FieldMetadataType.TS_VECTOR>[];
 } => {
  const { id: fieldMetadataToDeleteId } =
    trimAndRemoveDuplicatedWhitespacesFromObjectStringProperties(
@@ -187,11 +184,10 @@ export const fromDeleteFieldInputToFlatFieldMetadatasToDelete = ({
    },
  );

-  const { searchFieldMetadatasToDelete, flatSearchVectorFieldsToUpdate } =
+  const { searchFieldMetadatasToDelete } =
    computeSearchFieldMetadataDeletionForDeletedFields({
      flatFieldMetadatasToDelete,
      flatObjectMetadataMaps: existingFlatObjectMetadataMaps,
-      flatFieldMetadataMaps: existingFlatFieldMetadataMaps,
      flatSearchFieldMetadataMaps: existingFlatSearchFieldMetadataMaps,
    });

@@ -200,6 +196,5 @@ export const fromDeleteFieldInputToFlatFieldMetadatasToDelete = ({
    flatIndexesToDelete,
    flatIndexesToUpdate,
    searchFieldMetadatasToDelete,
-    flatSearchVectorFieldsToUpdate,
  };
 };
@@ -78,6 +78,12 @@ export const fromFieldMetadataEntityToFlatFieldMetadata = (
      fieldMetadataEntity.viewSorts?.map(
        ({ universalIdentifier }) => universalIdentifier,
      ) ?? [],
+    searchFieldMetadataIds:
+      fieldMetadataEntity.searchFieldMetadatas?.map(({ id }) => id) ?? [],
+    searchFieldMetadataUniversalIdentifiers:
+      fieldMetadataEntity.searchFieldMetadatas?.map(
+        ({ universalIdentifier }) => universalIdentifier,
+      ) ?? [],
    fieldPermissionUniversalIdentifiers:
      fieldMetadataEntity.fieldPermissions?.map(
        ({ universalIdentifier }) => universalIdentifier,
@@ -35,7 +35,6 @@ type FromUpdateFieldInputToFlatFieldMetadataArgs = {
  | 'flatViewGroupMaps'
  | 'flatViewMaps'
  | 'flatViewFieldMaps'
-  | 'flatSearchFieldMetadataMaps'
 >;

 type FlatFieldMetadataAndIndexToUpdate = {
@@ -52,7 +51,6 @@ export const fromUpdateFieldInputToFlatFieldMetadata = ({
  flatViewGroupMaps,
  flatViewMaps,
  flatViewFieldMaps,
-  flatSearchFieldMetadataMaps,
  isSystemBuild,
 }: FromUpdateFieldInputToFlatFieldMetadataArgs): FieldInputTranspilationResult<FlatFieldMetadataAndIndexToUpdate> => {
  const updateFieldInputInformalProperties =
@@ -152,7 +150,6 @@ export const fromUpdateFieldInputToFlatFieldMetadata = ({
      toFlatFieldMetadata,
      flatViewMaps,
      flatViewFieldMaps,
-      flatSearchFieldMetadataMaps,
      flatApplication,
    });

@@ -175,14 +172,12 @@ export const fromUpdateFieldInputToFlatFieldMetadata = ({
      flatViewsToDelete,
      flatViewFieldsToDelete,
      flatViewsToUpdate,
-      flatFieldMetadatasToUpdate: flatFieldMetadatasToUpdateFromSideEffect,
    } = sideEffectResult.result;

    return {
      flatFieldMetadatasToUpdate: [
        ...accumulator.flatFieldMetadatasToUpdate,
        toFlatFieldMetadata,
-        ...flatFieldMetadatasToUpdateFromSideEffect,
      ],
      flatIndexMetadatasToUpdate: [
        ...accumulator.flatIndexMetadatasToUpdate,
@@ -77,5 +77,6 @@ export const getDefaultFlatFieldMetadata = ({
    mainGroupByFieldMetadataViewUniversalIdentifiers: [],
    universalSettings: settings ?? null,
    viewSortUniversalIdentifiers: [],
+    searchFieldMetadataUniversalIdentifiers: [],
  } as const satisfies UniversalFlatFieldMetadata;
 };
@@ -1,9 +1,7 @@
 import { type FromTo } from 'twenty-shared/types';
-import { isDefined } from 'twenty-shared/utils';

 import { type FlatApplication } from 'src/engine/core-modules/application/types/flat-application.type';
 import { type AllFlatEntityMaps } from 'src/engine/metadata-modules/flat-entity/types/all-flat-entity-maps.type';
-import { findFlatEntityByIdInFlatEntityMapsOrThrow } from 'src/engine/metadata-modules/flat-entity/utils/find-flat-entity-by-id-in-flat-entity-maps-or-throw.util';
 import { type FieldInputTranspilationResult } from 'src/engine/metadata-modules/flat-field-metadata/types/field-input-transpilation-result.type';
 import { type FlatFieldMetadata } from 'src/engine/metadata-modules/flat-field-metadata/types/flat-field-metadata.type';
 import { handleEnumFlatFieldMetadataUpdateSideEffects } from 'src/engine/metadata-modules/flat-field-metadata/utils/handle-enum-flat-field-metadata-update-side-effects.util';
@@ -16,7 +14,6 @@ import {
  handleIndexChangesDuringFieldUpdate,
 } from 'src/engine/metadata-modules/flat-field-metadata/utils/handle-index-changes-during-field-update.util';
 import { isEnumFlatFieldMetadata } from 'src/engine/metadata-modules/flat-field-metadata/utils/is-enum-flat-field-metadata.util';
-import { recomputeSearchVectorOnFieldRename } from 'src/engine/metadata-modules/flat-field-metadata/utils/recompute-search-vector-on-field-rename.util';
 import { type FlatViewFiltersToDeleteAndUpdate } from 'src/engine/metadata-modules/flat-field-metadata/utils/recompute-view-filters-on-flat-field-metadata-options-update.util';
 import { type FlatViewGroupsToDeleteUpdateAndCreate } from 'src/engine/metadata-modules/flat-field-metadata/utils/recompute-view-groups-on-flat-field-metadata-options-update.util';

@@ -24,9 +21,7 @@ export type FlatFieldMetadataUpdateSideEffects =
  FlatViewFiltersToDeleteAndUpdate &
    FlatViewGroupsToDeleteUpdateAndCreate &
    FieldMetadataUpdateIndexSideEffect &
-    FieldMetadataDeactivationSideEffect & {
-      flatFieldMetadatasToUpdate: FlatFieldMetadata[];
-    };
+    FieldMetadataDeactivationSideEffect;

 type HandleFlatFieldMetadataUpdateSideEffectArgs = FromTo<
  FlatFieldMetadata,
@@ -41,7 +36,6 @@ type HandleFlatFieldMetadataUpdateSideEffectArgs = FromTo<
    | 'flatViewGroupMaps'
    | 'flatViewMaps'
    | 'flatViewFieldMaps'
-    | 'flatSearchFieldMetadataMaps'
  > & {
    flatApplication: FlatApplication;
  };
@@ -59,7 +53,6 @@ export const FLAT_FIELD_METADATA_UPDATE_EMPTY_SIDE_EFFECTS: FlatFieldMetadataUpd
    flatViewsToDelete: [],
    flatViewFieldsToDelete: [],
    flatViewsToUpdate: [],
-    flatFieldMetadatasToUpdate: [],
  };

 export const handleFlatFieldMetadataUpdateSideEffect = ({
@@ -72,7 +65,6 @@ export const handleFlatFieldMetadataUpdateSideEffect = ({
  flatViewGroupMaps,
  flatViewMaps,
  flatViewFieldMaps,
-  flatSearchFieldMetadataMaps,
  flatApplication,
 }: HandleFlatFieldMetadataUpdateSideEffectArgs): FieldInputTranspilationResult<FlatFieldMetadataUpdateSideEffects> => {
  const sideEffectResult = structuredClone(
@@ -140,25 +132,6 @@ export const handleFlatFieldMetadataUpdateSideEffect = ({
    return indexChangesSideEffectResult;
  }

-  const flatObjectMetadata = findFlatEntityByIdInFlatEntityMapsOrThrow({
-    flatEntityMaps: flatObjectMetadataMaps,
-    flatEntityId: fromFlatFieldMetadata.objectMetadataId,
-  });
-
-  const { flatSearchVectorFieldToUpdate } = recomputeSearchVectorOnFieldRename({
-    fromFlatFieldMetadata,
-    toFlatFieldMetadata,
-    flatObjectMetadata,
-    flatFieldMetadataMaps,
-    flatSearchFieldMetadataMaps,
-  });
-
-  if (isDefined(flatSearchVectorFieldToUpdate)) {
-    sideEffectResult.flatFieldMetadatasToUpdate.push(
-      flatSearchVectorFieldToUpdate,
-    );
-  }
-
  const {
    flatIndexMetadatasToUpdate,
    flatIndexMetadatasToCreate,
@@ -1,68 +0,0 @@
-import { type FieldMetadataType, type FromTo } from 'twenty-shared/types';
-
-import { type AllFlatEntityMaps } from 'src/engine/metadata-modules/flat-entity/types/all-flat-entity-maps.type';
-import { findManyFlatEntityByIdInFlatEntityMapsOrThrow } from 'src/engine/metadata-modules/flat-entity/utils/find-many-flat-entity-by-id-in-flat-entity-maps-or-throw.util';
-import { type FlatFieldMetadata } from 'src/engine/metadata-modules/flat-field-metadata/types/flat-field-metadata.type';
-import { type FlatObjectMetadata } from 'src/engine/metadata-modules/flat-object-metadata/types/flat-object-metadata.type';
-import { recomputeSearchVectorFieldFromSearchFieldMetadatas } from 'src/engine/metadata-modules/flat-object-metadata/utils/recompute-search-vector-field-from-search-field-metadatas.util';
-import { type FlatSearchFieldMetadata } from 'src/engine/metadata-modules/flat-search-field-metadata/types/flat-search-field-metadata.type';
-
-export type FieldMetadataUpdateSearchVectorSideEffect = {
-  flatSearchVectorFieldToUpdate:
-    | FlatFieldMetadata<FieldMetadataType.TS_VECTOR>
-    | undefined;
-};
-
-type RecomputeSearchVectorOnFieldRenameArgs = {
-  flatObjectMetadata: FlatObjectMetadata;
-} & FromTo<FlatFieldMetadata, 'flatFieldMetadata'> &
-  Pick<
-    AllFlatEntityMaps,
-    'flatFieldMetadataMaps' | 'flatSearchFieldMetadataMaps'
-  >;
-
-const EMPTY_SIDE_EFFECT: FieldMetadataUpdateSearchVectorSideEffect = {
-  flatSearchVectorFieldToUpdate: undefined,
-};
-
-export const recomputeSearchVectorOnFieldRename = ({
-  fromFlatFieldMetadata,
-  toFlatFieldMetadata,
-  flatObjectMetadata,
-  flatFieldMetadataMaps,
-  flatSearchFieldMetadataMaps,
-}: RecomputeSearchVectorOnFieldRenameArgs): FieldMetadataUpdateSearchVectorSideEffect => {
-  if (!flatObjectMetadata.isSearchable) {
-    return EMPTY_SIDE_EFFECT;
-  }
-
-  const objectSearchFieldMetadatas =
-    findManyFlatEntityByIdInFlatEntityMapsOrThrow<FlatSearchFieldMetadata>({
-      flatEntityMaps: flatSearchFieldMetadataMaps,
-      flatEntityIds: flatObjectMetadata.searchFieldMetadataIds,
-    });
-
-  const isIndexedField = objectSearchFieldMetadatas.some(
-    (searchFieldMetadata) =>
-      searchFieldMetadata.fieldMetadataId === toFlatFieldMetadata.id,
-  );
-
-  const hasNameChanged =
-    fromFlatFieldMetadata.name !== toFlatFieldMetadata.name;
-
-  if (!isIndexedField || !hasNameChanged) {
-    return EMPTY_SIDE_EFFECT;
-  }
-
-  return {
-    flatSearchVectorFieldToUpdate:
-      recomputeSearchVectorFieldFromSearchFieldMetadatas({
-        flatObjectMetadata,
-        flatFieldMetadataMaps,
-        searchFieldMetadatas: objectSearchFieldMetadatas,
-        overrideFlatFieldMetadataById: new Map([
-          [toFlatFieldMetadata.id, toFlatFieldMetadata],
-        ]),
-      }),
-  };
-};