Wire up search field metadata (#21964)

## Part 1 - Exact scope of the current PR (#21964) close https://github.com/twentyhq/core-team-issues/issues/2586 This PR introduces `searchFieldMetadata` as a first-class flat metadata entity and migrates the existing search surface onto it, with **no change to which records are searchable** (ISO with `main`). In scope (what the PR does): - New flat entity `searchFieldMetadata` (universalIdentifier, applicationId, **`position`**, maps, conversions), registered in the central flat-entity constants and the migration build orchestrator. - `searchVector.asExpression` is **derived server-side** from `searchFieldMetadata` rows (validated by `isSafeTsVectorExpression`); never trusted from client input. - **Derivation order is deterministic, driven by each row's `position`** ([compute-search-vector-as-expression-from-search-field-metadatas.util.ts](packages/twenty-server/src/engine/metadata-modules/flat-search-field-metadata/utils/compute-search-vector-as-expression-from-search-field-metadatas.util.ts)), replacing the previous non-deterministic `(createdAt, id)` sort. That sort collapsed to random UUIDs for standard fields (same `createdAt`), so any rename/relabel rewrote the `STORED` generated column to a logically-identical-but-textually-different expression and produced a permanent per-workspace diff vs the standard definition. Ordering now equals provisioning order; ties break on `universalIdentifier`. - Provisioning at object creation mirrors the existing surface exactly **and seeds `position`**: - custom objects -> the `name` field only, at `position: 0` ([build-default-search-field-metadatas-for-custom-object.util.ts](packages/twenty-server/src/engine/metadata-modules/object-metadata/utils/build-default-search-field-metadatas-for-custom-object.util.ts)) - standard objects -> their curated `SEARCH_FIELDS_FOR_*` sets, `position` = the curated index - Backfill (instance + workspace commands in `2-16`) provisions rows for existing workspaces with the same surface **and the same positions** (standard from the curated standard maps, custom `name` = `0`), scoped to the workspace's own custom application ([build-search-field-metadata-backfill-operations.util.ts](packages/twenty-server/src/database/commands/upgrade-version-command/2-16/utils/build-search-field-metadata-backfill-operations.util.ts)). The `position` column is added in the same `2-16` fast instance command as `universalIdentifier`/`applicationId`. - Field rename of an already-indexed field recomputes `asExpression` (positions preserved, so order is stable) ([recompute-search-vector-on-field-rename.util.ts](packages/twenty-server/src/engine/metadata-modules/flat-field-metadata/utils/recompute-search-vector-on-field-rename.util.ts)). - Field delete drops the matching row(s) and recomputes; remaining rows keep their relative order (no renumber) ([from-delete-field-input-to-flat-field-metadatas-to-delete.util.ts](packages/twenty-server/src/engine/metadata-modules/flat-field-metadata/utils/from-delete-field-input-to-flat-field-metadatas-to-delete.util.ts)). - Object relabel is **additive** and ISO/regression-fix only: it indexes the new label identifier **appended last (`position = max(existing) + 1`)** without dropping `name` ([recompute-search-vector-on-label-identifier-update.util.ts](packages/twenty-server/src/engine/metadata-modules/flat-object-metadata/utils/recompute-search-vector-on-label-identifier-update.util.ts)). This is a deliberate, temporary bridge. Explicitly OUT of scope (deferred): - No API to edit `searchFieldMetadata` (no user-facing search-field configuration, including `position` — it is internal and only written by provisioning/backfill/recompute). - No auto-indexing of arbitrary searchable fields. Creating a custom TEXT/EMAILS/etc. field does NOT add it to search (the `computeSearchFieldMetadataCreationForFields` behavior was removed in `e6820ad`). - No field-type-transition handling (field type is immutable - not in `FLAT_FIELD_METADATA_EDITABLE_PROPERTIES`, so that path was dead code). - No `position` validation (uniqueness/range) and no multi-vector / per-field `weight` config — deferred to the configurable-search follow-up (#1428). Net: `searchFieldMetadata` becomes the source of truth for the *same* surface as `main`. The only intentional divergences from `main` are "relabel preserves `name`" (additive) and the deterministic `position`-ordered `asExpression` (a correctness/perf fix that is byte-identical to provisioning order, so it does not change the searchable surface). --------- Co-authored-by: Félix Malfait <felix@twenty.com>
2026-06-23 16:27:13 +02:00
parent 99e7c2cde0
commit e9d5d71cd3
102 changed files with 3850 additions and 366 deletions
@@ -303,6 +303,12 @@ exports[`ALL_UNIVERSAL_FLAT_ENTITY_PROPERTIES_TO_COMPARE_AND_STRINGIFY should ma
    ],
    "propertiesToStringify": [],
  },
+  "searchFieldMetadata": {
+    "propertiesToCompare": [
+      "position",
+    ],
+    "propertiesToStringify": [],
+  },
  "skill": {
    "propertiesToCompare": [
      "name",
@@ -1769,6 +1769,33 @@ export const ALL_ENTITY_PROPERTIES_CONFIGURATION_BY_METADATA_NAME = {
      universalProperty: undefined,
    },
  },
+  searchFieldMetadata: {
+    objectMetadataId: {
+      toCompare: false,
+      toStringify: false,
+      universalProperty: 'objectMetadataUniversalIdentifier',
+    },
+    fieldMetadataId: {
+      toCompare: false,
+      toStringify: false,
+      universalProperty: 'fieldMetadataUniversalIdentifier',
+    },
+    position: {
+      toCompare: true,
+      toStringify: false,
+      universalProperty: undefined,
+    },
+    createdAt: {
+      toCompare: false,
+      toStringify: false,
+      universalProperty: undefined,
+    },
+    updatedAt: {
+      toCompare: false,
+      toStringify: false,
+      universalProperty: undefined,
+    },
+  },
 } as const satisfies {
  [P in AllMetadataName]: MetadataEntityPropertyConfiguration<P>;
 };
@@ -303,6 +303,16 @@ export const ALL_MANY_TO_ONE_METADATA_FOREIGN_KEY = {
    workspace: null,
    application: null,
  },
+  searchFieldMetadata: {
+    workspace: null,
+    application: null,
+    objectMetadata: {
+      foreignKey: 'objectMetadataId',
+    },
+    fieldMetadata: {
+      foreignKey: 'fieldMetadataId',
+    },
+  },
 } as const satisfies ManyToOneMetadataRelationsProperties;

 // satisfies with complex mapped types involving nested generics doesn't always catch missing required keys
@@ -531,6 +531,24 @@ export const ALL_MANY_TO_ONE_METADATA_RELATIONS = {
    workspace: null,
    application: null,
  },
+  searchFieldMetadata: {
+    workspace: null,
+    application: null,
+    objectMetadata: {
+      metadataName: 'objectMetadata',
+      foreignKey: 'objectMetadataId',
+      inverseOneToManyProperty: 'searchFieldMetadatas',
+      isNullable: false,
+      universalForeignKey: 'objectMetadataUniversalIdentifier',
+    },
+    fieldMetadata: {
+      metadataName: 'fieldMetadata',
+      foreignKey: 'fieldMetadataId',
+      inverseOneToManyProperty: null,
+      isNullable: false,
+      universalForeignKey: 'fieldMetadataUniversalIdentifier',
+    },
+  },
 } as const satisfies ManyToOneMetadataRelationsProperties;

 // satisfies with complex mapped types involving nested generics doesn't always catch missing required keys
@@ -31,6 +31,7 @@ import { ViewFilterEntity } from 'src/engine/metadata-modules/view-filter/entiti
 import { ViewGroupEntity } from 'src/engine/metadata-modules/view-group/entities/view-group.entity';
 import { ViewEntity } from 'src/engine/metadata-modules/view/entities/view.entity';
 import { ViewSortEntity } from 'src/engine/metadata-modules/view-sort/entities/view-sort.entity';
+import { SearchFieldMetadataEntity } from 'src/engine/metadata-modules/search-field-metadata/search-field-metadata.entity';

 export const ALL_METADATA_ENTITY_BY_METADATA_NAME = {
  viewField: ViewFieldEntity,
@@ -63,4 +64,5 @@ export const ALL_METADATA_ENTITY_BY_METADATA_NAME = {
  applicationVariable: ApplicationVariableEntity,
  viewSort: ViewSortEntity,
  connectionProvider: ConnectionProviderEntity,
+  searchFieldMetadata: SearchFieldMetadataEntity,
 } as const satisfies Record<AllMetadataName, EntityTarget<ObjectLiteral>>;
@@ -118,4 +118,8 @@ export const ALL_METADATA_REQUIRED_METADATA_FOR_VALIDATION = {
  webhook: {},
  applicationVariable: {},
  connectionProvider: {},
+  searchFieldMetadata: {
+    objectMetadata: true,
+    fieldMetadata: true,
+  },
 } as const satisfies MetadataRequiredForValidation;
@@ -53,6 +53,7 @@ export const ALL_METADATA_SERIALIZED_RELATION = {
  webhook: {},
  applicationVariable: {},
  connectionProvider: {},
+  searchFieldMetadata: {},
 } as const satisfies MetadataSerializedRelationProperties;

 // satisfies with complex mapped types involving nested generics doesn't always catch missing required keys
@@ -107,6 +107,12 @@ export const ALL_ONE_TO_MANY_METADATA_RELATIONS = {
      flatEntityForeignKeyAggregator: 'viewIds',
      universalFlatEntityForeignKeyAggregator: 'viewUniversalIdentifiers',
    },
+    searchFieldMetadatas: {
+      metadataName: 'searchFieldMetadata',
+      flatEntityForeignKeyAggregator: 'searchFieldMetadataIds',
+      universalFlatEntityForeignKeyAggregator:
+        'searchFieldMetadataUniversalIdentifiers',
+    },
  },
  view: {
    viewFields: {
@@ -254,6 +260,7 @@ export const ALL_ONE_TO_MANY_METADATA_RELATIONS = {
  applicationVariable: {},
  viewSort: {},
  connectionProvider: {},
+  searchFieldMetadata: {},
 } as const satisfies OneToManyMetadataRelationsProperties;

 // satisfies with complex mapped types involving nested generics doesn't always catch missing required keys
@@ -21,6 +21,7 @@ import { WorkspaceFlatViewFieldMapCacheService } from 'src/engine/metadata-modul
 import { WorkspaceFlatViewFilterGroupMapCacheService } from 'src/engine/metadata-modules/flat-view-filter-group/services/workspace-flat-view-filter-group-map-cache.service';
 import { WorkspaceFlatViewFilterMapCacheService } from 'src/engine/metadata-modules/flat-view-filter/services/workspace-flat-view-filter-map-cache.service';
 import { WorkspaceFlatViewGroupMapCacheService } from 'src/engine/metadata-modules/flat-view-group/services/workspace-flat-view-group-map-cache.service';
+import { WorkspaceFlatSearchFieldMetadataMapCacheService } from 'src/engine/metadata-modules/flat-search-field-metadata/services/workspace-flat-search-field-metadata-map-cache.service';
 import { WorkspaceFlatViewSortMapCacheService } from 'src/engine/metadata-modules/flat-view-sort/services/workspace-flat-view-sort-map-cache.service';
 import { WorkspaceFlatViewMapCacheService } from 'src/engine/metadata-modules/flat-view/services/workspace-flat-view-map-cache.service';
 import { FrontComponentEntity } from 'src/engine/metadata-modules/front-component/entities/front-component.entity';
@@ -37,6 +38,7 @@ import { RolePermissionFlagEntity } from 'src/engine/metadata-modules/role-permi
 import { RoleEntity } from 'src/engine/metadata-modules/role/role.entity';
 import { RowLevelPermissionPredicateGroupEntity } from 'src/engine/metadata-modules/row-level-permission-predicate/entities/row-level-permission-predicate-group.entity';
 import { RowLevelPermissionPredicateEntity } from 'src/engine/metadata-modules/row-level-permission-predicate/entities/row-level-permission-predicate.entity';
+import { SearchFieldMetadataEntity } from 'src/engine/metadata-modules/search-field-metadata/search-field-metadata.entity';
 import { ViewFieldGroupEntity } from 'src/engine/metadata-modules/view-field-group/entities/view-field-group.entity';
 import { ViewFieldEntity } from 'src/engine/metadata-modules/view-field/entities/view-field.entity';
 import { ViewFilterGroupEntity } from 'src/engine/metadata-modules/view-filter-group/entities/view-filter-group.entity';
@@ -74,6 +76,7 @@ import { WorkspaceCacheModule } from 'src/engine/workspace-cache/workspace-cache
      ApplicationEntity,
      RoleEntity,
      FrontComponentEntity,
+      SearchFieldMetadataEntity,
    ]),
  ],
  providers: [
@@ -92,12 +95,14 @@ import { WorkspaceCacheModule } from 'src/engine/workspace-cache/workspace-cache
    WorkspaceFlatRolePermissionFlagMapCacheService,
    WorkspaceFlatPermissionFlagMapCacheService,
    WorkspaceFlatViewSortMapCacheService,
+    WorkspaceFlatSearchFieldMetadataMapCacheService,
    WorkspaceFlatPageLayoutMapCacheService,
    WorkspaceFlatPageLayoutTabMapCacheService,
    WorkspaceFlatPageLayoutWidgetMapCacheService,
    WorkspaceFlatRowLevelPermissionPredicateMapCacheService,
    WorkspaceFlatRowLevelPermissionPredicateGroupMapCacheService,
    provideWorkspaceScopedRepository(PermissionFlagEntity),
+    provideWorkspaceScopedRepository(SearchFieldMetadataEntity),
    provideWorkspaceScopedRepository(IndexMetadataEntity),
    provideWorkspaceScopedRepository(ViewEntity),
    provideWorkspaceScopedRepository(ViewFieldEntity),
@@ -131,6 +136,7 @@ import { WorkspaceCacheModule } from 'src/engine/workspace-cache/workspace-cache
    WorkspaceFlatRolePermissionFlagMapCacheService,
    WorkspaceFlatPermissionFlagMapCacheService,
    WorkspaceFlatViewSortMapCacheService,
+    WorkspaceFlatSearchFieldMetadataMapCacheService,
    WorkspaceFlatPageLayoutMapCacheService,
    WorkspaceFlatPageLayoutTabMapCacheService,
    WorkspaceFlatPageLayoutWidgetMapCacheService,
@@ -45,6 +45,8 @@ import { type FlatViewGroupMaps } from 'src/engine/metadata-modules/flat-view-gr
 import { type FlatViewGroup } from 'src/engine/metadata-modules/flat-view-group/types/flat-view-group.type';
 import { type FlatViewSortMaps } from 'src/engine/metadata-modules/flat-view-sort/types/flat-view-sort-maps.type';
 import { type FlatViewSort } from 'src/engine/metadata-modules/flat-view-sort/types/flat-view-sort.type';
+import { type FlatSearchFieldMetadataMaps } from 'src/engine/metadata-modules/flat-search-field-metadata/types/flat-search-field-metadata-maps.type';
+import { type FlatSearchFieldMetadata } from 'src/engine/metadata-modules/flat-search-field-metadata/types/flat-search-field-metadata.type';
 import { type FlatViewMaps } from 'src/engine/metadata-modules/flat-view/types/flat-view-maps.type';
 import { type FlatView } from 'src/engine/metadata-modules/flat-view/types/flat-view.type';
 import { type FlatApplicationVariableMaps } from 'src/engine/metadata-modules/flat-application-variable/types/flat-application-variable-maps.type';
@@ -84,6 +86,7 @@ import { type UniversalFlatViewFilterGroup } from 'src/engine/workspace-manager/
 import { type UniversalFlatViewFilter } from 'src/engine/workspace-manager/workspace-migration/universal-flat-entity/types/universal-flat-view-filter.type';
 import { type UniversalFlatViewGroup } from 'src/engine/workspace-manager/workspace-migration/universal-flat-entity/types/universal-flat-view-group.type';
 import { type UniversalFlatViewSort } from 'src/engine/workspace-manager/workspace-migration/universal-flat-entity/types/universal-flat-view-sort.type';
+import { type UniversalFlatSearchFieldMetadata } from 'src/engine/workspace-manager/workspace-migration/universal-flat-entity/types/universal-flat-search-field-metadata.type';
 import { type UniversalFlatView } from 'src/engine/workspace-manager/workspace-migration/universal-flat-entity/types/universal-flat-view.type';
 import { type UniversalFlatApplicationVariable } from 'src/engine/workspace-manager/workspace-migration/universal-flat-entity/types/universal-flat-application-variable.type';
 import { type UniversalFlatWebhook } from 'src/engine/workspace-manager/workspace-migration/universal-flat-entity/types/universal-flat-webhook.type';
@@ -327,6 +330,14 @@ import {
  type UniversalDeleteWebhookAction,
  type UniversalUpdateWebhookAction,
 } from 'src/engine/workspace-manager/workspace-migration/workspace-migration-builder/builders/webhook/types/workspace-migration-webhook-action.type';
+import {
+  type FlatCreateSearchFieldMetadataAction,
+  type FlatDeleteSearchFieldMetadataAction,
+  type FlatUpdateSearchFieldMetadataAction,
+  type UniversalCreateSearchFieldMetadataAction,
+  type UniversalDeleteSearchFieldMetadataAction,
+  type UniversalUpdateSearchFieldMetadataAction,
+} from 'src/engine/workspace-manager/workspace-migration/workspace-migration-builder/builders/search-field-metadata/types/workspace-migration-search-field-metadata-action.type';

 export type AllFlatEntityTypesByMetadataName = {
  fieldMetadata: {
@@ -812,4 +823,20 @@ export type AllFlatEntityTypesByMetadataName = {
    universalFlatEntity: UniversalFlatConnectionProvider;
    entity: MetadataEntity<'connectionProvider'>;
  };
+  searchFieldMetadata: {
+    flatEntityMaps: FlatSearchFieldMetadataMaps;
+    universalActions: {
+      create: UniversalCreateSearchFieldMetadataAction;
+      update: UniversalUpdateSearchFieldMetadataAction;
+      delete: UniversalDeleteSearchFieldMetadataAction;
+    };
+    flatActions: {
+      create: FlatCreateSearchFieldMetadataAction;
+      update: FlatUpdateSearchFieldMetadataAction;
+      delete: FlatDeleteSearchFieldMetadataAction;
+    };
+    flatEntity: FlatSearchFieldMetadata;
+    universalFlatEntity: UniversalFlatSearchFieldMetadata;
+    entity: MetadataEntity<'searchFieldMetadata'>;
+  };
 };
@@ -60,6 +60,7 @@ exports[`getMetadataRelatedMetadataNames should return related metadata names fo
  "objectPermission",
  "fieldPermission",
  "view",
+  "searchFieldMetadata",
 ]
 `;

@@ -139,6 +140,13 @@ exports[`getMetadataRelatedMetadataNames should return related metadata names fo
 ]
 `;

+exports[`getMetadataRelatedMetadataNames should return related metadata names for searchFieldMetadata 1`] = `
+[
+  "objectMetadata",
+  "fieldMetadata",
+]
+`;
+
 exports[`getMetadataRelatedMetadataNames should return related metadata names for skill 1`] = `[]`;

 exports[`getMetadataRelatedMetadataNames should return related metadata names for view 1`] = `
@@ -11,6 +11,7 @@ exports[`sortMetadataNamesChildrenFirst should return metadata names sorted with
  "objectPermission",
  "pageLayoutWidget",
  "rolePermissionFlag",
+  "searchFieldMetadata",
  "viewSort",
  "index",
  "pageLayout",