Deprecate asExpression from field metadata search_vector (#22287)
## Summary Fully deprecates the cached `asExpression` / `generatedType` settings on `TS_VECTOR` (searchVector) fields. Previously the generated-column expression was stored in `FieldMetadataSettings` and kept in sync via imperative recompute side-effects. It is now **derived at DDL time** from the `searchFieldMetadata` rows that describe which fields feed the search vector, making `searchFieldMetadata` the single source of truth and removing a whole class of cache-drift bugs. This is delivered across the milestones tracked in #2587 and coordinates with the frontend migration (#1428). ## Why - The searchVector expression lived in two places (stored `settings.asExpression` + the actual generated column), kept consistent by bespoke side-effects (`recompute-search-vector-on-field-rename`, label-identifier recompute, etc.). - The frontend reconstructed the searchable-fields list by **regex-parsing** the stored `asExpression`. - Both are brittle. Deriving the expression from `searchFieldMetadata` rows at build/run time removes the cache and the parsing. ## What changed ### Server - data model & derivation - Introduce the `tsVectorFieldMetadata` relation on `searchFieldMetadata` (`tsVectorFieldMetadataId` / universal identifier) linking each searchable-field row to its target `TS_VECTOR` field. - New runtime derivation `deriveSearchVectorAsExpressionForTsVectorField` (`flat-search-field-metadata/utils/...`) used by the create-object and update-field handlers to generate the column expression from `searchFieldMetadata` rows. - Remove `asExpression` / `generatedType` from stored settings: `FieldMetadataSettings.TS_VECTOR` is now `null`; the column builder (`generate-column-definitions.util.ts`) hardcodes `generatedType: 'STORED'` and requires the derived expression. - Delete the imperative recompute side-effects and the `compute-search-vector-universal-settings-from-object-manifest` path; drop the `settings` block from all 28 standard `compute-*-standard-flat-field-metadata` utils. ### Server - migration runner - New `rebuildSearchVector` marker on `update-field` actions: the orchestrator synthesizes targeted column rebuilds (`compute-search-vector-rebuild-target-universal-identifiers.util.ts` + the deprioritize aggregator) only when a searchFieldMetadata change or indexed-field rename actually requires it - instead of rebuilding on every settings touch. - Deferrable FKs + in-flight ID resolution so a `searchFieldMetadata` row and its `TS_VECTOR` field can be created in the same transaction (deterministic UUIDs). ### Frontend (contract change, #1428) - New `SearchFieldMetadataDTO` + dataloader exposing `searchFieldMetadataList` on object metadata. - `SettingsObjectSearchSection` now reads `objectMetadataItem.searchFieldMetadatas` instead of parsing `asExpression`; new `SearchFieldMetadataItem` type, fragment, and mapping updates. ### Upgrade commands (2.18) - `2-18-instance-command-fast-...-add-ts-vector-field-metadata-id-to-search-field-metadata` - `2-18-instance-command-fast-...-make-search-field-metadata-fks-deferrable` - `2-18-instance-command-slow-...-backfill-ts-vector-field-metadata-id-on-search-field-metadata` (These were relocated from 2.16 to 2.18 and re-timestamped into an ordered block - add column -> make FK deferrable -> backfill data - since 2.16/2.17 are released.) ### Tests - Updated search-vector side-effect integration specs to assert behavior (search works) rather than the now-removed `asExpression`; removed the obsolete expression-validation specs; refreshed the application-sync snapshot (`universalSettings: null`). ## Upgrade / compatibility notes - Existing workspaces keep their stored `settings` until a later cleanup; nothing reads it anymore. The new derivation drives all DDL going forward. - Schema changes are gated behind the 2.18 instance commands above. ## Known follow-up (separate PR) https://github.com/twentyhq/core-team-issues/issues/2620 - The column rebuild (`DROP`/`ADD` of the `searchVector` STORED column) cascade-drops its GIN index and does not recreate it - a pre-existing regression on `main` inherited here. A follow-up PR will fix the rebuild handler to recreate the GIN index and add a 2.18 workspace command to recompute every search vector + strip the deprecated settings. (Planned.) ## Test plan - [ ] `npx nx typecheck twenty-server` / `twenty-front` - [ ] `npx nx lint:diff-with-main twenty-server` / `twenty-front` - [ ] Server integration: create/update/delete field, rename indexed field, update object - search returns expected records - [ ] Run the 2.18 instance commands on a seeded DB; verify `tsVectorFieldMetadataId` backfilled and FKs deferrable - [ ] Frontend: object Search settings tab lists the correct searchable fields (no `asExpression` parsing) close https://github.com/twentyhq/core-team-issues/issues/2587 <!-- This is an auto-generated description by cubic. --> <a href="https://cubic.dev/pr/twentyhq/twenty/pull/22287?utm_source=github" target="_blank" rel="noopener noreferrer" data-no-image-dialog="true"><picture><source media="(prefers-color-scheme: dark)" srcset="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"><source media="(prefers-color-scheme: light)" srcset="https://www.cubic.dev/buttons/review-in-cubic-light.svg"><img alt="Review in cubic" src="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"></picture></a> <!-- End of auto-generated description by cubic. -->
This commit is contained in:
Paul Rastoin
+19
-29
@@ -1,24 +1,20 @@
|
||||
import { type FieldMetadataType, type FromTo } from 'twenty-shared/types';
|
||||
import { type FromTo } from 'twenty-shared/types';
|
||||
import { isDefined, isSearchableFieldType } from 'twenty-shared/utils';
|
||||
|
||||
import { type AllFlatEntityMaps } from 'src/engine/metadata-modules/flat-entity/types/all-flat-entity-maps.type';
|
||||
import { findFlatEntityByIdInFlatEntityMaps } from 'src/engine/metadata-modules/flat-entity/utils/find-flat-entity-by-id-in-flat-entity-maps.util';
|
||||
import { findManyFlatEntityByIdInFlatEntityMapsOrThrow } from 'src/engine/metadata-modules/flat-entity/utils/find-many-flat-entity-by-id-in-flat-entity-maps-or-throw.util';
|
||||
import { type FlatFieldMetadata } from 'src/engine/metadata-modules/flat-field-metadata/types/flat-field-metadata.type';
|
||||
import { type FlatObjectMetadata } from 'src/engine/metadata-modules/flat-object-metadata/types/flat-object-metadata.type';
|
||||
import { recomputeSearchVectorFieldFromSearchFieldMetadatas } from 'src/engine/metadata-modules/flat-object-metadata/utils/recompute-search-vector-field-from-search-field-metadatas.util';
|
||||
import { type FlatSearchFieldMetadata } from 'src/engine/metadata-modules/flat-search-field-metadata/types/flat-search-field-metadata.type';
|
||||
import { buildFlatSearchFieldMetadataForField } from 'src/engine/metadata-modules/flat-search-field-metadata/utils/build-flat-search-field-metadata-for-field.util';
|
||||
import { findTsVectorFlatFieldMetadataForObject } from 'src/engine/metadata-modules/flat-search-field-metadata/utils/find-ts-vector-flat-field-metadata-for-object.util';
|
||||
import { type UniversalFlatSearchFieldMetadata } from 'src/engine/workspace-manager/workspace-migration/universal-flat-entity/types/universal-flat-search-field-metadata.type';
|
||||
|
||||
export type LabelIdentifierUpdateSearchVectorSideEffect = {
|
||||
flatSearchVectorFieldToUpdate:
|
||||
| FlatFieldMetadata<FieldMetadataType.TS_VECTOR>
|
||||
| undefined;
|
||||
export type LabelIdentifierUpdateSearchFieldMetadataSideEffect = {
|
||||
searchFieldMetadatasToCreate: UniversalFlatSearchFieldMetadata[];
|
||||
};
|
||||
|
||||
type RecomputeSearchVectorOnLabelIdentifierUpdateArgs = FromTo<
|
||||
type ComputeSearchFieldMetadataToCreateOnLabelIdentifierUpdateArgs = FromTo<
|
||||
FlatObjectMetadata,
|
||||
'flatObjectMetadata'
|
||||
> &
|
||||
@@ -27,21 +23,20 @@ type RecomputeSearchVectorOnLabelIdentifierUpdateArgs = FromTo<
|
||||
'flatFieldMetadataMaps' | 'flatSearchFieldMetadataMaps'
|
||||
>;
|
||||
|
||||
const EMPTY_SIDE_EFFECT: LabelIdentifierUpdateSearchVectorSideEffect = {
|
||||
flatSearchVectorFieldToUpdate: undefined,
|
||||
const EMPTY_SIDE_EFFECT: LabelIdentifierUpdateSearchFieldMetadataSideEffect = {
|
||||
searchFieldMetadatasToCreate: [],
|
||||
};
|
||||
|
||||
// Relabeling is additive: it indexes the new label identifier field while preserving the
|
||||
// existing search surface (e.g. the provisioned name row). We never drop a row here so the
|
||||
// previous label identifier stays searchable. Removal of fields from the search surface is
|
||||
// deferred to a follow-up PR.
|
||||
export const recomputeSearchVectorOnLabelIdentifierUpdate = ({
|
||||
// previous label identifier stays searchable. The searchVector column itself is rebuilt
|
||||
// from the resulting rows by the rebuildSearchVector marker in the migration runner.
|
||||
export const computeSearchFieldMetadataToCreateOnLabelIdentifierUpdate = ({
|
||||
fromFlatObjectMetadata,
|
||||
toFlatObjectMetadata,
|
||||
flatFieldMetadataMaps,
|
||||
flatSearchFieldMetadataMaps,
|
||||
}: RecomputeSearchVectorOnLabelIdentifierUpdateArgs): LabelIdentifierUpdateSearchVectorSideEffect => {
|
||||
}: ComputeSearchFieldMetadataToCreateOnLabelIdentifierUpdateArgs): LabelIdentifierUpdateSearchFieldMetadataSideEffect => {
|
||||
const newLabelIdentifierFieldMetadataId =
|
||||
toFlatObjectMetadata.labelIdentifierFieldMetadataId;
|
||||
|
||||
@@ -79,6 +74,15 @@ export const recomputeSearchVectorOnLabelIdentifierUpdate = ({
|
||||
return EMPTY_SIDE_EFFECT;
|
||||
}
|
||||
|
||||
const tsVectorFlatFieldMetadata = findTsVectorFlatFieldMetadataForObject({
|
||||
fieldUniversalIdentifiers: fromFlatObjectMetadata.fieldUniversalIdentifiers,
|
||||
flatFieldMetadataMaps,
|
||||
});
|
||||
|
||||
if (!isDefined(tsVectorFlatFieldMetadata)) {
|
||||
return EMPTY_SIDE_EFFECT;
|
||||
}
|
||||
|
||||
const newLabelIdentifierPosition =
|
||||
existingSearchFieldMetadatas.reduce(
|
||||
(maxPosition, searchFieldMetadata) =>
|
||||
@@ -89,25 +93,11 @@ export const recomputeSearchVectorOnLabelIdentifierUpdate = ({
|
||||
const newSearchFieldMetadata = buildFlatSearchFieldMetadataForField({
|
||||
flatObjectMetadata: fromFlatObjectMetadata,
|
||||
flatFieldMetadata: newLabelIdentifierField,
|
||||
tsVectorFlatFieldMetadata,
|
||||
position: newLabelIdentifierPosition,
|
||||
});
|
||||
|
||||
const nextSearchFieldMetadatas = [
|
||||
...existingSearchFieldMetadatas,
|
||||
{
|
||||
fieldMetadataId: newLabelIdentifierFieldMetadataId,
|
||||
position: newLabelIdentifierPosition,
|
||||
universalIdentifier: newSearchFieldMetadata.universalIdentifier,
|
||||
},
|
||||
];
|
||||
|
||||
return {
|
||||
flatSearchVectorFieldToUpdate:
|
||||
recomputeSearchVectorFieldFromSearchFieldMetadatas({
|
||||
flatObjectMetadata: fromFlatObjectMetadata,
|
||||
flatFieldMetadataMaps,
|
||||
searchFieldMetadatas: nextSearchFieldMetadatas,
|
||||
}),
|
||||
searchFieldMetadatasToCreate: [newSearchFieldMetadata],
|
||||
};
|
||||
};
|
||||
-2
@@ -102,7 +102,6 @@ export const fromUpdateObjectInputToFlatObjectMetadataAndRelatedFlatEntities =
|
||||
flatViewFieldsToCreate,
|
||||
flatViewFieldsToUpdate,
|
||||
otherObjectFlatFieldMetadatasToUpdate,
|
||||
sameObjectFlatFieldMetadatasToUpdate,
|
||||
searchFieldMetadatasToCreate,
|
||||
} = handleFlatObjectMetadataUpdateSideEffect({
|
||||
fromFlatObjectMetadata: existingFlatObjectMetadata,
|
||||
@@ -121,7 +120,6 @@ export const fromUpdateObjectInputToFlatObjectMetadataAndRelatedFlatEntities =
|
||||
flatViewFieldsToCreate,
|
||||
flatViewFieldsToUpdate,
|
||||
otherObjectFlatFieldMetadatasToUpdate,
|
||||
sameObjectFlatFieldMetadatasToUpdate,
|
||||
searchFieldMetadatasToCreate,
|
||||
};
|
||||
};
|
||||
|
||||
+3
-12
@@ -2,10 +2,9 @@ import { type FromTo } from 'twenty-shared/types';
|
||||
import { isDefined } from 'twenty-shared/utils';
|
||||
|
||||
import { type AllFlatEntityMaps } from 'src/engine/metadata-modules/flat-entity/types/all-flat-entity-maps.type';
|
||||
import { type FlatFieldMetadata } from 'src/engine/metadata-modules/flat-field-metadata/types/flat-field-metadata.type';
|
||||
import { type FlatObjectMetadata } from 'src/engine/metadata-modules/flat-object-metadata/types/flat-object-metadata.type';
|
||||
import { computeSearchFieldMetadataToCreateOnLabelIdentifierUpdate } from 'src/engine/metadata-modules/flat-object-metadata/utils/compute-search-field-metadata-to-create-on-label-identifier-update.util';
|
||||
import { recomputeIndexAfterFlatObjectMetadataSingularNameUpdate } from 'src/engine/metadata-modules/flat-object-metadata/utils/recompute-index-after-flat-object-metadata-singular-name-update.util';
|
||||
import { recomputeSearchVectorOnLabelIdentifierUpdate } from 'src/engine/metadata-modules/flat-object-metadata/utils/recompute-search-vector-on-label-identifier-update.util';
|
||||
import { recomputeViewFieldIdentifierAfterFlatObjectIdentifierUpdate } from 'src/engine/metadata-modules/flat-object-metadata/utils/recompute-view-field-identifier-after-flat-object-identifier-update.util';
|
||||
import { renameRelatedMorphFieldOnObjectNamesUpdate } from 'src/engine/metadata-modules/flat-object-metadata/utils/rename-related-morph-field-on-object-names-update.util';
|
||||
import { type UniversalFlatFieldMetadata } from 'src/engine/workspace-manager/workspace-migration/universal-flat-entity/types/universal-flat-field-metadata.type';
|
||||
@@ -15,7 +14,6 @@ import { type UniversalFlatViewField } from 'src/engine/workspace-manager/worksp
|
||||
|
||||
export type FlatObjectMetadataUpdateSideEffects = {
|
||||
otherObjectFlatFieldMetadatasToUpdate: UniversalFlatFieldMetadata[];
|
||||
sameObjectFlatFieldMetadatasToUpdate: UniversalFlatFieldMetadata[];
|
||||
flatViewFieldsToUpdate: UniversalFlatViewField[];
|
||||
flatViewFieldsToCreate: UniversalFlatViewField[];
|
||||
flatIndexMetadatasToUpdate: UniversalFlatIndexMetadata[];
|
||||
@@ -89,20 +87,14 @@ export const handleFlatObjectMetadataUpdateSideEffect = ({
|
||||
flatViewFieldsToUpdate: [],
|
||||
};
|
||||
|
||||
const sameObjectFlatFieldMetadatasToUpdate: FlatFieldMetadata[] = [];
|
||||
|
||||
const { flatSearchVectorFieldToUpdate, searchFieldMetadatasToCreate } =
|
||||
recomputeSearchVectorOnLabelIdentifierUpdate({
|
||||
const { searchFieldMetadatasToCreate } =
|
||||
computeSearchFieldMetadataToCreateOnLabelIdentifierUpdate({
|
||||
fromFlatObjectMetadata,
|
||||
toFlatObjectMetadata,
|
||||
flatFieldMetadataMaps,
|
||||
flatSearchFieldMetadataMaps,
|
||||
});
|
||||
|
||||
if (isDefined(flatSearchVectorFieldToUpdate)) {
|
||||
sameObjectFlatFieldMetadatasToUpdate.push(flatSearchVectorFieldToUpdate);
|
||||
}
|
||||
|
||||
return {
|
||||
flatIndexMetadatasToUpdate: [
|
||||
...morphRelatedFlatIndexesToUpdate,
|
||||
@@ -111,7 +103,6 @@ export const handleFlatObjectMetadataUpdateSideEffect = ({
|
||||
flatViewFieldsToCreate,
|
||||
flatViewFieldsToUpdate,
|
||||
otherObjectFlatFieldMetadatasToUpdate: morphFlatFieldMetadatasToUpdate,
|
||||
sameObjectFlatFieldMetadatasToUpdate,
|
||||
searchFieldMetadatasToCreate,
|
||||
};
|
||||
};
|
||||
|
||||
-112
@@ -1,112 +0,0 @@
|
||||
import { FieldMetadataType } from 'twenty-shared/types';
|
||||
import { isDefined } from 'twenty-shared/utils';
|
||||
|
||||
import { type AllFlatEntityMaps } from 'src/engine/metadata-modules/flat-entity/types/all-flat-entity-maps.type';
|
||||
import { findFlatEntityByIdInFlatEntityMaps } from 'src/engine/metadata-modules/flat-entity/utils/find-flat-entity-by-id-in-flat-entity-maps.util';
|
||||
import { findManyFlatEntityByIdInFlatEntityMapsOrThrow } from 'src/engine/metadata-modules/flat-entity/utils/find-many-flat-entity-by-id-in-flat-entity-maps-or-throw.util';
|
||||
import { type FlatFieldMetadata } from 'src/engine/metadata-modules/flat-field-metadata/types/flat-field-metadata.type';
|
||||
import { isFlatFieldMetadataOfType } from 'src/engine/metadata-modules/flat-field-metadata/utils/is-flat-field-metadata-of-type.util';
|
||||
import { type FlatObjectMetadata } from 'src/engine/metadata-modules/flat-object-metadata/types/flat-object-metadata.type';
|
||||
import { type FlatSearchFieldMetadata } from 'src/engine/metadata-modules/flat-search-field-metadata/types/flat-search-field-metadata.type';
|
||||
import {
|
||||
ObjectMetadataException,
|
||||
ObjectMetadataExceptionCode,
|
||||
} from 'src/engine/metadata-modules/object-metadata/object-metadata.exception';
|
||||
import {
|
||||
buildSearchVectorTargetField,
|
||||
computeSearchVectorAsExpressionFromSearchFieldMetadatas,
|
||||
} from 'src/engine/metadata-modules/flat-search-field-metadata/utils/compute-search-vector-as-expression-from-search-field-metadatas.util';
|
||||
import { SEARCH_VECTOR_FIELD } from 'src/engine/metadata-modules/search-field-metadata/constants/search-vector-field.constants';
|
||||
|
||||
type RecomputeSearchVectorFieldFromSearchFieldMetadatasArgs = {
|
||||
flatObjectMetadata: FlatObjectMetadata;
|
||||
// Post-change searchFieldMetadata rows for the object, carrying the field id and the
|
||||
// per-object position that drives the deterministic asExpression order.
|
||||
searchFieldMetadatas: Pick<
|
||||
FlatSearchFieldMetadata,
|
||||
'fieldMetadataId' | 'position' | 'universalIdentifier'
|
||||
>[];
|
||||
// Takes precedence over the flat maps when resolving an id — for a field created or
|
||||
// renamed in the same migration, whose maps entry is absent or stale.
|
||||
overrideFlatFieldMetadataById?: Map<string, FlatFieldMetadata>;
|
||||
} & Pick<AllFlatEntityMaps, 'flatFieldMetadataMaps'>;
|
||||
|
||||
export const recomputeSearchVectorFieldFromSearchFieldMetadatas = ({
|
||||
flatObjectMetadata,
|
||||
flatFieldMetadataMaps,
|
||||
searchFieldMetadatas,
|
||||
overrideFlatFieldMetadataById,
|
||||
}: RecomputeSearchVectorFieldFromSearchFieldMetadatasArgs):
|
||||
| FlatFieldMetadata<FieldMetadataType.TS_VECTOR>
|
||||
| undefined => {
|
||||
const objectFlatFieldMetadatas =
|
||||
findManyFlatEntityByIdInFlatEntityMapsOrThrow({
|
||||
flatEntityMaps: flatFieldMetadataMaps,
|
||||
flatEntityIds: flatObjectMetadata.fieldIds,
|
||||
});
|
||||
|
||||
const searchVectorField = objectFlatFieldMetadatas.find(
|
||||
(field) => field.name === SEARCH_VECTOR_FIELD.name,
|
||||
);
|
||||
|
||||
if (
|
||||
!isDefined(searchVectorField) ||
|
||||
!isFlatFieldMetadataOfType(searchVectorField, FieldMetadataType.TS_VECTOR)
|
||||
) {
|
||||
throw new ObjectMetadataException(
|
||||
`Search vector field not found for object metadata ${flatObjectMetadata.id}`,
|
||||
ObjectMetadataExceptionCode.INVALID_OBJECT_INPUT,
|
||||
);
|
||||
}
|
||||
|
||||
const targetSearchableFields = searchFieldMetadatas.map(
|
||||
(searchFieldMetadata) => {
|
||||
const { fieldMetadataId, position, universalIdentifier } =
|
||||
searchFieldMetadata;
|
||||
|
||||
const overriddenFlatFieldMetadata =
|
||||
overrideFlatFieldMetadataById?.get(fieldMetadataId);
|
||||
|
||||
const flatFieldMetadata =
|
||||
overriddenFlatFieldMetadata ??
|
||||
findFlatEntityByIdInFlatEntityMaps({
|
||||
flatEntityId: fieldMetadataId,
|
||||
flatEntityMaps: flatFieldMetadataMaps,
|
||||
});
|
||||
|
||||
if (!isDefined(flatFieldMetadata)) {
|
||||
throw new ObjectMetadataException(
|
||||
`Field metadata ${fieldMetadataId} referenced by searchFieldMetadata not found for object metadata ${flatObjectMetadata.id}`,
|
||||
ObjectMetadataExceptionCode.INVALID_OBJECT_INPUT,
|
||||
);
|
||||
}
|
||||
|
||||
return buildSearchVectorTargetField(
|
||||
flatFieldMetadata,
|
||||
position,
|
||||
universalIdentifier,
|
||||
);
|
||||
},
|
||||
);
|
||||
|
||||
try {
|
||||
const newAsExpression =
|
||||
computeSearchVectorAsExpressionFromSearchFieldMetadatas(
|
||||
targetSearchableFields,
|
||||
);
|
||||
|
||||
return {
|
||||
...searchVectorField,
|
||||
universalSettings: {
|
||||
...searchVectorField.universalSettings,
|
||||
asExpression: newAsExpression,
|
||||
generatedType: 'STORED',
|
||||
},
|
||||
};
|
||||
} catch (error) {
|
||||
throw new ObjectMetadataException(
|
||||
`Failed to compute search vector column expression for object metadata ${flatObjectMetadata.id}: ${error instanceof Error ? error.message : String(error)}`,
|
||||
ObjectMetadataExceptionCode.INVALID_OBJECT_INPUT,
|
||||
);
|
||||
}
|
||||
};
|
||||
Reference in New Issue
Block a user