Deprecate asExpression from field metadata search_vector (#22287)
## Summary Fully deprecates the cached `asExpression` / `generatedType` settings on `TS_VECTOR` (searchVector) fields. Previously the generated-column expression was stored in `FieldMetadataSettings` and kept in sync via imperative recompute side-effects. It is now **derived at DDL time** from the `searchFieldMetadata` rows that describe which fields feed the search vector, making `searchFieldMetadata` the single source of truth and removing a whole class of cache-drift bugs. This is delivered across the milestones tracked in #2587 and coordinates with the frontend migration (#1428). ## Why - The searchVector expression lived in two places (stored `settings.asExpression` + the actual generated column), kept consistent by bespoke side-effects (`recompute-search-vector-on-field-rename`, label-identifier recompute, etc.). - The frontend reconstructed the searchable-fields list by **regex-parsing** the stored `asExpression`. - Both are brittle. Deriving the expression from `searchFieldMetadata` rows at build/run time removes the cache and the parsing. ## What changed ### Server - data model & derivation - Introduce the `tsVectorFieldMetadata` relation on `searchFieldMetadata` (`tsVectorFieldMetadataId` / universal identifier) linking each searchable-field row to its target `TS_VECTOR` field. - New runtime derivation `deriveSearchVectorAsExpressionForTsVectorField` (`flat-search-field-metadata/utils/...`) used by the create-object and update-field handlers to generate the column expression from `searchFieldMetadata` rows. - Remove `asExpression` / `generatedType` from stored settings: `FieldMetadataSettings.TS_VECTOR` is now `null`; the column builder (`generate-column-definitions.util.ts`) hardcodes `generatedType: 'STORED'` and requires the derived expression. - Delete the imperative recompute side-effects and the `compute-search-vector-universal-settings-from-object-manifest` path; drop the `settings` block from all 28 standard `compute-*-standard-flat-field-metadata` utils. ### Server - migration runner - New `rebuildSearchVector` marker on `update-field` actions: the orchestrator synthesizes targeted column rebuilds (`compute-search-vector-rebuild-target-universal-identifiers.util.ts` + the deprioritize aggregator) only when a searchFieldMetadata change or indexed-field rename actually requires it - instead of rebuilding on every settings touch. - Deferrable FKs + in-flight ID resolution so a `searchFieldMetadata` row and its `TS_VECTOR` field can be created in the same transaction (deterministic UUIDs). ### Frontend (contract change, #1428) - New `SearchFieldMetadataDTO` + dataloader exposing `searchFieldMetadataList` on object metadata. - `SettingsObjectSearchSection` now reads `objectMetadataItem.searchFieldMetadatas` instead of parsing `asExpression`; new `SearchFieldMetadataItem` type, fragment, and mapping updates. ### Upgrade commands (2.18) - `2-18-instance-command-fast-...-add-ts-vector-field-metadata-id-to-search-field-metadata` - `2-18-instance-command-fast-...-make-search-field-metadata-fks-deferrable` - `2-18-instance-command-slow-...-backfill-ts-vector-field-metadata-id-on-search-field-metadata` (These were relocated from 2.16 to 2.18 and re-timestamped into an ordered block - add column -> make FK deferrable -> backfill data - since 2.16/2.17 are released.) ### Tests - Updated search-vector side-effect integration specs to assert behavior (search works) rather than the now-removed `asExpression`; removed the obsolete expression-validation specs; refreshed the application-sync snapshot (`universalSettings: null`). ## Upgrade / compatibility notes - Existing workspaces keep their stored `settings` until a later cleanup; nothing reads it anymore. The new derivation drives all DDL going forward. - Schema changes are gated behind the 2.18 instance commands above. ## Known follow-up (separate PR) https://github.com/twentyhq/core-team-issues/issues/2620 - The column rebuild (`DROP`/`ADD` of the `searchVector` STORED column) cascade-drops its GIN index and does not recreate it - a pre-existing regression on `main` inherited here. A follow-up PR will fix the rebuild handler to recreate the GIN index and add a 2.18 workspace command to recompute every search vector + strip the deprecated settings. (Planned.) ## Test plan - [ ] `npx nx typecheck twenty-server` / `twenty-front` - [ ] `npx nx lint:diff-with-main twenty-server` / `twenty-front` - [ ] Server integration: create/update/delete field, rename indexed field, update object - search returns expected records - [ ] Run the 2.18 instance commands on a seeded DB; verify `tsVectorFieldMetadataId` backfilled and FKs deferrable - [ ] Frontend: object Search settings tab lists the correct searchable fields (no `asExpression` parsing) close https://github.com/twentyhq/core-team-issues/issues/2587 <!-- This is an auto-generated description by cubic. --> <a href="https://cubic.dev/pr/twentyhq/twenty/pull/22287?utm_source=github" target="_blank" rel="noopener noreferrer" data-no-image-dialog="true"><picture><source media="(prefers-color-scheme: dark)" srcset="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"><source media="(prefers-color-scheme: light)" srcset="https://www.cubic.dev/buttons/review-in-cubic-light.svg"><img alt="Review in cubic" src="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"></picture></a> <!-- End of auto-generated description by cubic. -->
This commit is contained in:
Paul Rastoin
+1
@@ -114,6 +114,7 @@ export const fromFieldManifestToUniversalFlatFieldMetadata = ({
|
||||
calendarViewUniversalIdentifiers: [],
|
||||
mainGroupByFieldMetadataViewUniversalIdentifiers: [],
|
||||
viewSortUniversalIdentifiers: [],
|
||||
searchFieldMetadataUniversalIdentifiers: [],
|
||||
createdAt: now,
|
||||
updatedAt: now,
|
||||
};
|
||||
|
||||
+4
-17
@@ -2,7 +2,6 @@ import { Injectable } from '@nestjs/common';
|
||||
|
||||
import { type Manifest } from 'twenty-shared/application';
|
||||
import { MAX_CUSTOM_INDEXES_PER_OBJECT } from 'twenty-shared/constants';
|
||||
import { FieldMetadataType } from 'twenty-shared/types';
|
||||
import { isDefined } from 'twenty-shared/utils';
|
||||
|
||||
import { fromApplicationVariableManifestToUniversalFlatApplicationVariable } from 'src/engine/core-modules/application/application-manifest/converters/from-application-variable-manifest-to-universal-flat-application-variable.util';
|
||||
@@ -25,7 +24,6 @@ import { fromRoleManifestToUniversalFlatRole } from 'src/engine/core-modules/app
|
||||
import { fromRowLevelPermissionPredicateGroupManifestToUniversalFlatRowLevelPermissionPredicateGroup } from 'src/engine/core-modules/application/application-manifest/converters/from-row-level-permission-predicate-group-manifest-to-universal-flat-row-level-permission-predicate-group.util';
|
||||
import { fromRowLevelPermissionPredicateManifestToUniversalFlatRowLevelPermissionPredicate } from 'src/engine/core-modules/application/application-manifest/converters/from-row-level-permission-predicate-manifest-to-universal-flat-row-level-permission-predicate.util';
|
||||
import { fromSkillManifestToUniversalFlatSkill } from 'src/engine/core-modules/application/application-manifest/converters/from-skill-manifest-to-universal-flat-skill.util';
|
||||
import { computeSearchVectorUniversalSettingsFromObjectManifest } from 'src/engine/core-modules/application/application-manifest/utils/compute-search-vector-universal-settings-from-object-manifest.util';
|
||||
import { fromViewFieldGroupManifestToUniversalFlatViewFieldGroup } from 'src/engine/core-modules/application/application-manifest/converters/from-view-field-group-manifest-to-universal-flat-view-field-group.util';
|
||||
import { fromViewFieldManifestToUniversalFlatViewField } from 'src/engine/core-modules/application/application-manifest/converters/from-view-field-manifest-to-universal-flat-view-field.util';
|
||||
import { fromViewFilterGroupManifestToUniversalFlatViewFilterGroup } from 'src/engine/core-modules/application/application-manifest/converters/from-view-filter-group-manifest-to-universal-flat-view-filter-group.util';
|
||||
@@ -95,21 +93,10 @@ export class ComputeApplicationManifestAllUniversalFlatEntityMapsService {
|
||||
});
|
||||
|
||||
for (const fieldManifest of objectManifest.fields) {
|
||||
const enrichedFieldManifest =
|
||||
fieldManifest.type === FieldMetadataType.TS_VECTOR &&
|
||||
!isDefined(fieldManifest.universalSettings)
|
||||
? {
|
||||
...fieldManifest,
|
||||
objectUniversalIdentifier: objectManifest.universalIdentifier,
|
||||
universalSettings:
|
||||
computeSearchVectorUniversalSettingsFromObjectManifest({
|
||||
objectManifest,
|
||||
}),
|
||||
}
|
||||
: {
|
||||
...fieldManifest,
|
||||
objectUniversalIdentifier: objectManifest.universalIdentifier,
|
||||
};
|
||||
const enrichedFieldManifest = {
|
||||
...fieldManifest,
|
||||
objectUniversalIdentifier: objectManifest.universalIdentifier,
|
||||
};
|
||||
|
||||
const flatFieldMetadata = fromFieldManifestToUniversalFlatFieldMetadata(
|
||||
{
|
||||
|
||||
-149
@@ -1,149 +0,0 @@
|
||||
import { type ObjectManifest } from 'twenty-shared/application';
|
||||
import { FieldMetadataType } from 'twenty-shared/types';
|
||||
|
||||
import { computeSearchVectorUniversalSettingsFromObjectManifest } from 'src/engine/core-modules/application/application-manifest/utils/compute-search-vector-universal-settings-from-object-manifest.util';
|
||||
|
||||
const buildObjectManifest = (
|
||||
overrides: Partial<ObjectManifest> & {
|
||||
fields: ObjectManifest['fields'];
|
||||
labelIdentifierFieldMetadataUniversalIdentifier: string;
|
||||
},
|
||||
): ObjectManifest => ({
|
||||
universalIdentifier: 'obj-uuid-1',
|
||||
nameSingular: 'testObject',
|
||||
namePlural: 'testObjects',
|
||||
labelSingular: 'Test Object',
|
||||
labelPlural: 'Test Objects',
|
||||
...overrides,
|
||||
});
|
||||
|
||||
describe('computeSearchVectorUniversalSettingsFromObjectManifest', () => {
|
||||
it('should return asExpression and generatedType for a TEXT label identifier field', () => {
|
||||
const result = computeSearchVectorUniversalSettingsFromObjectManifest({
|
||||
objectManifest: buildObjectManifest({
|
||||
labelIdentifierFieldMetadataUniversalIdentifier: 'field-uuid-name',
|
||||
fields: [
|
||||
{
|
||||
universalIdentifier: 'field-uuid-name',
|
||||
name: 'name',
|
||||
label: 'Name',
|
||||
type: FieldMetadataType.TEXT,
|
||||
},
|
||||
],
|
||||
}),
|
||||
});
|
||||
|
||||
expect(result?.generatedType).toBe('STORED');
|
||||
expect(result?.asExpression).toContain("to_tsvector('simple'");
|
||||
expect(result?.asExpression).toContain('"name"');
|
||||
});
|
||||
|
||||
it('should return asExpression for a FULL_NAME label identifier field', () => {
|
||||
const result = computeSearchVectorUniversalSettingsFromObjectManifest({
|
||||
objectManifest: buildObjectManifest({
|
||||
labelIdentifierFieldMetadataUniversalIdentifier: 'field-uuid-name',
|
||||
fields: [
|
||||
{
|
||||
universalIdentifier: 'field-uuid-name',
|
||||
name: 'name',
|
||||
label: 'Name',
|
||||
type: FieldMetadataType.FULL_NAME,
|
||||
},
|
||||
],
|
||||
}),
|
||||
});
|
||||
|
||||
expect(result?.generatedType).toBe('STORED');
|
||||
expect(result?.asExpression).toContain("to_tsvector('simple'");
|
||||
expect(result?.asExpression).toContain('"nameFirstName"');
|
||||
expect(result?.asExpression).toContain('"nameLastName"');
|
||||
});
|
||||
|
||||
it('should return asExpression for an EMAILS label identifier field', () => {
|
||||
const result = computeSearchVectorUniversalSettingsFromObjectManifest({
|
||||
objectManifest: buildObjectManifest({
|
||||
labelIdentifierFieldMetadataUniversalIdentifier: 'field-uuid-email',
|
||||
fields: [
|
||||
{
|
||||
universalIdentifier: 'field-uuid-email',
|
||||
name: 'email',
|
||||
label: 'Email',
|
||||
type: FieldMetadataType.EMAILS,
|
||||
},
|
||||
],
|
||||
}),
|
||||
});
|
||||
|
||||
expect(result?.generatedType).toBe('STORED');
|
||||
expect(result?.asExpression).toContain("to_tsvector('simple'");
|
||||
expect(result?.asExpression).toContain('"emailPrimaryEmail"');
|
||||
});
|
||||
|
||||
it('should return asExpression for a UUID label identifier field', () => {
|
||||
const result = computeSearchVectorUniversalSettingsFromObjectManifest({
|
||||
objectManifest: buildObjectManifest({
|
||||
labelIdentifierFieldMetadataUniversalIdentifier: 'field-uuid-id',
|
||||
fields: [
|
||||
{
|
||||
universalIdentifier: 'field-uuid-id',
|
||||
name: 'externalId',
|
||||
label: 'External ID',
|
||||
type: FieldMetadataType.UUID,
|
||||
},
|
||||
],
|
||||
}),
|
||||
});
|
||||
|
||||
expect(result?.generatedType).toBe('STORED');
|
||||
expect(result?.asExpression).toContain("to_tsvector('simple'");
|
||||
expect(result?.asExpression).toContain('"externalId"');
|
||||
});
|
||||
|
||||
it('should return null when label identifier field is not found in fields', () => {
|
||||
const result = computeSearchVectorUniversalSettingsFromObjectManifest({
|
||||
objectManifest: buildObjectManifest({
|
||||
labelIdentifierFieldMetadataUniversalIdentifier:
|
||||
'non-existent-field-uuid',
|
||||
fields: [
|
||||
{
|
||||
universalIdentifier: 'field-uuid-name',
|
||||
name: 'name',
|
||||
label: 'Name',
|
||||
type: FieldMetadataType.TEXT,
|
||||
},
|
||||
],
|
||||
}),
|
||||
});
|
||||
|
||||
expect(result).toBeNull();
|
||||
});
|
||||
|
||||
it('should return null when label identifier field has a non-searchable type', () => {
|
||||
const result = computeSearchVectorUniversalSettingsFromObjectManifest({
|
||||
objectManifest: buildObjectManifest({
|
||||
labelIdentifierFieldMetadataUniversalIdentifier: 'field-uuid-number',
|
||||
fields: [
|
||||
{
|
||||
universalIdentifier: 'field-uuid-number',
|
||||
name: 'amount',
|
||||
label: 'Amount',
|
||||
type: FieldMetadataType.NUMBER,
|
||||
},
|
||||
],
|
||||
}),
|
||||
});
|
||||
|
||||
expect(result).toBeNull();
|
||||
});
|
||||
|
||||
it('should return null when fields array is empty', () => {
|
||||
const result = computeSearchVectorUniversalSettingsFromObjectManifest({
|
||||
objectManifest: buildObjectManifest({
|
||||
labelIdentifierFieldMetadataUniversalIdentifier: 'field-uuid-name',
|
||||
fields: [],
|
||||
}),
|
||||
});
|
||||
|
||||
expect(result).toBeNull();
|
||||
});
|
||||
});
|
||||
-37
@@ -1,37 +0,0 @@
|
||||
import { type ObjectManifest } from 'twenty-shared/application';
|
||||
import {
|
||||
type FieldMetadataUniversalSettings,
|
||||
FieldMetadataType,
|
||||
} from 'twenty-shared/types';
|
||||
|
||||
import { getTsVectorColumnExpressionFromFields } from 'src/engine/workspace-manager/utils/get-ts-vector-column-expression.util';
|
||||
import { isDefined, isSearchableFieldType } from 'twenty-shared/utils';
|
||||
|
||||
export const computeSearchVectorUniversalSettingsFromObjectManifest = ({
|
||||
objectManifest,
|
||||
}: {
|
||||
objectManifest: ObjectManifest;
|
||||
}): FieldMetadataUniversalSettings<FieldMetadataType.TS_VECTOR> => {
|
||||
const labelIdentifierField = objectManifest.fields.find(
|
||||
(field) =>
|
||||
field.universalIdentifier ===
|
||||
objectManifest.labelIdentifierFieldMetadataUniversalIdentifier,
|
||||
);
|
||||
|
||||
if (
|
||||
!isDefined(labelIdentifierField) ||
|
||||
!isSearchableFieldType(labelIdentifierField.type)
|
||||
) {
|
||||
return null;
|
||||
}
|
||||
|
||||
return {
|
||||
asExpression: getTsVectorColumnExpressionFromFields([
|
||||
{
|
||||
name: labelIdentifierField.name,
|
||||
type: labelIdentifierField.type,
|
||||
},
|
||||
]),
|
||||
generatedType: 'STORED',
|
||||
};
|
||||
};
|
||||
Reference in New Issue
Block a user