Centralize system field side effects + search field metadata (#22594)
## Introduction Closes twentyhq/core-team-issues#2635 and twentyhq/core-team-issues#2642 and twentyhq/core-team-issues#2589 Object system fields (`searchVector` + its GIN index + `searchFieldMetadata`, the reserved system fields, default relations) were provisioned through several scattered, path-specific code paths. As a result the **app-manifest sync path** authored objects with an empty/`NULL` `searchVector` and **zero `searchFieldMetadata`**, so app-owned objects shipped a broken generated search column (see #22657). The generation logic also lived partly in imperative services rather than in the metadata side-effect engine, and relied on non-deterministic (`v4`) universal identifiers that `twenty apply` could not converge, destroying manually backfilled rows. This PR centralizes every object-creation system side effect into the **metadata side-effect engine**, extends the engine to keep search metadata consistent on field delete and object relabel, makes the standard app's search identifiers deterministic, and ships upgrade commands to reconcile existing workspaces. ## What changed ### Side effects moved into the metadata side-effect engine New dedicated, self-contained handlers — so every write path (API and app manifest) gets identical results, and side effects never trigger other side effects. **Object create / delete** (`handlers/object-metadata`) * **`objectSystemFieldsOnCreate`** — generates the 7 reserved system fields (`id`, `createdAt`, `updatedAt`, `deletedAt`, `createdBy`, `updatedBy`, `position`). * **`objectSearchVectorOnCreate`** — provisions the full-text search surface as one unit: the `searchVector` `TS_VECTOR` field, its backing GIN index, and the `searchFieldMetadata` row (for searchable objects whose label identifier is a searchable field) that keeps `searchVector` populated instead of `NULL`. * **`objectSystemSideEffectsOnDelete`** — tears the above down on object deletion. **Search-metadata consistency on relabel / field delete** (new — these are what close the manifest-path gaps) * **`objectSearchVectorOnUpdate`** (`handlers/object-metadata`) — when a searchable object is relabeled onto a new searchable field, provisions the `searchFieldMetadata` row that indexes it. Relabeling is **additive**: existing rows (e.g. the provisioned `name` row) are preserved, so the previous label identifier stays searchable. Mirrors the API update path so a manifest re-sync that changes the label identifier reaches search parity. No-ops for junction objects (`id` label identifier) and non-searchable field types. * **`fieldSearchFieldMetadataOnDelete`** (`handlers/field-metadata`) — when a field is deleted, cascade-deletes every `searchFieldMetadata` row that indexes it. `searchFieldMetadata` is excluded from manifest deletion inference, so this explicit cascade is what covers **both the API and manifest paths** (the object-scoped DB cascade only fires on object deletion). Uses the `searchFieldMetadataUniversalIdentifiers` aggregator on the flat field for an O(k) lookup instead of scanning all rows. The **default `name` field and default relations are now caller-provided default fields** (SDK autocomplete on the manifest path, input transpiler on the API path) rather than system side effects — removing duplicate name generation, the imperative `build-default-*-for-custom-object` utilities, and the ad-hoc system-field integrity validator. ### Deterministic identifiers for the standard app The twenty-standard search GIN index and `searchFieldMetadata` now derive deterministic universal identifiers (`getIndexUniversalIdentifier` / `getSearchFieldUniversalIdentifier`) instead of `v4`, so `twenty apply` converges instead of recreating. ### Upgrade commands (`2-20`) to reconcile existing workspaces **Instance commands** (run once per instance; ordered fast → slow → workspace): 1. **`AddIsSystemSideEffectToSearchFieldMetadata`** (fast) — adds the `isSystemSideEffect` column to `core.searchFieldMetadata`. Defaults to `true`, which also correctly backfills every existing row since `searchFieldMetadata` is always system-derived (never user-authored). 2. **`BackfillNameFieldIsSystemSideEffect`** (slow) — re-flags existing `name` fields from `isSystemSideEffect: true` → `false`, since the default `name` field is now a caller-provided default like any other user-owned field (it was provisioned as `true` in 2.15 → 2.19). This is a pure data backfill, so the bulk `UPDATE` lives in `runDataMigration()` rather than `up()` — keeping it out of the fast schema transaction avoids holding an `ACCESS EXCLUSIVE` lock that could stall reads during the deploy. Slow instance commands still run before every workspace command of the version, so the fresh value is in place before the search-reconcile workspace commands recompute the `fieldMetadata` flat-entity cache. Scoping by name alone is safe (no engine-owned field is named `name`); `down()` is best-effort (pre-2.15 `false` rows are indistinguishable from flipped ones). **Workspace commands** (idempotent, dry-run supported): 1. **`reconcile-search-vector-gin-index-universal-identifier`** — re-owns every searchVector GIN index UID to its deterministic value (all applications), then backfills the missing GIN index for installed-app objects. 2. **`reconcile-search-field-metadata`** — re-owns every `searchFieldMetadata` UID (all applications), then backfills the missing rows for installed-app searchable objects. 3. **`rebuild-installed-app-search-vectors`** — rebuilds the `searchVector` column of every installed-app `TS_VECTOR` field, once the index and rows exist. Design notes: * **Re-own is global** (twenty-standard, workspace-custom, installed) — a UID convergence keyed on each row's own application. * **Backfill is installed-app only** — standard/custom objects already have these rows via the manifest funnel. * Re-own runs **before** backfill and is transaction-guarded; a failure aborts that workspace to avoid a unique-identifier collision. ## Tests * Integration: app manifest sync now asserts system fields + searchable objects (searchVector, GIN index, searchFieldMetadata) are created; a new relabel suite drives three manifest syncs and asserts records stay searchable through the old + new label identifiers and lose searchability when a field is removed; removed the obsolete system-fields-integrity suite/snapshots. * Unit: per-handler side-effect specs (including the new `objectSearchVectorOnUpdate` and `fieldSearchFieldMetadataOnDelete` handlers), and per-util specs for the re-own / backfill operation builders and the GIN-index classifier. ## Upgrade / migration notes * Existing workspaces converge on the next upgrade run via the `2-20` instance + workspace commands (idempotent, dry-run supported). * Backfill and rebuild go through the workspace-migration runner (automatic cache invalidation); the re-own step invalidates only the affected flat-entity maps directly. * The cross-version upgrade CI now flushes the cache before running the upgrade, so the new version recomputes every flat-entity map from the database instead of reading blobs the old version serialized in an older shape. ## Follow-up * `object-metadata.service.ts` still carries a `TODO: remove once default view fields move to the metadata side effect engine` — default view fields are the next candidate to move into the engine. * A single manifest sync cannot yet both create a field and relabel the object onto it, because `objectMetadata.update` is ordered before `fieldMetadata.create` in the migration runner. Tracked in twentyhq/core-team-issues#2655; to be fixed in a follow-up.
This commit is contained in:
Paul Rastoin
+25
-39
@@ -1,5 +1,7 @@
|
||||
import { getFieldUniversalIdentifier } from 'twenty-shared/application';
|
||||
import {
|
||||
capitalize,
|
||||
isDefined,
|
||||
trimAndRemoveDuplicatedWhitespacesFromObjectStringProperties,
|
||||
} from 'twenty-shared/utils';
|
||||
import { v4 } from 'uuid';
|
||||
@@ -7,20 +9,18 @@ import { v4 } from 'uuid';
|
||||
import { type FlatApplication } from 'src/engine/core-modules/application/types/flat-application.type';
|
||||
import { type AllFlatEntityMaps } from 'src/engine/metadata-modules/flat-entity/types/all-flat-entity-maps.type';
|
||||
import { type CreateObjectInput } from 'src/engine/metadata-modules/object-metadata/dtos/create-object.input';
|
||||
import { buildDefaultFlatFieldMetadatasForCustomObject } from 'src/engine/metadata-modules/object-metadata/utils/build-default-flat-field-metadatas-for-custom-object.util';
|
||||
import { buildDefaultIndexesForCustomObject } from 'src/engine/metadata-modules/object-metadata/utils/build-default-index-for-custom-object.util';
|
||||
import { buildNameFlatFieldMetadataForCustomObject } from 'src/engine/metadata-modules/object-metadata/utils/build-name-flat-field-metadata-for-custom-object.util';
|
||||
import { buildDefaultRelationFlatFieldMetadatasForCustomObject } from 'src/engine/metadata-modules/object-metadata/utils/build-default-relation-flat-field-metadatas-for-custom-object.util';
|
||||
import { buildDefaultSearchFieldMetadatasForCustomObject } from 'src/engine/metadata-modules/object-metadata/utils/build-default-search-field-metadatas-for-custom-object.util';
|
||||
import { type UniversalFlatFieldMetadata } from 'src/engine/workspace-manager/workspace-migration/universal-flat-entity/types/universal-flat-field-metadata.type';
|
||||
import { type UniversalFlatIndexMetadata } from 'src/engine/workspace-manager/workspace-migration/universal-flat-entity/types/universal-flat-index-metadata.type';
|
||||
import { type UniversalFlatObjectMetadata } from 'src/engine/workspace-manager/workspace-migration/universal-flat-entity/types/universal-flat-object-metadata.type';
|
||||
import { type UniversalFlatSearchFieldMetadata } from 'src/engine/workspace-manager/workspace-migration/universal-flat-entity/types/universal-flat-search-field-metadata.type';
|
||||
|
||||
type FromCreateObjectInputToFlatObjectMetadataAndFlatFieldMetadatasToCreateArgs =
|
||||
{
|
||||
createObjectInput: CreateObjectInput;
|
||||
flatApplication: FlatApplication;
|
||||
} & Pick<AllFlatEntityMaps, 'flatObjectMetadataMaps'>;
|
||||
|
||||
export const fromCreateObjectInputToFlatObjectMetadataAndFlatFieldMetadatasToCreate =
|
||||
({
|
||||
createObjectInput: rawCreateObjectInput,
|
||||
@@ -28,10 +28,9 @@ export const fromCreateObjectInputToFlatObjectMetadataAndFlatFieldMetadatasToCre
|
||||
flatObjectMetadataMaps: existingFlatObjectMetadataMaps,
|
||||
}: FromCreateObjectInputToFlatObjectMetadataAndFlatFieldMetadatasToCreateArgs): {
|
||||
flatObjectMetadataToCreate: UniversalFlatObjectMetadata & { id: string };
|
||||
relationTargetFlatFieldMetadataToCreate: UniversalFlatFieldMetadata[];
|
||||
flatFieldMetadataToCreateOnObject: UniversalFlatFieldMetadata[];
|
||||
relationTargetFlatFieldMetadataToCreate: UniversalFlatFieldMetadata[];
|
||||
flatIndexMetadataToCreate: UniversalFlatIndexMetadata[];
|
||||
flatSearchFieldMetadataToCreate: UniversalFlatSearchFieldMetadata[];
|
||||
} => {
|
||||
const createObjectInput =
|
||||
trimAndRemoveDuplicatedWhitespacesFromObjectStringProperties(
|
||||
@@ -49,21 +48,14 @@ export const fromCreateObjectInputToFlatObjectMetadataAndFlatFieldMetadatasToCre
|
||||
|
||||
const objectMetadataId = v4();
|
||||
const universalIdentifier = createObjectInput.universalIdentifier ?? v4();
|
||||
const defaultFlatFieldForCustomObjectMaps =
|
||||
buildDefaultFlatFieldMetadatasForCustomObject({
|
||||
flatObjectMetadata: {
|
||||
applicationUniversalIdentifier: flatApplication.universalIdentifier,
|
||||
universalIdentifier,
|
||||
},
|
||||
skipNameField: createObjectInput.skipNameField,
|
||||
});
|
||||
const createdAt = new Date().toISOString();
|
||||
|
||||
// Use nameField.id if it exists, otherwise use idField.id (for junction tables without name)
|
||||
const nameField = defaultFlatFieldForCustomObjectMaps.fields.nameField;
|
||||
const labelIdentifierFieldMetadataUniversalIdentifier =
|
||||
nameField?.universalIdentifier ??
|
||||
defaultFlatFieldForCustomObjectMaps.fields.id.universalIdentifier;
|
||||
getFieldUniversalIdentifier({
|
||||
applicationUniversalIdentifier: flatApplication.universalIdentifier,
|
||||
objectUniversalIdentifier: universalIdentifier,
|
||||
name: createObjectInput.skipNameField ? 'id' : 'name',
|
||||
});
|
||||
|
||||
const universalFlatObjectMetadataToCreate: UniversalFlatObjectMetadata & {
|
||||
id: string;
|
||||
@@ -102,6 +94,17 @@ export const fromCreateObjectInputToFlatObjectMetadataAndFlatFieldMetadatasToCre
|
||||
imageIdentifierFieldMetadataUniversalIdentifier: null,
|
||||
};
|
||||
|
||||
const nameFlatFieldMetadata =
|
||||
createObjectInput.skipNameField === true
|
||||
? null
|
||||
: buildNameFlatFieldMetadataForCustomObject({
|
||||
flatObjectMetadata: {
|
||||
applicationUniversalIdentifier:
|
||||
flatApplication.universalIdentifier,
|
||||
universalIdentifier,
|
||||
},
|
||||
});
|
||||
|
||||
const {
|
||||
standardSourceFlatFieldMetadatas,
|
||||
standardTargetFlatFieldMetadatas,
|
||||
@@ -112,32 +115,15 @@ export const fromCreateObjectInputToFlatObjectMetadataAndFlatFieldMetadatasToCre
|
||||
flatApplication,
|
||||
});
|
||||
|
||||
const objectFlatFieldMetadatas: UniversalFlatFieldMetadata[] = [
|
||||
...Object.values(defaultFlatFieldForCustomObjectMaps.fields),
|
||||
const flatFieldMetadataToCreateOnObject: UniversalFlatFieldMetadata[] = [
|
||||
...(isDefined(nameFlatFieldMetadata) ? [nameFlatFieldMetadata] : []),
|
||||
...standardSourceFlatFieldMetadatas,
|
||||
];
|
||||
|
||||
const defaultIndexesForCustomObject = buildDefaultIndexesForCustomObject({
|
||||
objectFlatFieldMetadatas,
|
||||
defaultFlatFieldForCustomObjectMaps,
|
||||
flatObjectMetadata: universalFlatObjectMetadataToCreate,
|
||||
});
|
||||
|
||||
const defaultSearchFieldMetadatasForCustomObject =
|
||||
buildDefaultSearchFieldMetadatasForCustomObject({
|
||||
defaultFlatFieldForCustomObjectMaps,
|
||||
flatObjectMetadata: universalFlatObjectMetadataToCreate,
|
||||
});
|
||||
|
||||
return {
|
||||
flatObjectMetadataToCreate: universalFlatObjectMetadataToCreate,
|
||||
flatIndexMetadataToCreate: [
|
||||
...Object.values(defaultIndexesForCustomObject.indexes),
|
||||
...standardTargetFlatIndexMetadatas,
|
||||
],
|
||||
flatSearchFieldMetadataToCreate:
|
||||
defaultSearchFieldMetadatasForCustomObject.searchFieldMetadatas,
|
||||
flatFieldMetadataToCreateOnObject,
|
||||
relationTargetFlatFieldMetadataToCreate: standardTargetFlatFieldMetadatas,
|
||||
flatFieldMetadataToCreateOnObject: objectFlatFieldMetadatas,
|
||||
flatIndexMetadataToCreate: standardTargetFlatIndexMetadatas,
|
||||
};
|
||||
};
|
||||
|
||||
-9
@@ -3,7 +3,6 @@ import { isDefined } from 'twenty-shared/utils';
|
||||
import { findFlatEntityByUniversalIdentifierOrThrow } from 'src/engine/metadata-modules/flat-entity/utils/find-flat-entity-by-universal-identifier-or-throw.util';
|
||||
import { findManyFlatEntityByUniversalIdentifierInUniversalFlatEntityMapsOrThrow } from 'src/engine/metadata-modules/flat-entity/utils/find-many-flat-entity-by-universal-identifier-in-universal-flat-entity-maps-or-throw.util';
|
||||
import { validateFlatObjectMetadataIdentifiers } from 'src/engine/metadata-modules/flat-object-metadata/validators/utils/validate-flat-object-metadata-identifiers.util';
|
||||
import { validateObjectMetadataSystemFieldsIntegrity } from 'src/engine/metadata-modules/flat-object-metadata/validators/utils/validate-object-metadata-system-fields-integrity.util';
|
||||
import {
|
||||
type OrchestratorActionsReport,
|
||||
type OrchestratorFailureReport,
|
||||
@@ -57,14 +56,6 @@ export const validateObjectMetadataCrossEntity = ({
|
||||
},
|
||||
);
|
||||
|
||||
createFailedFlatEntityValidations.errors.push(
|
||||
...validateObjectMetadataSystemFieldsIntegrity({
|
||||
universalFlatFieldMetadataMaps:
|
||||
optimisticUniversalFlatMaps.flatFieldMetadataMaps,
|
||||
universalFlatObjectMetadata,
|
||||
}),
|
||||
);
|
||||
|
||||
createFailedFlatEntityValidations.errors.push(
|
||||
...validateFlatObjectMetadataIdentifiers({
|
||||
universalFlatObjectMetadata,
|
||||
|
||||
-98
@@ -1,98 +0,0 @@
|
||||
import { msg } from '@lingui/core/macro';
|
||||
import { getFieldUniversalIdentifier } from 'twenty-shared/application';
|
||||
import { isDefined } from 'twenty-shared/utils';
|
||||
|
||||
import { findFlatEntityByUniversalIdentifierOrThrow } from 'src/engine/metadata-modules/flat-entity/utils/find-flat-entity-by-universal-identifier-or-throw.util';
|
||||
import { PARTIAL_SYSTEM_FLAT_FIELD_METADATAS } from 'src/engine/metadata-modules/object-metadata/constants/partial-system-flat-field-metadatas.constant';
|
||||
import { ObjectMetadataExceptionCode } from 'src/engine/metadata-modules/object-metadata/object-metadata.exception';
|
||||
import { type AllUniversalFlatEntityMaps } from 'src/engine/workspace-manager/workspace-migration/universal-flat-entity/types/all-universal-flat-entity-maps.type';
|
||||
import { type UniversalFlatFieldMetadata } from 'src/engine/workspace-manager/workspace-migration/universal-flat-entity/types/universal-flat-field-metadata.type';
|
||||
import { type UniversalFlatObjectMetadata } from 'src/engine/workspace-manager/workspace-migration/universal-flat-entity/types/universal-flat-object-metadata.type';
|
||||
import { type FlatEntityValidationError } from 'src/engine/workspace-manager/workspace-migration/workspace-migration-builder/builders/types/failed-flat-entity-validation.type';
|
||||
import { buildUniversalFlatObjectFieldByNameAndJoinColumnMaps } from 'src/engine/workspace-manager/workspace-migration/workspace-migration-builder/utils/build-universal-flat-object-field-by-name-and-join-column-maps.util';
|
||||
|
||||
type ValidateObjectMetadataSystemFieldsIntegrityArgs = {
|
||||
universalFlatObjectMetadata: UniversalFlatObjectMetadata;
|
||||
universalFlatFieldMetadataMaps: AllUniversalFlatEntityMaps['flatFieldMetadataMaps'];
|
||||
};
|
||||
export const validateObjectMetadataSystemFieldsIntegrity = ({
|
||||
universalFlatFieldMetadataMaps,
|
||||
universalFlatObjectMetadata,
|
||||
}: ValidateObjectMetadataSystemFieldsIntegrityArgs): FlatEntityValidationError[] => {
|
||||
const errors: FlatEntityValidationError[] = [];
|
||||
|
||||
const { fieldUniversalIdentifierByName } =
|
||||
buildUniversalFlatObjectFieldByNameAndJoinColumnMaps({
|
||||
flatFieldMetadataMaps: universalFlatFieldMetadataMaps,
|
||||
flatObjectMetadata: universalFlatObjectMetadata,
|
||||
});
|
||||
|
||||
for (const expectedSystemField of Object.values(
|
||||
PARTIAL_SYSTEM_FLAT_FIELD_METADATAS,
|
||||
)) {
|
||||
const matchingFieldUniversalIdentifier =
|
||||
fieldUniversalIdentifierByName[expectedSystemField.name];
|
||||
|
||||
const expectedFieldName = expectedSystemField.name;
|
||||
|
||||
if (!isDefined(matchingFieldUniversalIdentifier)) {
|
||||
errors.push({
|
||||
code: ObjectMetadataExceptionCode.MISSING_SYSTEM_FIELD,
|
||||
message: `System field ${expectedFieldName} is missing`,
|
||||
userFriendlyMessage: msg`System field ${expectedFieldName} is missing`,
|
||||
value: expectedFieldName,
|
||||
});
|
||||
} else {
|
||||
const universalFlatFieldMetadata =
|
||||
findFlatEntityByUniversalIdentifierOrThrow({
|
||||
flatEntityMaps: universalFlatFieldMetadataMaps,
|
||||
universalIdentifier: matchingFieldUniversalIdentifier,
|
||||
});
|
||||
|
||||
const propertiesToValidate = [
|
||||
'type',
|
||||
'isSystem',
|
||||
] as const satisfies (keyof UniversalFlatFieldMetadata)[];
|
||||
|
||||
for (const property of propertiesToValidate) {
|
||||
const expectedValue = expectedSystemField[property];
|
||||
const actualValue = universalFlatFieldMetadata[property];
|
||||
|
||||
if (actualValue !== expectedValue) {
|
||||
errors.push({
|
||||
code: ObjectMetadataExceptionCode.INVALID_SYSTEM_FIELD,
|
||||
message: `System field ${expectedFieldName} has invalid ${property}: expected ${String(expectedValue)}, got ${String(actualValue)}`,
|
||||
userFriendlyMessage: msg`System field ${expectedFieldName} has invalid ${property}`,
|
||||
value: actualValue,
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
// System field universal identifiers are server-owned and must match
|
||||
// the deterministic derivation; clients cannot provide custom ones.
|
||||
// TODO: remove once system fields are generated server side only by
|
||||
// the metadata side effect engine and stripped from client inputs.
|
||||
const expectedUniversalIdentifier = getFieldUniversalIdentifier({
|
||||
applicationUniversalIdentifier:
|
||||
universalFlatObjectMetadata.applicationUniversalIdentifier,
|
||||
objectUniversalIdentifier:
|
||||
universalFlatObjectMetadata.universalIdentifier,
|
||||
name: expectedFieldName,
|
||||
});
|
||||
|
||||
if (
|
||||
universalFlatFieldMetadata.universalIdentifier !==
|
||||
expectedUniversalIdentifier
|
||||
) {
|
||||
errors.push({
|
||||
code: ObjectMetadataExceptionCode.INVALID_SYSTEM_FIELD,
|
||||
message: `System field ${expectedFieldName} has invalid universalIdentifier: expected ${expectedUniversalIdentifier}, got ${universalFlatFieldMetadata.universalIdentifier}`,
|
||||
userFriendlyMessage: msg`System field ${expectedFieldName} universal identifier is not deterministic; it is derived by the server and cannot be customized`,
|
||||
value: universalFlatFieldMetadata.universalIdentifier,
|
||||
});
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
return errors;
|
||||
};
|
||||
Reference in New Issue
Block a user