60fd322b49
## Introduction Closes twentyhq/core-team-issues#2635 and twentyhq/core-team-issues#2642 and twentyhq/core-team-issues#2589 Object system fields (`searchVector` + its GIN index + `searchFieldMetadata`, the reserved system fields, default relations) were provisioned through several scattered, path-specific code paths. As a result the **app-manifest sync path** authored objects with an empty/`NULL` `searchVector` and **zero `searchFieldMetadata`**, so app-owned objects shipped a broken generated search column (see #22657). The generation logic also lived partly in imperative services rather than in the metadata side-effect engine, and relied on non-deterministic (`v4`) universal identifiers that `twenty apply` could not converge, destroying manually backfilled rows. This PR centralizes every object-creation system side effect into the **metadata side-effect engine**, extends the engine to keep search metadata consistent on field delete and object relabel, makes the standard app's search identifiers deterministic, and ships upgrade commands to reconcile existing workspaces. ## What changed ### Side effects moved into the metadata side-effect engine New dedicated, self-contained handlers — so every write path (API and app manifest) gets identical results, and side effects never trigger other side effects. **Object create / delete** (`handlers/object-metadata`) * **`objectSystemFieldsOnCreate`** — generates the 7 reserved system fields (`id`, `createdAt`, `updatedAt`, `deletedAt`, `createdBy`, `updatedBy`, `position`). * **`objectSearchVectorOnCreate`** — provisions the full-text search surface as one unit: the `searchVector` `TS_VECTOR` field, its backing GIN index, and the `searchFieldMetadata` row (for searchable objects whose label identifier is a searchable field) that keeps `searchVector` populated instead of `NULL`. * **`objectSystemSideEffectsOnDelete`** — tears the above down on object deletion. **Search-metadata consistency on relabel / field delete** (new — these are what close the manifest-path gaps) * **`objectSearchVectorOnUpdate`** (`handlers/object-metadata`) — when a searchable object is relabeled onto a new searchable field, provisions the `searchFieldMetadata` row that indexes it. Relabeling is **additive**: existing rows (e.g. the provisioned `name` row) are preserved, so the previous label identifier stays searchable. Mirrors the API update path so a manifest re-sync that changes the label identifier reaches search parity. No-ops for junction objects (`id` label identifier) and non-searchable field types. * **`fieldSearchFieldMetadataOnDelete`** (`handlers/field-metadata`) — when a field is deleted, cascade-deletes every `searchFieldMetadata` row that indexes it. `searchFieldMetadata` is excluded from manifest deletion inference, so this explicit cascade is what covers **both the API and manifest paths** (the object-scoped DB cascade only fires on object deletion). Uses the `searchFieldMetadataUniversalIdentifiers` aggregator on the flat field for an O(k) lookup instead of scanning all rows. The **default `name` field and default relations are now caller-provided default fields** (SDK autocomplete on the manifest path, input transpiler on the API path) rather than system side effects — removing duplicate name generation, the imperative `build-default-*-for-custom-object` utilities, and the ad-hoc system-field integrity validator. ### Deterministic identifiers for the standard app The twenty-standard search GIN index and `searchFieldMetadata` now derive deterministic universal identifiers (`getIndexUniversalIdentifier` / `getSearchFieldUniversalIdentifier`) instead of `v4`, so `twenty apply` converges instead of recreating. ### Upgrade commands (`2-20`) to reconcile existing workspaces **Instance commands** (run once per instance; ordered fast → slow → workspace): 1. **`AddIsSystemSideEffectToSearchFieldMetadata`** (fast) — adds the `isSystemSideEffect` column to `core.searchFieldMetadata`. Defaults to `true`, which also correctly backfills every existing row since `searchFieldMetadata` is always system-derived (never user-authored). 2. **`BackfillNameFieldIsSystemSideEffect`** (slow) — re-flags existing `name` fields from `isSystemSideEffect: true` → `false`, since the default `name` field is now a caller-provided default like any other user-owned field (it was provisioned as `true` in 2.15 → 2.19). This is a pure data backfill, so the bulk `UPDATE` lives in `runDataMigration()` rather than `up()` — keeping it out of the fast schema transaction avoids holding an `ACCESS EXCLUSIVE` lock that could stall reads during the deploy. Slow instance commands still run before every workspace command of the version, so the fresh value is in place before the search-reconcile workspace commands recompute the `fieldMetadata` flat-entity cache. Scoping by name alone is safe (no engine-owned field is named `name`); `down()` is best-effort (pre-2.15 `false` rows are indistinguishable from flipped ones). **Workspace commands** (idempotent, dry-run supported): 1. **`reconcile-search-vector-gin-index-universal-identifier`** — re-owns every searchVector GIN index UID to its deterministic value (all applications), then backfills the missing GIN index for installed-app objects. 2. **`reconcile-search-field-metadata`** — re-owns every `searchFieldMetadata` UID (all applications), then backfills the missing rows for installed-app searchable objects. 3. **`rebuild-installed-app-search-vectors`** — rebuilds the `searchVector` column of every installed-app `TS_VECTOR` field, once the index and rows exist. Design notes: * **Re-own is global** (twenty-standard, workspace-custom, installed) — a UID convergence keyed on each row's own application. * **Backfill is installed-app only** — standard/custom objects already have these rows via the manifest funnel. * Re-own runs **before** backfill and is transaction-guarded; a failure aborts that workspace to avoid a unique-identifier collision. ## Tests * Integration: app manifest sync now asserts system fields + searchable objects (searchVector, GIN index, searchFieldMetadata) are created; a new relabel suite drives three manifest syncs and asserts records stay searchable through the old + new label identifiers and lose searchability when a field is removed; removed the obsolete system-fields-integrity suite/snapshots. * Unit: per-handler side-effect specs (including the new `objectSearchVectorOnUpdate` and `fieldSearchFieldMetadataOnDelete` handlers), and per-util specs for the re-own / backfill operation builders and the GIN-index classifier. ## Upgrade / migration notes * Existing workspaces converge on the next upgrade run via the `2-20` instance + workspace commands (idempotent, dry-run supported). * Backfill and rebuild go through the workspace-migration runner (automatic cache invalidation); the re-own step invalidates only the affected flat-entity maps directly. * The cross-version upgrade CI now flushes the cache before running the upgrade, so the new version recomputes every flat-entity map from the database instead of reading blobs the old version serialized in an older shape. ## Follow-up * `object-metadata.service.ts` still carries a `TODO: remove once default view fields move to the metadata side effect engine` — default view fields are the next candidate to move into the engine. * A single manifest sync cannot yet both create a field and relabel the object onto it, because `objectMetadata.update` is ordered before `fieldMetadata.create` in the migration runner. Tracked in twentyhq/core-team-issues#2655; to be fixed in a follow-up.
138 lines
4.4 KiB
TypeScript
138 lines
4.4 KiB
TypeScript
import { buildBaseManifest } from 'test/integration/metadata/suites/application/utils/build-base-manifest.util';
|
|
import { cleanupApplicationAndAppRegistration } from 'test/integration/metadata/suites/application/utils/cleanup-application-and-app-registration.util';
|
|
import { setupApplicationForSync } from 'test/integration/metadata/suites/application/utils/setup-application-for-sync.util';
|
|
import { syncApplication } from 'test/integration/metadata/suites/application/utils/sync-application.util';
|
|
import { findManyFieldsMetadata } from 'test/integration/metadata/suites/field-metadata/utils/find-many-fields-metadata.util';
|
|
import { findManyObjectMetadata } from 'test/integration/metadata/suites/object-metadata/utils/find-many-object-metadata.util';
|
|
import {
|
|
getFieldUniversalIdentifier,
|
|
type ObjectManifest,
|
|
} from 'twenty-shared/application';
|
|
import { FieldMetadataType } from 'twenty-shared/types';
|
|
import { isDefined } from 'twenty-shared/utils';
|
|
import { v4 as uuidv4 } from 'uuid';
|
|
|
|
const SYSTEM_FIELD_NAMES = [
|
|
'id',
|
|
'createdAt',
|
|
'updatedAt',
|
|
'deletedAt',
|
|
'createdBy',
|
|
'updatedBy',
|
|
'position',
|
|
'searchVector',
|
|
];
|
|
|
|
const TEST_APP_UNIVERSAL_IDENTIFIER = uuidv4();
|
|
const TEST_ROLE_UNIVERSAL_IDENTIFIER = uuidv4();
|
|
const OBJECT_UNIVERSAL_IDENTIFIER = uuidv4();
|
|
const NAME_FIELD_UNIVERSAL_IDENTIFIER = uuidv4();
|
|
|
|
const OBJECT_NAME_SINGULAR = 'rocketForManifestUniversalIdentifier';
|
|
|
|
const TEST_OBJECT: ObjectManifest = {
|
|
universalIdentifier: OBJECT_UNIVERSAL_IDENTIFIER,
|
|
labelIdentifierFieldMetadataUniversalIdentifier:
|
|
NAME_FIELD_UNIVERSAL_IDENTIFIER,
|
|
nameSingular: OBJECT_NAME_SINGULAR,
|
|
namePlural: `${OBJECT_NAME_SINGULAR}s`,
|
|
labelSingular: 'Rocket For Manifest Universal Identifier',
|
|
labelPlural: 'Rockets For Manifest Universal Identifier',
|
|
description: 'A rocket synced through the manifest funnel',
|
|
icon: 'IconRocket',
|
|
fields: [
|
|
{
|
|
universalIdentifier: NAME_FIELD_UNIVERSAL_IDENTIFIER,
|
|
type: FieldMetadataType.TEXT,
|
|
name: 'name',
|
|
label: 'Name',
|
|
},
|
|
],
|
|
};
|
|
|
|
type FetchedField = {
|
|
id: string;
|
|
name: string;
|
|
universalIdentifier: string;
|
|
};
|
|
|
|
describe('Application manifest sync deterministic system field universal identifiers', () => {
|
|
let objectUniversalIdentifier: string;
|
|
let fetchedFields: FetchedField[];
|
|
|
|
beforeAll(async () => {
|
|
await setupApplicationForSync({
|
|
applicationUniversalIdentifier: TEST_APP_UNIVERSAL_IDENTIFIER,
|
|
name: 'Test Application',
|
|
description: 'A test application',
|
|
sourcePath: 'test-sync-deterministic',
|
|
});
|
|
|
|
await syncApplication({
|
|
expectToFail: false,
|
|
manifest: buildBaseManifest({
|
|
appId: TEST_APP_UNIVERSAL_IDENTIFIER,
|
|
roleId: TEST_ROLE_UNIVERSAL_IDENTIFIER,
|
|
overrides: { objects: [TEST_OBJECT] },
|
|
}),
|
|
});
|
|
|
|
const { objects } = await findManyObjectMetadata({
|
|
expectToFail: false,
|
|
input: { filter: {}, paging: { first: 100 } },
|
|
gqlFields: 'id nameSingular universalIdentifier',
|
|
});
|
|
|
|
const syncedObject = objects.find(
|
|
(object) => object.universalIdentifier === OBJECT_UNIVERSAL_IDENTIFIER,
|
|
);
|
|
|
|
if (!isDefined(syncedObject)) {
|
|
throw new Error(
|
|
'Could not resolve the object synced through the manifest funnel',
|
|
);
|
|
}
|
|
|
|
objectUniversalIdentifier = syncedObject.universalIdentifier;
|
|
|
|
const { fields } = await findManyFieldsMetadata({
|
|
expectToFail: false,
|
|
input: {
|
|
filter: { objectMetadataId: { eq: syncedObject.id } },
|
|
paging: { first: 100 },
|
|
},
|
|
gqlFields: `
|
|
id
|
|
name
|
|
universalIdentifier
|
|
`,
|
|
});
|
|
|
|
fetchedFields = fields.map((edge: { node: FetchedField }) => edge.node);
|
|
}, 60000);
|
|
|
|
afterAll(async () => {
|
|
await cleanupApplicationAndAppRegistration({
|
|
applicationUniversalIdentifier: TEST_APP_UNIVERSAL_IDENTIFIER,
|
|
});
|
|
});
|
|
|
|
it.each(SYSTEM_FIELD_NAMES)(
|
|
'should derive the %s system field universal identifier deterministically through the manifest sync funnel',
|
|
(systemFieldName) => {
|
|
const systemField = fetchedFields.find(
|
|
(field) => field.name === systemFieldName,
|
|
);
|
|
|
|
expect(systemField).toBeDefined();
|
|
expect(systemField?.universalIdentifier).toBe(
|
|
getFieldUniversalIdentifier({
|
|
applicationUniversalIdentifier: TEST_APP_UNIVERSAL_IDENTIFIER,
|
|
objectUniversalIdentifier,
|
|
name: systemFieldName,
|
|
}),
|
|
);
|
|
},
|
|
);
|
|
});
|