Files
twenty/packages/twenty-server/test/integration/metadata/suites/application/successful-sync-application-deterministic-field-universal-identifiers.integration-spec.ts
T
Paul Rastoin 60fd322b49 Centralize system field side effects + search field metadata (#22594)
## Introduction

Closes twentyhq/core-team-issues#2635 and twentyhq/core-team-issues#2642
and twentyhq/core-team-issues#2589

Object system fields (`searchVector` + its GIN index +
`searchFieldMetadata`, the reserved system fields, default relations)
were provisioned through several scattered, path-specific code paths. As
a result the **app-manifest sync path** authored objects with an
empty/`NULL` `searchVector` and **zero `searchFieldMetadata`**, so
app-owned objects shipped a broken generated search column (see #22657).
The generation logic also lived partly in imperative services rather
than in the metadata side-effect engine, and relied on non-deterministic
(`v4`) universal identifiers that `twenty apply` could not converge,
destroying manually backfilled rows.

This PR centralizes every object-creation system side effect into the
**metadata side-effect engine**, extends the engine to keep search
metadata consistent on field delete and object relabel, makes the
standard app's search identifiers deterministic, and ships upgrade
commands to reconcile existing workspaces.

## What changed

### Side effects moved into the metadata side-effect engine

New dedicated, self-contained handlers — so every write path (API and
app manifest) gets identical results, and side effects never trigger
other side effects.

**Object create / delete** (`handlers/object-metadata`)

* **`objectSystemFieldsOnCreate`** — generates the 7 reserved system
fields (`id`, `createdAt`, `updatedAt`, `deletedAt`, `createdBy`,
`updatedBy`, `position`).
* **`objectSearchVectorOnCreate`** — provisions the full-text search
surface as one unit: the `searchVector` `TS_VECTOR` field, its backing
GIN index, and the `searchFieldMetadata` row (for searchable objects
whose label identifier is a searchable field) that keeps `searchVector`
populated instead of `NULL`.
* **`objectSystemSideEffectsOnDelete`** — tears the above down on object
deletion.

**Search-metadata consistency on relabel / field delete** (new — these
are what close the manifest-path gaps)

* **`objectSearchVectorOnUpdate`** (`handlers/object-metadata`) — when a
searchable object is relabeled onto a new searchable field, provisions
the `searchFieldMetadata` row that indexes it. Relabeling is
**additive**: existing rows (e.g. the provisioned `name` row) are
preserved, so the previous label identifier stays searchable. Mirrors
the API update path so a manifest re-sync that changes the label
identifier reaches search parity. No-ops for junction objects (`id`
label identifier) and non-searchable field types.
* **`fieldSearchFieldMetadataOnDelete`** (`handlers/field-metadata`) —
when a field is deleted, cascade-deletes every `searchFieldMetadata` row
that indexes it. `searchFieldMetadata` is excluded from manifest
deletion inference, so this explicit cascade is what covers **both the
API and manifest paths** (the object-scoped DB cascade only fires on
object deletion). Uses the `searchFieldMetadataUniversalIdentifiers`
aggregator on the flat field for an O(k) lookup instead of scanning all
rows.

The **default `name` field and default relations are now caller-provided
default fields** (SDK autocomplete on the manifest path, input
transpiler on the API path) rather than system side effects — removing
duplicate name generation, the imperative
`build-default-*-for-custom-object` utilities, and the ad-hoc
system-field integrity validator.

### Deterministic identifiers for the standard app

The twenty-standard search GIN index and `searchFieldMetadata` now
derive deterministic universal identifiers
(`getIndexUniversalIdentifier` / `getSearchFieldUniversalIdentifier`)
instead of `v4`, so `twenty apply` converges instead of recreating.

### Upgrade commands (`2-20`) to reconcile existing workspaces

**Instance commands** (run once per instance; ordered fast → slow →
workspace):

1. **`AddIsSystemSideEffectToSearchFieldMetadata`** (fast) — adds the
`isSystemSideEffect` column to `core.searchFieldMetadata`. Defaults to
`true`, which also correctly backfills every existing row since
`searchFieldMetadata` is always system-derived (never user-authored).
2. **`BackfillNameFieldIsSystemSideEffect`** (slow) — re-flags existing
`name` fields from `isSystemSideEffect: true` → `false`, since the
default `name` field is now a caller-provided default like any other
user-owned field (it was provisioned as `true` in 2.15 → 2.19). This is
a pure data backfill, so the bulk `UPDATE` lives in `runDataMigration()`
rather than `up()` — keeping it out of the fast schema transaction
avoids holding an `ACCESS EXCLUSIVE` lock that could stall reads during
the deploy. Slow instance commands still run before every workspace
command of the version, so the fresh value is in place before the
search-reconcile workspace commands recompute the `fieldMetadata`
flat-entity cache. Scoping by name alone is safe (no engine-owned field
is named `name`); `down()` is best-effort (pre-2.15 `false` rows are
indistinguishable from flipped ones).

**Workspace commands** (idempotent, dry-run supported):

1. **`reconcile-search-vector-gin-index-universal-identifier`** —
re-owns every searchVector GIN index UID to its deterministic value (all
applications), then backfills the missing GIN index for installed-app
objects.
2. **`reconcile-search-field-metadata`** — re-owns every
`searchFieldMetadata` UID (all applications), then backfills the missing
rows for installed-app searchable objects.
3. **`rebuild-installed-app-search-vectors`** — rebuilds the
`searchVector` column of every installed-app `TS_VECTOR` field, once the
index and rows exist.

Design notes:

* **Re-own is global** (twenty-standard, workspace-custom, installed) —
a UID convergence keyed on each row's own application.
* **Backfill is installed-app only** — standard/custom objects already
have these rows via the manifest funnel.
* Re-own runs **before** backfill and is transaction-guarded; a failure
aborts that workspace to avoid a unique-identifier collision.

## Tests

* Integration: app manifest sync now asserts system fields + searchable
objects (searchVector, GIN index, searchFieldMetadata) are created; a
new relabel suite drives three manifest syncs and asserts records stay
searchable through the old + new label identifiers and lose
searchability when a field is removed; removed the obsolete
system-fields-integrity suite/snapshots.
* Unit: per-handler side-effect specs (including the new
`objectSearchVectorOnUpdate` and `fieldSearchFieldMetadataOnDelete`
handlers), and per-util specs for the re-own / backfill operation
builders and the GIN-index classifier.

## Upgrade / migration notes

* Existing workspaces converge on the next upgrade run via the `2-20`
instance + workspace commands (idempotent, dry-run supported).
* Backfill and rebuild go through the workspace-migration runner
(automatic cache invalidation); the re-own step invalidates only the
affected flat-entity maps directly.
* The cross-version upgrade CI now flushes the cache before running the
upgrade, so the new version recomputes every flat-entity map from the
database instead of reading blobs the old version serialized in an older
shape.

## Follow-up

* `object-metadata.service.ts` still carries a `TODO: remove once
default view fields move to the metadata side effect engine` — default
view fields are the next candidate to move into the engine.
* A single manifest sync cannot yet both create a field and relabel the
object onto it, because `objectMetadata.update` is ordered before
`fieldMetadata.create` in the migration runner. Tracked in
twentyhq/core-team-issues#2655; to be fixed in a follow-up.
2026-07-09 16:59:54 +02:00

138 lines
4.4 KiB
TypeScript

import { buildBaseManifest } from 'test/integration/metadata/suites/application/utils/build-base-manifest.util';
import { cleanupApplicationAndAppRegistration } from 'test/integration/metadata/suites/application/utils/cleanup-application-and-app-registration.util';
import { setupApplicationForSync } from 'test/integration/metadata/suites/application/utils/setup-application-for-sync.util';
import { syncApplication } from 'test/integration/metadata/suites/application/utils/sync-application.util';
import { findManyFieldsMetadata } from 'test/integration/metadata/suites/field-metadata/utils/find-many-fields-metadata.util';
import { findManyObjectMetadata } from 'test/integration/metadata/suites/object-metadata/utils/find-many-object-metadata.util';
import {
getFieldUniversalIdentifier,
type ObjectManifest,
} from 'twenty-shared/application';
import { FieldMetadataType } from 'twenty-shared/types';
import { isDefined } from 'twenty-shared/utils';
import { v4 as uuidv4 } from 'uuid';
const SYSTEM_FIELD_NAMES = [
'id',
'createdAt',
'updatedAt',
'deletedAt',
'createdBy',
'updatedBy',
'position',
'searchVector',
];
const TEST_APP_UNIVERSAL_IDENTIFIER = uuidv4();
const TEST_ROLE_UNIVERSAL_IDENTIFIER = uuidv4();
const OBJECT_UNIVERSAL_IDENTIFIER = uuidv4();
const NAME_FIELD_UNIVERSAL_IDENTIFIER = uuidv4();
const OBJECT_NAME_SINGULAR = 'rocketForManifestUniversalIdentifier';
const TEST_OBJECT: ObjectManifest = {
universalIdentifier: OBJECT_UNIVERSAL_IDENTIFIER,
labelIdentifierFieldMetadataUniversalIdentifier:
NAME_FIELD_UNIVERSAL_IDENTIFIER,
nameSingular: OBJECT_NAME_SINGULAR,
namePlural: `${OBJECT_NAME_SINGULAR}s`,
labelSingular: 'Rocket For Manifest Universal Identifier',
labelPlural: 'Rockets For Manifest Universal Identifier',
description: 'A rocket synced through the manifest funnel',
icon: 'IconRocket',
fields: [
{
universalIdentifier: NAME_FIELD_UNIVERSAL_IDENTIFIER,
type: FieldMetadataType.TEXT,
name: 'name',
label: 'Name',
},
],
};
type FetchedField = {
id: string;
name: string;
universalIdentifier: string;
};
describe('Application manifest sync deterministic system field universal identifiers', () => {
let objectUniversalIdentifier: string;
let fetchedFields: FetchedField[];
beforeAll(async () => {
await setupApplicationForSync({
applicationUniversalIdentifier: TEST_APP_UNIVERSAL_IDENTIFIER,
name: 'Test Application',
description: 'A test application',
sourcePath: 'test-sync-deterministic',
});
await syncApplication({
expectToFail: false,
manifest: buildBaseManifest({
appId: TEST_APP_UNIVERSAL_IDENTIFIER,
roleId: TEST_ROLE_UNIVERSAL_IDENTIFIER,
overrides: { objects: [TEST_OBJECT] },
}),
});
const { objects } = await findManyObjectMetadata({
expectToFail: false,
input: { filter: {}, paging: { first: 100 } },
gqlFields: 'id nameSingular universalIdentifier',
});
const syncedObject = objects.find(
(object) => object.universalIdentifier === OBJECT_UNIVERSAL_IDENTIFIER,
);
if (!isDefined(syncedObject)) {
throw new Error(
'Could not resolve the object synced through the manifest funnel',
);
}
objectUniversalIdentifier = syncedObject.universalIdentifier;
const { fields } = await findManyFieldsMetadata({
expectToFail: false,
input: {
filter: { objectMetadataId: { eq: syncedObject.id } },
paging: { first: 100 },
},
gqlFields: `
id
name
universalIdentifier
`,
});
fetchedFields = fields.map((edge: { node: FetchedField }) => edge.node);
}, 60000);
afterAll(async () => {
await cleanupApplicationAndAppRegistration({
applicationUniversalIdentifier: TEST_APP_UNIVERSAL_IDENTIFIER,
});
});
it.each(SYSTEM_FIELD_NAMES)(
'should derive the %s system field universal identifier deterministically through the manifest sync funnel',
(systemFieldName) => {
const systemField = fetchedFields.find(
(field) => field.name === systemFieldName,
);
expect(systemField).toBeDefined();
expect(systemField?.universalIdentifier).toBe(
getFieldUniversalIdentifier({
applicationUniversalIdentifier: TEST_APP_UNIVERSAL_IDENTIFIER,
objectUniversalIdentifier,
name: systemFieldName,
}),
);
},
);
});