twenty

houseassassin/twenty

Fork 0

Commit Graph

Author	SHA1	Message	Date
Paul Rastoin	1be5a0e54a	System side effect relations (#22882 ) Closes twentyhq/core-team-issues#2667 ## What Default relations to the standard relation objects (`timelineActivities`, `attachments`, `noteTargets`, `taskTargets`) are now fully owned by the metadata side-effect engine. Neither the API transpilers nor the SDK manifest builder provision them anymore: any object creation, rename or deletion — regardless of the caller — goes through the same engine handlers. ## Why - Provisioning was duplicated across the API path and the SDK manifest builder, with diverging behavior. - Universal identifiers of relation fields were derived from object names, so renaming an object mutated them and forced lossy delete+create cycles on manifest sync. ## How ### Engine-owned lifecycle (side-effect handlers) - `objectSystemRelationsOnCreate`: provisions the 8 forward/reverse relation fields (+ join column indexes) when an object is created. - `objectSystemRelationsOnUpdate`: renames the reverse morph fields (`target<ObjectName>`) when their host object is renamed — a lossless `fieldMetadata.update`. - `objectSystemSideEffectsOnDelete`: cascades deletion of engine-owned fields/indexes when the object is deleted. - The API transpilers and the SDK `buildManifest` no longer inject these fields; `isSystemSideEffect: true` marks engine-owned entities, guarded by a granular property allowlist (only `isActive` is user-editable) and excluded from manifest deletion inference. ### Name-free deterministic universal identifiers New `getSystemRelationFieldUniversalIdentifier({ applicationUniversalIdentifier, objectUniversalIdentifier, relationTargetObjectUniversalIdentifier })` in `twenty-shared`, exported from `twenty-sdk/define`. The identifier is keyed on the two object identifiers instead of field names (direction encoded by argument order), so object renames never mutate relation field identifiers. It cannot collide with the name-based `getFieldUniversalIdentifier` derivation (field names cannot contain `:`). ### twenty-standard re-owned All 48 forward/reverse system relation field declarations in `STANDARD_OBJECTS` now pin the derived name-free identifiers (computed inline via the shared util) and carry `isSystemSideEffect: true`, with labels/icons declared explicitly (translated via `msg`). `twenty-standard` is projected as if the engine had generated these fields itself. ### 2.23 upgrade commands - `reconcile-system-relation-field-universal-identifier`: structurally matches existing default relation fields per workspace and backfills the derived universal identifiers, `isSystemSideEffect` flags, and standard labels/icons. - `upgrade-people-data-labs-application`: upgrades installed PDL apps to `1.0.7` right after the backfill to close the desync window (its views reference the re-derived identifiers). ### Misc - `people-data-labs` `1.0.7`: views temporarily pin the new derived identifiers (TODO: import from the next released `twenty-sdk`). - `UpgradeStatusModule` split out of `UpgradeModule` so the application module cluster can consume upgrade status/migration services without importing the versioned command bundles (fixes a require cycle that crashed boot). - Docs: `system-fields.mdx` documents the system relation fields and their resolver; `sync-and-recovery.mdx` plan example no longer shows auto-injected relations. ## Known red CI `people-data-labs (dockerhub-latest)` fails by design until the 2.23 server image is published: the app pins the new identifiers which only exist on a 2.23 server. The `local` leg (server built from this branch) is green. ## System fields are no longer manifest-authorable (accepted regression) The manifest converter no longer derives `isSystem` / `isSystemSideEffect` from field names. Reserved-system-named manifest fields (`id`, `createdAt`, `updatedAt`, `deletedAt`, `createdBy`, `updatedBy`, `position`, `searchVector`) are now skipped at conversion time when they carry the exact derived universal identifier (keeps manifests built with older SDKs installable), and rejected with `INVALID_INPUT` when they pin any other identifier. System fields are therefore fully engine-canonical: nothing a manifest carries can produce a system-flagged entity anymore. Accepted regression: a manifest can no longer influence system field properties at all. Previously a (legacy) re-declaration could shape them at creation — which actually produced broken system fields, e.g. a nullable, non-unique `id` — and could still toggle the allowlisted `isActive` / `universalSettings` afterwards. We consider this acceptable for now: per-app granularity over system fields will be reintroduced later through the override framework, which will also settle update semantics by forbidding direct updates over `isSystemSideEffect: true` entities and expressing divergence as overrides. `isSystemSideEffect`-only entities (the default relation fields provisioned by this PR) still have no engine-level update guard (see Follow-up below); that part is unchanged and also lands with the overrides refactor. ## Follow-up `isSystemSideEffect` field update/delete guards intentionally live at the API layer (`sanitize-raw-update-field-input.ts`, `from-delete-field-input-...util.ts`) rather than in the engine-level `FlatFieldMetadataValidatorService`. Moving them into the validator requires threading operation-origin (direct field mutation vs engine cascade) through the migration matrix, otherwise legitimate object rename/delete cascades (which carry `isSystemBuild=false`) would be rejected. Tracked in twentyhq/core-team-issues#2671. <!-- This is an auto-generated description by cubic. --> <a href="https://cubic.dev/pr/twentyhq/twenty/pull/22882?utm_source=github" target="_blank" rel="noopener noreferrer" data-no-image-dialog="true"><picture><source media="(prefers-color-scheme: dark)" srcset="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"><source media="(prefers-color-scheme: light)" srcset="https://www.cubic.dev/buttons/review-in-cubic-light.svg"><img alt="Review in cubic" src="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"></picture></a> <!-- End of auto-generated description by cubic. -->	2026-07-20 18:53:24 +02:00
Paul Rastoin	60fd322b49	Centralize system field side effects + search field metadata (#22594 ) ## Introduction Closes twentyhq/core-team-issues#2635 and twentyhq/core-team-issues#2642 and twentyhq/core-team-issues#2589 Object system fields (`searchVector` + its GIN index + `searchFieldMetadata`, the reserved system fields, default relations) were provisioned through several scattered, path-specific code paths. As a result the app-manifest sync path authored objects with an empty/`NULL` `searchVector` and zero `searchFieldMetadata`, so app-owned objects shipped a broken generated search column (see #22657). The generation logic also lived partly in imperative services rather than in the metadata side-effect engine, and relied on non-deterministic (`v4`) universal identifiers that `twenty apply` could not converge, destroying manually backfilled rows. This PR centralizes every object-creation system side effect into the metadata side-effect engine, extends the engine to keep search metadata consistent on field delete and object relabel, makes the standard app's search identifiers deterministic, and ships upgrade commands to reconcile existing workspaces. ## What changed ### Side effects moved into the metadata side-effect engine New dedicated, self-contained handlers — so every write path (API and app manifest) gets identical results, and side effects never trigger other side effects. Object create / delete (`handlers/object-metadata`) * `objectSystemFieldsOnCreate` — generates the 7 reserved system fields (`id`, `createdAt`, `updatedAt`, `deletedAt`, `createdBy`, `updatedBy`, `position`). * `objectSearchVectorOnCreate` — provisions the full-text search surface as one unit: the `searchVector` `TS_VECTOR` field, its backing GIN index, and the `searchFieldMetadata` row (for searchable objects whose label identifier is a searchable field) that keeps `searchVector` populated instead of `NULL`. * `objectSystemSideEffectsOnDelete` — tears the above down on object deletion. Search-metadata consistency on relabel / field delete (new — these are what close the manifest-path gaps) * `objectSearchVectorOnUpdate` (`handlers/object-metadata`) — when a searchable object is relabeled onto a new searchable field, provisions the `searchFieldMetadata` row that indexes it. Relabeling is additive: existing rows (e.g. the provisioned `name` row) are preserved, so the previous label identifier stays searchable. Mirrors the API update path so a manifest re-sync that changes the label identifier reaches search parity. No-ops for junction objects (`id` label identifier) and non-searchable field types. * `fieldSearchFieldMetadataOnDelete` (`handlers/field-metadata`) — when a field is deleted, cascade-deletes every `searchFieldMetadata` row that indexes it. `searchFieldMetadata` is excluded from manifest deletion inference, so this explicit cascade is what covers both the API and manifest paths (the object-scoped DB cascade only fires on object deletion). Uses the `searchFieldMetadataUniversalIdentifiers` aggregator on the flat field for an O(k) lookup instead of scanning all rows. The default `name` field and default relations are now caller-provided default fields (SDK autocomplete on the manifest path, input transpiler on the API path) rather than system side effects — removing duplicate name generation, the imperative `build-default--for-custom-object` utilities, and the ad-hoc system-field integrity validator. ### Deterministic identifiers for the standard app The twenty-standard search GIN index and `searchFieldMetadata` now derive deterministic universal identifiers (`getIndexUniversalIdentifier` / `getSearchFieldUniversalIdentifier`) instead of `v4`, so `twenty apply` converges instead of recreating. ### Upgrade commands (`2-20`) to reconcile existing workspaces Instance commands* (run once per instance; ordered fast → slow → workspace): 1. `AddIsSystemSideEffectToSearchFieldMetadata` (fast) — adds the `isSystemSideEffect` column to `core.searchFieldMetadata`. Defaults to `true`, which also correctly backfills every existing row since `searchFieldMetadata` is always system-derived (never user-authored). 2. `BackfillNameFieldIsSystemSideEffect` (slow) — re-flags existing `name` fields from `isSystemSideEffect: true` → `false`, since the default `name` field is now a caller-provided default like any other user-owned field (it was provisioned as `true` in 2.15 → 2.19). This is a pure data backfill, so the bulk `UPDATE` lives in `runDataMigration()` rather than `up()` — keeping it out of the fast schema transaction avoids holding an `ACCESS EXCLUSIVE` lock that could stall reads during the deploy. Slow instance commands still run before every workspace command of the version, so the fresh value is in place before the search-reconcile workspace commands recompute the `fieldMetadata` flat-entity cache. Scoping by name alone is safe (no engine-owned field is named `name`); `down()` is best-effort (pre-2.15 `false` rows are indistinguishable from flipped ones). Workspace commands (idempotent, dry-run supported): 1. `reconcile-search-vector-gin-index-universal-identifier` — re-owns every searchVector GIN index UID to its deterministic value (all applications), then backfills the missing GIN index for installed-app objects. 2. `reconcile-search-field-metadata` — re-owns every `searchFieldMetadata` UID (all applications), then backfills the missing rows for installed-app searchable objects. 3. `rebuild-installed-app-search-vectors` — rebuilds the `searchVector` column of every installed-app `TS_VECTOR` field, once the index and rows exist. Design notes: * Re-own is global (twenty-standard, workspace-custom, installed) — a UID convergence keyed on each row's own application. * Backfill is installed-app only — standard/custom objects already have these rows via the manifest funnel. * Re-own runs before backfill and is transaction-guarded; a failure aborts that workspace to avoid a unique-identifier collision. ## Tests * Integration: app manifest sync now asserts system fields + searchable objects (searchVector, GIN index, searchFieldMetadata) are created; a new relabel suite drives three manifest syncs and asserts records stay searchable through the old + new label identifiers and lose searchability when a field is removed; removed the obsolete system-fields-integrity suite/snapshots. * Unit: per-handler side-effect specs (including the new `objectSearchVectorOnUpdate` and `fieldSearchFieldMetadataOnDelete` handlers), and per-util specs for the re-own / backfill operation builders and the GIN-index classifier. ## Upgrade / migration notes * Existing workspaces converge on the next upgrade run via the `2-20` instance + workspace commands (idempotent, dry-run supported). * Backfill and rebuild go through the workspace-migration runner (automatic cache invalidation); the re-own step invalidates only the affected flat-entity maps directly. * The cross-version upgrade CI now flushes the cache before running the upgrade, so the new version recomputes every flat-entity map from the database instead of reading blobs the old version serialized in an older shape. ## Follow-up * `object-metadata.service.ts` still carries a `TODO: remove once default view fields move to the metadata side effect engine` — default view fields are the next candidate to move into the engine. * A single manifest sync cannot yet both create a field and relabel the object onto it, because `objectMetadata.update` is ordered before `fieldMetadata.create` in the migration runner. Tracked in twentyhq/core-team-issues#2655; to be fixed in a follow-up.	2026-07-09 16:59:54 +02:00
Paul Rastoin	43730d7748	Centralized side effects devxp basis (#22295 ) # Introduction This PR introduces a centralized, strictly-typed metadata side-effect engine that unifies how system metadata side effects are derived and applied across both metadata entry points — the metadata GraphQL API and the application sync / manifest flow — and migrates the first side effect end-to-end: a unique scalar field owns its backing single-field `UNIQUE` index (full create / update / delete lifecycle). ## New conventions - Engine-owned companions: metadata flagged `isSystemSideEffect: true` is owned by the engine. Its deletion is never inferred from absence in a manifest — it results from PG-level cascade or from a delete side effect (a side effect always has a cause, its parent metadata). - Reserved deterministic identifiers: apps cannot declare metadata reusing an engine-owned deterministic `universalIdentifier`. Doing so fails validation with `RESERVED_SYSTEM_UNIVERSAL_IDENTIFIER` (until an explicit override API exists). - Record-native operation matrix: the operation matrix is keyed by `universalIdentifier` (`AllFlatEntityOperationRecordByMetadataName`) instead of arrays, making parent resolution and deduplication O(1). Array-based API callers are transpiled to records at the validate-build-and-run boundary. - Twenty-sdk user-facing experience with system fields will only be related to overrides. # What this PR does ## 1. Side-effect engine (foundation) - `MetadataSideEffectEngineService.expandWithSideEffects(...)` takes the intention-carrying record matrix and returns it expanded with derived side effects, or a structured failure. - Handlers are registered via a typed decorator + registry pattern (`MetadataSideEffectHandler({ operation, metadataName, name, description })`), with runtime duplicate-name detection. Multiple handlers per (operation, metadataName) are supported. - Handler contract mirrors the validator pattern: - receives the trigger flat entity, the live record matrix, and strictly-typed related flat entity maps (`MetadataFlatEntityAndRelatedFlatEntityMapsForSideEffect<P>`, derived from declared companion metadata names — no loose `Partial<AllFlatEntityMaps>` context) - returns `MetadataSideEffectResult`: `success` (operations record) \| `noop` \| `fail` (structured failure) - Non-recursion is structural: triggers are read from the original caller input, never from the expanded matrix, so a side effect can never trigger another side effect. - Deduplication + collision detection: side effects are deduped by `universalIdentifier` per operation; a caller-declared entity colliding with an engine-owned deterministic identifier is recorded as a collision. - Unified failure channel: handler failures and reserved-identifier collisions are merged into the same `OrchestratorFailureReport` contract as builder validation errors, and the run short-circuits (fail-closed, nothing is applied). ## 2. First migrated side effect — unique field → backing unique index Three handlers own the complete lifecycle of the deterministic single-field `UNIQUE` index backing a unique scalar field: - create: unique scalar field → generate the deterministic backing index (`fieldUniqueBackingIndexOnCreate`) - update: `isUnique` flips and renames of still-unique fields (the index name — and therefore its deterministic identifier — derives from the field name, so a rename drops the stale index and recreates the deterministic one) (`fieldUniqueBackingIndexOnUpdate`) - delete: cascade-delete the backing index (`fieldUniqueBackingIndexOnDelete`) Supporting rules: - The primary key `id` field never spawns a backing index (uniqueness comes from the PK constraint) — explicit `isPrimaryKeyFlatFieldMetadata` guard. - Parent object resolution is optimistic-first: an object created or updated in the same batch wins over the workspace cache (so e.g. renaming an object while flipping a field to unique builds the index from the post-rename object), resolved in O(1) via the record matrix. - A missing parent object is reported as a structured side-effect failure, never silently skipped. ## 3. Path convergence — manifest and API share one flow - The manifest sync now derives a from→to record matrix from the cache and feeds `validateBuildAndRunWorkspaceMigrationFromRecord`, the same flow the API uses — both paths converge on the engine. - Manifest-side unique-index generation and API transpiler system-unique-index handling were removed (declared/composite/relation indexes stay untouched). - New `WorkspaceMigrationFlatEntityMapsService` mutualizes flat-entity-maps computation between the side-effect engine and the builder: cache keys are derived from the caller metadata names (+ validation- and side-effect-related closures) instead of hardcoded loads. - App-scoping and pruning are folded into one shared primitive (`getSubAllFlatEntityMapsByApplicationIdsOrThrow`): slicing dependency maps to the involved applications always prunes dangling one-to-many aggregators — callers can no longer forget it. - Behavior change: an app extending another app's view with a view field now syncs successfully (cross-app view-field extension), covered by a dedicated integration test. ## 4. Backfill upgrade command (2.19) `upgrade:2-19:backfill-system-unique-index-universal-identifier` rewrites legacy system unique-index `universalIdentifier`s to their deterministic value so the engine can own pre-existing indexes. The backfill is driven from `isUnique: true` fields (mirroring the engine ownership predicate — excludes PK / morph / relation fields) and resolves each field's backing index in O(1). # Bugs fixed along the way - `database:reset` seeding failed with `INDEX_FIELD_INVALID_DEFAULT_VALUE`: the engine derived a backing `UNIQUE` index for the default `id` primary key. Fixed with the explicit primary-key guard. - `isUnique` updates on system-flagged standard fields (e.g. auto-created `name`) did not trigger the backing-index side effect. - Manifest sync crashed with "Could not find flat entity with universal identifier ..." when app-scoped slices left dangling aggregator references — fixed by centralizing pruning in the shared slice primitive	2026-07-03 18:13:20 +02:00

Author

SHA1

Message

Date

Paul Rastoin

1be5a0e54a

System side effect relations (#22882 )

Closes twentyhq/core-team-issues#2667

## What

Default relations to the standard relation objects
(`timelineActivities`, `attachments`, `noteTargets`, `taskTargets`) are
now fully owned by the **metadata side-effect engine**. Neither the API
transpilers nor the SDK manifest builder provision them anymore: any
object creation, rename or deletion — regardless of the caller — goes
through the same engine handlers.

## Why

- Provisioning was duplicated across the API path and the SDK manifest
builder, with diverging behavior.
- Universal identifiers of relation fields were derived from object
**names**, so renaming an object mutated them and forced lossy
delete+create cycles on manifest sync.

## How

### Engine-owned lifecycle (side-effect handlers)

- `objectSystemRelationsOnCreate`: provisions the 8 forward/reverse
relation fields (+ join column indexes) when an object is created.
- `objectSystemRelationsOnUpdate`: renames the reverse morph fields
(`target<ObjectName>`) when their host object is renamed — a lossless
`fieldMetadata.update`.
- `objectSystemSideEffectsOnDelete`: cascades deletion of engine-owned
fields/indexes when the object is deleted.
- The API transpilers and the SDK `buildManifest` no longer inject these
fields; `isSystemSideEffect: true` marks engine-owned entities, guarded
by a granular property allowlist (only `isActive` is user-editable) and
excluded from manifest deletion inference.

### Name-free deterministic universal identifiers

New `getSystemRelationFieldUniversalIdentifier({
applicationUniversalIdentifier, objectUniversalIdentifier,
relationTargetObjectUniversalIdentifier })` in `twenty-shared`, exported
from `twenty-sdk/define`. The identifier is keyed on the two **object**
identifiers instead of field names (direction encoded by argument
order), so object renames never mutate relation field identifiers. It
cannot collide with the name-based `getFieldUniversalIdentifier`
derivation (field names cannot contain `:`).

### twenty-standard re-owned

All 48 forward/reverse system relation field declarations in
`STANDARD_OBJECTS` now pin the derived name-free identifiers (computed
inline via the shared util) and carry `isSystemSideEffect: true`, with
labels/icons declared explicitly (translated via `msg`).
`twenty-standard` is projected as if the engine had generated these
fields itself.

### 2.23 upgrade commands

- `reconcile-system-relation-field-universal-identifier`: structurally
matches existing default relation fields per workspace and backfills the
derived universal identifiers, `isSystemSideEffect` flags, and standard
labels/icons.
- `upgrade-people-data-labs-application`: upgrades installed PDL apps to
`1.0.7` right after the backfill to close the desync window (its views
reference the re-derived identifiers).

### Misc

- `people-data-labs` `1.0.7`: views temporarily pin the new derived
identifiers (TODO: import from the next released `twenty-sdk`).
- `UpgradeStatusModule` split out of `UpgradeModule` so the application
module cluster can consume upgrade status/migration services without
importing the versioned command bundles (fixes a require cycle that
crashed boot).
- Docs: `system-fields.mdx` documents the system relation fields and
their resolver; `sync-and-recovery.mdx` plan example no longer shows
auto-injected relations.

## Known red CI

`people-data-labs (dockerhub-latest)` fails by design until the 2.23
server image is published: the app pins the new identifiers which only
exist on a 2.23 server. The `local` leg (server built from this branch)
is green.

## System fields are no longer manifest-authorable (accepted regression)

The manifest converter no longer derives `isSystem` /
`isSystemSideEffect` from field names. Reserved-system-named manifest
fields (`id`, `createdAt`, `updatedAt`, `deletedAt`, `createdBy`,
`updatedBy`, `position`, `searchVector`) are now skipped at conversion
time when they carry the exact derived universal identifier (keeps
manifests built with older SDKs installable), and rejected with
`INVALID_INPUT` when they pin any other identifier. System fields are
therefore fully engine-canonical: nothing a manifest carries can produce
a system-flagged entity anymore.

**Accepted regression**: a manifest can no longer influence system field
properties at all. Previously a (legacy) re-declaration could shape them
at creation — which actually produced broken system fields, e.g. a
nullable, non-unique `id` — and could still toggle the allowlisted
`isActive` / `universalSettings` afterwards. We consider this acceptable
for now: per-app granularity over system fields will be reintroduced
later through the **override framework**, which will also settle update
semantics by forbidding direct updates over `isSystemSideEffect: true`
entities and expressing divergence as overrides.

`isSystemSideEffect`-only entities (the default relation fields
provisioned by this PR) still have no engine-level update guard (see
Follow-up below); that part is unchanged and also lands with the
overrides refactor.

## Follow-up

`isSystemSideEffect` field update/delete guards intentionally live at
the API layer (`sanitize-raw-update-field-input.ts`,
`from-delete-field-input-...util.ts`) rather than in the engine-level
`FlatFieldMetadataValidatorService`. Moving them into the validator
requires threading operation-origin (direct field mutation vs engine
cascade) through the migration matrix, otherwise legitimate object
rename/delete cascades (which carry `isSystemBuild=false`) would be
rejected. Tracked in twentyhq/core-team-issues#2671.

<!-- This is an auto-generated description by cubic. -->
<a
href="https://cubic.dev/pr/twentyhq/twenty/pull/22882?utm_source=github"
target="_blank" rel="noopener noreferrer"
data-no-image-dialog="true"><picture><source
media="(prefers-color-scheme: dark)"
srcset="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"><source
media="(prefers-color-scheme: light)"
srcset="https://www.cubic.dev/buttons/review-in-cubic-light.svg"><img
alt="Review in cubic"
src="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"></picture></a>
<!-- End of auto-generated description by cubic. -->

2026-07-20 18:53:24 +02:00

Paul Rastoin

60fd322b49

Centralize system field side effects + search field metadata (#22594 )

## Introduction

Closes twentyhq/core-team-issues#2635 and twentyhq/core-team-issues#2642
and twentyhq/core-team-issues#2589

Object system fields (`searchVector` + its GIN index +
`searchFieldMetadata`, the reserved system fields, default relations)
were provisioned through several scattered, path-specific code paths. As
a result the **app-manifest sync path** authored objects with an
empty/`NULL` `searchVector` and **zero `searchFieldMetadata`**, so
app-owned objects shipped a broken generated search column (see #22657).
The generation logic also lived partly in imperative services rather
than in the metadata side-effect engine, and relied on non-deterministic
(`v4`) universal identifiers that `twenty apply` could not converge,
destroying manually backfilled rows.

This PR centralizes every object-creation system side effect into the
**metadata side-effect engine**, extends the engine to keep search
metadata consistent on field delete and object relabel, makes the
standard app's search identifiers deterministic, and ships upgrade
commands to reconcile existing workspaces.

## What changed

### Side effects moved into the metadata side-effect engine

New dedicated, self-contained handlers — so every write path (API and
app manifest) gets identical results, and side effects never trigger
other side effects.

**Object create / delete** (`handlers/object-metadata`)

* **`objectSystemFieldsOnCreate`** — generates the 7 reserved system
fields (`id`, `createdAt`, `updatedAt`, `deletedAt`, `createdBy`,
`updatedBy`, `position`).
* **`objectSearchVectorOnCreate`** — provisions the full-text search
surface as one unit: the `searchVector` `TS_VECTOR` field, its backing
GIN index, and the `searchFieldMetadata` row (for searchable objects
whose label identifier is a searchable field) that keeps `searchVector`
populated instead of `NULL`.
* **`objectSystemSideEffectsOnDelete`** — tears the above down on object
deletion.

**Search-metadata consistency on relabel / field delete** (new — these
are what close the manifest-path gaps)

* **`objectSearchVectorOnUpdate`** (`handlers/object-metadata`) — when a
searchable object is relabeled onto a new searchable field, provisions
the `searchFieldMetadata` row that indexes it. Relabeling is
**additive**: existing rows (e.g. the provisioned `name` row) are
preserved, so the previous label identifier stays searchable. Mirrors
the API update path so a manifest re-sync that changes the label
identifier reaches search parity. No-ops for junction objects (`id`
label identifier) and non-searchable field types.
* **`fieldSearchFieldMetadataOnDelete`** (`handlers/field-metadata`) —
when a field is deleted, cascade-deletes every `searchFieldMetadata` row
that indexes it. `searchFieldMetadata` is excluded from manifest
deletion inference, so this explicit cascade is what covers **both the
API and manifest paths** (the object-scoped DB cascade only fires on
object deletion). Uses the `searchFieldMetadataUniversalIdentifiers`
aggregator on the flat field for an O(k) lookup instead of scanning all
rows.

The **default `name` field and default relations are now caller-provided
default fields** (SDK autocomplete on the manifest path, input
transpiler on the API path) rather than system side effects — removing
duplicate name generation, the imperative
`build-default-*-for-custom-object` utilities, and the ad-hoc
system-field integrity validator.

### Deterministic identifiers for the standard app

The twenty-standard search GIN index and `searchFieldMetadata` now
derive deterministic universal identifiers
(`getIndexUniversalIdentifier` / `getSearchFieldUniversalIdentifier`)
instead of `v4`, so `twenty apply` converges instead of recreating.

### Upgrade commands (`2-20`) to reconcile existing workspaces

**Instance commands** (run once per instance; ordered fast → slow →
workspace):

1. **`AddIsSystemSideEffectToSearchFieldMetadata`** (fast) — adds the
`isSystemSideEffect` column to `core.searchFieldMetadata`. Defaults to
`true`, which also correctly backfills every existing row since
`searchFieldMetadata` is always system-derived (never user-authored).
2. **`BackfillNameFieldIsSystemSideEffect`** (slow) — re-flags existing
`name` fields from `isSystemSideEffect: true` → `false`, since the
default `name` field is now a caller-provided default like any other
user-owned field (it was provisioned as `true` in 2.15 → 2.19). This is
a pure data backfill, so the bulk `UPDATE` lives in `runDataMigration()`
rather than `up()` — keeping it out of the fast schema transaction
avoids holding an `ACCESS EXCLUSIVE` lock that could stall reads during
the deploy. Slow instance commands still run before every workspace
command of the version, so the fresh value is in place before the
search-reconcile workspace commands recompute the `fieldMetadata`
flat-entity cache. Scoping by name alone is safe (no engine-owned field
is named `name`); `down()` is best-effort (pre-2.15 `false` rows are
indistinguishable from flipped ones).

**Workspace commands** (idempotent, dry-run supported):

1. **`reconcile-search-vector-gin-index-universal-identifier`** —
re-owns every searchVector GIN index UID to its deterministic value (all
applications), then backfills the missing GIN index for installed-app
objects.
2. **`reconcile-search-field-metadata`** — re-owns every
`searchFieldMetadata` UID (all applications), then backfills the missing
rows for installed-app searchable objects.
3. **`rebuild-installed-app-search-vectors`** — rebuilds the
`searchVector` column of every installed-app `TS_VECTOR` field, once the
index and rows exist.

Design notes:

* **Re-own is global** (twenty-standard, workspace-custom, installed) —
a UID convergence keyed on each row's own application.
* **Backfill is installed-app only** — standard/custom objects already
have these rows via the manifest funnel.
* Re-own runs **before** backfill and is transaction-guarded; a failure
aborts that workspace to avoid a unique-identifier collision.

## Tests

* Integration: app manifest sync now asserts system fields + searchable
objects (searchVector, GIN index, searchFieldMetadata) are created; a
new relabel suite drives three manifest syncs and asserts records stay
searchable through the old + new label identifiers and lose
searchability when a field is removed; removed the obsolete
system-fields-integrity suite/snapshots.
* Unit: per-handler side-effect specs (including the new
`objectSearchVectorOnUpdate` and `fieldSearchFieldMetadataOnDelete`
handlers), and per-util specs for the re-own / backfill operation
builders and the GIN-index classifier.

## Upgrade / migration notes

* Existing workspaces converge on the next upgrade run via the `2-20`
instance + workspace commands (idempotent, dry-run supported).
* Backfill and rebuild go through the workspace-migration runner
(automatic cache invalidation); the re-own step invalidates only the
affected flat-entity maps directly.
* The cross-version upgrade CI now flushes the cache before running the
upgrade, so the new version recomputes every flat-entity map from the
database instead of reading blobs the old version serialized in an older
shape.

## Follow-up

* `object-metadata.service.ts` still carries a `TODO: remove once
default view fields move to the metadata side effect engine` — default
view fields are the next candidate to move into the engine.
* A single manifest sync cannot yet both create a field and relabel the
object onto it, because `objectMetadata.update` is ordered before
`fieldMetadata.create` in the migration runner. Tracked in
twentyhq/core-team-issues#2655; to be fixed in a follow-up.

2026-07-09 16:59:54 +02:00

Paul Rastoin

43730d7748

Centralized side effects devxp basis (#22295 )

# Introduction

This PR introduces a centralized, strictly-typed **metadata side-effect
engine** that unifies how system metadata side effects are derived and
applied across both metadata entry points — the **metadata GraphQL API**
and the **application sync / manifest** flow — and migrates the first
side effect end-to-end: **a unique scalar field owns its backing
single-field `UNIQUE` index** (full create / update / delete lifecycle).

## New conventions

- **Engine-owned companions**: metadata flagged `isSystemSideEffect:
true` is owned by the engine. Its deletion is never inferred from
absence in a manifest — it results from PG-level cascade or from a
delete side effect (a side effect always has a cause, its parent
metadata).
- **Reserved deterministic identifiers**: apps cannot declare metadata
reusing an engine-owned deterministic `universalIdentifier`. Doing so
fails validation with `RESERVED_SYSTEM_UNIVERSAL_IDENTIFIER` (until an
explicit override API exists).
- **Record-native operation matrix**: the operation matrix is keyed by
`universalIdentifier` (`AllFlatEntityOperationRecordByMetadataName`)
instead of arrays, making parent resolution and deduplication O(1).
Array-based API callers are transpiled to records at the
validate-build-and-run boundary.
- Twenty-sdk user-facing experience with system fields will only be
related to overrides.

# What this PR does

## 1. Side-effect engine (foundation)

- `MetadataSideEffectEngineService.expandWithSideEffects(...)` takes the
intention-carrying record matrix and returns it expanded with derived
side effects, or a structured failure.
- Handlers are registered via a typed **decorator + registry** pattern
(`MetadataSideEffectHandler({ operation, metadataName, name, description
})`), with runtime duplicate-name detection. Multiple handlers per
(operation, metadataName) are supported.
- Handler contract mirrors the validator pattern:
- receives the trigger flat entity, the live record matrix, and
**strictly-typed related flat entity maps**
(`MetadataFlatEntityAndRelatedFlatEntityMapsForSideEffect<P>`, derived
from declared companion metadata names — no loose
`Partial<AllFlatEntityMaps>` context)
- returns `MetadataSideEffectResult`: `success` (operations record) |
`noop` | `fail` (structured failure)
- **Non-recursion is structural**: triggers are read from the original
caller input, never from the expanded matrix, so a side effect can never
trigger another side effect.
- **Deduplication + collision detection**: side effects are deduped by
`universalIdentifier` per operation; a caller-declared entity colliding
with an engine-owned deterministic identifier is recorded as a
collision.
- **Unified failure channel**: handler failures and reserved-identifier
collisions are merged into the same `OrchestratorFailureReport` contract
as builder validation errors, and the run short-circuits (fail-closed,
nothing is applied).

## 2. First migrated side effect — unique field → backing unique index

Three handlers own the complete lifecycle of the deterministic
single-field `UNIQUE` index backing a unique scalar field:

- **create**: unique scalar field → generate the deterministic backing
index (`fieldUniqueBackingIndexOnCreate`)
- **update**: `isUnique` flips and renames of still-unique fields (the
index name — and therefore its deterministic identifier — derives from
the field name, so a rename drops the stale index and recreates the
deterministic one) (`fieldUniqueBackingIndexOnUpdate`)
- **delete**: cascade-delete the backing index
(`fieldUniqueBackingIndexOnDelete`)

Supporting rules:
- The primary key `id` field never spawns a backing index (uniqueness
comes from the PK constraint) — explicit `isPrimaryKeyFlatFieldMetadata`
guard.
- Parent object resolution is **optimistic-first**: an object created or
updated in the same batch wins over the workspace cache (so e.g.
renaming an object while flipping a field to unique builds the index
from the post-rename object), resolved in O(1) via the record matrix.
- A missing parent object is reported as a structured side-effect
failure, never silently skipped.

## 3. Path convergence — manifest and API share one flow

- The manifest sync now derives a from→to **record matrix** from the
cache and feeds `validateBuildAndRunWorkspaceMigrationFromRecord`, the
same flow the API uses — both paths converge on the engine.
- Manifest-side unique-index generation and API transpiler
system-unique-index handling were removed (declared/composite/relation
indexes stay untouched).
- New `WorkspaceMigrationFlatEntityMapsService` mutualizes
flat-entity-maps computation between the side-effect engine and the
builder: cache keys are derived from the caller metadata names (+
validation- and side-effect-related closures) instead of hardcoded
loads.
- App-scoping and pruning are folded into one shared primitive
(`getSubAllFlatEntityMapsByApplicationIdsOrThrow`): slicing dependency
maps to the involved applications always prunes dangling one-to-many
aggregators — callers can no longer forget it.
- **Behavior change**: an app extending another app's view with a view
field now syncs successfully (cross-app view-field extension), covered
by a dedicated integration test.

## 4. Backfill upgrade command (2.19)

`upgrade:2-19:backfill-system-unique-index-universal-identifier`
rewrites legacy system unique-index `universalIdentifier`s to their
deterministic value so the engine can own pre-existing indexes. The
backfill is **driven from `isUnique: true` fields** (mirroring the
engine ownership predicate — excludes PK / morph / relation fields) and
resolves each field's backing index in O(1).

# Bugs fixed along the way

- `database:reset` seeding failed with
`INDEX_FIELD_INVALID_DEFAULT_VALUE`: the engine derived a backing
`UNIQUE` index for the default `id` primary key. Fixed with the explicit
primary-key guard.
- `isUnique` updates on system-flagged standard fields (e.g.
auto-created `name`) did not trigger the backing-index side effect.
- Manifest sync crashed with "Could not find flat entity with universal
identifier ..." when app-scoped slices left dangling aggregator
references — fixed by centralizing pruning in the shared slice primitive

2026-07-03 18:13:20 +02:00

3 Commits