## Introduction
Closes twentyhq/core-team-issues#2635 and twentyhq/core-team-issues#2642
and twentyhq/core-team-issues#2589
Object system fields (`searchVector` + its GIN index +
`searchFieldMetadata`, the reserved system fields, default relations)
were provisioned through several scattered, path-specific code paths. As
a result the **app-manifest sync path** authored objects with an
empty/`NULL` `searchVector` and **zero `searchFieldMetadata`**, so
app-owned objects shipped a broken generated search column (see #22657).
The generation logic also lived partly in imperative services rather
than in the metadata side-effect engine, and relied on non-deterministic
(`v4`) universal identifiers that `twenty apply` could not converge,
destroying manually backfilled rows.
This PR centralizes every object-creation system side effect into the
**metadata side-effect engine**, extends the engine to keep search
metadata consistent on field delete and object relabel, makes the
standard app's search identifiers deterministic, and ships upgrade
commands to reconcile existing workspaces.
## What changed
### Side effects moved into the metadata side-effect engine
New dedicated, self-contained handlers — so every write path (API and
app manifest) gets identical results, and side effects never trigger
other side effects.
**Object create / delete** (`handlers/object-metadata`)
* **`objectSystemFieldsOnCreate`** — generates the 7 reserved system
fields (`id`, `createdAt`, `updatedAt`, `deletedAt`, `createdBy`,
`updatedBy`, `position`).
* **`objectSearchVectorOnCreate`** — provisions the full-text search
surface as one unit: the `searchVector` `TS_VECTOR` field, its backing
GIN index, and the `searchFieldMetadata` row (for searchable objects
whose label identifier is a searchable field) that keeps `searchVector`
populated instead of `NULL`.
* **`objectSystemSideEffectsOnDelete`** — tears the above down on object
deletion.
**Search-metadata consistency on relabel / field delete** (new — these
are what close the manifest-path gaps)
* **`objectSearchVectorOnUpdate`** (`handlers/object-metadata`) — when a
searchable object is relabeled onto a new searchable field, provisions
the `searchFieldMetadata` row that indexes it. Relabeling is
**additive**: existing rows (e.g. the provisioned `name` row) are
preserved, so the previous label identifier stays searchable. Mirrors
the API update path so a manifest re-sync that changes the label
identifier reaches search parity. No-ops for junction objects (`id`
label identifier) and non-searchable field types.
* **`fieldSearchFieldMetadataOnDelete`** (`handlers/field-metadata`) —
when a field is deleted, cascade-deletes every `searchFieldMetadata` row
that indexes it. `searchFieldMetadata` is excluded from manifest
deletion inference, so this explicit cascade is what covers **both the
API and manifest paths** (the object-scoped DB cascade only fires on
object deletion). Uses the `searchFieldMetadataUniversalIdentifiers`
aggregator on the flat field for an O(k) lookup instead of scanning all
rows.
The **default `name` field and default relations are now caller-provided
default fields** (SDK autocomplete on the manifest path, input
transpiler on the API path) rather than system side effects — removing
duplicate name generation, the imperative
`build-default-*-for-custom-object` utilities, and the ad-hoc
system-field integrity validator.
### Deterministic identifiers for the standard app
The twenty-standard search GIN index and `searchFieldMetadata` now
derive deterministic universal identifiers
(`getIndexUniversalIdentifier` / `getSearchFieldUniversalIdentifier`)
instead of `v4`, so `twenty apply` converges instead of recreating.
### Upgrade commands (`2-20`) to reconcile existing workspaces
**Instance commands** (run once per instance; ordered fast → slow →
workspace):
1. **`AddIsSystemSideEffectToSearchFieldMetadata`** (fast) — adds the
`isSystemSideEffect` column to `core.searchFieldMetadata`. Defaults to
`true`, which also correctly backfills every existing row since
`searchFieldMetadata` is always system-derived (never user-authored).
2. **`BackfillNameFieldIsSystemSideEffect`** (slow) — re-flags existing
`name` fields from `isSystemSideEffect: true` → `false`, since the
default `name` field is now a caller-provided default like any other
user-owned field (it was provisioned as `true` in 2.15 → 2.19). This is
a pure data backfill, so the bulk `UPDATE` lives in `runDataMigration()`
rather than `up()` — keeping it out of the fast schema transaction
avoids holding an `ACCESS EXCLUSIVE` lock that could stall reads during
the deploy. Slow instance commands still run before every workspace
command of the version, so the fresh value is in place before the
search-reconcile workspace commands recompute the `fieldMetadata`
flat-entity cache. Scoping by name alone is safe (no engine-owned field
is named `name`); `down()` is best-effort (pre-2.15 `false` rows are
indistinguishable from flipped ones).
**Workspace commands** (idempotent, dry-run supported):
1. **`reconcile-search-vector-gin-index-universal-identifier`** —
re-owns every searchVector GIN index UID to its deterministic value (all
applications), then backfills the missing GIN index for installed-app
objects.
2. **`reconcile-search-field-metadata`** — re-owns every
`searchFieldMetadata` UID (all applications), then backfills the missing
rows for installed-app searchable objects.
3. **`rebuild-installed-app-search-vectors`** — rebuilds the
`searchVector` column of every installed-app `TS_VECTOR` field, once the
index and rows exist.
Design notes:
* **Re-own is global** (twenty-standard, workspace-custom, installed) —
a UID convergence keyed on each row's own application.
* **Backfill is installed-app only** — standard/custom objects already
have these rows via the manifest funnel.
* Re-own runs **before** backfill and is transaction-guarded; a failure
aborts that workspace to avoid a unique-identifier collision.
## Tests
* Integration: app manifest sync now asserts system fields + searchable
objects (searchVector, GIN index, searchFieldMetadata) are created; a
new relabel suite drives three manifest syncs and asserts records stay
searchable through the old + new label identifiers and lose
searchability when a field is removed; removed the obsolete
system-fields-integrity suite/snapshots.
* Unit: per-handler side-effect specs (including the new
`objectSearchVectorOnUpdate` and `fieldSearchFieldMetadataOnDelete`
handlers), and per-util specs for the re-own / backfill operation
builders and the GIN-index classifier.
## Upgrade / migration notes
* Existing workspaces converge on the next upgrade run via the `2-20`
instance + workspace commands (idempotent, dry-run supported).
* Backfill and rebuild go through the workspace-migration runner
(automatic cache invalidation); the re-own step invalidates only the
affected flat-entity maps directly.
* The cross-version upgrade CI now flushes the cache before running the
upgrade, so the new version recomputes every flat-entity map from the
database instead of reading blobs the old version serialized in an older
shape.
## Follow-up
* `object-metadata.service.ts` still carries a `TODO: remove once
default view fields move to the metadata side effect engine` — default
view fields are the next candidate to move into the engine.
* A single manifest sync cannot yet both create a field and relabel the
object onto it, because `objectMetadata.update` is ordered before
`fieldMetadata.create` in the migration runner. Tracked in
twentyhq/core-team-issues#2655; to be fixed in a follow-up.
Paul Rastoin
60fd322b49