45cbdef9308f8adfcc7e1c5c53971e4d00a5b59c
4 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Paul Rastoin
|
60fd322b49 |
Centralize system field side effects + search field metadata (#22594)
## Introduction Closes twentyhq/core-team-issues#2635 and twentyhq/core-team-issues#2642 and twentyhq/core-team-issues#2589 Object system fields (`searchVector` + its GIN index + `searchFieldMetadata`, the reserved system fields, default relations) were provisioned through several scattered, path-specific code paths. As a result the **app-manifest sync path** authored objects with an empty/`NULL` `searchVector` and **zero `searchFieldMetadata`**, so app-owned objects shipped a broken generated search column (see #22657). The generation logic also lived partly in imperative services rather than in the metadata side-effect engine, and relied on non-deterministic (`v4`) universal identifiers that `twenty apply` could not converge, destroying manually backfilled rows. This PR centralizes every object-creation system side effect into the **metadata side-effect engine**, extends the engine to keep search metadata consistent on field delete and object relabel, makes the standard app's search identifiers deterministic, and ships upgrade commands to reconcile existing workspaces. ## What changed ### Side effects moved into the metadata side-effect engine New dedicated, self-contained handlers — so every write path (API and app manifest) gets identical results, and side effects never trigger other side effects. **Object create / delete** (`handlers/object-metadata`) * **`objectSystemFieldsOnCreate`** — generates the 7 reserved system fields (`id`, `createdAt`, `updatedAt`, `deletedAt`, `createdBy`, `updatedBy`, `position`). * **`objectSearchVectorOnCreate`** — provisions the full-text search surface as one unit: the `searchVector` `TS_VECTOR` field, its backing GIN index, and the `searchFieldMetadata` row (for searchable objects whose label identifier is a searchable field) that keeps `searchVector` populated instead of `NULL`. * **`objectSystemSideEffectsOnDelete`** — tears the above down on object deletion. **Search-metadata consistency on relabel / field delete** (new — these are what close the manifest-path gaps) * **`objectSearchVectorOnUpdate`** (`handlers/object-metadata`) — when a searchable object is relabeled onto a new searchable field, provisions the `searchFieldMetadata` row that indexes it. Relabeling is **additive**: existing rows (e.g. the provisioned `name` row) are preserved, so the previous label identifier stays searchable. Mirrors the API update path so a manifest re-sync that changes the label identifier reaches search parity. No-ops for junction objects (`id` label identifier) and non-searchable field types. * **`fieldSearchFieldMetadataOnDelete`** (`handlers/field-metadata`) — when a field is deleted, cascade-deletes every `searchFieldMetadata` row that indexes it. `searchFieldMetadata` is excluded from manifest deletion inference, so this explicit cascade is what covers **both the API and manifest paths** (the object-scoped DB cascade only fires on object deletion). Uses the `searchFieldMetadataUniversalIdentifiers` aggregator on the flat field for an O(k) lookup instead of scanning all rows. The **default `name` field and default relations are now caller-provided default fields** (SDK autocomplete on the manifest path, input transpiler on the API path) rather than system side effects — removing duplicate name generation, the imperative `build-default-*-for-custom-object` utilities, and the ad-hoc system-field integrity validator. ### Deterministic identifiers for the standard app The twenty-standard search GIN index and `searchFieldMetadata` now derive deterministic universal identifiers (`getIndexUniversalIdentifier` / `getSearchFieldUniversalIdentifier`) instead of `v4`, so `twenty apply` converges instead of recreating. ### Upgrade commands (`2-20`) to reconcile existing workspaces **Instance commands** (run once per instance; ordered fast → slow → workspace): 1. **`AddIsSystemSideEffectToSearchFieldMetadata`** (fast) — adds the `isSystemSideEffect` column to `core.searchFieldMetadata`. Defaults to `true`, which also correctly backfills every existing row since `searchFieldMetadata` is always system-derived (never user-authored). 2. **`BackfillNameFieldIsSystemSideEffect`** (slow) — re-flags existing `name` fields from `isSystemSideEffect: true` → `false`, since the default `name` field is now a caller-provided default like any other user-owned field (it was provisioned as `true` in 2.15 → 2.19). This is a pure data backfill, so the bulk `UPDATE` lives in `runDataMigration()` rather than `up()` — keeping it out of the fast schema transaction avoids holding an `ACCESS EXCLUSIVE` lock that could stall reads during the deploy. Slow instance commands still run before every workspace command of the version, so the fresh value is in place before the search-reconcile workspace commands recompute the `fieldMetadata` flat-entity cache. Scoping by name alone is safe (no engine-owned field is named `name`); `down()` is best-effort (pre-2.15 `false` rows are indistinguishable from flipped ones). **Workspace commands** (idempotent, dry-run supported): 1. **`reconcile-search-vector-gin-index-universal-identifier`** — re-owns every searchVector GIN index UID to its deterministic value (all applications), then backfills the missing GIN index for installed-app objects. 2. **`reconcile-search-field-metadata`** — re-owns every `searchFieldMetadata` UID (all applications), then backfills the missing rows for installed-app searchable objects. 3. **`rebuild-installed-app-search-vectors`** — rebuilds the `searchVector` column of every installed-app `TS_VECTOR` field, once the index and rows exist. Design notes: * **Re-own is global** (twenty-standard, workspace-custom, installed) — a UID convergence keyed on each row's own application. * **Backfill is installed-app only** — standard/custom objects already have these rows via the manifest funnel. * Re-own runs **before** backfill and is transaction-guarded; a failure aborts that workspace to avoid a unique-identifier collision. ## Tests * Integration: app manifest sync now asserts system fields + searchable objects (searchVector, GIN index, searchFieldMetadata) are created; a new relabel suite drives three manifest syncs and asserts records stay searchable through the old + new label identifiers and lose searchability when a field is removed; removed the obsolete system-fields-integrity suite/snapshots. * Unit: per-handler side-effect specs (including the new `objectSearchVectorOnUpdate` and `fieldSearchFieldMetadataOnDelete` handlers), and per-util specs for the re-own / backfill operation builders and the GIN-index classifier. ## Upgrade / migration notes * Existing workspaces converge on the next upgrade run via the `2-20` instance + workspace commands (idempotent, dry-run supported). * Backfill and rebuild go through the workspace-migration runner (automatic cache invalidation); the re-own step invalidates only the affected flat-entity maps directly. * The cross-version upgrade CI now flushes the cache before running the upgrade, so the new version recomputes every flat-entity map from the database instead of reading blobs the old version serialized in an older shape. ## Follow-up * `object-metadata.service.ts` still carries a `TODO: remove once default view fields move to the metadata side effect engine` — default view fields are the next candidate to move into the engine. * A single manifest sync cannot yet both create a field and relabel the object onto it, because `objectMetadata.update` is ordered before `fieldMetadata.create` in the migration runner. Tracked in twentyhq/core-team-issues#2655; to be fixed in a follow-up. |
||
|
Paul Rastoin
|
8842a80a44 |
ci(server): cross-version upgrade check on PRs (v1.22 → from source) (#22065)
## What Adds a pre-merge CI check that proves a database created and seeded by the **oldest supported release** (`twentycrm/twenty:v1.22` from Docker Hub) can be upgraded by the **current version built from source**, and that the upgraded instance comes up healthy with its data still queryable. Runs only on PRs touching the upgrade path (`upgrade-version-command/**` + `core-modules/upgrade/**`), and blocks the PR via `ci-server-status-check`. ## How New reusable workflow `ci-cross-version-upgrade.yaml` (`workflow_call` + `workflow_dispatch`), called from `ci-server.yaml` after `server-build` so the build cache is populated in-run: 1. **Services** — `postgres:16` (prod parity) + `redis:7` on a docker network. 2. **Old version** — pull `twentycrm/twenty:v1.22`, boot it against the DB, `workspace:seed:dev`, sanity-check the seed via `psql`. 3. **New version (from source)** — restore the `server-build` nx cache (best-effort: a miss just cold-builds), `nx build`, run the `upgrade` command against the same DB, `start:ci`, poll `/healthz`. 4. **Smoke** — assert `upgrade:status` shows `Instance: Up to date` / `0 behind, 0 failed`, then run companies/people/metadata GraphQL queries. The job is always invoked but gated by a `skip` input (computed from the upgrade-paths `changed-files` check), with a `no-op` job reporting success when skipped — so the status check always resolves instead of leaving a dangling skipped job, mirroring the twenty-infra pattern. Unlike the equivalent post-merge gate in infra-twenty, this is **pre-merge**, uses **native PR path filtering** (no compare API), and **reuses the from-source build cache** instead of pulling an ECR image — no cross-repo plumbing, no skipped-commit gap. ## Security note No credentials are committed. `APP_SECRET` is generated fresh per run (`openssl rand`, `::add-mask::`'d) and shared between the old container and the from-source server within the job; the smoke-test API token is minted at runtime via `workspace:generate-api-key` against the upgraded server and masked in logs. ## Verified with a real run Validated end-to-end by temporarily touching the upgrade path to trigger the job (trigger commit since dropped), in [CI Server run `28097035272`](https://github.com/twentyhq/twenty/actions/runs/28097035272) → [`cross-version-upgrade` job](https://github.com/twentyhq/twenty/actions/runs/28097035272/job/83189453451) ✅ **all steps green**: - v1.22 container boot → `workspace:seed:dev` → `psql` seed sanity check ✅ - from-source build (nx cache restored) → `upgrade` → **56 workspace(s) succeeded, 0 failed** ✅ - server healthy → API token minted at runtime via `workspace:generate-api-key` ✅ - `upgrade:status` → `Instance: Up to date`, `0 behind, 0 failed` ✅ - companies / people / metadata GraphQL smoke queries ✅ - `no-op` job correctly skipped (real job ran because the gate matched) ✅ The three assumptions originally flagged for first-run all held; one bug was found and fixed in the process — `upgrade:status` colorizes via `chalk` even with `NO_COLOR`, so the assertion now strips ANSI escapes before grepping. > Note: the overall `ci-server` run shows a failure from an **unrelated flaky integration test** (`if-else-workflow.integration-spec.ts`, `column workspaceMember.region does not exist` in shard 11). The same trigger commit passed all 16 integration shards in the prior run — it's a pre-existing flake, not caused by this PR. ## Note Still keeping the equivalent one inside infra-twenty as an final bottleneck just in case <!-- This is an auto-generated description by cubic. --> <a href="https://cubic.dev/pr/twentyhq/twenty/pull/22065?utm_source=github" target="_blank" rel="noopener noreferrer" data-no-image-dialog="true"><picture><source media="(prefers-color-scheme: dark)" srcset="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"><source media="(prefers-color-scheme: light)" srcset="https://www.cubic.dev/buttons/review-in-cubic-light.svg"><img alt="Review in cubic" src="https://www.cubic.dev/buttons/review-in-cubic-light.svg"></picture></a> <!-- End of auto-generated description by cubic. --> |
||
|
Paul Rastoin
|
66857ca77b |
Remove cross version upgrade placeholder (#19940)
Moving to somewhere else to leverage docker cache |
||
|
Paul Rastoin
|
65e01400c0 |
Cross version ci placeholder (#19932)
Created this empty workflow so it appears on main and is pickable from a different branch to start testing the whole flow |
