cdeebb1a185e4d53186464fc5ffe576dfd3b6fd9
2 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Paul Rastoin
|
6c40c7b91a |
Deterministic system field universal identifier (#22565)
# Introduction Close twentyhq/core-team-issues#2641 Auto-provisioned field metadata used to get its `universalIdentifier` from three unrelated sources: random `v4()` on the server when creating custom objects, hardcoded values in `STANDARD_OBJECTS`, and an ad-hoc `v5` derivation in the SDK manifest build. This PR unifies all of them behind the shared `getFieldUniversalIdentifier` derivation: ``` universalIdentifier = f(applicationUniversalIdentifier, objectUniversalIdentifier, fieldName) ``` ## Ownership model The rollout is built on an explicit split of who owns a field's universal identifier: - **The 8 system fields** (`id`, `createdAt`, `updatedAt`, `deletedAt`, `createdBy`, `updatedBy`, `position`, `searchVector`) are **server-owned**. Their universal identifiers are always the deterministic derivation, on **every** application (standard, workspace-custom, installed). Clients cannot provide custom values: a temporary check in `validateObjectMetadataSystemFieldsIntegrity` rejects any non-derived system field identifier at migration build time. This check stands in until system fields are generated exclusively server side by the metadata side-effect engine and stripped from client inputs — at which point it becomes structurally impossible to send one. - **`name` is a default field, not a system field**: it is auto-provisioned when absent (server side for custom objects, SDK side for application objects) but authors can define their own. It is only derived where it is guaranteed to be auto-provisioned. In particular, standard objects keep their **historical hardcoded** `name` identifiers: the standard app authors its `name` fields like any installed app would, and moving those identifiers would break every installed application referencing them (e.g. views on `opportunity.name`). - **User-created and author-provided fields** keep random / explicit identifiers, untouched. ## Server - `validateObjectMetadataSystemFieldsIntegrity` now validates, on top of the existing type/`isSystem` checks, that each system field's `universalIdentifier` equals the deterministic derivation. Runs for every object creation going through the migration orchestrator: app sync, custom object creation, standard provisioning - `build-default-flat-field-metadatas-for-custom-object.util.ts` derives the system field identifiers (and the auto-provisioned `name`) with `getFieldUniversalIdentifier` instead of `v4()` - `build-default-relation-flat-field-metadatas-for-custom-object.util.ts` derives both the forward and the reverse default relation field identifiers deterministically - `generateMorphOrRelationFlatFieldMetadataPair` accepts optional `sourceFieldUniversalIdentifier` / `targetFieldUniversalIdentifier` so callers can inject deterministic values; user-created relations still default to `v4()` ## twenty-shared - `STANDARD_OBJECTS` system field identifiers (the 8) are now computed at module load via `buildStandardObjectSystemFields`; `name` and every other identifier keep their hardcoded values - New snapshot test pinning **every** universal identifier of `STANDARD_OBJECTS`: any identifier change now requires an explicit snapshot update and should ship with a coordinated backfill ## SDK (breaking, pre-GA) - `generateDefaultFieldUniversalIdentifier` delegates to `getFieldUniversalIdentifier` and now requires `applicationUniversalIdentifier` - Reverse default relation field identifiers are derived from the field's real coordinates (standard object UID + actual field name, e.g. `targetRocket` on `attachment`) instead of the legacy custom-object UID + synthetic `${fieldName}Inverse` hash input. Field *names* are unchanged - The manifest build threads the application universal identifier through default field injection (two-pass over object configs) - `twenty dev:add` now resolves the application universal identifier upfront and refuses to scaffold anything until `defineApplication` declares one — no more `fill-later` placeholder for the app UID in generated files ## Upgrade A 2.19 **workspace command** backfills existing `fieldMetadata.universalIdentifier` rows to the deterministic derivation. Coverage follows the ownership model: - **The 8 system fields**: taken over for **every application**, whatever value they currently hold. This is both safe and required now that sync rejects non-derived values — leaving a row unconverged would make its application unsyncable - **`name`**: workspace-custom app → always taken over (server-generated, no author to clobber); installed applications → only rows still carrying the legacy SDK derivation are recomputed, author-provided identifiers are never touched; standard app → never touched (hardcoded in `STANDARD_OBJECTS`) - **Default relation fields**: workspace-custom app → forward fields on custom objects and reverse fields on the standard relation objects; installed applications → legacy-derivation probe only All identifiers of a workspace are updated inside a single transaction, then the command flushes the field-metadata-related workspace caches and bumps the metadata version. Stored `applicationRegistration.manifest` snapshots are intentionally **not** rewritten: installs and upgrades always sync from the `manifest.json` inside the resolved package (npm/tarball), the stored column is only used for display/marketplace purposes. ## Breaking behavior for old packages (fail closed) Packages built with an older SDK carry legacy system field identifiers in their tarball `manifest.json`. Installing or upgrading such a package now fails with an explicit `INVALID_SYSTEM_FIELD` validation error ("universal identifier is not deterministic") instead of silently mismatching against the backfilled rows and triggering a destructive delete+create. The remediation is to rebuild the package with the new SDK; the backfill has already converged the installed rows, so the rebuilt manifest syncs cleanly. ## Test plan - [x] `twenty-sdk` unit tests (526 tests) and typecheck - [x] `twenty-shared` unit tests (1635 tests) including the `STANDARD_OBJECTS` snapshot; `name` identifiers verified byte-for-byte identical to `main` - [x] Lint and typecheck clean on all touched packages - [x] Integration: create a custom object and verify system + default relation field identifiers match the deterministic derivation (`create-one-object-metadata-deterministic-field-universal-identifiers`, 13 assertions passing) - [x] Integration: `failing-sync-application-object-system-fields` extended with a non-derived system field identifier case; all identifiers in the spec pinned deterministically so snapshots embedding expected/actual values are stable across runs (verified with a double run) - [x] Integration: all application sync suites pass with the derived system field identifiers now required by the `buildDefaultObjectManifest` test helper (9 suites, 20 tests) - [x] Full test-database reset: standard app provisioning and seeded workspaces pass the new validation - [x] SDK manifest build verified on the postcard example app: all auto-generated default field identifiers match the derivation - [ ] Run `upgrade:2-19:backfill-deterministic-field-universal-identifiers` (dry-run then real) on a seeded workspace and verify identifier convergence with a rebuilt app manifest |
||
|
Charles Bochet
|
fcaf2b4d9b |
chore(twenty-server): temporary instrumentation for app-install 504 (#21365)
## Why App installs on cloud intermittently fail with a 504, surfacing in Sentry as `Migration action 'update' for 'logicFunction' failed` + `Failed to rollback transaction: Query runner already released`. This is **temporary instrumentation** to pin down where the time goes — to be reverted once the bottleneck is fixed. Everything is greppable via `[install-perf]` and marked `// TODO(install-perf)`. ## What the local repro already told us I instrumented the manifest-sync/migration path and ran a local harness (new skipped spec) installing **1 / 8 / 30 logic functions**, for both create and the checksum-bump **update** (the incident path): | stage (N=30, update) | ms | |---|---| | flat-maps recompute | ~1 | | build migration | ~11 | | transaction (all actions + commit) | ~79 | | post-commit cache invalidate | ~6 | | **full sync** | **~135** | Nothing approached 1s, let alone 10s; no slow queries logged. So the migration/cache code is **not** the algorithmic cause. Given the in-transaction `UPDATE ... WHERE id=?` is intrinsically fast, a >10s in prod almost certainly means it was **blocked on a lock**, and the 10s node-pg `query_timeout` (`core.datasource.ts`) then killed the connection → the observed errors + 504. Local can't reproduce prod lock contention / table sizes, hence this instrumentation. ## What this adds (all `TODO`-marked) - **hrtime per-stage timing** — flat-maps recompute, build vs run, per-action (`>50ms`), transaction summary, post-commit cache invalidation. Uses `process.hrtime` because the integration harness enables fake timers (so `Date.now()` is useless there). - **`maxQueryExecutionTime`** slow-query logging on the core datasource (logs the offending SQL). - **Scoped `SET LOCAL lock_timeout = '8s'`** on the migration transaction (below the 10s `query_timeout`) → a blocked action fails fast with a clear *"canceling statement due to lock timeout"* instead of the opaque connection kill. - **Best-effort `pg_stat_activity` snapshot on failure** (on a fresh pooled connection) to identify the blocking session, plus a **guarded rollback** so a released connection stops masking the real error. - **Skipped local perf harness** (`logic-function-install-performance.integration-spec.ts`) — run manually with `nx test:integration:with-db-reset -- --testPathPattern "logic-function-install-performance"`. ## How we'll use it Deploy, reproduce the failing install, and read the `[install-perf]` logs: the per-action timing names the action, the `lock_timeout` message + `pg_stat_activity` snapshot name the **blocking** query/PID. Then revert this PR and fix the actual contention. Typecheck (`nx typecheck twenty-server`) is clean. |