houseassassin/twenty
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
14,034 Commits 1 Branch 0 Tags
cdeebb1a185e4d53186464fc5ffe576dfd3b6fd9
Commit Graph

2 Commits

Author SHA1 Message Date
Paul Rastoin 6c40c7b91a Deterministic system field universal identifier (#22565) 
# Introduction

Close twentyhq/core-team-issues#2641

Auto-provisioned field metadata used to get its `universalIdentifier`
from three unrelated sources: random `v4()` on the server when creating
custom objects, hardcoded values in `STANDARD_OBJECTS`, and an ad-hoc
`v5` derivation in the SDK manifest build. This PR unifies all of them
behind the shared `getFieldUniversalIdentifier` derivation:

```
universalIdentifier = f(applicationUniversalIdentifier, objectUniversalIdentifier, fieldName)
```

## Ownership model

The rollout is built on an explicit split of who owns a field's
universal identifier:

- **The 8 system fields** (`id`, `createdAt`, `updatedAt`, `deletedAt`,
`createdBy`, `updatedBy`, `position`, `searchVector`) are
**server-owned**. Their universal identifiers are always the
deterministic derivation, on **every** application (standard,
workspace-custom, installed). Clients cannot provide custom values: a
temporary check in `validateObjectMetadataSystemFieldsIntegrity` rejects
any non-derived system field identifier at migration build time. This
check stands in until system fields are generated exclusively server
side by the metadata side-effect engine and stripped from client inputs
— at which point it becomes structurally impossible to send one.
- **`name` is a default field, not a system field**: it is
auto-provisioned when absent (server side for custom objects, SDK side
for application objects) but authors can define their own. It is only
derived where it is guaranteed to be auto-provisioned. In particular,
standard objects keep their **historical hardcoded** `name` identifiers:
the standard app authors its `name` fields like any installed app would,
and moving those identifiers would break every installed application
referencing them (e.g. views on `opportunity.name`).
- **User-created and author-provided fields** keep random / explicit
identifiers, untouched.

## Server

- `validateObjectMetadataSystemFieldsIntegrity` now validates, on top of
the existing type/`isSystem` checks, that each system field's
`universalIdentifier` equals the deterministic derivation. Runs for
every object creation going through the migration orchestrator: app
sync, custom object creation, standard provisioning
- `build-default-flat-field-metadatas-for-custom-object.util.ts` derives
the system field identifiers (and the auto-provisioned `name`) with
`getFieldUniversalIdentifier` instead of `v4()`
-
`build-default-relation-flat-field-metadatas-for-custom-object.util.ts`
derives both the forward and the reverse default relation field
identifiers deterministically
- `generateMorphOrRelationFlatFieldMetadataPair` accepts optional
`sourceFieldUniversalIdentifier` / `targetFieldUniversalIdentifier` so
callers can inject deterministic values; user-created relations still
default to `v4()`

## twenty-shared

- `STANDARD_OBJECTS` system field identifiers (the 8) are now computed
at module load via `buildStandardObjectSystemFields`; `name` and every
other identifier keep their hardcoded values
- New snapshot test pinning **every** universal identifier of
`STANDARD_OBJECTS`: any identifier change now requires an explicit
snapshot update and should ship with a coordinated backfill

## SDK (breaking, pre-GA)

- `generateDefaultFieldUniversalIdentifier` delegates to
`getFieldUniversalIdentifier` and now requires
`applicationUniversalIdentifier`
- Reverse default relation field identifiers are derived from the
field's real coordinates (standard object UID + actual field name, e.g.
`targetRocket` on `attachment`) instead of the legacy custom-object UID
+ synthetic `${fieldName}Inverse` hash input. Field *names* are
unchanged
- The manifest build threads the application universal identifier
through default field injection (two-pass over object configs)
- `twenty dev:add` now resolves the application universal identifier
upfront and refuses to scaffold anything until `defineApplication`
declares one — no more `fill-later` placeholder for the app UID in
generated files

## Upgrade

A 2.19 **workspace command** backfills existing
`fieldMetadata.universalIdentifier` rows to the deterministic
derivation. Coverage follows the ownership model:

- **The 8 system fields**: taken over for **every application**,
whatever value they currently hold. This is both safe and required now
that sync rejects non-derived values — leaving a row unconverged would
make its application unsyncable
- **`name`**: workspace-custom app → always taken over
(server-generated, no author to clobber); installed applications → only
rows still carrying the legacy SDK derivation are recomputed,
author-provided identifiers are never touched; standard app → never
touched (hardcoded in `STANDARD_OBJECTS`)
- **Default relation fields**: workspace-custom app → forward fields on
custom objects and reverse fields on the standard relation objects;
installed applications → legacy-derivation probe only

All identifiers of a workspace are updated inside a single transaction,
then the command flushes the field-metadata-related workspace caches and
bumps the metadata version.

Stored `applicationRegistration.manifest` snapshots are intentionally
**not** rewritten: installs and upgrades always sync from the
`manifest.json` inside the resolved package (npm/tarball), the stored
column is only used for display/marketplace purposes.

## Breaking behavior for old packages (fail closed)

Packages built with an older SDK carry legacy system field identifiers
in their tarball `manifest.json`. Installing or upgrading such a package
now fails with an explicit `INVALID_SYSTEM_FIELD` validation error
("universal identifier is not deterministic") instead of silently
mismatching against the backfilled rows and triggering a destructive
delete+create. The remediation is to rebuild the package with the new
SDK; the backfill has already converged the installed rows, so the
rebuilt manifest syncs cleanly.

## Test plan

- [x] `twenty-sdk` unit tests (526 tests) and typecheck
- [x] `twenty-shared` unit tests (1635 tests) including the
`STANDARD_OBJECTS` snapshot; `name` identifiers verified byte-for-byte
identical to `main`
- [x] Lint and typecheck clean on all touched packages
- [x] Integration: create a custom object and verify system + default
relation field identifiers match the deterministic derivation
(`create-one-object-metadata-deterministic-field-universal-identifiers`,
13 assertions passing)
- [x] Integration: `failing-sync-application-object-system-fields`
extended with a non-derived system field identifier case; all
identifiers in the spec pinned deterministically so snapshots embedding
expected/actual values are stable across runs (verified with a double
run)
- [x] Integration: all application sync suites pass with the derived
system field identifiers now required by the
`buildDefaultObjectManifest` test helper (9 suites, 20 tests)
- [x] Full test-database reset: standard app provisioning and seeded
workspaces pass the new validation
- [x] SDK manifest build verified on the postcard example app: all
auto-generated default field identifiers match the derivation
- [ ] Run
`upgrade:2-19:backfill-deterministic-field-universal-identifiers`
(dry-run then real) on a seeded workspace and verify identifier
convergence with a rebuilt app manifest
 2026-07-06 13:34:33 +00:00
Paul Rastoin 47689e676b feat(server): improve traceability of flat-entity map mutation errors (#22396) 
## Context

cc @rashad 

Twenty applies metadata changes optimistically to in-memory *flat entity
maps* before persisting them. The utils that mutate these maps throw
`FlatEntityMapsException` on invariant violations, which surface in
Sentry (e.g. during `InstallApplication`) as a **hardcoded, generic
message with no identifying data**:

```
GraphQLError: addUniversalFlatEntityToUniversalFlatEntityMapsThroughMutationOrThrow: flat entity to add already exists
```

There was no way to know *which* entity collided — making triage
impossible.

## What this does (two layers)

**Layer 1 — leaf utils emit identifiers**
- `FlatEntityMapsException` gains an optional structured `context`
(`universalIdentifier` / `id` / `applicationId` / `metadataName` /
`relatedMetadataName` / `operation`), read by the Sentry driver's
existing `'context' in exception` → `setExtra` channel.
- All **9 leaf throw sites** append their in-scope identifiers to the
message **and** populate `context`.

**Propagation — context survives the re-wraps**
- On the install path the collision throws in the (unwrapped)
`compute()` step, so the raw exception + context reaches app-sync
intact.
- For the run/build-phase paths, the migration runner and
build-orchestrator re-wraps copy only `.message`; they now also
**forward `context`** so structured data survives there too.

**Layer 2 — human installation error**
- `synchronizeFromManifest` catches flat-entity failures, resolves the
offending `universalIdentifier` to a manifest **object/field label**,
and rethrows `ApplicationException(APPLICATION_INSTALLATION_FAILED)`
with a safe, human `userFriendlyMessage`.
- The leaf `userFriendlyMessage` stays `STANDARD_ERROR_MESSAGE` — the
detailed message never leaks to end users.
- `APPLICATION_INSTALLATION_FAILED` surfaces with the dedicated
`ErrorCode.APPLICATION_INSTALLATION_FAILED` GraphQL code (mirroring the
workspace-migration runner formatter), not `INTERNAL_SERVER_ERROR`.

### Result — client-facing GraphQL error envelope
```json
{
  "extensions": {
    "code": "APPLICATION_INSTALLATION_FAILED",
    "subCode": "APPLICATION_INSTALLATION_FAILED",
    "userFriendlyMessage": "We couldn't install \"Test Application\". Its Invoice could not be applied to your workspace."
  },
  "message": "Installing application 'Test Application' failed [object: Invoice]: addUniversalFlatEntityToUniversalFlatEntityMapsThroughMutationOrThrow: flat entity to add already exists (universalIdentifier: ...)",
  "name": "GraphQLError"
}
```

## Where the identifier shows up (not just Sentry)

The offending `universalIdentifier` reaches every consumer, not only
Sentry:
- **Sentry (server):** structured `context` extras + the enriched
message (fingerprinted by `code`, so no issue fragmentation).
- **GraphQL response `message`:** un-masked (no `useMaskedErrors`; the
error-handler hook passes `BaseGraphQLError` through as-is), so it
travels over the wire.
- **App-author SDK/CLI terminal:** `twenty-sdk` captures
`errors[0].message`; for this error `formatManifestValidationErrors`
returns `null` (no `extensions.errors`/`summary`), so the orchestrator
falls back to printing the full message, e.g.:
  ```
✗ Sync failed with error: Installing application 'X' failed [object:
Invoice]: … already exists (universalIdentifier: b1b2c3d4-…)
ℹ Hint: a metadata conflict was detected. Preview the plan with `yarn
twenty dev --once --dry-run`; …
  ```
The `already exists` / `universalidentifier` substrings also trigger
`getSyncErrorRecoveryHint`, so the author gets an actionable next step.
- **End-user (CRM UI):** only the safe rendered `userFriendlyMessage`
(no UUIDs).

## Design note
`userFriendlyMessage` behaviour of the leaf exceptions is intentionally
unchanged (guardrail). Layer 2 resolves labels for **objects and
fields** (the bulk of metadata); other manifest entity kinds fall back
to an app-name-only human message to avoid brittle manifest-walking —
easy to extend. A future first-class option would be structured
`extensions` (like `METADATA_VALIDATION_FAILED`) + a dedicated SDK
formatter; deferred since the message path already surfaces the detail
in the terminal.

## Tests
- **Unit:** existing through-mutation + runner-exception specs still
pass (they assert on exception **code**, not message). Added a spec for
the enrichment util.
- **Response-format snapshot (verified, green):**
`application-exception-filter.spec.ts` runs the exception filter and
snapshots the exact client-facing GraphQL error envelope shown above.
- **Integration:**
`failing-sync-application-flat-entity-map-conflict.integration-spec.ts`
syncs a manifest whose two objects share a `universalIdentifier`
(collision during manifest map build, before validation) and snapshots
the GraphQL error response via
`expectOneNotInternalServerErrorSnapshot`.
- ⚠️ The integration `.snap` was authored from the identical
deterministic path (verified by the filter unit snapshot) because the
integration suite couldn't be executed in the authoring sandbox. Please
regenerate/confirm with `nx test:integration:with-db-reset` (or `-u`) in
a seeded env.

## Status
Draft — opening for review.
 2026-07-02 12:39:48 +00:00