Deterministic system field universal identifier (#22565)

# Introduction

Close twentyhq/core-team-issues#2641

Auto-provisioned field metadata used to get its `universalIdentifier`
from three unrelated sources: random `v4()` on the server when creating
custom objects, hardcoded values in `STANDARD_OBJECTS`, and an ad-hoc
`v5` derivation in the SDK manifest build. This PR unifies all of them
behind the shared `getFieldUniversalIdentifier` derivation:

```
universalIdentifier = f(applicationUniversalIdentifier, objectUniversalIdentifier, fieldName)
```

## Ownership model

The rollout is built on an explicit split of who owns a field's
universal identifier:

- **The 8 system fields** (`id`, `createdAt`, `updatedAt`, `deletedAt`,
`createdBy`, `updatedBy`, `position`, `searchVector`) are
**server-owned**. Their universal identifiers are always the
deterministic derivation, on **every** application (standard,
workspace-custom, installed). Clients cannot provide custom values: a
temporary check in `validateObjectMetadataSystemFieldsIntegrity` rejects
any non-derived system field identifier at migration build time. This
check stands in until system fields are generated exclusively server
side by the metadata side-effect engine and stripped from client inputs
— at which point it becomes structurally impossible to send one.
- **`name` is a default field, not a system field**: it is
auto-provisioned when absent (server side for custom objects, SDK side
for application objects) but authors can define their own. It is only
derived where it is guaranteed to be auto-provisioned. In particular,
standard objects keep their **historical hardcoded** `name` identifiers:
the standard app authors its `name` fields like any installed app would,
and moving those identifiers would break every installed application
referencing them (e.g. views on `opportunity.name`).
- **User-created and author-provided fields** keep random / explicit
identifiers, untouched.

## Server

- `validateObjectMetadataSystemFieldsIntegrity` now validates, on top of
the existing type/`isSystem` checks, that each system field's
`universalIdentifier` equals the deterministic derivation. Runs for
every object creation going through the migration orchestrator: app
sync, custom object creation, standard provisioning
- `build-default-flat-field-metadatas-for-custom-object.util.ts` derives
the system field identifiers (and the auto-provisioned `name`) with
`getFieldUniversalIdentifier` instead of `v4()`
-
`build-default-relation-flat-field-metadatas-for-custom-object.util.ts`
derives both the forward and the reverse default relation field
identifiers deterministically
- `generateMorphOrRelationFlatFieldMetadataPair` accepts optional
`sourceFieldUniversalIdentifier` / `targetFieldUniversalIdentifier` so
callers can inject deterministic values; user-created relations still
default to `v4()`

## twenty-shared

- `STANDARD_OBJECTS` system field identifiers (the 8) are now computed
at module load via `buildStandardObjectSystemFields`; `name` and every
other identifier keep their hardcoded values
- New snapshot test pinning **every** universal identifier of
`STANDARD_OBJECTS`: any identifier change now requires an explicit
snapshot update and should ship with a coordinated backfill

## SDK (breaking, pre-GA)

- `generateDefaultFieldUniversalIdentifier` delegates to
`getFieldUniversalIdentifier` and now requires
`applicationUniversalIdentifier`
- Reverse default relation field identifiers are derived from the
field's real coordinates (standard object UID + actual field name, e.g.
`targetRocket` on `attachment`) instead of the legacy custom-object UID
+ synthetic `${fieldName}Inverse` hash input. Field *names* are
unchanged
- The manifest build threads the application universal identifier
through default field injection (two-pass over object configs)
- `twenty dev:add` now resolves the application universal identifier
upfront and refuses to scaffold anything until `defineApplication`
declares one — no more `fill-later` placeholder for the app UID in
generated files

## Upgrade

A 2.19 **workspace command** backfills existing
`fieldMetadata.universalIdentifier` rows to the deterministic
derivation. Coverage follows the ownership model:

- **The 8 system fields**: taken over for **every application**,
whatever value they currently hold. This is both safe and required now
that sync rejects non-derived values — leaving a row unconverged would
make its application unsyncable
- **`name`**: workspace-custom app → always taken over
(server-generated, no author to clobber); installed applications → only
rows still carrying the legacy SDK derivation are recomputed,
author-provided identifiers are never touched; standard app → never
touched (hardcoded in `STANDARD_OBJECTS`)
- **Default relation fields**: workspace-custom app → forward fields on
custom objects and reverse fields on the standard relation objects;
installed applications → legacy-derivation probe only

All identifiers of a workspace are updated inside a single transaction,
then the command flushes the field-metadata-related workspace caches and
bumps the metadata version.

Stored `applicationRegistration.manifest` snapshots are intentionally
**not** rewritten: installs and upgrades always sync from the
`manifest.json` inside the resolved package (npm/tarball), the stored
column is only used for display/marketplace purposes.

## Breaking behavior for old packages (fail closed)

Packages built with an older SDK carry legacy system field identifiers
in their tarball `manifest.json`. Installing or upgrading such a package
now fails with an explicit `INVALID_SYSTEM_FIELD` validation error
("universal identifier is not deterministic") instead of silently
mismatching against the backfilled rows and triggering a destructive
delete+create. The remediation is to rebuild the package with the new
SDK; the backfill has already converged the installed rows, so the
rebuilt manifest syncs cleanly.

## Test plan

- [x] `twenty-sdk` unit tests (526 tests) and typecheck
- [x] `twenty-shared` unit tests (1635 tests) including the
`STANDARD_OBJECTS` snapshot; `name` identifiers verified byte-for-byte
identical to `main`
- [x] Lint and typecheck clean on all touched packages
- [x] Integration: create a custom object and verify system + default
relation field identifiers match the deterministic derivation
(`create-one-object-metadata-deterministic-field-universal-identifiers`,
13 assertions passing)
- [x] Integration: `failing-sync-application-object-system-fields`
extended with a non-derived system field identifier case; all
identifiers in the spec pinned deterministically so snapshots embedding
expected/actual values are stable across runs (verified with a double
run)
- [x] Integration: all application sync suites pass with the derived
system field identifiers now required by the
`buildDefaultObjectManifest` test helper (9 suites, 20 tests)
- [x] Full test-database reset: standard app provisioning and seeded
workspaces pass the new validation
- [x] SDK manifest build verified on the postcard example app: all
auto-generated default field identifiers match the derivation
- [ ] Run
`upgrade:2-19:backfill-deterministic-field-universal-identifiers`
(dry-run then real) on a seeded workspace and verify identifier
convergence with a rebuilt app manifest
This commit is contained in:
Paul Rastoin
2026-07-06 15:34:33 +02:00
committed by GitHub
parent 0706c7c1bc
commit 6c40c7b91a
68 changed files with 5060 additions and 1336 deletions
@@ -4,13 +4,16 @@ import { TypeOrmModule } from '@nestjs/typeorm';
import { WorkspaceIteratorModule } from 'src/database/commands/command-runners/workspace-iterator.module';
import { BackfillWorkspaceCustomApplicationRegistrationCommand } from 'src/database/commands/upgrade-version-command/2-19/2-19-workspace-command-1782853718000-backfill-workspace-custom-application-registration.command';
import { BackfillSystemUniqueIndexUniversalIdentifierCommand } from 'src/database/commands/upgrade-version-command/2-19/2-19-workspace-command-1783093620000-backfill-system-unique-index-universal-identifier.command';
import { BackfillDeterministicFieldUniversalIdentifiersCommand } from 'src/database/commands/upgrade-version-command/2-19/2-19-workspace-command-1783100000000-backfill-deterministic-field-universal-identifiers.command';
import { ApplicationEntity } from 'src/engine/core-modules/application/application.entity';
import { ApplicationModule } from 'src/engine/core-modules/application/application.module';
import { WorkspaceEntity } from 'src/engine/core-modules/workspace/workspace.entity';
import { FieldMetadataEntity } from 'src/engine/metadata-modules/field-metadata/field-metadata.entity';
import { IndexMetadataEntity } from 'src/engine/metadata-modules/index-metadata/index-metadata.entity';
import { WorkspaceMetadataVersionModule } from 'src/engine/metadata-modules/workspace-metadata-version/workspace-metadata-version.module';
import { WorkspaceCacheModule } from 'src/engine/workspace-cache/workspace-cache.module';
import { WorkspaceMigrationModule } from 'src/engine/workspace-manager/workspace-migration/workspace-migration.module';
import { WorkspaceMigrationRunnerModule } from 'src/engine/workspace-manager/workspace-migration/workspace-migration-runner/workspace-migration-runner.module';
@Module({
imports: [
@@ -18,16 +21,19 @@ import { WorkspaceMigrationModule } from 'src/engine/workspace-manager/workspace
TypeOrmModule.forFeature([
WorkspaceEntity,
ApplicationEntity,
FieldMetadataEntity,
IndexMetadataEntity,
]),
WorkspaceIteratorModule,
WorkspaceCacheModule,
WorkspaceMetadataVersionModule,
WorkspaceMigrationModule,
WorkspaceMigrationRunnerModule,
],
providers: [
BackfillWorkspaceCustomApplicationRegistrationCommand,
BackfillSystemUniqueIndexUniversalIdentifierCommand,
BackfillDeterministicFieldUniversalIdentifiersCommand,
],
})
export class V2_19_UpgradeVersionCommandModule {}
@@ -0,0 +1,368 @@
import { InjectRepository } from '@nestjs/typeorm';
import { Command } from 'nest-commander';
import { getFieldUniversalIdentifier } from 'twenty-shared/application';
import { STANDARD_OBJECTS } from 'twenty-shared/metadata';
import { FieldMetadataType } from 'twenty-shared/types';
import { capitalize, isDefined } from 'twenty-shared/utils';
import { Repository } from 'typeorm';
import { v5 } from 'uuid';
import { ActiveOrSuspendedWorkspaceCommandRunner } from 'src/database/commands/command-runners/active-or-suspended-workspace.command-runner';
import { WorkspaceIteratorService } from 'src/database/commands/command-runners/workspace-iterator.service';
import { type RunOnWorkspaceArgs } from 'src/database/commands/command-runners/workspace.command-runner';
import { ApplicationEntity } from 'src/engine/core-modules/application/application.entity';
import { ApplicationService } from 'src/engine/core-modules/application/application.service';
import { RegisteredWorkspaceCommand } from 'src/engine/core-modules/upgrade/decorators/registered-workspace-command.decorator';
import { FieldMetadataEntity } from 'src/engine/metadata-modules/field-metadata/field-metadata.entity';
import { type FlatEntityMaps } from 'src/engine/metadata-modules/flat-entity/types/flat-entity-maps.type';
import { findFlatEntityByIdInFlatEntityMaps } from 'src/engine/metadata-modules/flat-entity/utils/find-flat-entity-by-id-in-flat-entity-maps.util';
import { getMetadataFlatEntityMapsKey } from 'src/engine/metadata-modules/flat-entity/utils/get-metadata-flat-entity-maps-key.util';
import { getMetadataRelatedMetadataNames } from 'src/engine/metadata-modules/flat-entity/utils/get-metadata-related-metadata-names.util';
import { getMetadataSerializedRelationNames } from 'src/engine/metadata-modules/flat-entity/utils/get-metadata-serialized-relation-names.util';
import { type FlatFieldMetadata } from 'src/engine/metadata-modules/flat-field-metadata/types/flat-field-metadata.type';
import { type FlatObjectMetadata } from 'src/engine/metadata-modules/flat-object-metadata/types/flat-object-metadata.type';
import { PARTIAL_SYSTEM_FLAT_FIELD_METADATAS } from 'src/engine/metadata-modules/object-metadata/constants/partial-system-flat-field-metadatas.constant';
import { WorkspaceCacheService } from 'src/engine/workspace-cache/services/workspace-cache.service';
import { WorkspaceMigrationRunnerService } from 'src/engine/workspace-manager/workspace-migration/workspace-migration-runner/services/workspace-migration-runner.service';
// Server-owned system field names: their universal identifiers are taken
// over for every application, whatever value they currently hold, since
// validateObjectMetadataSystemFieldsIntegrity now rejects any non-derived
// value at sync time. Fields are picked explicitly so system fields added to
// PARTIAL_SYSTEM_FLAT_FIELD_METADATAS later never alter this shipped
// migration.
const SYSTEM_FIELD_NAMES = new Set<string>([
PARTIAL_SYSTEM_FLAT_FIELD_METADATAS.id.name,
PARTIAL_SYSTEM_FLAT_FIELD_METADATAS.createdAt.name,
PARTIAL_SYSTEM_FLAT_FIELD_METADATAS.updatedAt.name,
PARTIAL_SYSTEM_FLAT_FIELD_METADATAS.deletedAt.name,
PARTIAL_SYSTEM_FLAT_FIELD_METADATAS.createdBy.name,
PARTIAL_SYSTEM_FLAT_FIELD_METADATAS.updatedBy.name,
PARTIAL_SYSTEM_FLAT_FIELD_METADATAS.position.name,
PARTIAL_SYSTEM_FLAT_FIELD_METADATAS.searchVector.name,
]);
// The name field is a default field, not a system field: it is only taken
// over where it is guaranteed to be auto-provisioned (workspace-custom
// objects). For installed applications authors may define it themselves, so
// only legacy SDK-derived values are converged; for the standard application
// it is author-provided in STANDARD_OBJECTS and keeps its hardcoded value.
const NAME_FIELD_NAME = 'name';
const DEFAULT_RELATION_FORWARD_FIELD_NAMES = new Set([
'timelineActivities',
'attachments',
'noteTargets',
'taskTargets',
]);
const DEFAULT_RELATION_STANDARD_OBJECT_UNIVERSAL_IDENTIFIERS = new Set<string>([
STANDARD_OBJECTS.timelineActivity.universalIdentifier,
STANDARD_OBJECTS.attachment.universalIdentifier,
STANDARD_OBJECTS.noteTarget.universalIdentifier,
STANDARD_OBJECTS.taskTarget.universalIdentifier,
]);
// Namespace the SDK used before default field universal identifiers were
// aligned with getFieldUniversalIdentifier; only rows still carrying a
// legacy-derived value are backfilled for installed applications, so
// author-provided identifiers are left untouched.
const LEGACY_SDK_UNIVERSAL_IDENTIFIER_NAMESPACE =
'142046f0-4d80-48b5-ad56-26ad410e895c';
const computeLegacySdkDefaultFieldUniversalIdentifier = ({
objectUniversalIdentifier,
fieldName,
}: {
objectUniversalIdentifier: string;
fieldName: string;
}) =>
v5(
`${objectUniversalIdentifier}-${fieldName}`,
LEGACY_SDK_UNIVERSAL_IDENTIFIER_NAMESPACE,
);
const isMorphOrRelationFieldMetadataType = (type: FieldMetadataType) =>
type === FieldMetadataType.RELATION ||
type === FieldMetadataType.MORPH_RELATION;
@RegisteredWorkspaceCommand('2.19.0', 1783100000000)
@Command({
name: 'upgrade:2-19:backfill-deterministic-field-universal-identifiers',
description:
'Recompute the universal identifier of auto-provisioned field metadata (system fields and default relation fields) to the deterministic getFieldUniversalIdentifier derivation.',
})
export class BackfillDeterministicFieldUniversalIdentifiersCommand extends ActiveOrSuspendedWorkspaceCommandRunner {
constructor(
protected readonly workspaceIteratorService: WorkspaceIteratorService,
private readonly applicationService: ApplicationService,
private readonly workspaceCacheService: WorkspaceCacheService,
private readonly workspaceMigrationRunnerService: WorkspaceMigrationRunnerService,
@InjectRepository(ApplicationEntity)
private readonly applicationRepository: Repository<ApplicationEntity>,
@InjectRepository(FieldMetadataEntity)
private readonly fieldMetadataRepository: Repository<FieldMetadataEntity>,
) {
super(workspaceIteratorService);
}
override async runOnWorkspace({
workspaceId,
options,
}: RunOnWorkspaceArgs): Promise<void> {
const isDryRun = options.dryRun ?? false;
const { twentyStandardFlatApplication, workspaceCustomFlatApplication } =
await this.applicationService.findWorkspaceTwentyStandardAndCustomApplicationOrThrow(
{ workspaceId },
);
const applications = await this.applicationRepository.find({
select: ['id', 'universalIdentifier'],
where: { workspaceId },
withDeleted: true,
});
const applicationUniversalIdentifierById = new Map(
applications.map((application) => [
application.id,
application.universalIdentifier,
]),
);
const { flatFieldMetadataMaps, flatObjectMetadataMaps } =
await this.workspaceCacheService.getOrRecompute(workspaceId, [
'flatFieldMetadataMaps',
'flatObjectMetadataMaps',
]);
const updates: { id: string; newUniversalIdentifier: string }[] = [];
for (const flatFieldMetadata of Object.values(
flatFieldMetadataMaps.byUniversalIdentifier,
)) {
if (!isDefined(flatFieldMetadata)) {
continue;
}
const flatObjectMetadata = findFlatEntityByIdInFlatEntityMaps({
flatEntityMaps: flatObjectMetadataMaps,
flatEntityId: flatFieldMetadata.objectMetadataId,
});
const applicationUniversalIdentifier =
applicationUniversalIdentifierById.get(flatFieldMetadata.applicationId);
if (
!isDefined(flatObjectMetadata) ||
!isDefined(applicationUniversalIdentifier)
) {
this.logger.warn(
`Missing object or application for field ${flatFieldMetadata.name} (${flatFieldMetadata.id}) in workspace ${workspaceId}, skipping`,
);
continue;
}
const shouldBackfill = this.shouldBackfillFieldMetadata({
flatFieldMetadata,
flatObjectMetadata,
flatObjectMetadataMaps,
flatFieldMetadataMaps,
twentyStandardApplicationId: twentyStandardFlatApplication.id,
workspaceCustomApplicationId: workspaceCustomFlatApplication.id,
});
if (!shouldBackfill) {
continue;
}
const newUniversalIdentifier = getFieldUniversalIdentifier({
applicationUniversalIdentifier,
objectUniversalIdentifier: flatObjectMetadata.universalIdentifier,
name: flatFieldMetadata.name,
});
if (newUniversalIdentifier === flatFieldMetadata.universalIdentifier) {
continue;
}
updates.push({ id: flatFieldMetadata.id, newUniversalIdentifier });
}
if (updates.length === 0) {
this.logger.log(
`No field universal identifiers to backfill for workspace ${workspaceId}`,
);
return;
}
this.logger.log(
`${isDryRun ? '[DRY RUN] ' : ''}Backfilling ${updates.length} field universal identifier(s) for workspace ${workspaceId}`,
);
if (isDryRun) {
return;
}
await this.fieldMetadataRepository.manager.transaction(
async (entityManager) => {
const transactionalFieldMetadataRepository =
entityManager.getRepository(FieldMetadataEntity);
for (const { id, newUniversalIdentifier } of updates) {
await transactionalFieldMetadataRepository.update(
{ id, workspaceId },
{ universalIdentifier: newUniversalIdentifier },
);
}
},
);
const fieldMetadataRelatedNames = [
'fieldMetadata',
...getMetadataRelatedMetadataNames('fieldMetadata'),
...getMetadataSerializedRelationNames('fieldMetadata'),
'index',
] as const;
const allFlatEntityMapsKeys = [
...new Set(fieldMetadataRelatedNames.map(getMetadataFlatEntityMapsKey)),
];
await this.workspaceMigrationRunnerService.invalidateCache({
allFlatEntityMapsKeys,
workspaceId,
});
this.logger.log(
`Backfilled ${updates.length} field universal identifier(s) for workspace ${workspaceId}`,
);
}
private shouldBackfillFieldMetadata({
flatFieldMetadata,
flatObjectMetadata,
flatObjectMetadataMaps,
flatFieldMetadataMaps,
twentyStandardApplicationId,
workspaceCustomApplicationId,
}: {
flatFieldMetadata: FlatFieldMetadata;
flatObjectMetadata: FlatObjectMetadata;
flatObjectMetadataMaps: FlatEntityMaps<FlatObjectMetadata>;
flatFieldMetadataMaps: FlatEntityMaps<FlatFieldMetadata>;
twentyStandardApplicationId: string;
workspaceCustomApplicationId: string;
}): boolean {
// System field universal identifiers are server-owned: they are taken
// over for every application. Sync now rejects non-derived values
// (validateObjectMetadataSystemFieldsIntegrity), so converging every row
// here is both safe and required.
if (SYSTEM_FIELD_NAMES.has(flatFieldMetadata.name)) {
return true;
}
// Standard application: every non-system standard field (including name)
// is author-provided in STANDARD_OBJECTS and keeps its hardcoded value.
if (flatFieldMetadata.applicationId === twentyStandardApplicationId) {
return false;
}
const relationTargetFlatObjectMetadata = isDefined(
flatFieldMetadata.relationTargetObjectMetadataId,
)
? findFlatEntityByIdInFlatEntityMaps({
flatEntityMaps: flatObjectMetadataMaps,
flatEntityId: flatFieldMetadata.relationTargetObjectMetadataId,
})
: undefined;
if (flatFieldMetadata.applicationId === workspaceCustomApplicationId) {
if (flatFieldMetadata.name === NAME_FIELD_NAME) {
return true;
}
if (!isMorphOrRelationFieldMetadataType(flatFieldMetadata.type)) {
return false;
}
// Forward default relation fields on custom objects (attachments,
// noteTargets, taskTargets, timelineActivities).
if (
DEFAULT_RELATION_FORWARD_FIELD_NAMES.has(flatFieldMetadata.name) &&
isDefined(relationTargetFlatObjectMetadata) &&
DEFAULT_RELATION_STANDARD_OBJECT_UNIVERSAL_IDENTIFIERS.has(
relationTargetFlatObjectMetadata.universalIdentifier,
)
) {
return true;
}
// Reverse default relation fields living on the standard relation
// objects and pointing back at the custom object.
if (
DEFAULT_RELATION_STANDARD_OBJECT_UNIVERSAL_IDENTIFIERS.has(
flatObjectMetadata.universalIdentifier,
) &&
isDefined(relationTargetFlatObjectMetadata) &&
(flatFieldMetadata.name ===
`target${capitalize(relationTargetFlatObjectMetadata.nameSingular)}` ||
flatFieldMetadata.name ===
relationTargetFlatObjectMetadata.nameSingular)
) {
return true;
}
return false;
}
// Installed applications, remaining auto-provisioned fields (injected
// name field and default relation fields): only rows still carrying an
// SDK-auto-generated (legacy derivation) identifier are backfilled;
// author-provided identifiers must keep matching the application source
// code.
const legacyDefaultUniversalIdentifier =
computeLegacySdkDefaultFieldUniversalIdentifier({
objectUniversalIdentifier: flatObjectMetadata.universalIdentifier,
fieldName: flatFieldMetadata.name,
});
if (
flatFieldMetadata.universalIdentifier === legacyDefaultUniversalIdentifier
) {
return true;
}
// Legacy reverse default relation fields were derived from the custom
// object universal identifier and the forward field name suffixed with
// "Inverse".
const relationTargetFlatFieldMetadata = isDefined(
flatFieldMetadata.relationTargetFieldMetadataId,
)
? findFlatEntityByIdInFlatEntityMaps({
flatEntityMaps: flatFieldMetadataMaps,
flatEntityId: flatFieldMetadata.relationTargetFieldMetadataId,
})
: undefined;
if (
isDefined(relationTargetFlatObjectMetadata) &&
isDefined(relationTargetFlatFieldMetadata)
) {
const legacyReverseUniversalIdentifier =
computeLegacySdkDefaultFieldUniversalIdentifier({
objectUniversalIdentifier:
relationTargetFlatObjectMetadata.universalIdentifier,
fieldName: `${relationTargetFlatFieldMetadata.name}Inverse`,
});
if (
flatFieldMetadata.universalIdentifier ===
legacyReverseUniversalIdentifier
) {
return true;
}
}
return false;
}
}