Centralized side effects devxp basis (#22295)

# Introduction

This PR introduces a centralized, strictly-typed **metadata side-effect
engine** that unifies how system metadata side effects are derived and
applied across both metadata entry points — the **metadata GraphQL API**
and the **application sync / manifest** flow — and migrates the first
side effect end-to-end: **a unique scalar field owns its backing
single-field `UNIQUE` index** (full create / update / delete lifecycle).

## New conventions

- **Engine-owned companions**: metadata flagged `isSystemSideEffect:
true` is owned by the engine. Its deletion is never inferred from
absence in a manifest — it results from PG-level cascade or from a
delete side effect (a side effect always has a cause, its parent
metadata).
- **Reserved deterministic identifiers**: apps cannot declare metadata
reusing an engine-owned deterministic `universalIdentifier`. Doing so
fails validation with `RESERVED_SYSTEM_UNIVERSAL_IDENTIFIER` (until an
explicit override API exists).
- **Record-native operation matrix**: the operation matrix is keyed by
`universalIdentifier` (`AllFlatEntityOperationRecordByMetadataName`)
instead of arrays, making parent resolution and deduplication O(1).
Array-based API callers are transpiled to records at the
validate-build-and-run boundary.
- Twenty-sdk user-facing experience with system fields will only be
related to overrides.

# What this PR does

## 1. Side-effect engine (foundation)

- `MetadataSideEffectEngineService.expandWithSideEffects(...)` takes the
intention-carrying record matrix and returns it expanded with derived
side effects, or a structured failure.
- Handlers are registered via a typed **decorator + registry** pattern
(`MetadataSideEffectHandler({ operation, metadataName, name, description
})`), with runtime duplicate-name detection. Multiple handlers per
(operation, metadataName) are supported.
- Handler contract mirrors the validator pattern:
- receives the trigger flat entity, the live record matrix, and
**strictly-typed related flat entity maps**
(`MetadataFlatEntityAndRelatedFlatEntityMapsForSideEffect<P>`, derived
from declared companion metadata names — no loose
`Partial<AllFlatEntityMaps>` context)
- returns `MetadataSideEffectResult`: `success` (operations record) |
`noop` | `fail` (structured failure)
- **Non-recursion is structural**: triggers are read from the original
caller input, never from the expanded matrix, so a side effect can never
trigger another side effect.
- **Deduplication + collision detection**: side effects are deduped by
`universalIdentifier` per operation; a caller-declared entity colliding
with an engine-owned deterministic identifier is recorded as a
collision.
- **Unified failure channel**: handler failures and reserved-identifier
collisions are merged into the same `OrchestratorFailureReport` contract
as builder validation errors, and the run short-circuits (fail-closed,
nothing is applied).

## 2. First migrated side effect — unique field → backing unique index

Three handlers own the complete lifecycle of the deterministic
single-field `UNIQUE` index backing a unique scalar field:

- **create**: unique scalar field → generate the deterministic backing
index (`fieldUniqueBackingIndexOnCreate`)
- **update**: `isUnique` flips and renames of still-unique fields (the
index name — and therefore its deterministic identifier — derives from
the field name, so a rename drops the stale index and recreates the
deterministic one) (`fieldUniqueBackingIndexOnUpdate`)
- **delete**: cascade-delete the backing index
(`fieldUniqueBackingIndexOnDelete`)

Supporting rules:
- The primary key `id` field never spawns a backing index (uniqueness
comes from the PK constraint) — explicit `isPrimaryKeyFlatFieldMetadata`
guard.
- Parent object resolution is **optimistic-first**: an object created or
updated in the same batch wins over the workspace cache (so e.g.
renaming an object while flipping a field to unique builds the index
from the post-rename object), resolved in O(1) via the record matrix.
- A missing parent object is reported as a structured side-effect
failure, never silently skipped.

## 3. Path convergence — manifest and API share one flow

- The manifest sync now derives a from→to **record matrix** from the
cache and feeds `validateBuildAndRunWorkspaceMigrationFromRecord`, the
same flow the API uses — both paths converge on the engine.
- Manifest-side unique-index generation and API transpiler
system-unique-index handling were removed (declared/composite/relation
indexes stay untouched).
- New `WorkspaceMigrationFlatEntityMapsService` mutualizes
flat-entity-maps computation between the side-effect engine and the
builder: cache keys are derived from the caller metadata names (+
validation- and side-effect-related closures) instead of hardcoded
loads.
- App-scoping and pruning are folded into one shared primitive
(`getSubAllFlatEntityMapsByApplicationIdsOrThrow`): slicing dependency
maps to the involved applications always prunes dangling one-to-many
aggregators — callers can no longer forget it.
- **Behavior change**: an app extending another app's view with a view
field now syncs successfully (cross-app view-field extension), covered
by a dedicated integration test.

## 4. Backfill upgrade command (2.19)

`upgrade:2-19:backfill-system-unique-index-universal-identifier`
rewrites legacy system unique-index `universalIdentifier`s to their
deterministic value so the engine can own pre-existing indexes. The
backfill is **driven from `isUnique: true` fields** (mirroring the
engine ownership predicate — excludes PK / morph / relation fields) and
resolves each field's backing index in O(1).

# Bugs fixed along the way

- `database:reset` seeding failed with
`INDEX_FIELD_INVALID_DEFAULT_VALUE`: the engine derived a backing
`UNIQUE` index for the default `id` primary key. Fixed with the explicit
primary-key guard.
- `isUnique` updates on system-flagged standard fields (e.g.
auto-created `name`) did not trigger the backing-index side effect.
- Manifest sync crashed with "Could not find flat entity with universal
identifier ..." when app-scoped slices left dangling aggregator
references — fixed by centralizing pruning in the shared slice primitive
This commit is contained in:
Paul Rastoin
2026-07-03 18:13:20 +02:00
committed by GitHub
parent 566c3b6629
commit 43730d7748
59 changed files with 2840 additions and 691 deletions
@@ -5,6 +5,7 @@ import { ALL_METADATA_NAME } from 'twenty-shared/metadata';
import { isDefined } from 'twenty-shared/utils';
import { ComputeApplicationManifestAllUniversalFlatEntityMapsService } from 'src/engine/core-modules/application/application-manifest/services/compute-application-manifest-all-universal-flat-entity-maps.service';
import { buildAllFlatEntityOperationRecordByMetadataNameFromFromTo } from 'src/engine/core-modules/application/application-manifest/utils/build-all-flat-entity-operation-record-by-metadata-name-from-from-to.util';
import { buildFromToAllUniversalFlatEntityMaps } from 'src/engine/core-modules/application/application-manifest/utils/build-from-to-all-universal-flat-entity-maps.util';
import { getApplicationSubAllFlatEntityMaps } from 'src/engine/core-modules/application/application-manifest/utils/get-application-sub-all-flat-entity-maps.util';
import {
@@ -173,11 +174,6 @@ export class ApplicationManifestMigrationService {
}> {
const now = new Date().toISOString();
const { twentyStandardFlatApplication } =
await this.applicationService.findWorkspaceTwentyStandardAndCustomApplicationOrThrow(
{ workspaceId },
);
const recomputeStart = performance.now();
const cacheResult = await this.workspaceCacheService.getOrRecompute(
workspaceId,
@@ -193,7 +189,8 @@ export class ApplicationManifestMigrationService {
ApplicationManifestMigrationService.name,
);
const { featureFlagsMap, ...existingAllFlatEntityMaps } = cacheResult;
const { featureFlagsMap: _featureFlagsMap, ...existingAllFlatEntityMaps } =
cacheResult;
const fromAllFlatEntityMaps = getApplicationSubAllFlatEntityMaps({
applicationIds: [ownerFlatApplication.id],
@@ -208,32 +205,27 @@ export class ApplicationManifestMigrationService {
workspaceId,
});
const dependencyAllFlatEntityMaps = getApplicationSubAllFlatEntityMaps({
applicationIds:
ownerFlatApplication.universalIdentifier ===
TWENTY_STANDARD_APPLICATION.universalIdentifier
? [twentyStandardFlatApplication.id]
: [ownerFlatApplication.id, twentyStandardFlatApplication.id],
fromAllFlatEntityMaps: existingAllFlatEntityMaps,
});
const allFlatEntityOperationRecordByMetadataName =
buildAllFlatEntityOperationRecordByMetadataNameFromFromTo({
fromAllFlatEntityMaps,
toAllUniversalFlatEntityMaps,
buildOptions: {
isSystemBuild: false,
inferDeletionFromMissingEntities: true,
applicationUniversalIdentifier:
ownerFlatApplication.universalIdentifier,
},
});
const validateBuildRunStart = performance.now();
const validateAndBuildResult =
await this.workspaceMigrationValidateBuildAndRunService.validateBuildAndRunWorkspaceMigrationFromTo(
await this.workspaceMigrationValidateBuildAndRunService.validateBuildAndRunWorkspaceMigrationFromRecord(
{
buildOptions: {
isSystemBuild: false,
inferDeletionFromMissingEntities: true,
applicationUniversalIdentifier:
ownerFlatApplication.universalIdentifier,
},
fromToAllFlatEntityMaps: buildFromToAllUniversalFlatEntityMaps({
fromAllFlatEntityMaps,
toAllUniversalFlatEntityMaps,
}),
allFlatEntityOperationRecordByMetadataName,
workspaceId,
dependencyAllFlatEntityMaps,
additionalCacheDataMaps: { featureFlagsMap },
isSystemBuild: false,
applicationUniversalIdentifier:
ownerFlatApplication.universalIdentifier,
dryRun,
},
);
@@ -40,7 +40,6 @@ import { fromAgentManifestToUniversalFlatAgent } from 'src/engine/core-modules/a
import { type EncryptedString } from 'src/engine/core-modules/secret-encryption/branded-strings/encrypted-string.type';
import { type PlaintextString } from 'src/engine/core-modules/secret-encryption/branded-strings/plaintext-string.type';
import { SecretEncryptionService } from 'src/engine/core-modules/secret-encryption/secret-encryption.service';
import { generateIndexForFlatFieldMetadata } from 'src/engine/metadata-modules/flat-field-metadata/utils/generate-index-for-flat-field-metadata.util';
import { createEmptyAllFlatEntityMaps } from 'src/engine/metadata-modules/flat-entity/constant/create-empty-all-flat-entity-maps.constant';
import { type AllFlatEntityMaps } from 'src/engine/metadata-modules/flat-entity/types/all-flat-entity-maps.type';
import { type UniversalFlatFieldMetadata } from 'src/engine/workspace-manager/workspace-migration/universal-flat-entity/types/universal-flat-field-metadata.type';
@@ -115,19 +114,6 @@ export class ComputeApplicationManifestAllUniversalFlatEntityMapsService {
universalFlatEntityMapsToMutate:
allUniversalFlatEntityMaps.flatFieldMetadataMaps,
});
if (flatFieldMetadata.isUnique) {
addUniversalFlatEntityToUniversalFlatEntityMapsThroughMutationOrThrow(
{
universalFlatEntity: generateIndexForFlatFieldMetadata({
flatFieldMetadata,
flatObjectMetadata,
}),
universalFlatEntityMapsToMutate:
allUniversalFlatEntityMaps.flatIndexMaps,
},
);
}
}
}
@@ -143,27 +129,6 @@ export class ComputeApplicationManifestAllUniversalFlatEntityMapsService {
universalFlatEntityMapsToMutate:
allUniversalFlatEntityMaps.flatFieldMetadataMaps,
});
if (flatFieldMetadata.isUnique) {
const flatObjectMetadata =
allUniversalFlatEntityMaps.flatObjectMetadataMaps
.byUniversalIdentifier[
flatFieldMetadata.objectMetadataUniversalIdentifier
];
if (isDefined(flatObjectMetadata)) {
addUniversalFlatEntityToUniversalFlatEntityMapsThroughMutationOrThrow(
{
universalFlatEntity: generateIndexForFlatFieldMetadata({
flatFieldMetadata,
flatObjectMetadata,
}),
universalFlatEntityMapsToMutate:
allUniversalFlatEntityMaps.flatIndexMaps,
},
);
}
}
}
const indexCountByObjectUniversalIdentifier = new Map<string, number>();
@@ -0,0 +1,149 @@
import {
ALL_METADATA_NAME,
type AllMetadataName,
} from 'twenty-shared/metadata';
import { isDefined } from 'twenty-shared/utils';
import { type AllFlatEntityMaps } from 'src/engine/metadata-modules/flat-entity/types/all-flat-entity-maps.type';
import {
type AllFlatEntityOperationRecordByMetadataName,
type FlatEntityOperationRecord,
} from 'src/engine/metadata-modules/flat-entity/types/all-flat-entity-operation-record-by-metadata-name.type';
import { type MetadataUniversalFlatEntity } from 'src/engine/metadata-modules/flat-entity/types/metadata-universal-flat-entity.type';
import { getMetadataFlatEntityMapsKey } from 'src/engine/metadata-modules/flat-entity/utils/get-metadata-flat-entity-maps-key.util';
import { isSystemUniqueFlatIndexMetadata } from 'src/engine/metadata-modules/flat-index-metadata/utils/is-system-unique-flat-index-metadata.util';
import { type MetadataUniversalFlatEntityMaps } from 'src/engine/workspace-manager/workspace-migration/universal-flat-entity/types/metadata-universal-flat-entity-maps.type';
import { compareTwoFlatEntity } from 'src/engine/workspace-manager/workspace-migration/universal-flat-entity/utils/compare-two-universal-flat-entity.util';
import { shouldInferDeletionFromMissingEntities } from 'src/engine/workspace-manager/workspace-migration/utils/should-infer-deletion-from-missing-entities.util';
import { type WorkspaceMigrationBuilderOptions } from 'src/engine/workspace-manager/workspace-migration/workspace-migration-builder/types/workspace-migration-builder-options.type';
const toRecordByUniversalIdentifier = <T extends AllMetadataName>(
flatEntities: MetadataUniversalFlatEntity<T>[],
): Record<string, MetadataUniversalFlatEntity<T>> =>
Object.fromEntries(
flatEntities.map((flatEntity) => [
flatEntity.universalIdentifier,
flatEntity,
]),
);
const buildFlatEntityOperationRecordForMetadata = <T extends AllMetadataName>({
metadataName,
fromFlatEntityMaps,
toFlatEntityMaps,
buildOptions,
}: {
metadataName: T;
fromFlatEntityMaps: MetadataUniversalFlatEntityMaps<T>;
toFlatEntityMaps: MetadataUniversalFlatEntityMaps<T>;
buildOptions: WorkspaceMigrationBuilderOptions;
}): FlatEntityOperationRecord<T> => {
const fromByUniversalIdentifier = fromFlatEntityMaps.byUniversalIdentifier;
const toByUniversalIdentifier = toFlatEntityMaps.byUniversalIdentifier;
const flatEntityToCreate = Object.values(toByUniversalIdentifier)
.filter(isDefined)
.filter(
(toFlatEntity) =>
!isDefined(fromByUniversalIdentifier[toFlatEntity.universalIdentifier]),
);
const flatEntityToDelete = shouldInferDeletionFromMissingEntities({
buildOptions,
metadataName,
})
? Object.values(fromByUniversalIdentifier)
.filter(isDefined)
.filter(
(fromFlatEntity) =>
!isDefined(
toByUniversalIdentifier[fromFlatEntity.universalIdentifier],
),
)
.filter((fromFlatEntity) => {
if (metadataName !== ALL_METADATA_NAME.index) {
return true;
}
return !isSystemUniqueFlatIndexMetadata(
fromFlatEntity as unknown as {
isSystemSideEffect: boolean;
isUnique: boolean;
},
);
})
: [];
const flatEntityToUpdate = Object.values(fromByUniversalIdentifier)
.filter(isDefined)
.map((fromFlatEntity) => {
const toFlatEntity =
toByUniversalIdentifier[fromFlatEntity.universalIdentifier];
if (!isDefined(toFlatEntity)) {
return undefined;
}
const update = compareTwoFlatEntity({
fromUniversalFlatEntity: fromFlatEntity,
toUniversalFlatEntity: toFlatEntity,
metadataName,
});
return isDefined(update) ? toFlatEntity : undefined;
})
.filter(isDefined);
return {
flatEntityToCreate: toRecordByUniversalIdentifier(flatEntityToCreate),
flatEntityToUpdate: toRecordByUniversalIdentifier(flatEntityToUpdate),
flatEntityToDelete: toRecordByUniversalIdentifier(flatEntityToDelete),
};
};
export const buildAllFlatEntityOperationRecordByMetadataNameFromFromTo = ({
fromAllFlatEntityMaps,
toAllUniversalFlatEntityMaps,
buildOptions,
}: {
fromAllFlatEntityMaps: AllFlatEntityMaps;
toAllUniversalFlatEntityMaps: AllFlatEntityMaps;
buildOptions: WorkspaceMigrationBuilderOptions;
}): AllFlatEntityOperationRecordByMetadataName => {
const allFlatEntityOperationRecordByMetadataName: AllFlatEntityOperationRecordByMetadataName =
{};
for (const metadataName of Object.values(ALL_METADATA_NAME)) {
const flatEntityMapsKey = getMetadataFlatEntityMapsKey(metadataName);
const flatEntityOperationRecord = buildFlatEntityOperationRecordForMetadata(
{
metadataName,
fromFlatEntityMaps: fromAllFlatEntityMaps[
flatEntityMapsKey
] as unknown as MetadataUniversalFlatEntityMaps<typeof metadataName>,
toFlatEntityMaps: toAllUniversalFlatEntityMaps[
flatEntityMapsKey
] as unknown as MetadataUniversalFlatEntityMaps<typeof metadataName>,
buildOptions,
},
);
if (
Object.keys(flatEntityOperationRecord.flatEntityToCreate).length === 0 &&
Object.keys(flatEntityOperationRecord.flatEntityToUpdate).length === 0 &&
Object.keys(flatEntityOperationRecord.flatEntityToDelete).length === 0
) {
continue;
}
(
allFlatEntityOperationRecordByMetadataName as Record<
string,
FlatEntityOperationRecord<AllMetadataName>
>
)[metadataName] = flatEntityOperationRecord;
}
return allFlatEntityOperationRecordByMetadataName;
};
@@ -1,11 +1,7 @@
import { ALL_METADATA_NAME } from 'twenty-shared/metadata';
import { createEmptyAllFlatEntityMaps } from 'src/engine/metadata-modules/flat-entity/constant/create-empty-all-flat-entity-maps.constant';
import { type AllFlatEntityMaps } from 'src/engine/metadata-modules/flat-entity/types/all-flat-entity-maps.type';
import { type MetadataFlatEntity } from 'src/engine/metadata-modules/flat-entity/types/metadata-flat-entity.type';
import { getMetadataFlatEntityMapsKey } from 'src/engine/metadata-modules/flat-entity/utils/get-metadata-flat-entity-maps-key.util';
import { getSubFlatEntityMapsByApplicationIdsOrThrow } from 'src/engine/metadata-modules/flat-entity/utils/get-sub-flat-entity-maps-by-application-ids-or-throw.util';
import { pruneDanglingForeignKeyAggregatorsInAllFlatEntityMapsThroughMutation } from 'src/engine/metadata-modules/flat-entity/utils/prune-dangling-foreign-key-aggregators-in-all-flat-entity-maps-through-mutation.util';
import { getSubAllFlatEntityMapsByApplicationIdsOrThrow } from 'src/engine/metadata-modules/flat-entity/utils/get-sub-all-flat-entity-maps-by-application-ids-or-throw.util';
export const getApplicationSubAllFlatEntityMaps = ({
applicationIds,
@@ -13,28 +9,9 @@ export const getApplicationSubAllFlatEntityMaps = ({
}: {
applicationIds: string[];
fromAllFlatEntityMaps: AllFlatEntityMaps;
}): AllFlatEntityMaps => {
const emptyAllFlatEntityMaps = createEmptyAllFlatEntityMaps();
for (const metadataName of Object.values(ALL_METADATA_NAME)) {
const flatEntityMapsKey = getMetadataFlatEntityMapsKey(metadataName);
const fromFlatEntityMaps = fromAllFlatEntityMaps[flatEntityMapsKey];
const applicationSubFlatEntityMaps =
getSubFlatEntityMapsByApplicationIdsOrThrow<
MetadataFlatEntity<typeof metadataName>
>({
applicationIds,
flatEntityMaps: fromFlatEntityMaps,
});
// @ts-expect-error Metadata flat entity maps cache key and metadataName colliding
emptyAllFlatEntityMaps[flatEntityMapsKey] = applicationSubFlatEntityMaps;
}
pruneDanglingForeignKeyAggregatorsInAllFlatEntityMapsThroughMutation({
allFlatEntityMapsToMutate: emptyAllFlatEntityMaps,
});
return emptyAllFlatEntityMaps;
};
}): AllFlatEntityMaps =>
getSubAllFlatEntityMapsByApplicationIdsOrThrow({
applicationIds,
metadataNames: Object.values(ALL_METADATA_NAME),
fromAllFlatEntityMaps,
}) as AllFlatEntityMaps;