Centralized side effects devxp basis (#22295)

# Introduction

This PR introduces a centralized, strictly-typed **metadata side-effect
engine** that unifies how system metadata side effects are derived and
applied across both metadata entry points — the **metadata GraphQL API**
and the **application sync / manifest** flow — and migrates the first
side effect end-to-end: **a unique scalar field owns its backing
single-field `UNIQUE` index** (full create / update / delete lifecycle).

## New conventions

- **Engine-owned companions**: metadata flagged `isSystemSideEffect:
true` is owned by the engine. Its deletion is never inferred from
absence in a manifest — it results from PG-level cascade or from a
delete side effect (a side effect always has a cause, its parent
metadata).
- **Reserved deterministic identifiers**: apps cannot declare metadata
reusing an engine-owned deterministic `universalIdentifier`. Doing so
fails validation with `RESERVED_SYSTEM_UNIVERSAL_IDENTIFIER` (until an
explicit override API exists).
- **Record-native operation matrix**: the operation matrix is keyed by
`universalIdentifier` (`AllFlatEntityOperationRecordByMetadataName`)
instead of arrays, making parent resolution and deduplication O(1).
Array-based API callers are transpiled to records at the
validate-build-and-run boundary.
- Twenty-sdk user-facing experience with system fields will only be
related to overrides.

# What this PR does

## 1. Side-effect engine (foundation)

- `MetadataSideEffectEngineService.expandWithSideEffects(...)` takes the
intention-carrying record matrix and returns it expanded with derived
side effects, or a structured failure.
- Handlers are registered via a typed **decorator + registry** pattern
(`MetadataSideEffectHandler({ operation, metadataName, name, description
})`), with runtime duplicate-name detection. Multiple handlers per
(operation, metadataName) are supported.
- Handler contract mirrors the validator pattern:
- receives the trigger flat entity, the live record matrix, and
**strictly-typed related flat entity maps**
(`MetadataFlatEntityAndRelatedFlatEntityMapsForSideEffect<P>`, derived
from declared companion metadata names — no loose
`Partial<AllFlatEntityMaps>` context)
- returns `MetadataSideEffectResult`: `success` (operations record) |
`noop` | `fail` (structured failure)
- **Non-recursion is structural**: triggers are read from the original
caller input, never from the expanded matrix, so a side effect can never
trigger another side effect.
- **Deduplication + collision detection**: side effects are deduped by
`universalIdentifier` per operation; a caller-declared entity colliding
with an engine-owned deterministic identifier is recorded as a
collision.
- **Unified failure channel**: handler failures and reserved-identifier
collisions are merged into the same `OrchestratorFailureReport` contract
as builder validation errors, and the run short-circuits (fail-closed,
nothing is applied).

## 2. First migrated side effect — unique field → backing unique index

Three handlers own the complete lifecycle of the deterministic
single-field `UNIQUE` index backing a unique scalar field:

- **create**: unique scalar field → generate the deterministic backing
index (`fieldUniqueBackingIndexOnCreate`)
- **update**: `isUnique` flips and renames of still-unique fields (the
index name — and therefore its deterministic identifier — derives from
the field name, so a rename drops the stale index and recreates the
deterministic one) (`fieldUniqueBackingIndexOnUpdate`)
- **delete**: cascade-delete the backing index
(`fieldUniqueBackingIndexOnDelete`)

Supporting rules:
- The primary key `id` field never spawns a backing index (uniqueness
comes from the PK constraint) — explicit `isPrimaryKeyFlatFieldMetadata`
guard.
- Parent object resolution is **optimistic-first**: an object created or
updated in the same batch wins over the workspace cache (so e.g.
renaming an object while flipping a field to unique builds the index
from the post-rename object), resolved in O(1) via the record matrix.
- A missing parent object is reported as a structured side-effect
failure, never silently skipped.

## 3. Path convergence — manifest and API share one flow

- The manifest sync now derives a from→to **record matrix** from the
cache and feeds `validateBuildAndRunWorkspaceMigrationFromRecord`, the
same flow the API uses — both paths converge on the engine.
- Manifest-side unique-index generation and API transpiler
system-unique-index handling were removed (declared/composite/relation
indexes stay untouched).
- New `WorkspaceMigrationFlatEntityMapsService` mutualizes
flat-entity-maps computation between the side-effect engine and the
builder: cache keys are derived from the caller metadata names (+
validation- and side-effect-related closures) instead of hardcoded
loads.
- App-scoping and pruning are folded into one shared primitive
(`getSubAllFlatEntityMapsByApplicationIdsOrThrow`): slicing dependency
maps to the involved applications always prunes dangling one-to-many
aggregators — callers can no longer forget it.
- **Behavior change**: an app extending another app's view with a view
field now syncs successfully (cross-app view-field extension), covered
by a dedicated integration test.

## 4. Backfill upgrade command (2.19)

`upgrade:2-19:backfill-system-unique-index-universal-identifier`
rewrites legacy system unique-index `universalIdentifier`s to their
deterministic value so the engine can own pre-existing indexes. The
backfill is **driven from `isUnique: true` fields** (mirroring the
engine ownership predicate — excludes PK / morph / relation fields) and
resolves each field's backing index in O(1).

# Bugs fixed along the way

- `database:reset` seeding failed with
`INDEX_FIELD_INVALID_DEFAULT_VALUE`: the engine derived a backing
`UNIQUE` index for the default `id` primary key. Fixed with the explicit
primary-key guard.
- `isUnique` updates on system-flagged standard fields (e.g.
auto-created `name`) did not trigger the backing-index side effect.
- Manifest sync crashed with "Could not find flat entity with universal
identifier ..." when app-scoped slices left dangling aggregator
references — fixed by centralizing pruning in the shared slice primitive
This commit is contained in:
Paul Rastoin
2026-07-03 18:13:20 +02:00
committed by GitHub
parent 566c3b6629
commit 43730d7748
59 changed files with 2840 additions and 691 deletions
@@ -0,0 +1,2 @@
export const METADATA_SIDE_EFFECT_HANDLER_METADATA_KEY =
'METADATA_SIDE_EFFECT_HANDLER_METADATA_KEY';
@@ -0,0 +1,4 @@
export enum MetadataSideEffectExceptionCode {
RESERVED_SYSTEM_UNIVERSAL_IDENTIFIER = 'RESERVED_SYSTEM_UNIVERSAL_IDENTIFIER',
SIDE_EFFECT_PARENT_METADATA_NOT_FOUND = 'SIDE_EFFECT_PARENT_METADATA_NOT_FOUND',
}
@@ -0,0 +1,74 @@
import { Injectable } from '@nestjs/common';
import { isDefined } from 'twenty-shared/utils';
import { generateDeterministicIndexForFlatFieldMetadataOrThrow } from 'src/engine/metadata-modules/flat-field-metadata/utils/generate-deterministic-index-for-flat-field-metadata-or-throw.util';
import { isMorphOrRelationUniversalFlatFieldMetadata } from 'src/engine/metadata-modules/flat-field-metadata/utils/is-morph-or-relation-flat-field-metadata.util';
import { isPrimaryKeyFlatFieldMetadata } from 'src/engine/metadata-modules/flat-field-metadata/utils/is-primary-key-flat-field-metadata.util';
import { buildFieldSideEffectParentNotFoundFailure } from 'src/engine/metadata-modules/metadata-side-effect/handlers/field-metadata/utils/build-field-side-effect-parent-not-found-failure.util';
import { resolveParentFlatObjectMetadataAfterStateForFieldSideEffect } from 'src/engine/metadata-modules/metadata-side-effect/handlers/field-metadata/utils/resolve-parent-flat-object-metadata-after-state-for-field-side-effect.util';
import {
type BuildSideEffectsArgs,
MetadataSideEffectHandler,
} from 'src/engine/metadata-modules/metadata-side-effect/interfaces/base-metadata-side-effect-handler.service';
import { type MetadataSideEffectResult } from 'src/engine/metadata-modules/metadata-side-effect/types/metadata-side-effect-result.type';
@Injectable()
export class FieldUniqueBackingIndexOnCreateSideEffectHandlerService extends MetadataSideEffectHandler(
{
operation: 'create',
metadataName: 'fieldMetadata',
name: 'fieldUniqueBackingIndexOnCreate',
description:
'When a unique scalar field is created, generate the single-field UNIQUE index that enforces its uniqueness constraint at the database level.',
},
) {
buildSideEffects({
flatEntity: flatFieldMetadata,
allFlatEntityOperationRecordByMetadataName,
relatedFlatEntityMaps,
}: BuildSideEffectsArgs<'fieldMetadata'>): MetadataSideEffectResult {
if (
flatFieldMetadata.isUnique !== true ||
isMorphOrRelationUniversalFlatFieldMetadata(flatFieldMetadata)
) {
return { status: 'noop' };
}
if (isPrimaryKeyFlatFieldMetadata(flatFieldMetadata)) {
return { status: 'noop' };
}
const parentFlatObjectMetadata =
resolveParentFlatObjectMetadataAfterStateForFieldSideEffect({
objectMetadataUniversalIdentifier:
flatFieldMetadata.objectMetadataUniversalIdentifier,
allFlatEntityOperationRecordByMetadataName,
relatedFlatEntityMaps,
});
if (!isDefined(parentFlatObjectMetadata)) {
return buildFieldSideEffectParentNotFoundFailure({
flatFieldMetadata,
operation: 'create',
});
}
const flatIndexMetadata =
generateDeterministicIndexForFlatFieldMetadataOrThrow({
flatFieldMetadata,
flatObjectMetadata: parentFlatObjectMetadata,
});
return {
status: 'success',
operations: {
index: {
flatEntityToCreate: {
[flatIndexMetadata.universalIdentifier]: flatIndexMetadata,
},
},
},
};
}
}
@@ -0,0 +1,74 @@
import { Injectable } from '@nestjs/common';
import { isDefined } from 'twenty-shared/utils';
import { generateDeterministicIndexForFlatFieldMetadataOrThrow } from 'src/engine/metadata-modules/flat-field-metadata/utils/generate-deterministic-index-for-flat-field-metadata-or-throw.util';
import { isMorphOrRelationUniversalFlatFieldMetadata } from 'src/engine/metadata-modules/flat-field-metadata/utils/is-morph-or-relation-flat-field-metadata.util';
import { buildFieldSideEffectParentNotFoundFailure } from 'src/engine/metadata-modules/metadata-side-effect/handlers/field-metadata/utils/build-field-side-effect-parent-not-found-failure.util';
import {
type BuildSideEffectsArgs,
MetadataSideEffectHandler,
} from 'src/engine/metadata-modules/metadata-side-effect/interfaces/base-metadata-side-effect-handler.service';
import { type MetadataSideEffectResult } from 'src/engine/metadata-modules/metadata-side-effect/types/metadata-side-effect-result.type';
@Injectable()
export class FieldUniqueBackingIndexOnDeleteSideEffectHandlerService extends MetadataSideEffectHandler(
{
operation: 'delete',
metadataName: 'fieldMetadata',
name: 'fieldUniqueBackingIndexOnDelete',
description:
'When a unique scalar field is deleted, cascade-delete the single-field UNIQUE index that backed its uniqueness constraint.',
},
) {
buildSideEffects({
flatEntity: flatFieldMetadata,
relatedFlatEntityMaps,
}: BuildSideEffectsArgs<'fieldMetadata'>): MetadataSideEffectResult {
if (
flatFieldMetadata.isUnique !== true ||
isMorphOrRelationUniversalFlatFieldMetadata(flatFieldMetadata)
) {
return { status: 'noop' };
}
const parentFlatObjectMetadata =
relatedFlatEntityMaps.flatObjectMetadataMaps.byUniversalIdentifier[
flatFieldMetadata.objectMetadataUniversalIdentifier
];
if (!isDefined(parentFlatObjectMetadata)) {
return buildFieldSideEffectParentNotFoundFailure({
flatFieldMetadata,
operation: 'delete',
});
}
const flatIndexMetadataToDelete =
generateDeterministicIndexForFlatFieldMetadataOrThrow({
flatFieldMetadata,
flatObjectMetadata: parentFlatObjectMetadata,
});
const indexExistsInWorkspace = isDefined(
relatedFlatEntityMaps.flatIndexMaps.byUniversalIdentifier[
flatIndexMetadataToDelete.universalIdentifier
],
);
if (!indexExistsInWorkspace) {
return { status: 'noop' };
}
return {
status: 'success',
operations: {
index: {
flatEntityToDelete: {
[flatIndexMetadataToDelete.universalIdentifier]:
flatIndexMetadataToDelete,
},
},
},
};
}
}
@@ -0,0 +1,149 @@
import { Injectable } from '@nestjs/common';
import { isDefined } from 'twenty-shared/utils';
import { generateDeterministicIndexForFlatFieldMetadataOrThrow } from 'src/engine/metadata-modules/flat-field-metadata/utils/generate-deterministic-index-for-flat-field-metadata-or-throw.util';
import { isMorphOrRelationUniversalFlatFieldMetadata } from 'src/engine/metadata-modules/flat-field-metadata/utils/is-morph-or-relation-flat-field-metadata.util';
import { isPrimaryKeyFlatFieldMetadata } from 'src/engine/metadata-modules/flat-field-metadata/utils/is-primary-key-flat-field-metadata.util';
import { buildFieldSideEffectParentNotFoundFailure } from 'src/engine/metadata-modules/metadata-side-effect/handlers/field-metadata/utils/build-field-side-effect-parent-not-found-failure.util';
import { resolveParentFlatObjectMetadataAfterStateForFieldSideEffect } from 'src/engine/metadata-modules/metadata-side-effect/handlers/field-metadata/utils/resolve-parent-flat-object-metadata-after-state-for-field-side-effect.util';
import {
type BuildSideEffectsArgs,
MetadataSideEffectHandler,
} from 'src/engine/metadata-modules/metadata-side-effect/interfaces/base-metadata-side-effect-handler.service';
import { type MetadataSideEffectResult } from 'src/engine/metadata-modules/metadata-side-effect/types/metadata-side-effect-result.type';
@Injectable()
export class FieldUniqueBackingIndexOnUpdateSideEffectHandlerService extends MetadataSideEffectHandler(
{
operation: 'update',
metadataName: 'fieldMetadata',
name: 'fieldUniqueBackingIndexOnUpdate',
description:
"Keep a unique scalar field's backing UNIQUE index in sync when its `isUnique` flag flips or the field is renamed (drop the stale index and recreate the deterministic one).",
},
) {
buildSideEffects({
flatEntity: flatFieldMetadata,
allFlatEntityOperationRecordByMetadataName,
relatedFlatEntityMaps,
}: BuildSideEffectsArgs<'fieldMetadata'>): MetadataSideEffectResult {
if (isMorphOrRelationUniversalFlatFieldMetadata(flatFieldMetadata)) {
return { status: 'noop' };
}
if (isPrimaryKeyFlatFieldMetadata(flatFieldMetadata)) {
return { status: 'noop' };
}
const existingFlatFieldMetadata =
relatedFlatEntityMaps.flatFieldMetadataMaps.byUniversalIdentifier[
flatFieldMetadata.universalIdentifier
];
if (!isDefined(existingFlatFieldMetadata)) {
return { status: 'noop' };
}
const wasRenamed =
existingFlatFieldMetadata.name !== flatFieldMetadata.name;
const uniquenessHasFlipped =
existingFlatFieldMetadata.isUnique !== flatFieldMetadata.isUnique;
const backingIndexMustFollowRename =
existingFlatFieldMetadata.isUnique === true &&
flatFieldMetadata.isUnique === true &&
wasRenamed;
if (!uniquenessHasFlipped && !backingIndexMustFollowRename) {
return { status: 'noop' };
}
const existingFlatObjectMetadata =
relatedFlatEntityMaps.flatObjectMetadataMaps.byUniversalIdentifier[
flatFieldMetadata.objectMetadataUniversalIdentifier
];
const optimisticFlatObjectMetadata =
resolveParentFlatObjectMetadataAfterStateForFieldSideEffect({
objectMetadataUniversalIdentifier:
flatFieldMetadata.objectMetadataUniversalIdentifier,
allFlatEntityOperationRecordByMetadataName,
relatedFlatEntityMaps,
});
const backingIndexMustBeDeleted =
existingFlatFieldMetadata.isUnique === true;
const backingIndexMustBeCreated = flatFieldMetadata.isUnique === true;
if (
(backingIndexMustBeDeleted && !isDefined(existingFlatObjectMetadata)) ||
(backingIndexMustBeCreated && !isDefined(optimisticFlatObjectMetadata))
) {
return buildFieldSideEffectParentNotFoundFailure({
flatFieldMetadata,
operation: 'update',
});
}
const previousFlatIndexMetadata =
backingIndexMustBeDeleted && isDefined(existingFlatObjectMetadata)
? generateDeterministicIndexForFlatFieldMetadataOrThrow({
flatFieldMetadata: {
...flatFieldMetadata,
name: existingFlatFieldMetadata.name,
isUnique: true,
},
flatObjectMetadata: existingFlatObjectMetadata,
})
: undefined;
const flatIndexMetadataToDelete =
isDefined(previousFlatIndexMetadata) &&
isDefined(
relatedFlatEntityMaps.flatIndexMaps.byUniversalIdentifier[
previousFlatIndexMetadata.universalIdentifier
],
)
? previousFlatIndexMetadata
: undefined;
const flatIndexMetadataToCreate =
backingIndexMustBeCreated && isDefined(optimisticFlatObjectMetadata)
? generateDeterministicIndexForFlatFieldMetadataOrThrow({
flatFieldMetadata: { ...flatFieldMetadata, isUnique: true },
flatObjectMetadata: optimisticFlatObjectMetadata,
})
: undefined;
if (
!isDefined(flatIndexMetadataToCreate) &&
!isDefined(flatIndexMetadataToDelete)
) {
return { status: 'noop' };
}
return {
status: 'success',
operations: {
index: {
...(isDefined(flatIndexMetadataToCreate)
? {
flatEntityToCreate: {
[flatIndexMetadataToCreate.universalIdentifier]:
flatIndexMetadataToCreate,
},
}
: {}),
...(isDefined(flatIndexMetadataToDelete)
? {
flatEntityToDelete: {
[flatIndexMetadataToDelete.universalIdentifier]:
flatIndexMetadataToDelete,
},
}
: {}),
},
},
};
}
}
@@ -0,0 +1,31 @@
import { msg, t } from '@lingui/core/macro';
import { type AllMetadataName } from 'twenty-shared/metadata';
import { type MetadataFlatEntity } from 'src/engine/metadata-modules/flat-entity/types/metadata-flat-entity.type';
import { type MetadataUniversalFlatEntity } from 'src/engine/metadata-modules/flat-entity/types/metadata-universal-flat-entity.type';
import { type WorkspaceMigrationActionType } from 'src/engine/metadata-modules/flat-entity/types/metadata-workspace-migration-action.type';
import { MetadataSideEffectExceptionCode } from 'src/engine/metadata-modules/metadata-side-effect/exceptions/metadata-side-effect-exception-code';
import { type MetadataSideEffectFailure } from 'src/engine/metadata-modules/metadata-side-effect/types/metadata-side-effect-result.type';
export const buildFieldSideEffectParentNotFoundFailure = ({
flatFieldMetadata,
operation,
}: {
flatFieldMetadata: MetadataUniversalFlatEntity<'fieldMetadata'>;
operation: WorkspaceMigrationActionType;
}): MetadataSideEffectFailure => ({
status: 'fail',
type: operation,
metadataName: 'fieldMetadata',
flatEntityMinimalInformation: {
universalIdentifier: flatFieldMetadata.universalIdentifier,
name: flatFieldMetadata.name,
} as Partial<MetadataFlatEntity<AllMetadataName>>,
errors: [
{
code: MetadataSideEffectExceptionCode.SIDE_EFFECT_PARENT_METADATA_NOT_FOUND,
message: t`Could not resolve parent object metadata "${flatFieldMetadata.objectMetadataUniversalIdentifier}" for field unique index side effect`,
userFriendlyMessage: msg`This field references an object that could not be found`,
},
],
});
@@ -0,0 +1,26 @@
import { type AllFlatEntityOperationRecordByMetadataName } from 'src/engine/metadata-modules/flat-entity/types/all-flat-entity-operation-record-by-metadata-name.type';
import { type MetadataFlatEntityAndRelatedFlatEntityMapsForSideEffect } from 'src/engine/metadata-modules/flat-entity/types/metadata-flat-entity-and-related-flat-entity-maps-for-side-effect.type';
import { type UniversalFlatObjectMetadata } from 'src/engine/workspace-manager/workspace-migration/universal-flat-entity/types/universal-flat-object-metadata.type';
export const resolveParentFlatObjectMetadataAfterStateForFieldSideEffect = ({
objectMetadataUniversalIdentifier,
allFlatEntityOperationRecordByMetadataName,
relatedFlatEntityMaps,
}: {
objectMetadataUniversalIdentifier: string;
allFlatEntityOperationRecordByMetadataName: AllFlatEntityOperationRecordByMetadataName;
relatedFlatEntityMaps: MetadataFlatEntityAndRelatedFlatEntityMapsForSideEffect<'fieldMetadata'>;
}): UniversalFlatObjectMetadata | undefined => {
const pendingFlatObjectMetadata =
allFlatEntityOperationRecordByMetadataName.objectMetadata
?.flatEntityToUpdate[objectMetadataUniversalIdentifier] ??
allFlatEntityOperationRecordByMetadataName.objectMetadata
?.flatEntityToCreate[objectMetadataUniversalIdentifier];
return (
pendingFlatObjectMetadata ??
relatedFlatEntityMaps.flatObjectMetadataMaps.byUniversalIdentifier[
objectMetadataUniversalIdentifier
]
);
};
@@ -0,0 +1,14 @@
import { Module } from '@nestjs/common';
import { FieldUniqueBackingIndexOnCreateSideEffectHandlerService } from 'src/engine/metadata-modules/metadata-side-effect/handlers/field-metadata/services/field-unique-backing-index-on-create-side-effect-handler.service';
import { FieldUniqueBackingIndexOnDeleteSideEffectHandlerService } from 'src/engine/metadata-modules/metadata-side-effect/handlers/field-metadata/services/field-unique-backing-index-on-delete-side-effect-handler.service';
import { FieldUniqueBackingIndexOnUpdateSideEffectHandlerService } from 'src/engine/metadata-modules/metadata-side-effect/handlers/field-metadata/services/field-unique-backing-index-on-update-side-effect-handler.service';
@Module({
providers: [
FieldUniqueBackingIndexOnCreateSideEffectHandlerService,
FieldUniqueBackingIndexOnUpdateSideEffectHandlerService,
FieldUniqueBackingIndexOnDeleteSideEffectHandlerService,
],
})
export class MetadataSideEffectHandlersModule {}
@@ -0,0 +1,61 @@
import { SetMetadata } from '@nestjs/common';
import { type AllMetadataName } from 'twenty-shared/metadata';
import { type AllFlatEntityOperationRecordByMetadataName } from 'src/engine/metadata-modules/flat-entity/types/all-flat-entity-operation-record-by-metadata-name.type';
import { type MetadataFlatEntityAndRelatedFlatEntityMapsForSideEffect } from 'src/engine/metadata-modules/flat-entity/types/metadata-flat-entity-and-related-flat-entity-maps-for-side-effect.type';
import { type MetadataUniversalFlatEntity } from 'src/engine/metadata-modules/flat-entity/types/metadata-universal-flat-entity.type';
import { METADATA_SIDE_EFFECT_HANDLER_METADATA_KEY } from 'src/engine/metadata-modules/metadata-side-effect/constants/metadata-side-effect-handler-metadata-key.constant';
import { type MetadataSideEffectContext } from 'src/engine/metadata-modules/metadata-side-effect/types/metadata-side-effect-context.type';
import { type MetadataSideEffectOperation } from 'src/engine/metadata-modules/metadata-side-effect/types/metadata-side-effect-operation.type';
import { type MetadataSideEffectResult } from 'src/engine/metadata-modules/metadata-side-effect/types/metadata-side-effect-result.type';
export type BuildSideEffectsArgs<P extends AllMetadataName> = {
flatEntity: MetadataUniversalFlatEntity<P>;
allFlatEntityOperationRecordByMetadataName: AllFlatEntityOperationRecordByMetadataName;
relatedFlatEntityMaps: MetadataFlatEntityAndRelatedFlatEntityMapsForSideEffect<P>;
context: MetadataSideEffectContext;
};
export abstract class BaseMetadataSideEffectHandlerService<
P extends AllMetadataName,
> {
public operation: MetadataSideEffectOperation;
public metadataName: P;
public sideEffectName: string;
public sideEffectDescription: string;
abstract buildSideEffects(
args: BuildSideEffectsArgs<P>,
): MetadataSideEffectResult;
}
type MetadataSideEffectHandlerDeclaration<P extends AllMetadataName> = {
operation: MetadataSideEffectOperation;
metadataName: P;
name: string;
description: string;
};
export function MetadataSideEffectHandler<P extends AllMetadataName>({
operation,
metadataName,
name,
description,
}: MetadataSideEffectHandlerDeclaration<P>): typeof BaseMetadataSideEffectHandlerService<P> {
abstract class SideEffectHandlerService extends BaseMetadataSideEffectHandlerService<P> {
operation = operation;
metadataName = metadataName;
sideEffectName = name;
sideEffectDescription = description;
}
SetMetadata(METADATA_SIDE_EFFECT_HANDLER_METADATA_KEY, {
operation,
metadataName,
name,
description,
})(SideEffectHandlerService);
return SideEffectHandlerService;
}
@@ -0,0 +1,16 @@
import { Module } from '@nestjs/common';
import { DiscoveryModule } from '@nestjs/core';
import { MetadataSideEffectHandlersModule } from 'src/engine/metadata-modules/metadata-side-effect/handlers/metadata-side-effect-handlers.module';
import { MetadataSideEffectHandlerRegistryService } from 'src/engine/metadata-modules/metadata-side-effect/registry/metadata-side-effect-handler-registry.service';
import { MetadataSideEffectEngineService } from 'src/engine/metadata-modules/metadata-side-effect/services/metadata-side-effect-engine.service';
@Module({
imports: [DiscoveryModule, MetadataSideEffectHandlersModule],
providers: [
MetadataSideEffectHandlerRegistryService,
MetadataSideEffectEngineService,
],
exports: [MetadataSideEffectEngineService],
})
export class MetadataSideEffectModule {}
@@ -0,0 +1,109 @@
import { Injectable, OnModuleInit } from '@nestjs/common';
import { DiscoveryService } from '@nestjs/core';
import { type AllMetadataName } from 'twenty-shared/metadata';
import { isDefined } from 'twenty-shared/utils';
import { METADATA_SIDE_EFFECT_HANDLER_METADATA_KEY } from 'src/engine/metadata-modules/metadata-side-effect/constants/metadata-side-effect-handler-metadata-key.constant';
import { MetadataSideEffectHandlersModule } from 'src/engine/metadata-modules/metadata-side-effect/handlers/metadata-side-effect-handlers.module';
import { type BaseMetadataSideEffectHandlerService } from 'src/engine/metadata-modules/metadata-side-effect/interfaces/base-metadata-side-effect-handler.service';
import {
buildMetadataSideEffectHandlerKey,
type MetadataSideEffectHandlerDescriptor,
type MetadataSideEffectHandlerKey,
type MetadataSideEffectOperation,
} from 'src/engine/metadata-modules/metadata-side-effect/types/metadata-side-effect-operation.type';
type RegisteredSideEffectHandler =
BaseMetadataSideEffectHandlerService<AllMetadataName>;
export type RegisteredMetadataSideEffectHandlerKey = {
operation: MetadataSideEffectOperation;
metadataName: AllMetadataName;
};
@Injectable()
export class MetadataSideEffectHandlerRegistryService implements OnModuleInit {
private readonly handlersByKey = new Map<
MetadataSideEffectHandlerKey,
RegisteredSideEffectHandler[]
>();
private readonly registeredHandlerKeys: RegisteredMetadataSideEffectHandlerKey[] =
[];
private readonly registeredSideEffectNames = new Set<string>();
constructor(private readonly discoveryService: DiscoveryService) {}
onModuleInit() {
this.discoverAndRegisterHandlers();
}
private discoverAndRegisterHandlers(): void {
const providers = this.discoveryService.getProviders({
include: [MetadataSideEffectHandlersModule],
});
providers.forEach((wrapper) => {
const { instance, metatype } = wrapper;
if (!instance || !metatype) return;
const descriptor: MetadataSideEffectHandlerDescriptor | undefined =
Reflect.getMetadata(
METADATA_SIDE_EFFECT_HANDLER_METADATA_KEY,
metatype,
);
if (
!isDefined(descriptor) ||
typeof instance.buildSideEffects !== 'function'
) {
return;
}
this.registerHandler(instance);
});
}
private registerHandler(instance: RegisteredSideEffectHandler): void {
if (this.registeredSideEffectNames.has(instance.sideEffectName)) {
throw new Error(
`Duplicate metadata side-effect name "${instance.sideEffectName}". Side-effect names must be unique.`,
);
}
this.registeredSideEffectNames.add(instance.sideEffectName);
const handlerKey = buildMetadataSideEffectHandlerKey(
instance.operation,
instance.metadataName,
);
const existingHandlers = this.handlersByKey.get(handlerKey);
if (isDefined(existingHandlers)) {
existingHandlers.push(instance);
return;
}
this.handlersByKey.set(handlerKey, [instance]);
this.registeredHandlerKeys.push({
operation: instance.operation,
metadataName: instance.metadataName,
});
}
getHandlers(
operation: MetadataSideEffectOperation,
metadataName: AllMetadataName,
): RegisteredSideEffectHandler[] {
return (
this.handlersByKey.get(
buildMetadataSideEffectHandlerKey(operation, metadataName),
) ?? []
);
}
getRegisteredHandlerKeys(): RegisteredMetadataSideEffectHandlerKey[] {
return this.registeredHandlerKeys;
}
}
@@ -0,0 +1,329 @@
import { Injectable } from '@nestjs/common';
import { type AllMetadataName } from 'twenty-shared/metadata';
import { isDefined } from 'twenty-shared/utils';
import { type AllFlatEntityMaps } from 'src/engine/metadata-modules/flat-entity/types/all-flat-entity-maps.type';
import { type AllFlatEntityOperationRecordByMetadataName } from 'src/engine/metadata-modules/flat-entity/types/all-flat-entity-operation-record-by-metadata-name.type';
import { type MetadataFlatEntityAndRelatedFlatEntityMapsForSideEffect } from 'src/engine/metadata-modules/flat-entity/types/metadata-flat-entity-and-related-flat-entity-maps-for-side-effect.type';
import { type MetadataUniversalFlatEntity } from 'src/engine/metadata-modules/flat-entity/types/metadata-universal-flat-entity.type';
import { getMetadataManyToOneRelatedNames } from 'src/engine/metadata-modules/flat-entity/utils/get-metadata-many-to-one-related-names.util';
import { getMetadataSideEffectCompanionNames } from 'src/engine/metadata-modules/flat-entity/utils/get-metadata-side-effect-companion-names.util';
import { isSystemSideEffectFlatEntity } from 'src/engine/metadata-modules/flat-entity/utils/is-system-side-effect-flat-entity.util';
import { MetadataSideEffectHandlerRegistryService } from 'src/engine/metadata-modules/metadata-side-effect/registry/metadata-side-effect-handler-registry.service';
import { type MetadataSideEffectContext } from 'src/engine/metadata-modules/metadata-side-effect/types/metadata-side-effect-context.type';
import { type MetadataSideEffectExpansionResult } from 'src/engine/metadata-modules/metadata-side-effect/types/metadata-side-effect-expansion-result.type';
import {
METADATA_SIDE_EFFECT_OPERATIONS,
type MetadataSideEffectOperation,
} from 'src/engine/metadata-modules/metadata-side-effect/types/metadata-side-effect-operation.type';
import { type MetadataSideEffectFailure } from 'src/engine/metadata-modules/metadata-side-effect/types/metadata-side-effect-result.type';
import { type SystemSideEffectUniversalIdentifierCollision } from 'src/engine/metadata-modules/metadata-side-effect/types/system-side-effect-universal-identifier-collision.type';
import { mapSystemSideEffectCollisionToFailure } from 'src/engine/metadata-modules/metadata-side-effect/utils/map-system-side-effect-collision-to-failure.util';
import { EMPTY_ORCHESTRATOR_FAILURE_REPORT } from 'src/engine/workspace-manager/workspace-migration/constant/empty-orchestrator-failure-report.constant';
import { pushToOrchestratorFailureReport } from 'src/engine/workspace-manager/workspace-migration/utils/merge-orchestrator-failure-reports.util';
type GenericUniversalFlatEntity = { universalIdentifier: string };
type GenericFlatEntityOperationRecord = {
flatEntityToCreate: Record<string, GenericUniversalFlatEntity>;
flatEntityToUpdate: Record<string, GenericUniversalFlatEntity>;
flatEntityToDelete: Record<string, GenericUniversalFlatEntity>;
};
type GenericAllFlatEntityOperationRecordByMetadataName = Record<
string,
GenericFlatEntityOperationRecord | undefined
>;
type GenericPartialFlatEntityOperationRecord = {
flatEntityToCreate?: Record<string, GenericUniversalFlatEntity>;
flatEntityToUpdate?: Record<string, GenericUniversalFlatEntity>;
flatEntityToDelete?: Record<string, GenericUniversalFlatEntity>;
};
type GenericMetadataSideEffectOperationsByMetadataName = Record<
string,
GenericPartialFlatEntityOperationRecord | undefined
>;
const OPERATION_TO_FLAT_ENTITY_RECORD_KEY = {
create: 'flatEntityToCreate',
update: 'flatEntityToUpdate',
delete: 'flatEntityToDelete',
} as const satisfies Record<
MetadataSideEffectOperation,
keyof GenericFlatEntityOperationRecord
>;
@Injectable()
export class MetadataSideEffectEngineService {
constructor(
private readonly metadataSideEffectHandlerRegistryService: MetadataSideEffectHandlerRegistryService,
) {}
getSideEffectRelatedMetadataNames(
triggerMetadataNames: AllMetadataName[],
): AllMetadataName[] {
const relatedMetadataNames = new Set<AllMetadataName>();
for (const {
metadataName,
} of this.metadataSideEffectHandlerRegistryService.getRegisteredHandlerKeys()) {
if (!triggerMetadataNames.includes(metadataName)) {
continue;
}
for (const relatedMetadataName of [
metadataName,
...getMetadataManyToOneRelatedNames(metadataName),
...getMetadataSideEffectCompanionNames(metadataName),
]) {
relatedMetadataNames.add(relatedMetadataName);
}
}
return [...relatedMetadataNames];
}
expandWithSideEffects({
allFlatEntityOperationRecordByMetadataName,
sideEffectRelatedFlatEntityMaps,
context,
}: {
allFlatEntityOperationRecordByMetadataName: AllFlatEntityOperationRecordByMetadataName;
sideEffectRelatedFlatEntityMaps: Partial<AllFlatEntityMaps>;
context: MetadataSideEffectContext;
}): MetadataSideEffectExpansionResult {
const expandedMatrix = this.cloneMatrix(
allFlatEntityOperationRecordByMetadataName,
);
const systemSideEffectUniversalIdentifierCollisions: SystemSideEffectUniversalIdentifierCollision[] =
[];
const sideEffectFailures: MetadataSideEffectFailure[] = [];
const triggerMatrix =
allFlatEntityOperationRecordByMetadataName as unknown as GenericAllFlatEntityOperationRecordByMetadataName;
for (const {
operation,
metadataName,
} of this.metadataSideEffectHandlerRegistryService.getRegisteredHandlerKeys()) {
const handlers =
this.metadataSideEffectHandlerRegistryService.getHandlers(
operation,
metadataName,
);
if (handlers.length === 0) {
continue;
}
const triggerFlatEntities = Object.values(
triggerMatrix[metadataName]?.[
OPERATION_TO_FLAT_ENTITY_RECORD_KEY[operation]
] ?? {},
);
for (const triggerFlatEntity of triggerFlatEntities) {
for (const handler of handlers) {
const sideEffectResult = handler.buildSideEffects({
flatEntity:
triggerFlatEntity as unknown as MetadataUniversalFlatEntity<AllMetadataName>,
allFlatEntityOperationRecordByMetadataName:
expandedMatrix as unknown as AllFlatEntityOperationRecordByMetadataName,
relatedFlatEntityMaps:
sideEffectRelatedFlatEntityMaps as unknown as MetadataFlatEntityAndRelatedFlatEntityMapsForSideEffect<AllMetadataName>,
context,
});
if (sideEffectResult.status === 'fail') {
sideEffectFailures.push(sideEffectResult);
continue;
}
if (sideEffectResult.status === 'noop') {
continue;
}
this.mergeSideEffectsIntoMatrix({
expandedMatrix,
sideEffectOperations:
sideEffectResult.operations as unknown as GenericMetadataSideEffectOperationsByMetadataName,
systemSideEffectUniversalIdentifierCollisions,
});
}
}
}
const allSideEffectFailures: MetadataSideEffectFailure[] = [
...sideEffectFailures,
...systemSideEffectUniversalIdentifierCollisions.map(
mapSystemSideEffectCollisionToFailure,
),
];
if (allSideEffectFailures.length > 0) {
const report = EMPTY_ORCHESTRATOR_FAILURE_REPORT();
for (const sideEffectFailure of allSideEffectFailures) {
pushToOrchestratorFailureReport({
report,
metadataName: sideEffectFailure.metadataName,
items: [sideEffectFailure],
});
}
return {
status: 'fail',
report,
};
}
return {
status: 'success',
allFlatEntityOperationRecordByMetadataName:
expandedMatrix as unknown as AllFlatEntityOperationRecordByMetadataName,
};
}
private mergeSideEffectsIntoMatrix({
expandedMatrix,
sideEffectOperations,
systemSideEffectUniversalIdentifierCollisions,
}: {
expandedMatrix: GenericAllFlatEntityOperationRecordByMetadataName;
sideEffectOperations: GenericMetadataSideEffectOperationsByMetadataName;
systemSideEffectUniversalIdentifierCollisions: SystemSideEffectUniversalIdentifierCollision[];
}): void {
for (const metadataName of Object.keys(sideEffectOperations)) {
const operationBuckets = sideEffectOperations[metadataName];
if (!isDefined(operationBuckets)) {
continue;
}
for (const operation of METADATA_SIDE_EFFECT_OPERATIONS) {
const sideEffectFlatEntities = Object.values(
operationBuckets[OPERATION_TO_FLAT_ENTITY_RECORD_KEY[operation]] ??
{},
);
for (const sideEffectFlatEntity of sideEffectFlatEntities) {
this.addToOperationIfAbsent({
expandedMatrix,
operation,
metadataName,
flatEntity: sideEffectFlatEntity,
systemSideEffectUniversalIdentifierCollisions,
});
}
}
}
}
private cloneMatrix(
allFlatEntityOperationRecordByMetadataName: AllFlatEntityOperationRecordByMetadataName,
): GenericAllFlatEntityOperationRecordByMetadataName {
const genericMatrix =
allFlatEntityOperationRecordByMetadataName as unknown as GenericAllFlatEntityOperationRecordByMetadataName;
const clonedMatrix: GenericAllFlatEntityOperationRecordByMetadataName = {};
for (const metadataName of Object.keys(genericMatrix)) {
const operations = genericMatrix[metadataName];
if (!isDefined(operations)) {
continue;
}
clonedMatrix[metadataName] = {
flatEntityToCreate: { ...operations.flatEntityToCreate },
flatEntityToUpdate: { ...operations.flatEntityToUpdate },
flatEntityToDelete: { ...operations.flatEntityToDelete },
};
}
return clonedMatrix;
}
private addToOperationIfAbsent({
expandedMatrix,
operation,
metadataName,
flatEntity,
systemSideEffectUniversalIdentifierCollisions,
}: {
expandedMatrix: GenericAllFlatEntityOperationRecordByMetadataName;
operation: MetadataSideEffectOperation;
metadataName: string;
flatEntity: GenericUniversalFlatEntity;
systemSideEffectUniversalIdentifierCollisions: SystemSideEffectUniversalIdentifierCollision[];
}): void {
const operations = (expandedMatrix[metadataName] ??= {
flatEntityToCreate: {},
flatEntityToUpdate: {},
flatEntityToDelete: {},
});
const flatEntityRecordKey = OPERATION_TO_FLAT_ENTITY_RECORD_KEY[operation];
const flatEntityRecord = operations[flatEntityRecordKey];
const existingFlatEntity = flatEntityRecord[flatEntity.universalIdentifier];
if (isDefined(existingFlatEntity)) {
this.recordUniversalIdentifierCollisionIfNeeded({
existingFlatEntity,
operation,
metadataName,
flatEntity,
systemSideEffectUniversalIdentifierCollisions,
});
return;
}
flatEntityRecord[flatEntity.universalIdentifier] = flatEntity;
}
private recordUniversalIdentifierCollisionIfNeeded({
existingFlatEntity,
operation,
metadataName,
flatEntity,
systemSideEffectUniversalIdentifierCollisions,
}: {
existingFlatEntity: GenericUniversalFlatEntity;
operation: MetadataSideEffectOperation;
metadataName: string;
flatEntity: GenericUniversalFlatEntity;
systemSideEffectUniversalIdentifierCollisions: SystemSideEffectUniversalIdentifierCollision[];
}): void {
const isIncomingSystemSideEffect = isSystemSideEffectFlatEntity(
flatEntity as unknown as MetadataUniversalFlatEntity<AllMetadataName>,
);
if (!isIncomingSystemSideEffect) {
return;
}
if (
isSystemSideEffectFlatEntity(
existingFlatEntity as unknown as MetadataUniversalFlatEntity<AllMetadataName>,
)
) {
return;
}
systemSideEffectUniversalIdentifierCollisions.push({
metadataName: metadataName as AllMetadataName,
operation,
universalIdentifier: flatEntity.universalIdentifier,
name: this.extractFlatEntityName(flatEntity),
});
}
private extractFlatEntityName(
flatEntity: GenericUniversalFlatEntity,
): string | undefined {
const { name } = flatEntity as { name?: unknown };
return typeof name === 'string' ? name : undefined;
}
}
@@ -0,0 +1,5 @@
import { type WorkspaceMigrationBuilderOptions } from 'src/engine/workspace-manager/workspace-migration/workspace-migration-builder/types/workspace-migration-builder-options.type';
export type MetadataSideEffectContext = {
buildOptions: WorkspaceMigrationBuilderOptions;
};
@@ -0,0 +1,15 @@
import { type AllFlatEntityOperationRecordByMetadataName } from 'src/engine/metadata-modules/flat-entity/types/all-flat-entity-operation-record-by-metadata-name.type';
import { type OrchestratorFailureReport } from 'src/engine/workspace-manager/workspace-migration/types/workspace-migration-orchestrator.type';
// Same contract as the builder (WorkspaceMigrationOrchestratorFailedResult):
// collisions and handler failures are merged into a single OrchestratorFailureReport
// so callers handle side-effect and builder failures through one uniform channel.
export type MetadataSideEffectExpansionResult =
| {
status: 'success';
allFlatEntityOperationRecordByMetadataName: AllFlatEntityOperationRecordByMetadataName;
}
| {
status: 'fail';
report: OrchestratorFailureReport;
};
@@ -0,0 +1,24 @@
import { type AllMetadataName } from 'twenty-shared/metadata';
export type MetadataSideEffectOperation = 'create' | 'update' | 'delete';
export const METADATA_SIDE_EFFECT_OPERATIONS = [
'create',
'update',
'delete',
] as const satisfies readonly MetadataSideEffectOperation[];
export type MetadataSideEffectHandlerKey =
`${MetadataSideEffectOperation}:${AllMetadataName}`;
export const buildMetadataSideEffectHandlerKey = (
operation: MetadataSideEffectOperation,
metadataName: AllMetadataName,
): MetadataSideEffectHandlerKey => `${operation}:${metadataName}`;
export type MetadataSideEffectHandlerDescriptor = {
operation: MetadataSideEffectOperation;
metadataName: AllMetadataName;
name: string;
description: string;
};
@@ -0,0 +1,17 @@
import { type AllMetadataName } from 'twenty-shared/metadata';
import { type MetadataUniversalFlatEntity } from 'src/engine/metadata-modules/flat-entity/types/metadata-universal-flat-entity.type';
// Record-native, partial companion operations a handler emits. Each bucket is keyed by
// universalIdentifier so the engine merges/deduplicates against the matrix in O(1),
// matching the shape it consumes (AllFlatEntityOperationRecordByMetadataName).
export type MetadataSideEffectOperationsByMetadataName = {
[P in AllMetadataName]?: {
flatEntityToCreate?: Record<
string,
MetadataUniversalFlatEntity<P> & { id?: string }
>;
flatEntityToUpdate?: Record<string, MetadataUniversalFlatEntity<P>>;
flatEntityToDelete?: Record<string, MetadataUniversalFlatEntity<P>>;
};
};
@@ -0,0 +1,27 @@
import { type AllMetadataName } from 'twenty-shared/metadata';
import { type WorkspaceMigrationActionType } from 'src/engine/metadata-modules/flat-entity/types/metadata-workspace-migration-action.type';
import { type MetadataSideEffectOperationsByMetadataName } from 'src/engine/metadata-modules/metadata-side-effect/types/metadata-side-effect-operations-by-metadata-name.type';
import { type FailedFlatEntityValidation } from 'src/engine/workspace-manager/workspace-migration/workspace-migration-builder/builders/types/failed-flat-entity-validation.type';
// A single failure emitted by a side-effect handler, using the exact same shape
// as the builder's per-entity validation failure (a `status: 'fail'` tag spread
// over a FailedFlatEntityValidation), so both sources merge into one report.
export type MetadataSideEffectFailure = {
status: 'fail';
} & FailedFlatEntityValidation<AllMetadataName, WorkspaceMigrationActionType>;
// Discriminated union of the three handler outcomes: `noop` when there is
// nothing to do, `success` when it produced companion operations to merge, or
// `fail` (fail-slow) with a validation failure the engine aggregates instead of
// throwing. `success` therefore always carries operations, keeping the empty
// case explicit as `noop` rather than a success with `operations: {}`.
export type MetadataSideEffectResult =
| {
status: 'noop';
}
| {
status: 'success';
operations: MetadataSideEffectOperationsByMetadataName;
}
| MetadataSideEffectFailure;
@@ -0,0 +1,10 @@
import { type AllMetadataName } from 'twenty-shared/metadata';
import { type MetadataSideEffectOperation } from 'src/engine/metadata-modules/metadata-side-effect/types/metadata-side-effect-operation.type';
export type SystemSideEffectUniversalIdentifierCollision = {
metadataName: AllMetadataName;
operation: MetadataSideEffectOperation;
universalIdentifier: string;
name?: string;
};
@@ -0,0 +1,29 @@
import { msg, t } from '@lingui/core/macro';
import { type AllMetadataName } from 'twenty-shared/metadata';
import { type MetadataFlatEntity } from 'src/engine/metadata-modules/flat-entity/types/metadata-flat-entity.type';
import { MetadataSideEffectExceptionCode } from 'src/engine/metadata-modules/metadata-side-effect/exceptions/metadata-side-effect-exception-code';
import { type MetadataSideEffectFailure } from 'src/engine/metadata-modules/metadata-side-effect/types/metadata-side-effect-result.type';
import { type SystemSideEffectUniversalIdentifierCollision } from 'src/engine/metadata-modules/metadata-side-effect/types/system-side-effect-universal-identifier-collision.type';
// A reserved-identifier collision is just another side-effect failure: mapping it
// to the shared MetadataSideEffectFailure shape lets the engine merge collisions
// and handler failures into a single report.
export const mapSystemSideEffectCollisionToFailure = (
collision: SystemSideEffectUniversalIdentifierCollision,
): MetadataSideEffectFailure => ({
status: 'fail',
type: collision.operation,
metadataName: collision.metadataName,
flatEntityMinimalInformation: {
universalIdentifier: collision.universalIdentifier,
...(collision.name !== undefined ? { name: collision.name } : {}),
} as Partial<MetadataFlatEntity<AllMetadataName>>,
errors: [
{
code: MetadataSideEffectExceptionCode.RESERVED_SYSTEM_UNIVERSAL_IDENTIFIER,
message: t`Universal identifier is reserved for system-managed metadata`,
userFriendlyMessage: msg`This identifier is reserved by the system`,
},
],
});