Centralized side effects devxp basis (#22295)

# Introduction This PR introduces a centralized, strictly-typed **metadata side-effect engine** that unifies how system metadata side effects are derived and applied across both metadata entry points — the **metadata GraphQL API** and the **application sync / manifest** flow — and migrates the first side effect end-to-end: **a unique scalar field owns its backing single-field `UNIQUE` index** (full create / update / delete lifecycle). ## New conventions - **Engine-owned companions**: metadata flagged `isSystemSideEffect: true` is owned by the engine. Its deletion is never inferred from absence in a manifest — it results from PG-level cascade or from a delete side effect (a side effect always has a cause, its parent metadata). - **Reserved deterministic identifiers**: apps cannot declare metadata reusing an engine-owned deterministic `universalIdentifier`. Doing so fails validation with `RESERVED_SYSTEM_UNIVERSAL_IDENTIFIER` (until an explicit override API exists). - **Record-native operation matrix**: the operation matrix is keyed by `universalIdentifier` (`AllFlatEntityOperationRecordByMetadataName`) instead of arrays, making parent resolution and deduplication O(1). Array-based API callers are transpiled to records at the validate-build-and-run boundary. - Twenty-sdk user-facing experience with system fields will only be related to overrides. # What this PR does ## 1. Side-effect engine (foundation) - `MetadataSideEffectEngineService.expandWithSideEffects(...)` takes the intention-carrying record matrix and returns it expanded with derived side effects, or a structured failure. - Handlers are registered via a typed **decorator + registry** pattern (`MetadataSideEffectHandler({ operation, metadataName, name, description })`), with runtime duplicate-name detection. Multiple handlers per (operation, metadataName) are supported. - Handler contract mirrors the validator pattern: - receives the trigger flat entity, the live record matrix, and **strictly-typed related flat entity maps** (`MetadataFlatEntityAndRelatedFlatEntityMapsForSideEffect<P>`, derived from declared companion metadata names — no loose `Partial<AllFlatEntityMaps>` context) - returns `MetadataSideEffectResult`: `success` (operations record) | `noop` | `fail` (structured failure) - **Non-recursion is structural**: triggers are read from the original caller input, never from the expanded matrix, so a side effect can never trigger another side effect. - **Deduplication + collision detection**: side effects are deduped by `universalIdentifier` per operation; a caller-declared entity colliding with an engine-owned deterministic identifier is recorded as a collision. - **Unified failure channel**: handler failures and reserved-identifier collisions are merged into the same `OrchestratorFailureReport` contract as builder validation errors, and the run short-circuits (fail-closed, nothing is applied). ## 2. First migrated side effect — unique field → backing unique index Three handlers own the complete lifecycle of the deterministic single-field `UNIQUE` index backing a unique scalar field: - **create**: unique scalar field → generate the deterministic backing index (`fieldUniqueBackingIndexOnCreate`) - **update**: `isUnique` flips and renames of still-unique fields (the index name — and therefore its deterministic identifier — derives from the field name, so a rename drops the stale index and recreates the deterministic one) (`fieldUniqueBackingIndexOnUpdate`) - **delete**: cascade-delete the backing index (`fieldUniqueBackingIndexOnDelete`) Supporting rules: - The primary key `id` field never spawns a backing index (uniqueness comes from the PK constraint) — explicit `isPrimaryKeyFlatFieldMetadata` guard. - Parent object resolution is **optimistic-first**: an object created or updated in the same batch wins over the workspace cache (so e.g. renaming an object while flipping a field to unique builds the index from the post-rename object), resolved in O(1) via the record matrix. - A missing parent object is reported as a structured side-effect failure, never silently skipped. ## 3. Path convergence — manifest and API share one flow - The manifest sync now derives a from→to **record matrix** from the cache and feeds `validateBuildAndRunWorkspaceMigrationFromRecord`, the same flow the API uses — both paths converge on the engine. - Manifest-side unique-index generation and API transpiler system-unique-index handling were removed (declared/composite/relation indexes stay untouched). - New `WorkspaceMigrationFlatEntityMapsService` mutualizes flat-entity-maps computation between the side-effect engine and the builder: cache keys are derived from the caller metadata names (+ validation- and side-effect-related closures) instead of hardcoded loads. - App-scoping and pruning are folded into one shared primitive (`getSubAllFlatEntityMapsByApplicationIdsOrThrow`): slicing dependency maps to the involved applications always prunes dangling one-to-many aggregators — callers can no longer forget it. - **Behavior change**: an app extending another app's view with a view field now syncs successfully (cross-app view-field extension), covered by a dedicated integration test. ## 4. Backfill upgrade command (2.19) `upgrade:2-19:backfill-system-unique-index-universal-identifier` rewrites legacy system unique-index `universalIdentifier`s to their deterministic value so the engine can own pre-existing indexes. The backfill is **driven from `isUnique: true` fields** (mirroring the engine ownership predicate — excludes PK / morph / relation fields) and resolves each field's backing index in O(1). # Bugs fixed along the way - `database:reset` seeding failed with `INDEX_FIELD_INVALID_DEFAULT_VALUE`: the engine derived a backing `UNIQUE` index for the default `id` primary key. Fixed with the explicit primary-key guard. - `isUnique` updates on system-flagged standard fields (e.g. auto-created `name`) did not trigger the backing-index side effect. - Manifest sync crashed with "Could not find flat entity with universal identifier ..." when app-scoped slices left dangling aggregator references — fixed by centralizing pruning in the shared slice primitive
2026-07-03 18:13:20 +02:00
parent 566c3b6629
commit 43730d7748
59 changed files with 2840 additions and 691 deletions
@@ -19,7 +19,6 @@ import { findFlatEntityByIdInFlatEntityMaps } from 'src/engine/metadata-modules/
 import { type FieldInputTranspilationResult } from 'src/engine/metadata-modules/flat-field-metadata/types/field-input-transpilation-result.type';
 import { fromMorphRelationCreateFieldInputToFlatFieldMetadatas } from 'src/engine/metadata-modules/flat-field-metadata/utils/from-morph-relation-create-field-input-to-flat-field-metadatas.util';
 import { fromRelationCreateFieldInputToFlatFieldMetadatas } from 'src/engine/metadata-modules/flat-field-metadata/utils/from-relation-create-field-input-to-flat-field-metadatas.util';
-import { generateIndexForFlatFieldMetadata } from 'src/engine/metadata-modules/flat-field-metadata/utils/generate-index-for-flat-field-metadata.util';
 import { getDefaultFlatFieldMetadata } from 'src/engine/metadata-modules/flat-field-metadata/utils/get-default-flat-field-metadata-from-create-field-input.util';
 import { type UniversalFlatFieldMetadata } from 'src/engine/workspace-manager/workspace-migration/universal-flat-entity/types/universal-flat-field-metadata.type';
 import { type UniversalFlatIndexMetadata } from 'src/engine/workspace-manager/workspace-migration/universal-flat-entity/types/universal-flat-index-metadata.type';
@@ -186,17 +185,6 @@ export const fromCreateFieldInputToFlatFieldMetadatasToCreate = async ({
    case FieldMetadataType.RICH_TEXT:
    case FieldMetadataType.ACTOR:
    case FieldMetadataType.ARRAY: {
-      const indexMetadatas: UniversalFlatIndexMetadata[] = [];
-
-      if (commonFlatFieldMetadata.isUnique) {
-        indexMetadatas.push(
-          generateIndexForFlatFieldMetadata({
-            flatFieldMetadata: commonFlatFieldMetadata,
-            flatObjectMetadata: parentFlatObjectMetadata,
-          }),
-        );
-      }
-
      return {
        status: 'success',
        result: {
@@ -206,7 +194,7 @@ export const fromCreateFieldInputToFlatFieldMetadatasToCreate = async ({
              type: createFieldInput.type,
            },
          ],
-          indexMetadatas,
+          indexMetadatas: [],
        },
      };
    }
@@ -14,7 +14,9 @@ import { findFlatEntityByUniversalIdentifierOrThrow } from 'src/engine/metadata-
 import { findManyFlatEntityByUniversalIdentifierInUniversalFlatEntityMapsOrThrow } from 'src/engine/metadata-modules/flat-entity/utils/find-many-flat-entity-by-universal-identifier-in-universal-flat-entity-maps-or-throw.util';
 import { computeFlatFieldMetadataRelatedFlatFieldMetadata } from 'src/engine/metadata-modules/flat-field-metadata/utils/compute-flat-field-metadata-related-flat-field-metadata.util';
 import { computeSearchFieldMetadataDeletionForDeletedFields } from 'src/engine/metadata-modules/flat-field-metadata/utils/compute-search-field-metadata-deletion-for-deleted-fields.util';
+import { isMorphOrRelationFlatFieldMetadata } from 'src/engine/metadata-modules/flat-field-metadata/utils/is-morph-or-relation-flat-field-metadata.util';
 import { type FlatIndexMetadata } from 'src/engine/metadata-modules/flat-index-metadata/types/flat-index-metadata.type';
+import { isSystemUniqueFlatIndexMetadata } from 'src/engine/metadata-modules/flat-index-metadata/utils/is-system-unique-flat-index-metadata.util';
 import { generateFlatIndexMetadataWithNameOrThrow } from 'src/engine/metadata-modules/index-metadata/utils/generate-flat-index.util';
 import { belongsToTwentyStandardApp } from 'src/engine/metadata-modules/utils/belongs-to-twenty-standard-app.util';
 import { type UniversalFlatFieldMetadata } from 'src/engine/workspace-manager/workspace-migration/universal-flat-entity/types/universal-flat-field-metadata.type';
@@ -95,10 +97,38 @@ export const fromDeleteFieldInputToFlatFieldMetadatasToDelete = ({
    ...relatedFlatFieldMetadataToDelete,
  ];

+  const isEngineOwnedBackingUniqueIndex = (
+    flatIndexMetadata: FlatIndexMetadata,
+  ): boolean => {
+    if (
+      !isSystemUniqueFlatIndexMetadata(flatIndexMetadata) ||
+      flatIndexMetadata.flatIndexFieldMetadatas.length !== 1
+    ) {
+      return false;
+    }
+
+    const backingFlatFieldMetadata = findFlatEntityByIdInFlatEntityMaps({
+      flatEntityId:
+        flatIndexMetadata.flatIndexFieldMetadatas[0].fieldMetadataId,
+      flatEntityMaps: existingFlatFieldMetadataMaps,
+    });
+
+    return (
+      isDefined(backingFlatFieldMetadata) &&
+      backingFlatFieldMetadata.isUnique === true &&
+      !isMorphOrRelationFlatFieldMetadata(backingFlatFieldMetadata)
+    );
+  };
+
  const flatIndexMap = new Map<string, FlatIndexMetadata>();
  const allFlatIndexes = Object.values(
    existingFlatIndexMaps.byUniversalIdentifier,
-  ).filter(isDefined);
+  )
+    .filter(isDefined)
+    .filter(
+      (flatIndexMetadata) =>
+        !isEngineOwnedBackingUniqueIndex(flatIndexMetadata),
+    );

  for (const flatFieldMetadata of flatFieldMetadatasToDelete) {
    allFlatIndexes.forEach((flatIndex) => {
@@ -150,7 +150,6 @@ export const fromUpdateFieldInputToFlatFieldMetadata = ({
      toFlatFieldMetadata,
      flatViewMaps,
      flatViewFieldMaps,
-      flatApplication,
    });

    if (sideEffectResult.status === 'fail') {
@@ -0,0 +1,44 @@
+import { IndexType } from 'src/engine/metadata-modules/index-metadata/types/indexType.types';
+import { generateDeterministicFlatIndexMetadataOrThrow } from 'src/engine/metadata-modules/index-metadata/utils/generate-deterministic-flat-index.util';
+import { type UniversalFlatFieldMetadata } from 'src/engine/workspace-manager/workspace-migration/universal-flat-entity/types/universal-flat-field-metadata.type';
+import { type UniversalFlatIndexMetadata } from 'src/engine/workspace-manager/workspace-migration/universal-flat-entity/types/universal-flat-index-metadata.type';
+import { type UniversalFlatObjectMetadata } from 'src/engine/workspace-manager/workspace-migration/universal-flat-entity/types/universal-flat-object-metadata.type';
+
+type GenerateDeterministicIndexForFlatFieldMetadataOrThrowArgs = {
+  flatFieldMetadata: UniversalFlatFieldMetadata;
+  flatObjectMetadata: UniversalFlatObjectMetadata;
+};
+
+export const generateDeterministicIndexForFlatFieldMetadataOrThrow = ({
+  flatFieldMetadata,
+  flatObjectMetadata,
+}: GenerateDeterministicIndexForFlatFieldMetadataOrThrowArgs): UniversalFlatIndexMetadata => {
+  const createdAt = new Date().toISOString();
+
+  return generateDeterministicFlatIndexMetadataOrThrow({
+    flatObjectMetadata,
+    objectFlatFieldMetadatas: [flatFieldMetadata],
+    flatIndex: {
+      createdAt,
+      universalFlatIndexFieldMetadatas: [
+        {
+          createdAt,
+          fieldMetadataUniversalIdentifier:
+            flatFieldMetadata.universalIdentifier,
+          order: 0,
+          subFieldName: null,
+          updatedAt: createdAt,
+        },
+      ],
+      indexType: IndexType.BTREE,
+      indexWhereClause: null,
+      isCustom: true,
+      isUnique: flatFieldMetadata.isUnique ?? false,
+      isSystemSideEffect: true,
+      objectMetadataUniversalIdentifier: flatObjectMetadata.universalIdentifier,
+      updatedAt: createdAt,
+      applicationUniversalIdentifier:
+        flatFieldMetadata.applicationUniversalIdentifier,
+    },
+  });
+};
@@ -1,6 +1,5 @@
 import { type FromTo } from 'twenty-shared/types';

-import { type FlatApplication } from 'src/engine/core-modules/application/types/flat-application.type';
 import { type AllFlatEntityMaps } from 'src/engine/metadata-modules/flat-entity/types/all-flat-entity-maps.type';
 import { type FieldInputTranspilationResult } from 'src/engine/metadata-modules/flat-field-metadata/types/field-input-transpilation-result.type';
 import { type FlatFieldMetadata } from 'src/engine/metadata-modules/flat-field-metadata/types/flat-field-metadata.type';
@@ -36,9 +35,7 @@ type HandleFlatFieldMetadataUpdateSideEffectArgs = FromTo<
    | 'flatViewGroupMaps'
    | 'flatViewMaps'
    | 'flatViewFieldMaps'
-  > & {
-    flatApplication: FlatApplication;
-  };
+  >;

 export const FLAT_FIELD_METADATA_UPDATE_EMPTY_SIDE_EFFECTS: FlatFieldMetadataUpdateSideEffects =
  {
@@ -65,7 +62,6 @@ export const handleFlatFieldMetadataUpdateSideEffect = ({
  flatViewGroupMaps,
  flatViewMaps,
  flatViewFieldMaps,
-  flatApplication,
 }: HandleFlatFieldMetadataUpdateSideEffectArgs): FieldInputTranspilationResult<FlatFieldMetadataUpdateSideEffects> => {
  const sideEffectResult = structuredClone(
    FLAT_FIELD_METADATA_UPDATE_EMPTY_SIDE_EFFECTS,
@@ -125,7 +121,6 @@ export const handleFlatFieldMetadataUpdateSideEffect = ({
    flatIndexMaps,
    flatObjectMetadataMaps,
    flatFieldMetadataMaps,
-    flatApplication,
  });

  if (indexChangesSideEffectResult.status === 'fail') {
@@ -1,19 +1,12 @@
-import { msg } from '@lingui/core/macro';
 import { type FromTo } from 'twenty-shared/types';
-import { isDefined } from 'twenty-shared/utils';

-import { type FlatApplication } from 'src/engine/core-modules/application/types/flat-application.type';
-import { FieldMetadataExceptionCode } from 'src/engine/metadata-modules/field-metadata/field-metadata.exception';
 import { type AllFlatEntityMaps } from 'src/engine/metadata-modules/flat-entity/types/all-flat-entity-maps.type';
 import { findFlatEntityByIdInFlatEntityMapsOrThrow } from 'src/engine/metadata-modules/flat-entity/utils/find-flat-entity-by-id-in-flat-entity-maps-or-throw.util';
 import { type FieldInputTranspilationResult } from 'src/engine/metadata-modules/flat-field-metadata/types/field-input-transpilation-result.type';
 import { type FlatFieldMetadata } from 'src/engine/metadata-modules/flat-field-metadata/types/flat-field-metadata.type';
 import { findFieldRelatedIndexes } from 'src/engine/metadata-modules/flat-field-metadata/utils/find-field-related-index.util';
-import { generateIndexForFlatFieldMetadata } from 'src/engine/metadata-modules/flat-field-metadata/utils/generate-index-for-flat-field-metadata.util';
-import { isMorphOrRelationFlatFieldMetadata } from 'src/engine/metadata-modules/flat-field-metadata/utils/is-morph-or-relation-flat-field-metadata.util';
 import { recomputeIndexOnFlatFieldMetadataNameUpdate } from 'src/engine/metadata-modules/flat-field-metadata/utils/recompute-index-on-flat-field-metadata-name-update.util';
-import { type FlatIndexMetadata } from 'src/engine/metadata-modules/flat-index-metadata/types/flat-index-metadata.type';
-import { type FlatObjectMetadata } from 'src/engine/metadata-modules/flat-object-metadata/types/flat-object-metadata.type';
+import { isSystemUniqueFlatIndexMetadata } from 'src/engine/metadata-modules/flat-index-metadata/utils/is-system-unique-flat-index-metadata.util';
 import { type UniversalFlatIndexMetadata } from 'src/engine/workspace-manager/workspace-migration/universal-flat-entity/types/universal-flat-index-metadata.type';

 export type FieldMetadataUpdateIndexSideEffect = {
@@ -29,9 +22,8 @@ type FromToFlatFieldMetadataAndFlatEntityMaps = FromTo<
  Pick<
    AllFlatEntityMaps,
    'flatObjectMetadataMaps' | 'flatFieldMetadataMaps' | 'flatIndexMaps'
-  > & {
-    flatApplication: FlatApplication;
-  };
+  >;
+
 const FIELD_METADATA_UPDATE_INDEX_SIDE_EFFECT: FieldMetadataUpdateIndexSideEffect =
  {
    flatIndexMetadatasToUpdate: [],
@@ -45,14 +37,8 @@ export const handleIndexChangesDuringFieldUpdate = ({
  flatIndexMaps,
  flatObjectMetadataMaps,
  flatFieldMetadataMaps,
-  flatApplication,
 }: FromToFlatFieldMetadataAndFlatEntityMaps): FieldInputTranspilationResult<FieldMetadataUpdateIndexSideEffect> => {
-  if (
-    !hasIndexRelevantChanges({
-      fromFlatFieldMetadata,
-      toFlatFieldMetadata,
-    })
-  ) {
+  if (fromFlatFieldMetadata.name === toFlatFieldMetadata.name) {
    return {
      status: 'success',
      result: FIELD_METADATA_UPDATE_INDEX_SIDE_EFFECT,
@@ -68,119 +54,17 @@ export const handleIndexChangesDuringFieldUpdate = ({
    flatFieldMetadata: fromFlatFieldMetadata,
    flatObjectMetadata,
    flatIndexMaps,
-  });
+  }).filter(
+    (flatIndexMetadata) => !isSystemUniqueFlatIndexMetadata(flatIndexMetadata),
+  );

  if (relatedIndexes.length === 0) {
-    return handleNoExistingIndexes({
-      toFlatFieldMetadata,
-      flatObjectMetadata,
-    });
-  }
-
-  return handleExistingIndexes({
-    toFlatFieldMetadata,
-    fromFlatFieldMetadata,
-    relatedIndexes,
-    flatObjectMetadata,
-    flatFieldMetadataMaps,
-    flatApplication,
-  });
-};
-
-const hasIndexRelevantChanges = ({
-  fromFlatFieldMetadata,
-  toFlatFieldMetadata,
-}: FromTo<FlatFieldMetadata, 'flatFieldMetadata'>): boolean =>
-  fromFlatFieldMetadata.name !== toFlatFieldMetadata.name ||
-  fromFlatFieldMetadata.isUnique !== toFlatFieldMetadata.isUnique;
-
-const handleNoExistingIndexes = ({
-  toFlatFieldMetadata,
-  flatObjectMetadata,
-}: {
-  toFlatFieldMetadata: FlatFieldMetadata;
-  flatObjectMetadata: FlatObjectMetadata;
-}): FieldInputTranspilationResult<FieldMetadataUpdateIndexSideEffect> => {
-  if (!toFlatFieldMetadata.isUnique) {
    return {
      status: 'success',
      result: FIELD_METADATA_UPDATE_INDEX_SIDE_EFFECT,
    };
  }

-  const newIndex = generateIndexForFlatFieldMetadata({
-    flatFieldMetadata: toFlatFieldMetadata,
-    flatObjectMetadata,
-  });
-
-  return {
-    status: 'success',
-    result: {
-      ...FIELD_METADATA_UPDATE_INDEX_SIDE_EFFECT,
-      flatIndexMetadatasToCreate: [newIndex],
-    },
-  };
-};
-
-const handleExistingIndexes = ({
-  fromFlatFieldMetadata,
-  toFlatFieldMetadata,
-  relatedIndexes,
-  flatObjectMetadata,
-  flatFieldMetadataMaps,
-  flatApplication,
-}: {
-  relatedIndexes: FlatIndexMetadata[];
-  flatObjectMetadata: FlatObjectMetadata;
-  flatFieldMetadataMaps: AllFlatEntityMaps['flatFieldMetadataMaps'];
-} & FromTo<FlatFieldMetadata, 'flatFieldMetadata'> & {
-    flatApplication: FlatApplication;
-  }): FieldInputTranspilationResult<FieldMetadataUpdateIndexSideEffect> => {
-  if (
-    toFlatFieldMetadata.isUnique === false &&
-    !isMorphOrRelationFlatFieldMetadata(fromFlatFieldMetadata)
-  ) {
-    const expectedUniqueIndex = generateIndexForFlatFieldMetadata({
-      flatFieldMetadata: {
-        ...fromFlatFieldMetadata,
-        isUnique: true,
-      },
-      flatObjectMetadata,
-    });
-
-    const uniqueIndexToDelete = relatedIndexes.find(
-      (index) => index.name === expectedUniqueIndex.name,
-    );
-
-    if (
-      isDefined(uniqueIndexToDelete) &&
-      ((isDefined(uniqueIndexToDelete.applicationId) &&
-        uniqueIndexToDelete.applicationId !== flatApplication.id) ||
-        !uniqueIndexToDelete.isCustom)
-    ) {
-      return {
-        status: 'fail',
-        errors: [
-          {
-            code: FieldMetadataExceptionCode.INVALID_FIELD_INPUT,
-            message:
-              'Cannot delete unique index that have not been created by the workspace custom application',
-            userFriendlyMessage: msg`Cannot delete unique index that have not been created by the workspace custom application`,
-          },
-        ],
-      };
-    }
-
-    return {
-      status: 'success',
-      result: {
-        ...FIELD_METADATA_UPDATE_INDEX_SIDE_EFFECT,
-        flatIndexMetadatasToDelete: uniqueIndexToDelete
-          ? [uniqueIndexToDelete]
-          : [],
-      },
-    };
-  }
  const updatedIndexes = recomputeIndexOnFlatFieldMetadataNameUpdate({
    flatFieldMetadataMaps,
    flatObjectMetadata,
@@ -0,0 +1,7 @@
+import { type UniversalFlatFieldMetadata } from 'src/engine/workspace-manager/workspace-migration/universal-flat-entity/types/universal-flat-field-metadata.type';
+
+// The primary key already enforces uniqueness via its constraint, so it must not
+// spawn a redundant application-managed unique index.
+export const isPrimaryKeyFlatFieldMetadata = (
+  flatFieldMetadata: Pick<UniversalFlatFieldMetadata, 'name'>,
+): boolean => flatFieldMetadata.name === 'id';