From 43730d7748082bbfea31faca05861f3c1d990795 Mon Sep 17 00:00:00 2001 From: Paul Rastoin <45004772+prastoin@users.noreply.github.com> Date: Fri, 3 Jul 2026 18:13:20 +0200 Subject: [PATCH] Centralized side effects devxp basis (#22295) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit # Introduction This PR introduces a centralized, strictly-typed **metadata side-effect engine** that unifies how system metadata side effects are derived and applied across both metadata entry points — the **metadata GraphQL API** and the **application sync / manifest** flow — and migrates the first side effect end-to-end: **a unique scalar field owns its backing single-field `UNIQUE` index** (full create / update / delete lifecycle). ## New conventions - **Engine-owned companions**: metadata flagged `isSystemSideEffect: true` is owned by the engine. Its deletion is never inferred from absence in a manifest — it results from PG-level cascade or from a delete side effect (a side effect always has a cause, its parent metadata). - **Reserved deterministic identifiers**: apps cannot declare metadata reusing an engine-owned deterministic `universalIdentifier`. Doing so fails validation with `RESERVED_SYSTEM_UNIVERSAL_IDENTIFIER` (until an explicit override API exists). - **Record-native operation matrix**: the operation matrix is keyed by `universalIdentifier` (`AllFlatEntityOperationRecordByMetadataName`) instead of arrays, making parent resolution and deduplication O(1). Array-based API callers are transpiled to records at the validate-build-and-run boundary. - Twenty-sdk user-facing experience with system fields will only be related to overrides. # What this PR does ## 1. Side-effect engine (foundation) - `MetadataSideEffectEngineService.expandWithSideEffects(...)` takes the intention-carrying record matrix and returns it expanded with derived side effects, or a structured failure. - Handlers are registered via a typed **decorator + registry** pattern (`MetadataSideEffectHandler({ operation, metadataName, name, description })`), with runtime duplicate-name detection. Multiple handlers per (operation, metadataName) are supported. - Handler contract mirrors the validator pattern: - receives the trigger flat entity, the live record matrix, and **strictly-typed related flat entity maps** (`MetadataFlatEntityAndRelatedFlatEntityMapsForSideEffect

`, derived from declared companion metadata names — no loose `Partial` context) - returns `MetadataSideEffectResult`: `success` (operations record) | `noop` | `fail` (structured failure) - **Non-recursion is structural**: triggers are read from the original caller input, never from the expanded matrix, so a side effect can never trigger another side effect. - **Deduplication + collision detection**: side effects are deduped by `universalIdentifier` per operation; a caller-declared entity colliding with an engine-owned deterministic identifier is recorded as a collision. - **Unified failure channel**: handler failures and reserved-identifier collisions are merged into the same `OrchestratorFailureReport` contract as builder validation errors, and the run short-circuits (fail-closed, nothing is applied). ## 2. First migrated side effect — unique field → backing unique index Three handlers own the complete lifecycle of the deterministic single-field `UNIQUE` index backing a unique scalar field: - **create**: unique scalar field → generate the deterministic backing index (`fieldUniqueBackingIndexOnCreate`) - **update**: `isUnique` flips and renames of still-unique fields (the index name — and therefore its deterministic identifier — derives from the field name, so a rename drops the stale index and recreates the deterministic one) (`fieldUniqueBackingIndexOnUpdate`) - **delete**: cascade-delete the backing index (`fieldUniqueBackingIndexOnDelete`) Supporting rules: - The primary key `id` field never spawns a backing index (uniqueness comes from the PK constraint) — explicit `isPrimaryKeyFlatFieldMetadata` guard. - Parent object resolution is **optimistic-first**: an object created or updated in the same batch wins over the workspace cache (so e.g. renaming an object while flipping a field to unique builds the index from the post-rename object), resolved in O(1) via the record matrix. - A missing parent object is reported as a structured side-effect failure, never silently skipped. ## 3. Path convergence — manifest and API share one flow - The manifest sync now derives a from→to **record matrix** from the cache and feeds `validateBuildAndRunWorkspaceMigrationFromRecord`, the same flow the API uses — both paths converge on the engine. - Manifest-side unique-index generation and API transpiler system-unique-index handling were removed (declared/composite/relation indexes stay untouched). - New `WorkspaceMigrationFlatEntityMapsService` mutualizes flat-entity-maps computation between the side-effect engine and the builder: cache keys are derived from the caller metadata names (+ validation- and side-effect-related closures) instead of hardcoded loads. - App-scoping and pruning are folded into one shared primitive (`getSubAllFlatEntityMapsByApplicationIdsOrThrow`): slicing dependency maps to the involved applications always prunes dangling one-to-many aggregators — callers can no longer forget it. - **Behavior change**: an app extending another app's view with a view field now syncs successfully (cross-app view-field extension), covered by a dedicated integration test. ## 4. Backfill upgrade command (2.19) `upgrade:2-19:backfill-system-unique-index-universal-identifier` rewrites legacy system unique-index `universalIdentifier`s to their deterministic value so the engine can own pre-existing indexes. The backfill is **driven from `isUnique: true` fields** (mirroring the engine ownership predicate — excludes PK / morph / relation fields) and resolves each field's backing index in O(1). # Bugs fixed along the way - `database:reset` seeding failed with `INDEX_FIELD_INVALID_DEFAULT_VALUE`: the engine derived a backing `UNIQUE` index for the default `id` primary key. Fixed with the explicit primary-key guard. - `isUnique` updates on system-flagged standard fields (e.g. auto-created `name`) did not trigger the backing-index side effect. - Manifest sync crashed with "Could not find flat entity with universal identifier ..." when app-scoped slices left dangling aggregator references — fixed by centralizing pruning in the shared slice primitive --- .../2-19-upgrade-version-command.module.ts | 21 +- ...ique-index-universal-identifier.command.ts | 175 ++++++++ .../application-manifest-migration.service.ts | 46 +- ...-all-universal-flat-entity-maps.service.ts | 35 -- ...cord-by-metadata-name-from-from-to.util.ts | 149 +++++++ ...plication-sub-all-flat-entity-maps.util.ts | 37 +- ...ffect-companion-metadata-names.constant.ts | 7 + ...-operation-record-by-metadata-name.type.ts | 15 + ...lat-entity-to-create-delete-update.type.ts | 4 + ...d-flat-entity-maps-for-side-effect.type.ts | 13 + ...side-effect-related-metadata-names.type.ts | 14 + ...tadata-side-effect-companion-names.util.ts | 16 + ...y-maps-by-application-ids-or-throw.util.ts | 51 +++ .../is-system-side-effect-flat-entity.util.ts | 8 + ...-flat-entity-maps-through-mutation.util.ts | 42 +- ...t-entity-operation-array-to-record.util.ts | 94 ++++ ...-to-flat-field-metadatas-to-create.util.ts | 14 +- ...-to-flat-field-metadatas-to-delete.util.ts | 32 +- ...field-input-to-flat-field-metadata.util.ts | 1 - ...x-for-flat-field-metadata-or-throw.util.ts | 44 ++ ...-field-metadata-update-side-effect.util.ts | 7 +- ...-index-changes-during-field-update.util.ts | 130 +----- ...is-primary-key-flat-field-metadata.util.ts | 7 + ...-system-unique-flat-index-metadata.util.ts | 6 + .../utils/compute-flat-index-name.util.ts | 96 +++++ .../generate-deterministic-flat-index.util.ts | 56 +++ ...s => generate-deterministic-index-name.ts} | 2 +- .../utils/generate-flat-index.util.ts | 78 +--- ...de-effect-handler-metadata-key.constant.ts | 2 + .../metadata-side-effect-exception-code.ts | 4 + ...x-on-create-side-effect-handler.service.ts | 74 ++++ ...x-on-delete-side-effect-handler.service.ts | 74 ++++ ...x-on-update-side-effect-handler.service.ts | 149 +++++++ ...de-effect-parent-not-found-failure.util.ts | 31 ++ ...-after-state-for-field-side-effect.util.ts | 26 ++ .../metadata-side-effect-handlers.module.ts | 14 + ...se-metadata-side-effect-handler.service.ts | 61 +++ .../metadata-side-effect.module.ts | 16 + ...ta-side-effect-handler-registry.service.ts | 109 +++++ .../metadata-side-effect-engine.service.ts | 329 ++++++++++++++ .../metadata-side-effect-context.type.ts | 5 + ...adata-side-effect-expansion-result.type.ts | 15 + .../metadata-side-effect-operation.type.ts | 24 ++ ...effect-operations-by-metadata-name.type.ts | 17 + .../types/metadata-side-effect-result.type.ts | 27 ++ ...ect-universal-identifier-collision.type.ts | 10 + ...m-side-effect-collision-to-failure.util.ts | 29 ++ ...mpany-standard-flat-index-metadata.util.ts | 1 + ...erson-standard-flat-index-metadata.util.ts | 1 + ...ember-standard-flat-index-metadata.util.ts | 1 + ...reate-standard-index-flat-metadata.util.ts | 79 ++-- ...pace-migration-flat-entity-maps.service.ts | 341 +++++++++++++++ ...igration-validate-build-and-run-service.ts | 405 ++++-------------- .../update-field-action-handler.service.ts | 4 +- .../workspace-migration.module.ts | 4 + ...-effect-collision.integration-spec.ts.snap | 37 ++ ...-side-effect-collision.integration-spec.ts | 119 +++++ ...-cross-app-view-field.integration-spec.ts} | 78 +++- ...on-unique-field-rename.integration-spec.ts | 245 +++++++++++ 59 files changed, 2840 insertions(+), 691 deletions(-) create mode 100644 packages/twenty-server/src/database/commands/upgrade-version-command/2-19/2-19-workspace-command-1783093620000-backfill-system-unique-index-universal-identifier.command.ts create mode 100644 packages/twenty-server/src/engine/core-modules/application/application-manifest/utils/build-all-flat-entity-operation-record-by-metadata-name-from-from-to.util.ts create mode 100644 packages/twenty-server/src/engine/metadata-modules/flat-entity/constant/all-metadata-side-effect-companion-metadata-names.constant.ts create mode 100644 packages/twenty-server/src/engine/metadata-modules/flat-entity/types/all-flat-entity-operation-record-by-metadata-name.type.ts create mode 100644 packages/twenty-server/src/engine/metadata-modules/flat-entity/types/metadata-flat-entity-and-related-flat-entity-maps-for-side-effect.type.ts create mode 100644 packages/twenty-server/src/engine/metadata-modules/flat-entity/types/metadata-side-effect-related-metadata-names.type.ts create mode 100644 packages/twenty-server/src/engine/metadata-modules/flat-entity/utils/get-metadata-side-effect-companion-names.util.ts create mode 100644 packages/twenty-server/src/engine/metadata-modules/flat-entity/utils/get-sub-all-flat-entity-maps-by-application-ids-or-throw.util.ts create mode 100644 packages/twenty-server/src/engine/metadata-modules/flat-entity/utils/is-system-side-effect-flat-entity.util.ts create mode 100644 packages/twenty-server/src/engine/metadata-modules/flat-entity/utils/transpile-flat-entity-operation-array-to-record.util.ts create mode 100644 packages/twenty-server/src/engine/metadata-modules/flat-field-metadata/utils/generate-deterministic-index-for-flat-field-metadata-or-throw.util.ts create mode 100644 packages/twenty-server/src/engine/metadata-modules/flat-field-metadata/utils/is-primary-key-flat-field-metadata.util.ts create mode 100644 packages/twenty-server/src/engine/metadata-modules/flat-index-metadata/utils/is-system-unique-flat-index-metadata.util.ts create mode 100644 packages/twenty-server/src/engine/metadata-modules/index-metadata/utils/compute-flat-index-name.util.ts create mode 100644 packages/twenty-server/src/engine/metadata-modules/index-metadata/utils/generate-deterministic-flat-index.util.ts rename packages/twenty-server/src/engine/metadata-modules/index-metadata/utils/{generate-deterministic-index-name-v2.ts => generate-deterministic-index-name.ts} (96%) create mode 100644 packages/twenty-server/src/engine/metadata-modules/metadata-side-effect/constants/metadata-side-effect-handler-metadata-key.constant.ts create mode 100644 packages/twenty-server/src/engine/metadata-modules/metadata-side-effect/exceptions/metadata-side-effect-exception-code.ts create mode 100644 packages/twenty-server/src/engine/metadata-modules/metadata-side-effect/handlers/field-metadata/services/field-unique-backing-index-on-create-side-effect-handler.service.ts create mode 100644 packages/twenty-server/src/engine/metadata-modules/metadata-side-effect/handlers/field-metadata/services/field-unique-backing-index-on-delete-side-effect-handler.service.ts create mode 100644 packages/twenty-server/src/engine/metadata-modules/metadata-side-effect/handlers/field-metadata/services/field-unique-backing-index-on-update-side-effect-handler.service.ts create mode 100644 packages/twenty-server/src/engine/metadata-modules/metadata-side-effect/handlers/field-metadata/utils/build-field-side-effect-parent-not-found-failure.util.ts create mode 100644 packages/twenty-server/src/engine/metadata-modules/metadata-side-effect/handlers/field-metadata/utils/resolve-parent-flat-object-metadata-after-state-for-field-side-effect.util.ts create mode 100644 packages/twenty-server/src/engine/metadata-modules/metadata-side-effect/handlers/metadata-side-effect-handlers.module.ts create mode 100644 packages/twenty-server/src/engine/metadata-modules/metadata-side-effect/interfaces/base-metadata-side-effect-handler.service.ts create mode 100644 packages/twenty-server/src/engine/metadata-modules/metadata-side-effect/metadata-side-effect.module.ts create mode 100644 packages/twenty-server/src/engine/metadata-modules/metadata-side-effect/registry/metadata-side-effect-handler-registry.service.ts create mode 100644 packages/twenty-server/src/engine/metadata-modules/metadata-side-effect/services/metadata-side-effect-engine.service.ts create mode 100644 packages/twenty-server/src/engine/metadata-modules/metadata-side-effect/types/metadata-side-effect-context.type.ts create mode 100644 packages/twenty-server/src/engine/metadata-modules/metadata-side-effect/types/metadata-side-effect-expansion-result.type.ts create mode 100644 packages/twenty-server/src/engine/metadata-modules/metadata-side-effect/types/metadata-side-effect-operation.type.ts create mode 100644 packages/twenty-server/src/engine/metadata-modules/metadata-side-effect/types/metadata-side-effect-operations-by-metadata-name.type.ts create mode 100644 packages/twenty-server/src/engine/metadata-modules/metadata-side-effect/types/metadata-side-effect-result.type.ts create mode 100644 packages/twenty-server/src/engine/metadata-modules/metadata-side-effect/types/system-side-effect-universal-identifier-collision.type.ts create mode 100644 packages/twenty-server/src/engine/metadata-modules/metadata-side-effect/utils/map-system-side-effect-collision-to-failure.util.ts create mode 100644 packages/twenty-server/src/engine/workspace-manager/workspace-migration/services/workspace-migration-flat-entity-maps.service.ts create mode 100644 packages/twenty-server/test/integration/metadata/suites/application/__snapshots__/failing-sync-application-unique-index-side-effect-collision.integration-spec.ts.snap create mode 100644 packages/twenty-server/test/integration/metadata/suites/application/failing-sync-application-unique-index-side-effect-collision.integration-spec.ts rename packages/twenty-server/test/integration/metadata/suites/application/{failing-sync-application-cross-app-view-field.integration-spec.ts => successful-sync-application-cross-app-view-field.integration-spec.ts} (57%) create mode 100644 packages/twenty-server/test/integration/metadata/suites/application/successful-sync-application-unique-field-rename.integration-spec.ts diff --git a/packages/twenty-server/src/database/commands/upgrade-version-command/2-19/2-19-upgrade-version-command.module.ts b/packages/twenty-server/src/database/commands/upgrade-version-command/2-19/2-19-upgrade-version-command.module.ts index c1a2920a46..659f4e0e6f 100644 --- a/packages/twenty-server/src/database/commands/upgrade-version-command/2-19/2-19-upgrade-version-command.module.ts +++ b/packages/twenty-server/src/database/commands/upgrade-version-command/2-19/2-19-upgrade-version-command.module.ts @@ -3,16 +3,31 @@ import { TypeOrmModule } from '@nestjs/typeorm'; import { WorkspaceIteratorModule } from 'src/database/commands/command-runners/workspace-iterator.module'; import { BackfillWorkspaceCustomApplicationRegistrationCommand } from 'src/database/commands/upgrade-version-command/2-19/2-19-workspace-command-1782853718000-backfill-workspace-custom-application-registration.command'; -import { ApplicationModule } from 'src/engine/core-modules/application/application.module'; +import { BackfillSystemUniqueIndexUniversalIdentifierCommand } from 'src/database/commands/upgrade-version-command/2-19/2-19-workspace-command-1783093620000-backfill-system-unique-index-universal-identifier.command'; import { ApplicationEntity } from 'src/engine/core-modules/application/application.entity'; +import { ApplicationModule } from 'src/engine/core-modules/application/application.module'; import { WorkspaceEntity } from 'src/engine/core-modules/workspace/workspace.entity'; +import { IndexMetadataEntity } from 'src/engine/metadata-modules/index-metadata/index-metadata.entity'; +import { WorkspaceMetadataVersionModule } from 'src/engine/metadata-modules/workspace-metadata-version/workspace-metadata-version.module'; +import { WorkspaceCacheModule } from 'src/engine/workspace-cache/workspace-cache.module'; +import { WorkspaceMigrationModule } from 'src/engine/workspace-manager/workspace-migration/workspace-migration.module'; @Module({ imports: [ ApplicationModule, - TypeOrmModule.forFeature([WorkspaceEntity, ApplicationEntity]), + TypeOrmModule.forFeature([ + WorkspaceEntity, + ApplicationEntity, + IndexMetadataEntity, + ]), WorkspaceIteratorModule, + WorkspaceCacheModule, + WorkspaceMetadataVersionModule, + WorkspaceMigrationModule, + ], + providers: [ + BackfillWorkspaceCustomApplicationRegistrationCommand, + BackfillSystemUniqueIndexUniversalIdentifierCommand, ], - providers: [BackfillWorkspaceCustomApplicationRegistrationCommand], }) export class V2_19_UpgradeVersionCommandModule {} diff --git a/packages/twenty-server/src/database/commands/upgrade-version-command/2-19/2-19-workspace-command-1783093620000-backfill-system-unique-index-universal-identifier.command.ts b/packages/twenty-server/src/database/commands/upgrade-version-command/2-19/2-19-workspace-command-1783093620000-backfill-system-unique-index-universal-identifier.command.ts new file mode 100644 index 0000000000..f4c18d6f34 --- /dev/null +++ b/packages/twenty-server/src/database/commands/upgrade-version-command/2-19/2-19-workspace-command-1783093620000-backfill-system-unique-index-universal-identifier.command.ts @@ -0,0 +1,175 @@ +import { Command } from 'nest-commander'; + +import { InjectRepository } from '@nestjs/typeorm'; +import { getIndexUniversalIdentifier } from 'twenty-shared/application'; +import { isDefined } from 'twenty-shared/utils'; +import { Repository } from 'typeorm'; + +import { ActiveOrSuspendedWorkspaceCommandRunner } from 'src/database/commands/command-runners/active-or-suspended-workspace.command-runner'; +import { WorkspaceIteratorService } from 'src/database/commands/command-runners/workspace-iterator.service'; +import { type RunOnWorkspaceArgs } from 'src/database/commands/command-runners/workspace.command-runner'; +import { RegisteredWorkspaceCommand } from 'src/engine/core-modules/upgrade/decorators/registered-workspace-command.decorator'; +import { findFlatEntityByIdInFlatEntityMaps } from 'src/engine/metadata-modules/flat-entity/utils/find-flat-entity-by-id-in-flat-entity-maps.util'; +import { getMetadataFlatEntityMapsKey } from 'src/engine/metadata-modules/flat-entity/utils/get-metadata-flat-entity-maps-key.util'; +import { getMetadataRelatedMetadataNames } from 'src/engine/metadata-modules/flat-entity/utils/get-metadata-related-metadata-names.util'; +import { getMetadataSerializedRelationNames } from 'src/engine/metadata-modules/flat-entity/utils/get-metadata-serialized-relation-names.util'; +import { isMorphOrRelationFlatFieldMetadata } from 'src/engine/metadata-modules/flat-field-metadata/utils/is-morph-or-relation-flat-field-metadata.util'; +import { isPrimaryKeyFlatFieldMetadata } from 'src/engine/metadata-modules/flat-field-metadata/utils/is-primary-key-flat-field-metadata.util'; +import { isSystemUniqueFlatIndexMetadata } from 'src/engine/metadata-modules/flat-index-metadata/utils/is-system-unique-flat-index-metadata.util'; +import { IndexMetadataEntity } from 'src/engine/metadata-modules/index-metadata/index-metadata.entity'; +import { WorkspaceMetadataVersionService } from 'src/engine/metadata-modules/workspace-metadata-version/services/workspace-metadata-version.service'; +import { WorkspaceCacheService } from 'src/engine/workspace-cache/services/workspace-cache.service'; + +@RegisteredWorkspaceCommand('2.19.0', 1783093620000) +@Command({ + name: 'upgrade:2-19:backfill-system-unique-index-universal-identifier', + description: + 'Backfill the deterministic universal identifier of system unique indexes (the index backing a unique scalar field) so the metadata side-effect engine can own their lifecycle.', +}) +export class BackfillSystemUniqueIndexUniversalIdentifierCommand extends ActiveOrSuspendedWorkspaceCommandRunner { + constructor( + protected readonly workspaceIteratorService: WorkspaceIteratorService, + private readonly workspaceCacheService: WorkspaceCacheService, + private readonly workspaceMetadataVersionService: WorkspaceMetadataVersionService, + // eslint-disable-next-line twenty/prefer-workspace-scoped-repository + @InjectRepository(IndexMetadataEntity) + private readonly indexMetadataRepository: Repository, + ) { + super(workspaceIteratorService); + } + + override async runOnWorkspace({ + workspaceId, + options, + }: RunOnWorkspaceArgs): Promise { + const isDryRun = options.dryRun ?? false; + + const { + flatObjectMetadataMaps, + flatFieldMetadataMaps, + flatIndexMaps, + flatApplicationMaps, + } = await this.workspaceCacheService.getOrRecompute(workspaceId, [ + 'flatObjectMetadataMaps', + 'flatFieldMetadataMaps', + 'flatIndexMaps', + 'flatApplicationMaps', + ]); + + const backingFlatIndexMetadataByFieldMetadataId = new Map( + Object.values(flatIndexMaps.byUniversalIdentifier) + .filter(isDefined) + .filter( + (flatIndexMetadata) => + isSystemUniqueFlatIndexMetadata(flatIndexMetadata) && + flatIndexMetadata.flatIndexFieldMetadatas.length === 1, + ) + .map((flatIndexMetadata): [string, typeof flatIndexMetadata] => [ + flatIndexMetadata.flatIndexFieldMetadatas[0].fieldMetadataId, + flatIndexMetadata, + ]), + ); + + const indexesToBackfill = Object.values( + flatFieldMetadataMaps.byUniversalIdentifier, + ) + .filter(isDefined) + .filter( + (flatFieldMetadata) => + flatFieldMetadata.isUnique === true && + !isMorphOrRelationFlatFieldMetadata(flatFieldMetadata) && + !isPrimaryKeyFlatFieldMetadata(flatFieldMetadata), + ) + .flatMap((flatFieldMetadata) => { + const flatObjectMetadata = findFlatEntityByIdInFlatEntityMaps({ + flatEntityId: flatFieldMetadata.objectMetadataId, + flatEntityMaps: flatObjectMetadataMaps, + }); + + if ( + !isDefined(flatObjectMetadata) || + !isDefined(flatFieldMetadata.applicationId) + ) { + return []; + } + + const applicationUniversalIdentifier = + flatApplicationMaps.byId[flatFieldMetadata.applicationId] + ?.universalIdentifier; + + if (!isDefined(applicationUniversalIdentifier)) { + return []; + } + + const backingFlatIndexMetadata = + backingFlatIndexMetadataByFieldMetadataId.get(flatFieldMetadata.id); + + if (!isDefined(backingFlatIndexMetadata)) { + return []; + } + + const deterministicUniversalIdentifier = getIndexUniversalIdentifier({ + applicationUniversalIdentifier, + objectUniversalIdentifier: flatObjectMetadata.universalIdentifier, + name: backingFlatIndexMetadata.name, + }); + + if ( + deterministicUniversalIdentifier === + backingFlatIndexMetadata.universalIdentifier + ) { + return []; + } + + return [ + { + id: backingFlatIndexMetadata.id, + name: backingFlatIndexMetadata.name, + deterministicUniversalIdentifier, + }, + ]; + }); + + if (indexesToBackfill.length === 0) { + this.logger.log( + `No system unique index universal identifier to backfill for workspace ${workspaceId}`, + ); + + return; + } + + this.logger.log( + `${isDryRun ? '[DRY RUN] ' : ''}Backfilling ${indexesToBackfill.length} system unique index universal identifier(s) for workspace ${workspaceId}: ${indexesToBackfill.map(({ name }) => name).join(', ')}`, + ); + + if (isDryRun) { + return; + } + + for (const { id, deterministicUniversalIdentifier } of indexesToBackfill) { + await this.indexMetadataRepository.update( + { id, workspaceId }, + { universalIdentifier: deterministicUniversalIdentifier }, + ); + } + + const indexRelatedMetadataNames = [ + 'index', + ...getMetadataRelatedMetadataNames('index'), + ...getMetadataSerializedRelationNames('index'), + ] as const; + const cacheKeysToFlush = [ + ...new Set(indexRelatedMetadataNames.map(getMetadataFlatEntityMapsKey)), + ]; + + await this.workspaceCacheService.flush(workspaceId, cacheKeysToFlush); + + await this.workspaceMetadataVersionService.incrementMetadataVersion( + workspaceId, + ); + + this.logger.log( + `Backfilled ${indexesToBackfill.length} system unique index universal identifier(s) for workspace ${workspaceId}`, + ); + } +} diff --git a/packages/twenty-server/src/engine/core-modules/application/application-manifest/application-manifest-migration.service.ts b/packages/twenty-server/src/engine/core-modules/application/application-manifest/application-manifest-migration.service.ts index ffeb9ea228..23ec8afd45 100644 --- a/packages/twenty-server/src/engine/core-modules/application/application-manifest/application-manifest-migration.service.ts +++ b/packages/twenty-server/src/engine/core-modules/application/application-manifest/application-manifest-migration.service.ts @@ -5,6 +5,7 @@ import { ALL_METADATA_NAME } from 'twenty-shared/metadata'; import { isDefined } from 'twenty-shared/utils'; import { ComputeApplicationManifestAllUniversalFlatEntityMapsService } from 'src/engine/core-modules/application/application-manifest/services/compute-application-manifest-all-universal-flat-entity-maps.service'; +import { buildAllFlatEntityOperationRecordByMetadataNameFromFromTo } from 'src/engine/core-modules/application/application-manifest/utils/build-all-flat-entity-operation-record-by-metadata-name-from-from-to.util'; import { buildFromToAllUniversalFlatEntityMaps } from 'src/engine/core-modules/application/application-manifest/utils/build-from-to-all-universal-flat-entity-maps.util'; import { getApplicationSubAllFlatEntityMaps } from 'src/engine/core-modules/application/application-manifest/utils/get-application-sub-all-flat-entity-maps.util'; import { @@ -173,11 +174,6 @@ export class ApplicationManifestMigrationService { }> { const now = new Date().toISOString(); - const { twentyStandardFlatApplication } = - await this.applicationService.findWorkspaceTwentyStandardAndCustomApplicationOrThrow( - { workspaceId }, - ); - const recomputeStart = performance.now(); const cacheResult = await this.workspaceCacheService.getOrRecompute( workspaceId, @@ -193,7 +189,8 @@ export class ApplicationManifestMigrationService { ApplicationManifestMigrationService.name, ); - const { featureFlagsMap, ...existingAllFlatEntityMaps } = cacheResult; + const { featureFlagsMap: _featureFlagsMap, ...existingAllFlatEntityMaps } = + cacheResult; const fromAllFlatEntityMaps = getApplicationSubAllFlatEntityMaps({ applicationIds: [ownerFlatApplication.id], @@ -208,32 +205,27 @@ export class ApplicationManifestMigrationService { workspaceId, }); - const dependencyAllFlatEntityMaps = getApplicationSubAllFlatEntityMaps({ - applicationIds: - ownerFlatApplication.universalIdentifier === - TWENTY_STANDARD_APPLICATION.universalIdentifier - ? [twentyStandardFlatApplication.id] - : [ownerFlatApplication.id, twentyStandardFlatApplication.id], - fromAllFlatEntityMaps: existingAllFlatEntityMaps, - }); + const allFlatEntityOperationRecordByMetadataName = + buildAllFlatEntityOperationRecordByMetadataNameFromFromTo({ + fromAllFlatEntityMaps, + toAllUniversalFlatEntityMaps, + buildOptions: { + isSystemBuild: false, + inferDeletionFromMissingEntities: true, + applicationUniversalIdentifier: + ownerFlatApplication.universalIdentifier, + }, + }); const validateBuildRunStart = performance.now(); const validateAndBuildResult = - await this.workspaceMigrationValidateBuildAndRunService.validateBuildAndRunWorkspaceMigrationFromTo( + await this.workspaceMigrationValidateBuildAndRunService.validateBuildAndRunWorkspaceMigrationFromRecord( { - buildOptions: { - isSystemBuild: false, - inferDeletionFromMissingEntities: true, - applicationUniversalIdentifier: - ownerFlatApplication.universalIdentifier, - }, - fromToAllFlatEntityMaps: buildFromToAllUniversalFlatEntityMaps({ - fromAllFlatEntityMaps, - toAllUniversalFlatEntityMaps, - }), + allFlatEntityOperationRecordByMetadataName, workspaceId, - dependencyAllFlatEntityMaps, - additionalCacheDataMaps: { featureFlagsMap }, + isSystemBuild: false, + applicationUniversalIdentifier: + ownerFlatApplication.universalIdentifier, dryRun, }, ); diff --git a/packages/twenty-server/src/engine/core-modules/application/application-manifest/services/compute-application-manifest-all-universal-flat-entity-maps.service.ts b/packages/twenty-server/src/engine/core-modules/application/application-manifest/services/compute-application-manifest-all-universal-flat-entity-maps.service.ts index ac5f52c969..522094dba5 100644 --- a/packages/twenty-server/src/engine/core-modules/application/application-manifest/services/compute-application-manifest-all-universal-flat-entity-maps.service.ts +++ b/packages/twenty-server/src/engine/core-modules/application/application-manifest/services/compute-application-manifest-all-universal-flat-entity-maps.service.ts @@ -40,7 +40,6 @@ import { fromAgentManifestToUniversalFlatAgent } from 'src/engine/core-modules/a import { type EncryptedString } from 'src/engine/core-modules/secret-encryption/branded-strings/encrypted-string.type'; import { type PlaintextString } from 'src/engine/core-modules/secret-encryption/branded-strings/plaintext-string.type'; import { SecretEncryptionService } from 'src/engine/core-modules/secret-encryption/secret-encryption.service'; -import { generateIndexForFlatFieldMetadata } from 'src/engine/metadata-modules/flat-field-metadata/utils/generate-index-for-flat-field-metadata.util'; import { createEmptyAllFlatEntityMaps } from 'src/engine/metadata-modules/flat-entity/constant/create-empty-all-flat-entity-maps.constant'; import { type AllFlatEntityMaps } from 'src/engine/metadata-modules/flat-entity/types/all-flat-entity-maps.type'; import { type UniversalFlatFieldMetadata } from 'src/engine/workspace-manager/workspace-migration/universal-flat-entity/types/universal-flat-field-metadata.type'; @@ -115,19 +114,6 @@ export class ComputeApplicationManifestAllUniversalFlatEntityMapsService { universalFlatEntityMapsToMutate: allUniversalFlatEntityMaps.flatFieldMetadataMaps, }); - - if (flatFieldMetadata.isUnique) { - addUniversalFlatEntityToUniversalFlatEntityMapsThroughMutationOrThrow( - { - universalFlatEntity: generateIndexForFlatFieldMetadata({ - flatFieldMetadata, - flatObjectMetadata, - }), - universalFlatEntityMapsToMutate: - allUniversalFlatEntityMaps.flatIndexMaps, - }, - ); - } } } @@ -143,27 +129,6 @@ export class ComputeApplicationManifestAllUniversalFlatEntityMapsService { universalFlatEntityMapsToMutate: allUniversalFlatEntityMaps.flatFieldMetadataMaps, }); - - if (flatFieldMetadata.isUnique) { - const flatObjectMetadata = - allUniversalFlatEntityMaps.flatObjectMetadataMaps - .byUniversalIdentifier[ - flatFieldMetadata.objectMetadataUniversalIdentifier - ]; - - if (isDefined(flatObjectMetadata)) { - addUniversalFlatEntityToUniversalFlatEntityMapsThroughMutationOrThrow( - { - universalFlatEntity: generateIndexForFlatFieldMetadata({ - flatFieldMetadata, - flatObjectMetadata, - }), - universalFlatEntityMapsToMutate: - allUniversalFlatEntityMaps.flatIndexMaps, - }, - ); - } - } } const indexCountByObjectUniversalIdentifier = new Map(); diff --git a/packages/twenty-server/src/engine/core-modules/application/application-manifest/utils/build-all-flat-entity-operation-record-by-metadata-name-from-from-to.util.ts b/packages/twenty-server/src/engine/core-modules/application/application-manifest/utils/build-all-flat-entity-operation-record-by-metadata-name-from-from-to.util.ts new file mode 100644 index 0000000000..e4d2817b0d --- /dev/null +++ b/packages/twenty-server/src/engine/core-modules/application/application-manifest/utils/build-all-flat-entity-operation-record-by-metadata-name-from-from-to.util.ts @@ -0,0 +1,149 @@ +import { + ALL_METADATA_NAME, + type AllMetadataName, +} from 'twenty-shared/metadata'; +import { isDefined } from 'twenty-shared/utils'; + +import { type AllFlatEntityMaps } from 'src/engine/metadata-modules/flat-entity/types/all-flat-entity-maps.type'; +import { + type AllFlatEntityOperationRecordByMetadataName, + type FlatEntityOperationRecord, +} from 'src/engine/metadata-modules/flat-entity/types/all-flat-entity-operation-record-by-metadata-name.type'; +import { type MetadataUniversalFlatEntity } from 'src/engine/metadata-modules/flat-entity/types/metadata-universal-flat-entity.type'; +import { getMetadataFlatEntityMapsKey } from 'src/engine/metadata-modules/flat-entity/utils/get-metadata-flat-entity-maps-key.util'; +import { isSystemUniqueFlatIndexMetadata } from 'src/engine/metadata-modules/flat-index-metadata/utils/is-system-unique-flat-index-metadata.util'; +import { type MetadataUniversalFlatEntityMaps } from 'src/engine/workspace-manager/workspace-migration/universal-flat-entity/types/metadata-universal-flat-entity-maps.type'; +import { compareTwoFlatEntity } from 'src/engine/workspace-manager/workspace-migration/universal-flat-entity/utils/compare-two-universal-flat-entity.util'; +import { shouldInferDeletionFromMissingEntities } from 'src/engine/workspace-manager/workspace-migration/utils/should-infer-deletion-from-missing-entities.util'; +import { type WorkspaceMigrationBuilderOptions } from 'src/engine/workspace-manager/workspace-migration/workspace-migration-builder/types/workspace-migration-builder-options.type'; + +const toRecordByUniversalIdentifier = ( + flatEntities: MetadataUniversalFlatEntity[], +): Record> => + Object.fromEntries( + flatEntities.map((flatEntity) => [ + flatEntity.universalIdentifier, + flatEntity, + ]), + ); + +const buildFlatEntityOperationRecordForMetadata = ({ + metadataName, + fromFlatEntityMaps, + toFlatEntityMaps, + buildOptions, +}: { + metadataName: T; + fromFlatEntityMaps: MetadataUniversalFlatEntityMaps; + toFlatEntityMaps: MetadataUniversalFlatEntityMaps; + buildOptions: WorkspaceMigrationBuilderOptions; +}): FlatEntityOperationRecord => { + const fromByUniversalIdentifier = fromFlatEntityMaps.byUniversalIdentifier; + const toByUniversalIdentifier = toFlatEntityMaps.byUniversalIdentifier; + + const flatEntityToCreate = Object.values(toByUniversalIdentifier) + .filter(isDefined) + .filter( + (toFlatEntity) => + !isDefined(fromByUniversalIdentifier[toFlatEntity.universalIdentifier]), + ); + + const flatEntityToDelete = shouldInferDeletionFromMissingEntities({ + buildOptions, + metadataName, + }) + ? Object.values(fromByUniversalIdentifier) + .filter(isDefined) + .filter( + (fromFlatEntity) => + !isDefined( + toByUniversalIdentifier[fromFlatEntity.universalIdentifier], + ), + ) + .filter((fromFlatEntity) => { + if (metadataName !== ALL_METADATA_NAME.index) { + return true; + } + + return !isSystemUniqueFlatIndexMetadata( + fromFlatEntity as unknown as { + isSystemSideEffect: boolean; + isUnique: boolean; + }, + ); + }) + : []; + + const flatEntityToUpdate = Object.values(fromByUniversalIdentifier) + .filter(isDefined) + .map((fromFlatEntity) => { + const toFlatEntity = + toByUniversalIdentifier[fromFlatEntity.universalIdentifier]; + + if (!isDefined(toFlatEntity)) { + return undefined; + } + + const update = compareTwoFlatEntity({ + fromUniversalFlatEntity: fromFlatEntity, + toUniversalFlatEntity: toFlatEntity, + metadataName, + }); + + return isDefined(update) ? toFlatEntity : undefined; + }) + .filter(isDefined); + + return { + flatEntityToCreate: toRecordByUniversalIdentifier(flatEntityToCreate), + flatEntityToUpdate: toRecordByUniversalIdentifier(flatEntityToUpdate), + flatEntityToDelete: toRecordByUniversalIdentifier(flatEntityToDelete), + }; +}; + +export const buildAllFlatEntityOperationRecordByMetadataNameFromFromTo = ({ + fromAllFlatEntityMaps, + toAllUniversalFlatEntityMaps, + buildOptions, +}: { + fromAllFlatEntityMaps: AllFlatEntityMaps; + toAllUniversalFlatEntityMaps: AllFlatEntityMaps; + buildOptions: WorkspaceMigrationBuilderOptions; +}): AllFlatEntityOperationRecordByMetadataName => { + const allFlatEntityOperationRecordByMetadataName: AllFlatEntityOperationRecordByMetadataName = + {}; + + for (const metadataName of Object.values(ALL_METADATA_NAME)) { + const flatEntityMapsKey = getMetadataFlatEntityMapsKey(metadataName); + + const flatEntityOperationRecord = buildFlatEntityOperationRecordForMetadata( + { + metadataName, + fromFlatEntityMaps: fromAllFlatEntityMaps[ + flatEntityMapsKey + ] as unknown as MetadataUniversalFlatEntityMaps, + toFlatEntityMaps: toAllUniversalFlatEntityMaps[ + flatEntityMapsKey + ] as unknown as MetadataUniversalFlatEntityMaps, + buildOptions, + }, + ); + + if ( + Object.keys(flatEntityOperationRecord.flatEntityToCreate).length === 0 && + Object.keys(flatEntityOperationRecord.flatEntityToUpdate).length === 0 && + Object.keys(flatEntityOperationRecord.flatEntityToDelete).length === 0 + ) { + continue; + } + + ( + allFlatEntityOperationRecordByMetadataName as Record< + string, + FlatEntityOperationRecord + > + )[metadataName] = flatEntityOperationRecord; + } + + return allFlatEntityOperationRecordByMetadataName; +}; diff --git a/packages/twenty-server/src/engine/core-modules/application/application-manifest/utils/get-application-sub-all-flat-entity-maps.util.ts b/packages/twenty-server/src/engine/core-modules/application/application-manifest/utils/get-application-sub-all-flat-entity-maps.util.ts index 5354d1af57..a16110e54f 100644 --- a/packages/twenty-server/src/engine/core-modules/application/application-manifest/utils/get-application-sub-all-flat-entity-maps.util.ts +++ b/packages/twenty-server/src/engine/core-modules/application/application-manifest/utils/get-application-sub-all-flat-entity-maps.util.ts @@ -1,11 +1,7 @@ import { ALL_METADATA_NAME } from 'twenty-shared/metadata'; -import { createEmptyAllFlatEntityMaps } from 'src/engine/metadata-modules/flat-entity/constant/create-empty-all-flat-entity-maps.constant'; import { type AllFlatEntityMaps } from 'src/engine/metadata-modules/flat-entity/types/all-flat-entity-maps.type'; -import { type MetadataFlatEntity } from 'src/engine/metadata-modules/flat-entity/types/metadata-flat-entity.type'; -import { getMetadataFlatEntityMapsKey } from 'src/engine/metadata-modules/flat-entity/utils/get-metadata-flat-entity-maps-key.util'; -import { getSubFlatEntityMapsByApplicationIdsOrThrow } from 'src/engine/metadata-modules/flat-entity/utils/get-sub-flat-entity-maps-by-application-ids-or-throw.util'; -import { pruneDanglingForeignKeyAggregatorsInAllFlatEntityMapsThroughMutation } from 'src/engine/metadata-modules/flat-entity/utils/prune-dangling-foreign-key-aggregators-in-all-flat-entity-maps-through-mutation.util'; +import { getSubAllFlatEntityMapsByApplicationIdsOrThrow } from 'src/engine/metadata-modules/flat-entity/utils/get-sub-all-flat-entity-maps-by-application-ids-or-throw.util'; export const getApplicationSubAllFlatEntityMaps = ({ applicationIds, @@ -13,28 +9,9 @@ export const getApplicationSubAllFlatEntityMaps = ({ }: { applicationIds: string[]; fromAllFlatEntityMaps: AllFlatEntityMaps; -}): AllFlatEntityMaps => { - const emptyAllFlatEntityMaps = createEmptyAllFlatEntityMaps(); - - for (const metadataName of Object.values(ALL_METADATA_NAME)) { - const flatEntityMapsKey = getMetadataFlatEntityMapsKey(metadataName); - const fromFlatEntityMaps = fromAllFlatEntityMaps[flatEntityMapsKey]; - - const applicationSubFlatEntityMaps = - getSubFlatEntityMapsByApplicationIdsOrThrow< - MetadataFlatEntity - >({ - applicationIds, - flatEntityMaps: fromFlatEntityMaps, - }); - - // @ts-expect-error Metadata flat entity maps cache key and metadataName colliding - emptyAllFlatEntityMaps[flatEntityMapsKey] = applicationSubFlatEntityMaps; - } - - pruneDanglingForeignKeyAggregatorsInAllFlatEntityMapsThroughMutation({ - allFlatEntityMapsToMutate: emptyAllFlatEntityMaps, - }); - - return emptyAllFlatEntityMaps; -}; +}): AllFlatEntityMaps => + getSubAllFlatEntityMapsByApplicationIdsOrThrow({ + applicationIds, + metadataNames: Object.values(ALL_METADATA_NAME), + fromAllFlatEntityMaps, + }) as AllFlatEntityMaps; diff --git a/packages/twenty-server/src/engine/metadata-modules/flat-entity/constant/all-metadata-side-effect-companion-metadata-names.constant.ts b/packages/twenty-server/src/engine/metadata-modules/flat-entity/constant/all-metadata-side-effect-companion-metadata-names.constant.ts new file mode 100644 index 0000000000..36c71c76be --- /dev/null +++ b/packages/twenty-server/src/engine/metadata-modules/flat-entity/constant/all-metadata-side-effect-companion-metadata-names.constant.ts @@ -0,0 +1,7 @@ +import { type AllMetadataName } from 'twenty-shared/metadata'; + +export const ALL_METADATA_SIDE_EFFECT_COMPANION_METADATA_NAMES = { + fieldMetadata: ['index'], +} as const satisfies Partial< + Record +>; diff --git a/packages/twenty-server/src/engine/metadata-modules/flat-entity/types/all-flat-entity-operation-record-by-metadata-name.type.ts b/packages/twenty-server/src/engine/metadata-modules/flat-entity/types/all-flat-entity-operation-record-by-metadata-name.type.ts new file mode 100644 index 0000000000..cbd0abc54d --- /dev/null +++ b/packages/twenty-server/src/engine/metadata-modules/flat-entity/types/all-flat-entity-operation-record-by-metadata-name.type.ts @@ -0,0 +1,15 @@ +import { type AllMetadataName } from 'twenty-shared/metadata'; + +import { type MetadataUniversalFlatEntity } from 'src/engine/metadata-modules/flat-entity/types/metadata-universal-flat-entity.type'; + +export type FlatEntityOperationRecord = { + flatEntityToCreate: Partial< + Record & { id?: string }> + >; + flatEntityToUpdate: Partial>>; + flatEntityToDelete: Partial>>; +}; + +export type AllFlatEntityOperationRecordByMetadataName = { + [P in AllMetadataName]?: FlatEntityOperationRecord

; +}; diff --git a/packages/twenty-server/src/engine/metadata-modules/flat-entity/types/flat-entity-to-create-delete-update.type.ts b/packages/twenty-server/src/engine/metadata-modules/flat-entity/types/flat-entity-to-create-delete-update.type.ts index 5656884315..6201e9b465 100644 --- a/packages/twenty-server/src/engine/metadata-modules/flat-entity/types/flat-entity-to-create-delete-update.type.ts +++ b/packages/twenty-server/src/engine/metadata-modules/flat-entity/types/flat-entity-to-create-delete-update.type.ts @@ -7,3 +7,7 @@ export type FlatEntityToCreateDeleteUpdate = { flatEntityToCreate: (MetadataUniversalFlatEntity & { id?: string })[]; flatEntityToDelete: MetadataUniversalFlatEntity[]; }; + +export type AllFlatEntityOperationByMetadataName = { + [P in AllMetadataName]?: FlatEntityToCreateDeleteUpdate

; +}; diff --git a/packages/twenty-server/src/engine/metadata-modules/flat-entity/types/metadata-flat-entity-and-related-flat-entity-maps-for-side-effect.type.ts b/packages/twenty-server/src/engine/metadata-modules/flat-entity/types/metadata-flat-entity-and-related-flat-entity-maps-for-side-effect.type.ts new file mode 100644 index 0000000000..1666998d8b --- /dev/null +++ b/packages/twenty-server/src/engine/metadata-modules/flat-entity/types/metadata-flat-entity-and-related-flat-entity-maps-for-side-effect.type.ts @@ -0,0 +1,13 @@ +import { type AllMetadataName } from 'twenty-shared/metadata'; + +import { type AllFlatEntityMaps } from 'src/engine/metadata-modules/flat-entity/types/all-flat-entity-maps.type'; +import { type MetadataSideEffectRelatedMetadataNames } from 'src/engine/metadata-modules/flat-entity/types/metadata-side-effect-related-metadata-names.type'; +import { type MetadataToFlatEntityMapsKey } from 'src/engine/metadata-modules/flat-entity/types/metadata-to-flat-entity-maps-key'; + +export type MetadataFlatEntityAndRelatedFlatEntityMapsForSideEffect< + T extends AllMetadataName, +> = Pick< + AllFlatEntityMaps, + | MetadataToFlatEntityMapsKey + | MetadataToFlatEntityMapsKey> +>; diff --git a/packages/twenty-server/src/engine/metadata-modules/flat-entity/types/metadata-side-effect-related-metadata-names.type.ts b/packages/twenty-server/src/engine/metadata-modules/flat-entity/types/metadata-side-effect-related-metadata-names.type.ts new file mode 100644 index 0000000000..cd532f93fc --- /dev/null +++ b/packages/twenty-server/src/engine/metadata-modules/flat-entity/types/metadata-side-effect-related-metadata-names.type.ts @@ -0,0 +1,14 @@ +import { type AllMetadataName } from 'twenty-shared/metadata'; + +import { type ALL_METADATA_SIDE_EFFECT_COMPANION_METADATA_NAMES } from 'src/engine/metadata-modules/flat-entity/constant/all-metadata-side-effect-companion-metadata-names.constant'; +import { type MetadataManyToOneRelatedMetadataNames } from 'src/engine/metadata-modules/flat-entity/types/metadata-many-to-one-related-metadata-names.type'; + +export type MetadataSideEffectCompanionMetadataNames< + T extends AllMetadataName, +> = T extends keyof typeof ALL_METADATA_SIDE_EFFECT_COMPANION_METADATA_NAMES + ? (typeof ALL_METADATA_SIDE_EFFECT_COMPANION_METADATA_NAMES)[T][number] + : never; + +export type MetadataSideEffectRelatedMetadataNames = + | MetadataManyToOneRelatedMetadataNames + | MetadataSideEffectCompanionMetadataNames; diff --git a/packages/twenty-server/src/engine/metadata-modules/flat-entity/utils/get-metadata-side-effect-companion-names.util.ts b/packages/twenty-server/src/engine/metadata-modules/flat-entity/utils/get-metadata-side-effect-companion-names.util.ts new file mode 100644 index 0000000000..0d77e5d526 --- /dev/null +++ b/packages/twenty-server/src/engine/metadata-modules/flat-entity/utils/get-metadata-side-effect-companion-names.util.ts @@ -0,0 +1,16 @@ +import { type AllMetadataName } from 'twenty-shared/metadata'; + +import { ALL_METADATA_SIDE_EFFECT_COMPANION_METADATA_NAMES } from 'src/engine/metadata-modules/flat-entity/constant/all-metadata-side-effect-companion-metadata-names.constant'; +import { type MetadataSideEffectCompanionMetadataNames } from 'src/engine/metadata-modules/flat-entity/types/metadata-side-effect-related-metadata-names.type'; + +export const getMetadataSideEffectCompanionNames = ( + metadataName: T, +): MetadataSideEffectCompanionMetadataNames[] => { + const companionMetadataNamesByMetadataName = + ALL_METADATA_SIDE_EFFECT_COMPANION_METADATA_NAMES as Partial< + Record + >; + + return (companionMetadataNamesByMetadataName[metadataName] ?? + []) as MetadataSideEffectCompanionMetadataNames[]; +}; diff --git a/packages/twenty-server/src/engine/metadata-modules/flat-entity/utils/get-sub-all-flat-entity-maps-by-application-ids-or-throw.util.ts b/packages/twenty-server/src/engine/metadata-modules/flat-entity/utils/get-sub-all-flat-entity-maps-by-application-ids-or-throw.util.ts new file mode 100644 index 0000000000..3f3efa4e37 --- /dev/null +++ b/packages/twenty-server/src/engine/metadata-modules/flat-entity/utils/get-sub-all-flat-entity-maps-by-application-ids-or-throw.util.ts @@ -0,0 +1,51 @@ +import { type AllMetadataName } from 'twenty-shared/metadata'; +import { isDefined } from 'twenty-shared/utils'; + +import { + FlatEntityMapsException, + FlatEntityMapsExceptionCode, +} from 'src/engine/metadata-modules/flat-entity/exceptions/flat-entity-maps.exception'; +import { type AllFlatEntityMaps } from 'src/engine/metadata-modules/flat-entity/types/all-flat-entity-maps.type'; +import { type MetadataFlatEntity } from 'src/engine/metadata-modules/flat-entity/types/metadata-flat-entity.type'; +import { getMetadataFlatEntityMapsKey } from 'src/engine/metadata-modules/flat-entity/utils/get-metadata-flat-entity-maps-key.util'; +import { getSubFlatEntityMapsByApplicationIdsOrThrow } from 'src/engine/metadata-modules/flat-entity/utils/get-sub-flat-entity-maps-by-application-ids-or-throw.util'; +import { pruneDanglingForeignKeyAggregatorsInAllFlatEntityMapsThroughMutation } from 'src/engine/metadata-modules/flat-entity/utils/prune-dangling-foreign-key-aggregators-in-all-flat-entity-maps-through-mutation.util'; + +export const getSubAllFlatEntityMapsByApplicationIdsOrThrow = ({ + applicationIds, + metadataNames, + fromAllFlatEntityMaps, +}: { + applicationIds: string[]; + metadataNames: AllMetadataName[]; + fromAllFlatEntityMaps: Partial; +}): Partial => { + const subAllFlatEntityMaps: Partial = {}; + + for (const metadataName of metadataNames) { + const flatEntityMapsKey = getMetadataFlatEntityMapsKey(metadataName); + const fromFlatEntityMaps = fromAllFlatEntityMaps[flatEntityMapsKey]; + + if (!isDefined(fromFlatEntityMaps)) { + throw new FlatEntityMapsException( + `Missing flat entity maps for metadata "${metadataName}" while building application-scoped slice`, + FlatEntityMapsExceptionCode.INTERNAL_SERVER_ERROR, + ); + } + + // @ts-expect-error Metadata flat entity maps cache key and metadataName colliding + subAllFlatEntityMaps[flatEntityMapsKey] = + getSubFlatEntityMapsByApplicationIdsOrThrow< + MetadataFlatEntity + >({ + applicationIds, + flatEntityMaps: fromFlatEntityMaps, + }); + } + + pruneDanglingForeignKeyAggregatorsInAllFlatEntityMapsThroughMutation({ + allFlatEntityMapsToMutate: subAllFlatEntityMaps, + }); + + return subAllFlatEntityMaps; +}; diff --git a/packages/twenty-server/src/engine/metadata-modules/flat-entity/utils/is-system-side-effect-flat-entity.util.ts b/packages/twenty-server/src/engine/metadata-modules/flat-entity/utils/is-system-side-effect-flat-entity.util.ts new file mode 100644 index 0000000000..b2581495ce --- /dev/null +++ b/packages/twenty-server/src/engine/metadata-modules/flat-entity/utils/is-system-side-effect-flat-entity.util.ts @@ -0,0 +1,8 @@ +import { type AllMetadataName } from 'twenty-shared/metadata'; + +import { type MetadataUniversalFlatEntity } from 'src/engine/metadata-modules/flat-entity/types/metadata-universal-flat-entity.type'; + +export const isSystemSideEffectFlatEntity = ( + flatEntity: MetadataUniversalFlatEntity, +): boolean => + 'isSystemSideEffect' in flatEntity && flatEntity.isSystemSideEffect === true; diff --git a/packages/twenty-server/src/engine/metadata-modules/flat-entity/utils/prune-dangling-foreign-key-aggregators-in-all-flat-entity-maps-through-mutation.util.ts b/packages/twenty-server/src/engine/metadata-modules/flat-entity/utils/prune-dangling-foreign-key-aggregators-in-all-flat-entity-maps-through-mutation.util.ts index 18e486981f..b014e54c12 100644 --- a/packages/twenty-server/src/engine/metadata-modules/flat-entity/utils/prune-dangling-foreign-key-aggregators-in-all-flat-entity-maps-through-mutation.util.ts +++ b/packages/twenty-server/src/engine/metadata-modules/flat-entity/utils/prune-dangling-foreign-key-aggregators-in-all-flat-entity-maps-through-mutation.util.ts @@ -74,35 +74,49 @@ export const pruneDanglingForeignKeyAggregatorsInAllFlatEntityMapsThroughMutatio ({ allFlatEntityMapsToMutate, }: { - allFlatEntityMapsToMutate: AllFlatEntityMaps; + allFlatEntityMapsToMutate: Partial; }): void => { const looseAllFlatEntityMaps = - allFlatEntityMapsToMutate as unknown as LooseAllFlatEntityMaps; + allFlatEntityMapsToMutate as unknown as Partial; for (const metadataName of Object.values(ALL_METADATA_NAME)) { + const parentFlatEntityMaps = + looseAllFlatEntityMaps[getMetadataFlatEntityMapsKey(metadataName)]; + + if (!isDefined(parentFlatEntityMaps)) { + continue; + } + const oneToManyRelations = Object.values( ALL_ONE_TO_MANY_METADATA_RELATIONS[metadataName], ) as OneToManyRelation[]; const aggregatorsWithChildIdentifiers: OneToManyAggregatorWithChildIdentifiers[] = - oneToManyRelations.filter(isDefined).map((relation) => ({ - aggregatorProperty: relation.universalFlatEntityForeignKeyAggregator, - childUniversalIdentifiers: new Set( - Object.keys( - looseAllFlatEntityMaps[ - getMetadataFlatEntityMapsKey(relation.metadataName) - ].byUniversalIdentifier, - ), - ), - })); + oneToManyRelations.filter(isDefined).flatMap((relation) => { + const childFlatEntityMaps = + looseAllFlatEntityMaps[ + getMetadataFlatEntityMapsKey(relation.metadataName) + ]; + + const childUniversalIdentifiers = isDefined(childFlatEntityMaps) + ? new Set(Object.keys(childFlatEntityMaps.byUniversalIdentifier)) + : new Set(); + + return [ + { + aggregatorProperty: + relation.universalFlatEntityForeignKeyAggregator, + childUniversalIdentifiers, + }, + ]; + }); if (aggregatorsWithChildIdentifiers.length === 0) { continue; } const flatEntityByUniversalIdentifier = - looseAllFlatEntityMaps[getMetadataFlatEntityMapsKey(metadataName)] - .byUniversalIdentifier; + parentFlatEntityMaps.byUniversalIdentifier; for (const [universalIdentifier, parentFlatEntity] of Object.entries( flatEntityByUniversalIdentifier, diff --git a/packages/twenty-server/src/engine/metadata-modules/flat-entity/utils/transpile-flat-entity-operation-array-to-record.util.ts b/packages/twenty-server/src/engine/metadata-modules/flat-entity/utils/transpile-flat-entity-operation-array-to-record.util.ts new file mode 100644 index 0000000000..9931a7b747 --- /dev/null +++ b/packages/twenty-server/src/engine/metadata-modules/flat-entity/utils/transpile-flat-entity-operation-array-to-record.util.ts @@ -0,0 +1,94 @@ +import { isDefined } from 'twenty-shared/utils'; + +import { + FlatEntityMapsException, + FlatEntityMapsExceptionCode, +} from 'src/engine/metadata-modules/flat-entity/exceptions/flat-entity-maps.exception'; +import { type AllFlatEntityOperationRecordByMetadataName } from 'src/engine/metadata-modules/flat-entity/types/all-flat-entity-operation-record-by-metadata-name.type'; +import { type AllFlatEntityOperationByMetadataName } from 'src/engine/metadata-modules/flat-entity/types/flat-entity-to-create-delete-update.type'; + +type GenericUniversalFlatEntity = { universalIdentifier: string }; + +const toRecordByUniversalIdentifierOrThrow = ({ + flatEntities, + metadataName, + operation, +}: { + flatEntities: GenericUniversalFlatEntity[]; + metadataName: string; + operation: 'flatEntityToCreate' | 'flatEntityToUpdate' | 'flatEntityToDelete'; +}): Record => { + const recordByUniversalIdentifier: Record< + string, + GenericUniversalFlatEntity + > = {}; + const seenUniversalIdentifiers = new Set(); + + for (const flatEntity of flatEntities) { + if (seenUniversalIdentifiers.has(flatEntity.universalIdentifier)) { + throw new FlatEntityMapsException( + `Duplicate universalIdentifier "${flatEntity.universalIdentifier}" in ${operation} for metadata "${metadataName}"`, + FlatEntityMapsExceptionCode.ENTITY_ALREADY_EXISTS, + ); + } + + seenUniversalIdentifiers.add(flatEntity.universalIdentifier); + recordByUniversalIdentifier[flatEntity.universalIdentifier] = flatEntity; + } + + return recordByUniversalIdentifier; +}; + +// Compat bridge: array-based API callers are transpiled into the canonical record +// matrix so the side-effect engine and downstream from/to computation stay record-native. +// To be removed once every caller produces records directly. +export const transpileFlatEntityOperationArrayToRecord = ( + allFlatEntityOperationByMetadataName: AllFlatEntityOperationByMetadataName, +): AllFlatEntityOperationRecordByMetadataName => { + const genericMatrix = + allFlatEntityOperationByMetadataName as unknown as Record< + string, + | { + flatEntityToCreate: GenericUniversalFlatEntity[]; + flatEntityToUpdate: GenericUniversalFlatEntity[]; + flatEntityToDelete: GenericUniversalFlatEntity[]; + } + | undefined + >; + const recordMatrix: Record< + string, + { + flatEntityToCreate: Record; + flatEntityToUpdate: Record; + flatEntityToDelete: Record; + } + > = {}; + + for (const metadataName of Object.keys(genericMatrix)) { + const operations = genericMatrix[metadataName]; + + if (!isDefined(operations)) { + continue; + } + + recordMatrix[metadataName] = { + flatEntityToCreate: toRecordByUniversalIdentifierOrThrow({ + flatEntities: operations.flatEntityToCreate, + metadataName, + operation: 'flatEntityToCreate', + }), + flatEntityToUpdate: toRecordByUniversalIdentifierOrThrow({ + flatEntities: operations.flatEntityToUpdate, + metadataName, + operation: 'flatEntityToUpdate', + }), + flatEntityToDelete: toRecordByUniversalIdentifierOrThrow({ + flatEntities: operations.flatEntityToDelete, + metadataName, + operation: 'flatEntityToDelete', + }), + }; + } + + return recordMatrix as unknown as AllFlatEntityOperationRecordByMetadataName; +}; diff --git a/packages/twenty-server/src/engine/metadata-modules/flat-field-metadata/utils/from-create-field-input-to-flat-field-metadatas-to-create.util.ts b/packages/twenty-server/src/engine/metadata-modules/flat-field-metadata/utils/from-create-field-input-to-flat-field-metadatas-to-create.util.ts index 39b8790d6d..978ea99587 100644 --- a/packages/twenty-server/src/engine/metadata-modules/flat-field-metadata/utils/from-create-field-input-to-flat-field-metadatas-to-create.util.ts +++ b/packages/twenty-server/src/engine/metadata-modules/flat-field-metadata/utils/from-create-field-input-to-flat-field-metadatas-to-create.util.ts @@ -19,7 +19,6 @@ import { findFlatEntityByIdInFlatEntityMaps } from 'src/engine/metadata-modules/ import { type FieldInputTranspilationResult } from 'src/engine/metadata-modules/flat-field-metadata/types/field-input-transpilation-result.type'; import { fromMorphRelationCreateFieldInputToFlatFieldMetadatas } from 'src/engine/metadata-modules/flat-field-metadata/utils/from-morph-relation-create-field-input-to-flat-field-metadatas.util'; import { fromRelationCreateFieldInputToFlatFieldMetadatas } from 'src/engine/metadata-modules/flat-field-metadata/utils/from-relation-create-field-input-to-flat-field-metadatas.util'; -import { generateIndexForFlatFieldMetadata } from 'src/engine/metadata-modules/flat-field-metadata/utils/generate-index-for-flat-field-metadata.util'; import { getDefaultFlatFieldMetadata } from 'src/engine/metadata-modules/flat-field-metadata/utils/get-default-flat-field-metadata-from-create-field-input.util'; import { type UniversalFlatFieldMetadata } from 'src/engine/workspace-manager/workspace-migration/universal-flat-entity/types/universal-flat-field-metadata.type'; import { type UniversalFlatIndexMetadata } from 'src/engine/workspace-manager/workspace-migration/universal-flat-entity/types/universal-flat-index-metadata.type'; @@ -186,17 +185,6 @@ export const fromCreateFieldInputToFlatFieldMetadatasToCreate = async ({ case FieldMetadataType.RICH_TEXT: case FieldMetadataType.ACTOR: case FieldMetadataType.ARRAY: { - const indexMetadatas: UniversalFlatIndexMetadata[] = []; - - if (commonFlatFieldMetadata.isUnique) { - indexMetadatas.push( - generateIndexForFlatFieldMetadata({ - flatFieldMetadata: commonFlatFieldMetadata, - flatObjectMetadata: parentFlatObjectMetadata, - }), - ); - } - return { status: 'success', result: { @@ -206,7 +194,7 @@ export const fromCreateFieldInputToFlatFieldMetadatasToCreate = async ({ type: createFieldInput.type, }, ], - indexMetadatas, + indexMetadatas: [], }, }; } diff --git a/packages/twenty-server/src/engine/metadata-modules/flat-field-metadata/utils/from-delete-field-input-to-flat-field-metadatas-to-delete.util.ts b/packages/twenty-server/src/engine/metadata-modules/flat-field-metadata/utils/from-delete-field-input-to-flat-field-metadatas-to-delete.util.ts index 11f5609e9e..f628cdaa22 100644 --- a/packages/twenty-server/src/engine/metadata-modules/flat-field-metadata/utils/from-delete-field-input-to-flat-field-metadatas-to-delete.util.ts +++ b/packages/twenty-server/src/engine/metadata-modules/flat-field-metadata/utils/from-delete-field-input-to-flat-field-metadatas-to-delete.util.ts @@ -14,7 +14,9 @@ import { findFlatEntityByUniversalIdentifierOrThrow } from 'src/engine/metadata- import { findManyFlatEntityByUniversalIdentifierInUniversalFlatEntityMapsOrThrow } from 'src/engine/metadata-modules/flat-entity/utils/find-many-flat-entity-by-universal-identifier-in-universal-flat-entity-maps-or-throw.util'; import { computeFlatFieldMetadataRelatedFlatFieldMetadata } from 'src/engine/metadata-modules/flat-field-metadata/utils/compute-flat-field-metadata-related-flat-field-metadata.util'; import { computeSearchFieldMetadataDeletionForDeletedFields } from 'src/engine/metadata-modules/flat-field-metadata/utils/compute-search-field-metadata-deletion-for-deleted-fields.util'; +import { isMorphOrRelationFlatFieldMetadata } from 'src/engine/metadata-modules/flat-field-metadata/utils/is-morph-or-relation-flat-field-metadata.util'; import { type FlatIndexMetadata } from 'src/engine/metadata-modules/flat-index-metadata/types/flat-index-metadata.type'; +import { isSystemUniqueFlatIndexMetadata } from 'src/engine/metadata-modules/flat-index-metadata/utils/is-system-unique-flat-index-metadata.util'; import { generateFlatIndexMetadataWithNameOrThrow } from 'src/engine/metadata-modules/index-metadata/utils/generate-flat-index.util'; import { belongsToTwentyStandardApp } from 'src/engine/metadata-modules/utils/belongs-to-twenty-standard-app.util'; import { type UniversalFlatFieldMetadata } from 'src/engine/workspace-manager/workspace-migration/universal-flat-entity/types/universal-flat-field-metadata.type'; @@ -95,10 +97,38 @@ export const fromDeleteFieldInputToFlatFieldMetadatasToDelete = ({ ...relatedFlatFieldMetadataToDelete, ]; + const isEngineOwnedBackingUniqueIndex = ( + flatIndexMetadata: FlatIndexMetadata, + ): boolean => { + if ( + !isSystemUniqueFlatIndexMetadata(flatIndexMetadata) || + flatIndexMetadata.flatIndexFieldMetadatas.length !== 1 + ) { + return false; + } + + const backingFlatFieldMetadata = findFlatEntityByIdInFlatEntityMaps({ + flatEntityId: + flatIndexMetadata.flatIndexFieldMetadatas[0].fieldMetadataId, + flatEntityMaps: existingFlatFieldMetadataMaps, + }); + + return ( + isDefined(backingFlatFieldMetadata) && + backingFlatFieldMetadata.isUnique === true && + !isMorphOrRelationFlatFieldMetadata(backingFlatFieldMetadata) + ); + }; + const flatIndexMap = new Map(); const allFlatIndexes = Object.values( existingFlatIndexMaps.byUniversalIdentifier, - ).filter(isDefined); + ) + .filter(isDefined) + .filter( + (flatIndexMetadata) => + !isEngineOwnedBackingUniqueIndex(flatIndexMetadata), + ); for (const flatFieldMetadata of flatFieldMetadatasToDelete) { allFlatIndexes.forEach((flatIndex) => { diff --git a/packages/twenty-server/src/engine/metadata-modules/flat-field-metadata/utils/from-update-field-input-to-flat-field-metadata.util.ts b/packages/twenty-server/src/engine/metadata-modules/flat-field-metadata/utils/from-update-field-input-to-flat-field-metadata.util.ts index 63abd96223..ecd56d1ad8 100644 --- a/packages/twenty-server/src/engine/metadata-modules/flat-field-metadata/utils/from-update-field-input-to-flat-field-metadata.util.ts +++ b/packages/twenty-server/src/engine/metadata-modules/flat-field-metadata/utils/from-update-field-input-to-flat-field-metadata.util.ts @@ -150,7 +150,6 @@ export const fromUpdateFieldInputToFlatFieldMetadata = ({ toFlatFieldMetadata, flatViewMaps, flatViewFieldMaps, - flatApplication, }); if (sideEffectResult.status === 'fail') { diff --git a/packages/twenty-server/src/engine/metadata-modules/flat-field-metadata/utils/generate-deterministic-index-for-flat-field-metadata-or-throw.util.ts b/packages/twenty-server/src/engine/metadata-modules/flat-field-metadata/utils/generate-deterministic-index-for-flat-field-metadata-or-throw.util.ts new file mode 100644 index 0000000000..96b62bc112 --- /dev/null +++ b/packages/twenty-server/src/engine/metadata-modules/flat-field-metadata/utils/generate-deterministic-index-for-flat-field-metadata-or-throw.util.ts @@ -0,0 +1,44 @@ +import { IndexType } from 'src/engine/metadata-modules/index-metadata/types/indexType.types'; +import { generateDeterministicFlatIndexMetadataOrThrow } from 'src/engine/metadata-modules/index-metadata/utils/generate-deterministic-flat-index.util'; +import { type UniversalFlatFieldMetadata } from 'src/engine/workspace-manager/workspace-migration/universal-flat-entity/types/universal-flat-field-metadata.type'; +import { type UniversalFlatIndexMetadata } from 'src/engine/workspace-manager/workspace-migration/universal-flat-entity/types/universal-flat-index-metadata.type'; +import { type UniversalFlatObjectMetadata } from 'src/engine/workspace-manager/workspace-migration/universal-flat-entity/types/universal-flat-object-metadata.type'; + +type GenerateDeterministicIndexForFlatFieldMetadataOrThrowArgs = { + flatFieldMetadata: UniversalFlatFieldMetadata; + flatObjectMetadata: UniversalFlatObjectMetadata; +}; + +export const generateDeterministicIndexForFlatFieldMetadataOrThrow = ({ + flatFieldMetadata, + flatObjectMetadata, +}: GenerateDeterministicIndexForFlatFieldMetadataOrThrowArgs): UniversalFlatIndexMetadata => { + const createdAt = new Date().toISOString(); + + return generateDeterministicFlatIndexMetadataOrThrow({ + flatObjectMetadata, + objectFlatFieldMetadatas: [flatFieldMetadata], + flatIndex: { + createdAt, + universalFlatIndexFieldMetadatas: [ + { + createdAt, + fieldMetadataUniversalIdentifier: + flatFieldMetadata.universalIdentifier, + order: 0, + subFieldName: null, + updatedAt: createdAt, + }, + ], + indexType: IndexType.BTREE, + indexWhereClause: null, + isCustom: true, + isUnique: flatFieldMetadata.isUnique ?? false, + isSystemSideEffect: true, + objectMetadataUniversalIdentifier: flatObjectMetadata.universalIdentifier, + updatedAt: createdAt, + applicationUniversalIdentifier: + flatFieldMetadata.applicationUniversalIdentifier, + }, + }); +}; diff --git a/packages/twenty-server/src/engine/metadata-modules/flat-field-metadata/utils/handle-flat-field-metadata-update-side-effect.util.ts b/packages/twenty-server/src/engine/metadata-modules/flat-field-metadata/utils/handle-flat-field-metadata-update-side-effect.util.ts index 7022ff4070..a8838c1559 100644 --- a/packages/twenty-server/src/engine/metadata-modules/flat-field-metadata/utils/handle-flat-field-metadata-update-side-effect.util.ts +++ b/packages/twenty-server/src/engine/metadata-modules/flat-field-metadata/utils/handle-flat-field-metadata-update-side-effect.util.ts @@ -1,6 +1,5 @@ import { type FromTo } from 'twenty-shared/types'; -import { type FlatApplication } from 'src/engine/core-modules/application/types/flat-application.type'; import { type AllFlatEntityMaps } from 'src/engine/metadata-modules/flat-entity/types/all-flat-entity-maps.type'; import { type FieldInputTranspilationResult } from 'src/engine/metadata-modules/flat-field-metadata/types/field-input-transpilation-result.type'; import { type FlatFieldMetadata } from 'src/engine/metadata-modules/flat-field-metadata/types/flat-field-metadata.type'; @@ -36,9 +35,7 @@ type HandleFlatFieldMetadataUpdateSideEffectArgs = FromTo< | 'flatViewGroupMaps' | 'flatViewMaps' | 'flatViewFieldMaps' - > & { - flatApplication: FlatApplication; - }; + >; export const FLAT_FIELD_METADATA_UPDATE_EMPTY_SIDE_EFFECTS: FlatFieldMetadataUpdateSideEffects = { @@ -65,7 +62,6 @@ export const handleFlatFieldMetadataUpdateSideEffect = ({ flatViewGroupMaps, flatViewMaps, flatViewFieldMaps, - flatApplication, }: HandleFlatFieldMetadataUpdateSideEffectArgs): FieldInputTranspilationResult => { const sideEffectResult = structuredClone( FLAT_FIELD_METADATA_UPDATE_EMPTY_SIDE_EFFECTS, @@ -125,7 +121,6 @@ export const handleFlatFieldMetadataUpdateSideEffect = ({ flatIndexMaps, flatObjectMetadataMaps, flatFieldMetadataMaps, - flatApplication, }); if (indexChangesSideEffectResult.status === 'fail') { diff --git a/packages/twenty-server/src/engine/metadata-modules/flat-field-metadata/utils/handle-index-changes-during-field-update.util.ts b/packages/twenty-server/src/engine/metadata-modules/flat-field-metadata/utils/handle-index-changes-during-field-update.util.ts index 6806acf177..16bc0afc2d 100644 --- a/packages/twenty-server/src/engine/metadata-modules/flat-field-metadata/utils/handle-index-changes-during-field-update.util.ts +++ b/packages/twenty-server/src/engine/metadata-modules/flat-field-metadata/utils/handle-index-changes-during-field-update.util.ts @@ -1,19 +1,12 @@ -import { msg } from '@lingui/core/macro'; import { type FromTo } from 'twenty-shared/types'; -import { isDefined } from 'twenty-shared/utils'; -import { type FlatApplication } from 'src/engine/core-modules/application/types/flat-application.type'; -import { FieldMetadataExceptionCode } from 'src/engine/metadata-modules/field-metadata/field-metadata.exception'; import { type AllFlatEntityMaps } from 'src/engine/metadata-modules/flat-entity/types/all-flat-entity-maps.type'; import { findFlatEntityByIdInFlatEntityMapsOrThrow } from 'src/engine/metadata-modules/flat-entity/utils/find-flat-entity-by-id-in-flat-entity-maps-or-throw.util'; import { type FieldInputTranspilationResult } from 'src/engine/metadata-modules/flat-field-metadata/types/field-input-transpilation-result.type'; import { type FlatFieldMetadata } from 'src/engine/metadata-modules/flat-field-metadata/types/flat-field-metadata.type'; import { findFieldRelatedIndexes } from 'src/engine/metadata-modules/flat-field-metadata/utils/find-field-related-index.util'; -import { generateIndexForFlatFieldMetadata } from 'src/engine/metadata-modules/flat-field-metadata/utils/generate-index-for-flat-field-metadata.util'; -import { isMorphOrRelationFlatFieldMetadata } from 'src/engine/metadata-modules/flat-field-metadata/utils/is-morph-or-relation-flat-field-metadata.util'; import { recomputeIndexOnFlatFieldMetadataNameUpdate } from 'src/engine/metadata-modules/flat-field-metadata/utils/recompute-index-on-flat-field-metadata-name-update.util'; -import { type FlatIndexMetadata } from 'src/engine/metadata-modules/flat-index-metadata/types/flat-index-metadata.type'; -import { type FlatObjectMetadata } from 'src/engine/metadata-modules/flat-object-metadata/types/flat-object-metadata.type'; +import { isSystemUniqueFlatIndexMetadata } from 'src/engine/metadata-modules/flat-index-metadata/utils/is-system-unique-flat-index-metadata.util'; import { type UniversalFlatIndexMetadata } from 'src/engine/workspace-manager/workspace-migration/universal-flat-entity/types/universal-flat-index-metadata.type'; export type FieldMetadataUpdateIndexSideEffect = { @@ -29,9 +22,8 @@ type FromToFlatFieldMetadataAndFlatEntityMaps = FromTo< Pick< AllFlatEntityMaps, 'flatObjectMetadataMaps' | 'flatFieldMetadataMaps' | 'flatIndexMaps' - > & { - flatApplication: FlatApplication; - }; + >; + const FIELD_METADATA_UPDATE_INDEX_SIDE_EFFECT: FieldMetadataUpdateIndexSideEffect = { flatIndexMetadatasToUpdate: [], @@ -45,14 +37,8 @@ export const handleIndexChangesDuringFieldUpdate = ({ flatIndexMaps, flatObjectMetadataMaps, flatFieldMetadataMaps, - flatApplication, }: FromToFlatFieldMetadataAndFlatEntityMaps): FieldInputTranspilationResult => { - if ( - !hasIndexRelevantChanges({ - fromFlatFieldMetadata, - toFlatFieldMetadata, - }) - ) { + if (fromFlatFieldMetadata.name === toFlatFieldMetadata.name) { return { status: 'success', result: FIELD_METADATA_UPDATE_INDEX_SIDE_EFFECT, @@ -68,119 +54,17 @@ export const handleIndexChangesDuringFieldUpdate = ({ flatFieldMetadata: fromFlatFieldMetadata, flatObjectMetadata, flatIndexMaps, - }); + }).filter( + (flatIndexMetadata) => !isSystemUniqueFlatIndexMetadata(flatIndexMetadata), + ); if (relatedIndexes.length === 0) { - return handleNoExistingIndexes({ - toFlatFieldMetadata, - flatObjectMetadata, - }); - } - - return handleExistingIndexes({ - toFlatFieldMetadata, - fromFlatFieldMetadata, - relatedIndexes, - flatObjectMetadata, - flatFieldMetadataMaps, - flatApplication, - }); -}; - -const hasIndexRelevantChanges = ({ - fromFlatFieldMetadata, - toFlatFieldMetadata, -}: FromTo): boolean => - fromFlatFieldMetadata.name !== toFlatFieldMetadata.name || - fromFlatFieldMetadata.isUnique !== toFlatFieldMetadata.isUnique; - -const handleNoExistingIndexes = ({ - toFlatFieldMetadata, - flatObjectMetadata, -}: { - toFlatFieldMetadata: FlatFieldMetadata; - flatObjectMetadata: FlatObjectMetadata; -}): FieldInputTranspilationResult => { - if (!toFlatFieldMetadata.isUnique) { return { status: 'success', result: FIELD_METADATA_UPDATE_INDEX_SIDE_EFFECT, }; } - const newIndex = generateIndexForFlatFieldMetadata({ - flatFieldMetadata: toFlatFieldMetadata, - flatObjectMetadata, - }); - - return { - status: 'success', - result: { - ...FIELD_METADATA_UPDATE_INDEX_SIDE_EFFECT, - flatIndexMetadatasToCreate: [newIndex], - }, - }; -}; - -const handleExistingIndexes = ({ - fromFlatFieldMetadata, - toFlatFieldMetadata, - relatedIndexes, - flatObjectMetadata, - flatFieldMetadataMaps, - flatApplication, -}: { - relatedIndexes: FlatIndexMetadata[]; - flatObjectMetadata: FlatObjectMetadata; - flatFieldMetadataMaps: AllFlatEntityMaps['flatFieldMetadataMaps']; -} & FromTo & { - flatApplication: FlatApplication; - }): FieldInputTranspilationResult => { - if ( - toFlatFieldMetadata.isUnique === false && - !isMorphOrRelationFlatFieldMetadata(fromFlatFieldMetadata) - ) { - const expectedUniqueIndex = generateIndexForFlatFieldMetadata({ - flatFieldMetadata: { - ...fromFlatFieldMetadata, - isUnique: true, - }, - flatObjectMetadata, - }); - - const uniqueIndexToDelete = relatedIndexes.find( - (index) => index.name === expectedUniqueIndex.name, - ); - - if ( - isDefined(uniqueIndexToDelete) && - ((isDefined(uniqueIndexToDelete.applicationId) && - uniqueIndexToDelete.applicationId !== flatApplication.id) || - !uniqueIndexToDelete.isCustom) - ) { - return { - status: 'fail', - errors: [ - { - code: FieldMetadataExceptionCode.INVALID_FIELD_INPUT, - message: - 'Cannot delete unique index that have not been created by the workspace custom application', - userFriendlyMessage: msg`Cannot delete unique index that have not been created by the workspace custom application`, - }, - ], - }; - } - - return { - status: 'success', - result: { - ...FIELD_METADATA_UPDATE_INDEX_SIDE_EFFECT, - flatIndexMetadatasToDelete: uniqueIndexToDelete - ? [uniqueIndexToDelete] - : [], - }, - }; - } const updatedIndexes = recomputeIndexOnFlatFieldMetadataNameUpdate({ flatFieldMetadataMaps, flatObjectMetadata, diff --git a/packages/twenty-server/src/engine/metadata-modules/flat-field-metadata/utils/is-primary-key-flat-field-metadata.util.ts b/packages/twenty-server/src/engine/metadata-modules/flat-field-metadata/utils/is-primary-key-flat-field-metadata.util.ts new file mode 100644 index 0000000000..6b2e038262 --- /dev/null +++ b/packages/twenty-server/src/engine/metadata-modules/flat-field-metadata/utils/is-primary-key-flat-field-metadata.util.ts @@ -0,0 +1,7 @@ +import { type UniversalFlatFieldMetadata } from 'src/engine/workspace-manager/workspace-migration/universal-flat-entity/types/universal-flat-field-metadata.type'; + +// The primary key already enforces uniqueness via its constraint, so it must not +// spawn a redundant application-managed unique index. +export const isPrimaryKeyFlatFieldMetadata = ( + flatFieldMetadata: Pick, +): boolean => flatFieldMetadata.name === 'id'; diff --git a/packages/twenty-server/src/engine/metadata-modules/flat-index-metadata/utils/is-system-unique-flat-index-metadata.util.ts b/packages/twenty-server/src/engine/metadata-modules/flat-index-metadata/utils/is-system-unique-flat-index-metadata.util.ts new file mode 100644 index 0000000000..5a25d4d8f4 --- /dev/null +++ b/packages/twenty-server/src/engine/metadata-modules/flat-index-metadata/utils/is-system-unique-flat-index-metadata.util.ts @@ -0,0 +1,6 @@ +export const isSystemUniqueFlatIndexMetadata = (flatIndexMetadata: { + isSystemSideEffect: boolean; + isUnique: boolean; +}): boolean => + flatIndexMetadata.isSystemSideEffect === true && + flatIndexMetadata.isUnique === true; diff --git a/packages/twenty-server/src/engine/metadata-modules/index-metadata/utils/compute-flat-index-name.util.ts b/packages/twenty-server/src/engine/metadata-modules/index-metadata/utils/compute-flat-index-name.util.ts new file mode 100644 index 0000000000..6a4c6d4bc4 --- /dev/null +++ b/packages/twenty-server/src/engine/metadata-modules/index-metadata/utils/compute-flat-index-name.util.ts @@ -0,0 +1,96 @@ +import { compositeTypeDefinitions, RelationType } from 'twenty-shared/types'; +import { isDefined } from 'twenty-shared/utils'; + +import { computeCompositeColumnName } from 'src/engine/metadata-modules/field-metadata/utils/compute-column-name.util'; +import { computeMorphOrRelationFieldJoinColumnName } from 'src/engine/metadata-modules/field-metadata/utils/compute-morph-or-relation-field-join-column-name.util'; +import { isCompositeFieldMetadataType } from 'src/engine/metadata-modules/field-metadata/utils/is-composite-field-metadata-type.util'; +import { + FlatEntityMapsException, + FlatEntityMapsExceptionCode, +} from 'src/engine/metadata-modules/flat-entity/exceptions/flat-entity-maps.exception'; +import { isMorphOrRelationUniversalFlatFieldMetadata } from 'src/engine/metadata-modules/flat-field-metadata/utils/is-morph-or-relation-flat-field-metadata.util'; +import { generateDeterministicIndexName } from 'src/engine/metadata-modules/index-metadata/utils/generate-deterministic-index-name'; +import { type UniversalFlatFieldMetadata } from 'src/engine/workspace-manager/workspace-migration/universal-flat-entity/types/universal-flat-field-metadata.type'; +import { type UniversalFlatIndexFieldMetadata } from 'src/engine/workspace-manager/workspace-migration/universal-flat-entity/types/universal-flat-index-metadata.type'; +import { type UniversalFlatObjectMetadata } from 'src/engine/workspace-manager/workspace-migration/universal-flat-entity/types/universal-flat-object-metadata.type'; + +type ComputeFlatIndexNameArgs = { + flatObjectMetadata: UniversalFlatObjectMetadata; + objectFlatFieldMetadatas: UniversalFlatFieldMetadata[]; + indexFields: Pick< + UniversalFlatIndexFieldMetadata, + 'order' | 'fieldMetadataUniversalIdentifier' | 'subFieldName' + >[]; + isUnique: boolean; + indexWhereClause: string | null; +}; + +export const computeFlatIndexNameOrThrow = ({ + flatObjectMetadata, + objectFlatFieldMetadatas, + indexFields, + isUnique, + indexWhereClause, +}: ComputeFlatIndexNameArgs): string => { + const orderedIndexColumnNames = [...indexFields] + .sort((a, b) => a.order - b.order) + .map((flatIndexField) => { + const relatedFlatFieldMetadata = objectFlatFieldMetadatas.find( + (flatFieldMetadata) => + flatFieldMetadata.universalIdentifier === + flatIndexField.fieldMetadataUniversalIdentifier, + ); + + if (!isDefined(relatedFlatFieldMetadata)) { + throw new FlatEntityMapsException( + 'Could not find flat index field related field in cache', + FlatEntityMapsExceptionCode.ENTITY_NOT_FOUND, + ); + } + + if ( + isCompositeFieldMetadataType(relatedFlatFieldMetadata.type) && + isDefined(flatIndexField.subFieldName) + ) { + const property = compositeTypeDefinitions + .get(relatedFlatFieldMetadata.type) + ?.properties.find( + (compositeProperty) => + compositeProperty.name === flatIndexField.subFieldName, + ); + + if (!isDefined(property)) { + throw new FlatEntityMapsException( + `Composite sub-field "${flatIndexField.subFieldName}" not found on ${relatedFlatFieldMetadata.name}`, + FlatEntityMapsExceptionCode.ENTITY_NOT_FOUND, + ); + } + + return computeCompositeColumnName( + { + name: relatedFlatFieldMetadata.name, + type: relatedFlatFieldMetadata.type, + }, + property, + ); + } + + const isManyToOneRelation = + isMorphOrRelationUniversalFlatFieldMetadata(relatedFlatFieldMetadata) && + relatedFlatFieldMetadata.universalSettings?.relationType === + RelationType.MANY_TO_ONE; + + return isManyToOneRelation + ? computeMorphOrRelationFieldJoinColumnName({ + name: relatedFlatFieldMetadata.name, + }) + : relatedFlatFieldMetadata.name; + }); + + return generateDeterministicIndexName({ + flatObjectMetadata, + orderedIndexColumnNames, + isUnique, + indexWhereClause, + }); +}; diff --git a/packages/twenty-server/src/engine/metadata-modules/index-metadata/utils/generate-deterministic-flat-index.util.ts b/packages/twenty-server/src/engine/metadata-modules/index-metadata/utils/generate-deterministic-flat-index.util.ts new file mode 100644 index 0000000000..b79ebc4b09 --- /dev/null +++ b/packages/twenty-server/src/engine/metadata-modules/index-metadata/utils/generate-deterministic-flat-index.util.ts @@ -0,0 +1,56 @@ +import { getIndexUniversalIdentifier } from 'twenty-shared/application'; + +import { computeFlatIndexNameOrThrow } from 'src/engine/metadata-modules/index-metadata/utils/compute-flat-index-name.util'; +import { type UniversalFlatFieldMetadata } from 'src/engine/workspace-manager/workspace-migration/universal-flat-entity/types/universal-flat-field-metadata.type'; +import { + type UniversalFlatIndexFieldMetadata, + type UniversalFlatIndexMetadata, +} from 'src/engine/workspace-manager/workspace-migration/universal-flat-entity/types/universal-flat-index-metadata.type'; +import { type UniversalFlatObjectMetadata } from 'src/engine/workspace-manager/workspace-migration/universal-flat-entity/types/universal-flat-object-metadata.type'; + +type GenerateDeterministicFlatIndexArgs = { + flatObjectMetadata: UniversalFlatObjectMetadata; + objectFlatFieldMetadatas: UniversalFlatFieldMetadata[]; + flatIndex: Omit< + UniversalFlatIndexMetadata, + 'name' | 'universalIdentifier' | 'universalFlatIndexFieldMetadatas' + > & { + universalFlatIndexFieldMetadatas: Omit< + UniversalFlatIndexFieldMetadata, + 'indexMetadataUniversalIdentifier' + >[]; + }; +}; + +export const generateDeterministicFlatIndexMetadataOrThrow = ({ + flatObjectMetadata, + objectFlatFieldMetadatas, + flatIndex, +}: GenerateDeterministicFlatIndexArgs): UniversalFlatIndexMetadata => { + const name = computeFlatIndexNameOrThrow({ + flatObjectMetadata, + objectFlatFieldMetadatas, + indexFields: flatIndex.universalFlatIndexFieldMetadatas, + isUnique: flatIndex.isUnique, + indexWhereClause: flatIndex.indexWhereClause, + }); + + const universalIdentifier = getIndexUniversalIdentifier({ + applicationUniversalIdentifier: flatIndex.applicationUniversalIdentifier, + objectUniversalIdentifier: flatObjectMetadata.universalIdentifier, + name, + }); + + return { + ...flatIndex, + name, + universalIdentifier, + universalFlatIndexFieldMetadatas: + flatIndex.universalFlatIndexFieldMetadatas.map( + (universalFlatIndexFieldMetadata) => ({ + ...universalFlatIndexFieldMetadata, + indexMetadataUniversalIdentifier: universalIdentifier, + }), + ), + }; +}; diff --git a/packages/twenty-server/src/engine/metadata-modules/index-metadata/utils/generate-deterministic-index-name-v2.ts b/packages/twenty-server/src/engine/metadata-modules/index-metadata/utils/generate-deterministic-index-name.ts similarity index 96% rename from packages/twenty-server/src/engine/metadata-modules/index-metadata/utils/generate-deterministic-index-name-v2.ts rename to packages/twenty-server/src/engine/metadata-modules/index-metadata/utils/generate-deterministic-index-name.ts index 20a7b78e38..9d8544f976 100644 --- a/packages/twenty-server/src/engine/metadata-modules/index-metadata/utils/generate-deterministic-index-name-v2.ts +++ b/packages/twenty-server/src/engine/metadata-modules/index-metadata/utils/generate-deterministic-index-name.ts @@ -13,7 +13,7 @@ type GenerateDeterministicIndexNameArgs = { // coexist; the unique-name constraint must not block that). indexWhereClause?: string | null; }; -export const generateDeterministicIndexNameV2 = ({ +export const generateDeterministicIndexName = ({ orderedIndexColumnNames, flatObjectMetadata, isUnique = false, diff --git a/packages/twenty-server/src/engine/metadata-modules/index-metadata/utils/generate-flat-index.util.ts b/packages/twenty-server/src/engine/metadata-modules/index-metadata/utils/generate-flat-index.util.ts index d6b78aef58..7e7fb348b1 100644 --- a/packages/twenty-server/src/engine/metadata-modules/index-metadata/utils/generate-flat-index.util.ts +++ b/packages/twenty-server/src/engine/metadata-modules/index-metadata/utils/generate-flat-index.util.ts @@ -1,15 +1,4 @@ -import { compositeTypeDefinitions, RelationType } from 'twenty-shared/types'; -import { isDefined } from 'twenty-shared/utils'; - -import { computeCompositeColumnName } from 'src/engine/metadata-modules/field-metadata/utils/compute-column-name.util'; -import { computeMorphOrRelationFieldJoinColumnName } from 'src/engine/metadata-modules/field-metadata/utils/compute-morph-or-relation-field-join-column-name.util'; -import { isCompositeFieldMetadataType } from 'src/engine/metadata-modules/field-metadata/utils/is-composite-field-metadata-type.util'; -import { - FlatEntityMapsException, - FlatEntityMapsExceptionCode, -} from 'src/engine/metadata-modules/flat-entity/exceptions/flat-entity-maps.exception'; -import { isMorphOrRelationUniversalFlatFieldMetadata } from 'src/engine/metadata-modules/flat-field-metadata/utils/is-morph-or-relation-flat-field-metadata.util'; -import { generateDeterministicIndexNameV2 } from 'src/engine/metadata-modules/index-metadata/utils/generate-deterministic-index-name-v2'; +import { computeFlatIndexNameOrThrow } from 'src/engine/metadata-modules/index-metadata/utils/compute-flat-index-name.util'; import { type UniversalFlatFieldMetadata } from 'src/engine/workspace-manager/workspace-migration/universal-flat-entity/types/universal-flat-field-metadata.type'; import { type UniversalFlatIndexMetadata } from 'src/engine/workspace-manager/workspace-migration/universal-flat-entity/types/universal-flat-index-metadata.type'; import { type UniversalFlatObjectMetadata } from 'src/engine/workspace-manager/workspace-migration/universal-flat-entity/types/universal-flat-object-metadata.type'; @@ -25,69 +14,10 @@ export const generateFlatIndexMetadataWithNameOrThrow = ({ objectFlatFieldMetadatas, flatIndex, }: GenerateFlatIndexArgs): UniversalFlatIndexMetadata => { - const orderedIndexColumnNames = flatIndex.universalFlatIndexFieldMetadatas - .sort((a, b) => a.order - b.order) - .map((flatIndexField) => { - const relatedFlatFieldMetadata = objectFlatFieldMetadatas.find( - (flatFieldMetadata) => - flatFieldMetadata.universalIdentifier === - flatIndexField.fieldMetadataUniversalIdentifier, - ); - - if (!isDefined(relatedFlatFieldMetadata)) { - throw new FlatEntityMapsException( - 'Could not find flat index field related field in cache', - FlatEntityMapsExceptionCode.ENTITY_NOT_FOUND, - ); - } - - // Composite parent with an explicit sub-field → single sub-column. - // Composite parent without sub-field falls through to the legacy - // scalar branch below, which produces a deterministic name based on - // the parent name (the runner handles the multi-column SQL expansion - // via isIncludedInUniqueConstraint). - if ( - isCompositeFieldMetadataType(relatedFlatFieldMetadata.type) && - isDefined(flatIndexField.subFieldName) - ) { - const property = compositeTypeDefinitions - .get(relatedFlatFieldMetadata.type) - ?.properties.find( - (compositeProperty) => - compositeProperty.name === flatIndexField.subFieldName, - ); - - if (!isDefined(property)) { - throw new FlatEntityMapsException( - `Composite sub-field "${flatIndexField.subFieldName}" not found on ${relatedFlatFieldMetadata.name}`, - FlatEntityMapsExceptionCode.ENTITY_NOT_FOUND, - ); - } - - return computeCompositeColumnName( - { - name: relatedFlatFieldMetadata.name, - type: relatedFlatFieldMetadata.type, - }, - property, - ); - } - - const isManyToOneRelation = - isMorphOrRelationUniversalFlatFieldMetadata(relatedFlatFieldMetadata) && - relatedFlatFieldMetadata.universalSettings?.relationType === - RelationType.MANY_TO_ONE; - - return isManyToOneRelation - ? computeMorphOrRelationFieldJoinColumnName({ - name: relatedFlatFieldMetadata.name, - }) - : relatedFlatFieldMetadata.name; - }); - - const name = generateDeterministicIndexNameV2({ + const name = computeFlatIndexNameOrThrow({ flatObjectMetadata, - orderedIndexColumnNames, + objectFlatFieldMetadatas, + indexFields: flatIndex.universalFlatIndexFieldMetadatas, isUnique: flatIndex.isUnique, indexWhereClause: flatIndex.indexWhereClause, }); diff --git a/packages/twenty-server/src/engine/metadata-modules/metadata-side-effect/constants/metadata-side-effect-handler-metadata-key.constant.ts b/packages/twenty-server/src/engine/metadata-modules/metadata-side-effect/constants/metadata-side-effect-handler-metadata-key.constant.ts new file mode 100644 index 0000000000..fbc8e010b0 --- /dev/null +++ b/packages/twenty-server/src/engine/metadata-modules/metadata-side-effect/constants/metadata-side-effect-handler-metadata-key.constant.ts @@ -0,0 +1,2 @@ +export const METADATA_SIDE_EFFECT_HANDLER_METADATA_KEY = + 'METADATA_SIDE_EFFECT_HANDLER_METADATA_KEY'; diff --git a/packages/twenty-server/src/engine/metadata-modules/metadata-side-effect/exceptions/metadata-side-effect-exception-code.ts b/packages/twenty-server/src/engine/metadata-modules/metadata-side-effect/exceptions/metadata-side-effect-exception-code.ts new file mode 100644 index 0000000000..e46ce72646 --- /dev/null +++ b/packages/twenty-server/src/engine/metadata-modules/metadata-side-effect/exceptions/metadata-side-effect-exception-code.ts @@ -0,0 +1,4 @@ +export enum MetadataSideEffectExceptionCode { + RESERVED_SYSTEM_UNIVERSAL_IDENTIFIER = 'RESERVED_SYSTEM_UNIVERSAL_IDENTIFIER', + SIDE_EFFECT_PARENT_METADATA_NOT_FOUND = 'SIDE_EFFECT_PARENT_METADATA_NOT_FOUND', +} diff --git a/packages/twenty-server/src/engine/metadata-modules/metadata-side-effect/handlers/field-metadata/services/field-unique-backing-index-on-create-side-effect-handler.service.ts b/packages/twenty-server/src/engine/metadata-modules/metadata-side-effect/handlers/field-metadata/services/field-unique-backing-index-on-create-side-effect-handler.service.ts new file mode 100644 index 0000000000..8f1e85fa39 --- /dev/null +++ b/packages/twenty-server/src/engine/metadata-modules/metadata-side-effect/handlers/field-metadata/services/field-unique-backing-index-on-create-side-effect-handler.service.ts @@ -0,0 +1,74 @@ +import { Injectable } from '@nestjs/common'; + +import { isDefined } from 'twenty-shared/utils'; + +import { generateDeterministicIndexForFlatFieldMetadataOrThrow } from 'src/engine/metadata-modules/flat-field-metadata/utils/generate-deterministic-index-for-flat-field-metadata-or-throw.util'; +import { isMorphOrRelationUniversalFlatFieldMetadata } from 'src/engine/metadata-modules/flat-field-metadata/utils/is-morph-or-relation-flat-field-metadata.util'; +import { isPrimaryKeyFlatFieldMetadata } from 'src/engine/metadata-modules/flat-field-metadata/utils/is-primary-key-flat-field-metadata.util'; +import { buildFieldSideEffectParentNotFoundFailure } from 'src/engine/metadata-modules/metadata-side-effect/handlers/field-metadata/utils/build-field-side-effect-parent-not-found-failure.util'; +import { resolveParentFlatObjectMetadataAfterStateForFieldSideEffect } from 'src/engine/metadata-modules/metadata-side-effect/handlers/field-metadata/utils/resolve-parent-flat-object-metadata-after-state-for-field-side-effect.util'; +import { + type BuildSideEffectsArgs, + MetadataSideEffectHandler, +} from 'src/engine/metadata-modules/metadata-side-effect/interfaces/base-metadata-side-effect-handler.service'; +import { type MetadataSideEffectResult } from 'src/engine/metadata-modules/metadata-side-effect/types/metadata-side-effect-result.type'; + +@Injectable() +export class FieldUniqueBackingIndexOnCreateSideEffectHandlerService extends MetadataSideEffectHandler( + { + operation: 'create', + metadataName: 'fieldMetadata', + name: 'fieldUniqueBackingIndexOnCreate', + description: + 'When a unique scalar field is created, generate the single-field UNIQUE index that enforces its uniqueness constraint at the database level.', + }, +) { + buildSideEffects({ + flatEntity: flatFieldMetadata, + allFlatEntityOperationRecordByMetadataName, + relatedFlatEntityMaps, + }: BuildSideEffectsArgs<'fieldMetadata'>): MetadataSideEffectResult { + if ( + flatFieldMetadata.isUnique !== true || + isMorphOrRelationUniversalFlatFieldMetadata(flatFieldMetadata) + ) { + return { status: 'noop' }; + } + + if (isPrimaryKeyFlatFieldMetadata(flatFieldMetadata)) { + return { status: 'noop' }; + } + + const parentFlatObjectMetadata = + resolveParentFlatObjectMetadataAfterStateForFieldSideEffect({ + objectMetadataUniversalIdentifier: + flatFieldMetadata.objectMetadataUniversalIdentifier, + allFlatEntityOperationRecordByMetadataName, + relatedFlatEntityMaps, + }); + + if (!isDefined(parentFlatObjectMetadata)) { + return buildFieldSideEffectParentNotFoundFailure({ + flatFieldMetadata, + operation: 'create', + }); + } + + const flatIndexMetadata = + generateDeterministicIndexForFlatFieldMetadataOrThrow({ + flatFieldMetadata, + flatObjectMetadata: parentFlatObjectMetadata, + }); + + return { + status: 'success', + operations: { + index: { + flatEntityToCreate: { + [flatIndexMetadata.universalIdentifier]: flatIndexMetadata, + }, + }, + }, + }; + } +} diff --git a/packages/twenty-server/src/engine/metadata-modules/metadata-side-effect/handlers/field-metadata/services/field-unique-backing-index-on-delete-side-effect-handler.service.ts b/packages/twenty-server/src/engine/metadata-modules/metadata-side-effect/handlers/field-metadata/services/field-unique-backing-index-on-delete-side-effect-handler.service.ts new file mode 100644 index 0000000000..f0ee9f77e3 --- /dev/null +++ b/packages/twenty-server/src/engine/metadata-modules/metadata-side-effect/handlers/field-metadata/services/field-unique-backing-index-on-delete-side-effect-handler.service.ts @@ -0,0 +1,74 @@ +import { Injectable } from '@nestjs/common'; + +import { isDefined } from 'twenty-shared/utils'; + +import { generateDeterministicIndexForFlatFieldMetadataOrThrow } from 'src/engine/metadata-modules/flat-field-metadata/utils/generate-deterministic-index-for-flat-field-metadata-or-throw.util'; +import { isMorphOrRelationUniversalFlatFieldMetadata } from 'src/engine/metadata-modules/flat-field-metadata/utils/is-morph-or-relation-flat-field-metadata.util'; +import { buildFieldSideEffectParentNotFoundFailure } from 'src/engine/metadata-modules/metadata-side-effect/handlers/field-metadata/utils/build-field-side-effect-parent-not-found-failure.util'; +import { + type BuildSideEffectsArgs, + MetadataSideEffectHandler, +} from 'src/engine/metadata-modules/metadata-side-effect/interfaces/base-metadata-side-effect-handler.service'; +import { type MetadataSideEffectResult } from 'src/engine/metadata-modules/metadata-side-effect/types/metadata-side-effect-result.type'; + +@Injectable() +export class FieldUniqueBackingIndexOnDeleteSideEffectHandlerService extends MetadataSideEffectHandler( + { + operation: 'delete', + metadataName: 'fieldMetadata', + name: 'fieldUniqueBackingIndexOnDelete', + description: + 'When a unique scalar field is deleted, cascade-delete the single-field UNIQUE index that backed its uniqueness constraint.', + }, +) { + buildSideEffects({ + flatEntity: flatFieldMetadata, + relatedFlatEntityMaps, + }: BuildSideEffectsArgs<'fieldMetadata'>): MetadataSideEffectResult { + if ( + flatFieldMetadata.isUnique !== true || + isMorphOrRelationUniversalFlatFieldMetadata(flatFieldMetadata) + ) { + return { status: 'noop' }; + } + const parentFlatObjectMetadata = + relatedFlatEntityMaps.flatObjectMetadataMaps.byUniversalIdentifier[ + flatFieldMetadata.objectMetadataUniversalIdentifier + ]; + + if (!isDefined(parentFlatObjectMetadata)) { + return buildFieldSideEffectParentNotFoundFailure({ + flatFieldMetadata, + operation: 'delete', + }); + } + + const flatIndexMetadataToDelete = + generateDeterministicIndexForFlatFieldMetadataOrThrow({ + flatFieldMetadata, + flatObjectMetadata: parentFlatObjectMetadata, + }); + + const indexExistsInWorkspace = isDefined( + relatedFlatEntityMaps.flatIndexMaps.byUniversalIdentifier[ + flatIndexMetadataToDelete.universalIdentifier + ], + ); + + if (!indexExistsInWorkspace) { + return { status: 'noop' }; + } + + return { + status: 'success', + operations: { + index: { + flatEntityToDelete: { + [flatIndexMetadataToDelete.universalIdentifier]: + flatIndexMetadataToDelete, + }, + }, + }, + }; + } +} diff --git a/packages/twenty-server/src/engine/metadata-modules/metadata-side-effect/handlers/field-metadata/services/field-unique-backing-index-on-update-side-effect-handler.service.ts b/packages/twenty-server/src/engine/metadata-modules/metadata-side-effect/handlers/field-metadata/services/field-unique-backing-index-on-update-side-effect-handler.service.ts new file mode 100644 index 0000000000..c858c02d71 --- /dev/null +++ b/packages/twenty-server/src/engine/metadata-modules/metadata-side-effect/handlers/field-metadata/services/field-unique-backing-index-on-update-side-effect-handler.service.ts @@ -0,0 +1,149 @@ +import { Injectable } from '@nestjs/common'; + +import { isDefined } from 'twenty-shared/utils'; + +import { generateDeterministicIndexForFlatFieldMetadataOrThrow } from 'src/engine/metadata-modules/flat-field-metadata/utils/generate-deterministic-index-for-flat-field-metadata-or-throw.util'; +import { isMorphOrRelationUniversalFlatFieldMetadata } from 'src/engine/metadata-modules/flat-field-metadata/utils/is-morph-or-relation-flat-field-metadata.util'; +import { isPrimaryKeyFlatFieldMetadata } from 'src/engine/metadata-modules/flat-field-metadata/utils/is-primary-key-flat-field-metadata.util'; +import { buildFieldSideEffectParentNotFoundFailure } from 'src/engine/metadata-modules/metadata-side-effect/handlers/field-metadata/utils/build-field-side-effect-parent-not-found-failure.util'; +import { resolveParentFlatObjectMetadataAfterStateForFieldSideEffect } from 'src/engine/metadata-modules/metadata-side-effect/handlers/field-metadata/utils/resolve-parent-flat-object-metadata-after-state-for-field-side-effect.util'; +import { + type BuildSideEffectsArgs, + MetadataSideEffectHandler, +} from 'src/engine/metadata-modules/metadata-side-effect/interfaces/base-metadata-side-effect-handler.service'; +import { type MetadataSideEffectResult } from 'src/engine/metadata-modules/metadata-side-effect/types/metadata-side-effect-result.type'; + +@Injectable() +export class FieldUniqueBackingIndexOnUpdateSideEffectHandlerService extends MetadataSideEffectHandler( + { + operation: 'update', + metadataName: 'fieldMetadata', + name: 'fieldUniqueBackingIndexOnUpdate', + description: + "Keep a unique scalar field's backing UNIQUE index in sync when its `isUnique` flag flips or the field is renamed (drop the stale index and recreate the deterministic one).", + }, +) { + buildSideEffects({ + flatEntity: flatFieldMetadata, + allFlatEntityOperationRecordByMetadataName, + relatedFlatEntityMaps, + }: BuildSideEffectsArgs<'fieldMetadata'>): MetadataSideEffectResult { + if (isMorphOrRelationUniversalFlatFieldMetadata(flatFieldMetadata)) { + return { status: 'noop' }; + } + + if (isPrimaryKeyFlatFieldMetadata(flatFieldMetadata)) { + return { status: 'noop' }; + } + + const existingFlatFieldMetadata = + relatedFlatEntityMaps.flatFieldMetadataMaps.byUniversalIdentifier[ + flatFieldMetadata.universalIdentifier + ]; + + if (!isDefined(existingFlatFieldMetadata)) { + return { status: 'noop' }; + } + + const wasRenamed = + existingFlatFieldMetadata.name !== flatFieldMetadata.name; + + const uniquenessHasFlipped = + existingFlatFieldMetadata.isUnique !== flatFieldMetadata.isUnique; + const backingIndexMustFollowRename = + existingFlatFieldMetadata.isUnique === true && + flatFieldMetadata.isUnique === true && + wasRenamed; + + if (!uniquenessHasFlipped && !backingIndexMustFollowRename) { + return { status: 'noop' }; + } + + const existingFlatObjectMetadata = + relatedFlatEntityMaps.flatObjectMetadataMaps.byUniversalIdentifier[ + flatFieldMetadata.objectMetadataUniversalIdentifier + ]; + const optimisticFlatObjectMetadata = + resolveParentFlatObjectMetadataAfterStateForFieldSideEffect({ + objectMetadataUniversalIdentifier: + flatFieldMetadata.objectMetadataUniversalIdentifier, + allFlatEntityOperationRecordByMetadataName, + relatedFlatEntityMaps, + }); + + const backingIndexMustBeDeleted = + existingFlatFieldMetadata.isUnique === true; + const backingIndexMustBeCreated = flatFieldMetadata.isUnique === true; + + if ( + (backingIndexMustBeDeleted && !isDefined(existingFlatObjectMetadata)) || + (backingIndexMustBeCreated && !isDefined(optimisticFlatObjectMetadata)) + ) { + return buildFieldSideEffectParentNotFoundFailure({ + flatFieldMetadata, + operation: 'update', + }); + } + + const previousFlatIndexMetadata = + backingIndexMustBeDeleted && isDefined(existingFlatObjectMetadata) + ? generateDeterministicIndexForFlatFieldMetadataOrThrow({ + flatFieldMetadata: { + ...flatFieldMetadata, + name: existingFlatFieldMetadata.name, + isUnique: true, + }, + flatObjectMetadata: existingFlatObjectMetadata, + }) + : undefined; + + const flatIndexMetadataToDelete = + isDefined(previousFlatIndexMetadata) && + isDefined( + relatedFlatEntityMaps.flatIndexMaps.byUniversalIdentifier[ + previousFlatIndexMetadata.universalIdentifier + ], + ) + ? previousFlatIndexMetadata + : undefined; + + const flatIndexMetadataToCreate = + backingIndexMustBeCreated && isDefined(optimisticFlatObjectMetadata) + ? generateDeterministicIndexForFlatFieldMetadataOrThrow({ + flatFieldMetadata: { ...flatFieldMetadata, isUnique: true }, + flatObjectMetadata: optimisticFlatObjectMetadata, + }) + : undefined; + + if ( + !isDefined(flatIndexMetadataToCreate) && + !isDefined(flatIndexMetadataToDelete) + ) { + return { status: 'noop' }; + } + + return { + status: 'success', + operations: { + index: { + ...(isDefined(flatIndexMetadataToCreate) + ? { + flatEntityToCreate: { + [flatIndexMetadataToCreate.universalIdentifier]: + flatIndexMetadataToCreate, + }, + } + : {}), + ...(isDefined(flatIndexMetadataToDelete) + ? { + flatEntityToDelete: { + [flatIndexMetadataToDelete.universalIdentifier]: + flatIndexMetadataToDelete, + }, + } + : {}), + }, + }, + }; + } +} diff --git a/packages/twenty-server/src/engine/metadata-modules/metadata-side-effect/handlers/field-metadata/utils/build-field-side-effect-parent-not-found-failure.util.ts b/packages/twenty-server/src/engine/metadata-modules/metadata-side-effect/handlers/field-metadata/utils/build-field-side-effect-parent-not-found-failure.util.ts new file mode 100644 index 0000000000..00ba89c6d1 --- /dev/null +++ b/packages/twenty-server/src/engine/metadata-modules/metadata-side-effect/handlers/field-metadata/utils/build-field-side-effect-parent-not-found-failure.util.ts @@ -0,0 +1,31 @@ +import { msg, t } from '@lingui/core/macro'; +import { type AllMetadataName } from 'twenty-shared/metadata'; + +import { type MetadataFlatEntity } from 'src/engine/metadata-modules/flat-entity/types/metadata-flat-entity.type'; +import { type MetadataUniversalFlatEntity } from 'src/engine/metadata-modules/flat-entity/types/metadata-universal-flat-entity.type'; +import { type WorkspaceMigrationActionType } from 'src/engine/metadata-modules/flat-entity/types/metadata-workspace-migration-action.type'; +import { MetadataSideEffectExceptionCode } from 'src/engine/metadata-modules/metadata-side-effect/exceptions/metadata-side-effect-exception-code'; +import { type MetadataSideEffectFailure } from 'src/engine/metadata-modules/metadata-side-effect/types/metadata-side-effect-result.type'; + +export const buildFieldSideEffectParentNotFoundFailure = ({ + flatFieldMetadata, + operation, +}: { + flatFieldMetadata: MetadataUniversalFlatEntity<'fieldMetadata'>; + operation: WorkspaceMigrationActionType; +}): MetadataSideEffectFailure => ({ + status: 'fail', + type: operation, + metadataName: 'fieldMetadata', + flatEntityMinimalInformation: { + universalIdentifier: flatFieldMetadata.universalIdentifier, + name: flatFieldMetadata.name, + } as Partial>, + errors: [ + { + code: MetadataSideEffectExceptionCode.SIDE_EFFECT_PARENT_METADATA_NOT_FOUND, + message: t`Could not resolve parent object metadata "${flatFieldMetadata.objectMetadataUniversalIdentifier}" for field unique index side effect`, + userFriendlyMessage: msg`This field references an object that could not be found`, + }, + ], +}); diff --git a/packages/twenty-server/src/engine/metadata-modules/metadata-side-effect/handlers/field-metadata/utils/resolve-parent-flat-object-metadata-after-state-for-field-side-effect.util.ts b/packages/twenty-server/src/engine/metadata-modules/metadata-side-effect/handlers/field-metadata/utils/resolve-parent-flat-object-metadata-after-state-for-field-side-effect.util.ts new file mode 100644 index 0000000000..de675f4c1d --- /dev/null +++ b/packages/twenty-server/src/engine/metadata-modules/metadata-side-effect/handlers/field-metadata/utils/resolve-parent-flat-object-metadata-after-state-for-field-side-effect.util.ts @@ -0,0 +1,26 @@ +import { type AllFlatEntityOperationRecordByMetadataName } from 'src/engine/metadata-modules/flat-entity/types/all-flat-entity-operation-record-by-metadata-name.type'; +import { type MetadataFlatEntityAndRelatedFlatEntityMapsForSideEffect } from 'src/engine/metadata-modules/flat-entity/types/metadata-flat-entity-and-related-flat-entity-maps-for-side-effect.type'; +import { type UniversalFlatObjectMetadata } from 'src/engine/workspace-manager/workspace-migration/universal-flat-entity/types/universal-flat-object-metadata.type'; + +export const resolveParentFlatObjectMetadataAfterStateForFieldSideEffect = ({ + objectMetadataUniversalIdentifier, + allFlatEntityOperationRecordByMetadataName, + relatedFlatEntityMaps, +}: { + objectMetadataUniversalIdentifier: string; + allFlatEntityOperationRecordByMetadataName: AllFlatEntityOperationRecordByMetadataName; + relatedFlatEntityMaps: MetadataFlatEntityAndRelatedFlatEntityMapsForSideEffect<'fieldMetadata'>; +}): UniversalFlatObjectMetadata | undefined => { + const pendingFlatObjectMetadata = + allFlatEntityOperationRecordByMetadataName.objectMetadata + ?.flatEntityToUpdate[objectMetadataUniversalIdentifier] ?? + allFlatEntityOperationRecordByMetadataName.objectMetadata + ?.flatEntityToCreate[objectMetadataUniversalIdentifier]; + + return ( + pendingFlatObjectMetadata ?? + relatedFlatEntityMaps.flatObjectMetadataMaps.byUniversalIdentifier[ + objectMetadataUniversalIdentifier + ] + ); +}; diff --git a/packages/twenty-server/src/engine/metadata-modules/metadata-side-effect/handlers/metadata-side-effect-handlers.module.ts b/packages/twenty-server/src/engine/metadata-modules/metadata-side-effect/handlers/metadata-side-effect-handlers.module.ts new file mode 100644 index 0000000000..8da389827c --- /dev/null +++ b/packages/twenty-server/src/engine/metadata-modules/metadata-side-effect/handlers/metadata-side-effect-handlers.module.ts @@ -0,0 +1,14 @@ +import { Module } from '@nestjs/common'; + +import { FieldUniqueBackingIndexOnCreateSideEffectHandlerService } from 'src/engine/metadata-modules/metadata-side-effect/handlers/field-metadata/services/field-unique-backing-index-on-create-side-effect-handler.service'; +import { FieldUniqueBackingIndexOnDeleteSideEffectHandlerService } from 'src/engine/metadata-modules/metadata-side-effect/handlers/field-metadata/services/field-unique-backing-index-on-delete-side-effect-handler.service'; +import { FieldUniqueBackingIndexOnUpdateSideEffectHandlerService } from 'src/engine/metadata-modules/metadata-side-effect/handlers/field-metadata/services/field-unique-backing-index-on-update-side-effect-handler.service'; + +@Module({ + providers: [ + FieldUniqueBackingIndexOnCreateSideEffectHandlerService, + FieldUniqueBackingIndexOnUpdateSideEffectHandlerService, + FieldUniqueBackingIndexOnDeleteSideEffectHandlerService, + ], +}) +export class MetadataSideEffectHandlersModule {} diff --git a/packages/twenty-server/src/engine/metadata-modules/metadata-side-effect/interfaces/base-metadata-side-effect-handler.service.ts b/packages/twenty-server/src/engine/metadata-modules/metadata-side-effect/interfaces/base-metadata-side-effect-handler.service.ts new file mode 100644 index 0000000000..4f44dae5db --- /dev/null +++ b/packages/twenty-server/src/engine/metadata-modules/metadata-side-effect/interfaces/base-metadata-side-effect-handler.service.ts @@ -0,0 +1,61 @@ +import { SetMetadata } from '@nestjs/common'; + +import { type AllMetadataName } from 'twenty-shared/metadata'; + +import { type AllFlatEntityOperationRecordByMetadataName } from 'src/engine/metadata-modules/flat-entity/types/all-flat-entity-operation-record-by-metadata-name.type'; +import { type MetadataFlatEntityAndRelatedFlatEntityMapsForSideEffect } from 'src/engine/metadata-modules/flat-entity/types/metadata-flat-entity-and-related-flat-entity-maps-for-side-effect.type'; +import { type MetadataUniversalFlatEntity } from 'src/engine/metadata-modules/flat-entity/types/metadata-universal-flat-entity.type'; +import { METADATA_SIDE_EFFECT_HANDLER_METADATA_KEY } from 'src/engine/metadata-modules/metadata-side-effect/constants/metadata-side-effect-handler-metadata-key.constant'; +import { type MetadataSideEffectContext } from 'src/engine/metadata-modules/metadata-side-effect/types/metadata-side-effect-context.type'; +import { type MetadataSideEffectOperation } from 'src/engine/metadata-modules/metadata-side-effect/types/metadata-side-effect-operation.type'; +import { type MetadataSideEffectResult } from 'src/engine/metadata-modules/metadata-side-effect/types/metadata-side-effect-result.type'; + +export type BuildSideEffectsArgs

= { + flatEntity: MetadataUniversalFlatEntity

; + allFlatEntityOperationRecordByMetadataName: AllFlatEntityOperationRecordByMetadataName; + relatedFlatEntityMaps: MetadataFlatEntityAndRelatedFlatEntityMapsForSideEffect

; + context: MetadataSideEffectContext; +}; + +export abstract class BaseMetadataSideEffectHandlerService< + P extends AllMetadataName, +> { + public operation: MetadataSideEffectOperation; + public metadataName: P; + public sideEffectName: string; + public sideEffectDescription: string; + + abstract buildSideEffects( + args: BuildSideEffectsArgs

, + ): MetadataSideEffectResult; +} + +type MetadataSideEffectHandlerDeclaration

= { + operation: MetadataSideEffectOperation; + metadataName: P; + name: string; + description: string; +}; + +export function MetadataSideEffectHandler

({ + operation, + metadataName, + name, + description, +}: MetadataSideEffectHandlerDeclaration

): typeof BaseMetadataSideEffectHandlerService

{ + abstract class SideEffectHandlerService extends BaseMetadataSideEffectHandlerService

{ + operation = operation; + metadataName = metadataName; + sideEffectName = name; + sideEffectDescription = description; + } + + SetMetadata(METADATA_SIDE_EFFECT_HANDLER_METADATA_KEY, { + operation, + metadataName, + name, + description, + })(SideEffectHandlerService); + + return SideEffectHandlerService; +} diff --git a/packages/twenty-server/src/engine/metadata-modules/metadata-side-effect/metadata-side-effect.module.ts b/packages/twenty-server/src/engine/metadata-modules/metadata-side-effect/metadata-side-effect.module.ts new file mode 100644 index 0000000000..af05dd9909 --- /dev/null +++ b/packages/twenty-server/src/engine/metadata-modules/metadata-side-effect/metadata-side-effect.module.ts @@ -0,0 +1,16 @@ +import { Module } from '@nestjs/common'; +import { DiscoveryModule } from '@nestjs/core'; + +import { MetadataSideEffectHandlersModule } from 'src/engine/metadata-modules/metadata-side-effect/handlers/metadata-side-effect-handlers.module'; +import { MetadataSideEffectHandlerRegistryService } from 'src/engine/metadata-modules/metadata-side-effect/registry/metadata-side-effect-handler-registry.service'; +import { MetadataSideEffectEngineService } from 'src/engine/metadata-modules/metadata-side-effect/services/metadata-side-effect-engine.service'; + +@Module({ + imports: [DiscoveryModule, MetadataSideEffectHandlersModule], + providers: [ + MetadataSideEffectHandlerRegistryService, + MetadataSideEffectEngineService, + ], + exports: [MetadataSideEffectEngineService], +}) +export class MetadataSideEffectModule {} diff --git a/packages/twenty-server/src/engine/metadata-modules/metadata-side-effect/registry/metadata-side-effect-handler-registry.service.ts b/packages/twenty-server/src/engine/metadata-modules/metadata-side-effect/registry/metadata-side-effect-handler-registry.service.ts new file mode 100644 index 0000000000..baf507b030 --- /dev/null +++ b/packages/twenty-server/src/engine/metadata-modules/metadata-side-effect/registry/metadata-side-effect-handler-registry.service.ts @@ -0,0 +1,109 @@ +import { Injectable, OnModuleInit } from '@nestjs/common'; +import { DiscoveryService } from '@nestjs/core'; + +import { type AllMetadataName } from 'twenty-shared/metadata'; +import { isDefined } from 'twenty-shared/utils'; + +import { METADATA_SIDE_EFFECT_HANDLER_METADATA_KEY } from 'src/engine/metadata-modules/metadata-side-effect/constants/metadata-side-effect-handler-metadata-key.constant'; +import { MetadataSideEffectHandlersModule } from 'src/engine/metadata-modules/metadata-side-effect/handlers/metadata-side-effect-handlers.module'; +import { type BaseMetadataSideEffectHandlerService } from 'src/engine/metadata-modules/metadata-side-effect/interfaces/base-metadata-side-effect-handler.service'; +import { + buildMetadataSideEffectHandlerKey, + type MetadataSideEffectHandlerDescriptor, + type MetadataSideEffectHandlerKey, + type MetadataSideEffectOperation, +} from 'src/engine/metadata-modules/metadata-side-effect/types/metadata-side-effect-operation.type'; + +type RegisteredSideEffectHandler = + BaseMetadataSideEffectHandlerService; + +export type RegisteredMetadataSideEffectHandlerKey = { + operation: MetadataSideEffectOperation; + metadataName: AllMetadataName; +}; + +@Injectable() +export class MetadataSideEffectHandlerRegistryService implements OnModuleInit { + private readonly handlersByKey = new Map< + MetadataSideEffectHandlerKey, + RegisteredSideEffectHandler[] + >(); + private readonly registeredHandlerKeys: RegisteredMetadataSideEffectHandlerKey[] = + []; + private readonly registeredSideEffectNames = new Set(); + + constructor(private readonly discoveryService: DiscoveryService) {} + + onModuleInit() { + this.discoverAndRegisterHandlers(); + } + + private discoverAndRegisterHandlers(): void { + const providers = this.discoveryService.getProviders({ + include: [MetadataSideEffectHandlersModule], + }); + + providers.forEach((wrapper) => { + const { instance, metatype } = wrapper; + + if (!instance || !metatype) return; + + const descriptor: MetadataSideEffectHandlerDescriptor | undefined = + Reflect.getMetadata( + METADATA_SIDE_EFFECT_HANDLER_METADATA_KEY, + metatype, + ); + + if ( + !isDefined(descriptor) || + typeof instance.buildSideEffects !== 'function' + ) { + return; + } + + this.registerHandler(instance); + }); + } + + private registerHandler(instance: RegisteredSideEffectHandler): void { + if (this.registeredSideEffectNames.has(instance.sideEffectName)) { + throw new Error( + `Duplicate metadata side-effect name "${instance.sideEffectName}". Side-effect names must be unique.`, + ); + } + this.registeredSideEffectNames.add(instance.sideEffectName); + + const handlerKey = buildMetadataSideEffectHandlerKey( + instance.operation, + instance.metadataName, + ); + const existingHandlers = this.handlersByKey.get(handlerKey); + + if (isDefined(existingHandlers)) { + existingHandlers.push(instance); + + return; + } + + this.handlersByKey.set(handlerKey, [instance]); + this.registeredHandlerKeys.push({ + operation: instance.operation, + metadataName: instance.metadataName, + }); + } + + getHandlers( + operation: MetadataSideEffectOperation, + metadataName: AllMetadataName, + ): RegisteredSideEffectHandler[] { + return ( + this.handlersByKey.get( + buildMetadataSideEffectHandlerKey(operation, metadataName), + ) ?? [] + ); + } + + getRegisteredHandlerKeys(): RegisteredMetadataSideEffectHandlerKey[] { + return this.registeredHandlerKeys; + } +} diff --git a/packages/twenty-server/src/engine/metadata-modules/metadata-side-effect/services/metadata-side-effect-engine.service.ts b/packages/twenty-server/src/engine/metadata-modules/metadata-side-effect/services/metadata-side-effect-engine.service.ts new file mode 100644 index 0000000000..1d9b6dc7d2 --- /dev/null +++ b/packages/twenty-server/src/engine/metadata-modules/metadata-side-effect/services/metadata-side-effect-engine.service.ts @@ -0,0 +1,329 @@ +import { Injectable } from '@nestjs/common'; + +import { type AllMetadataName } from 'twenty-shared/metadata'; +import { isDefined } from 'twenty-shared/utils'; + +import { type AllFlatEntityMaps } from 'src/engine/metadata-modules/flat-entity/types/all-flat-entity-maps.type'; +import { type AllFlatEntityOperationRecordByMetadataName } from 'src/engine/metadata-modules/flat-entity/types/all-flat-entity-operation-record-by-metadata-name.type'; +import { type MetadataFlatEntityAndRelatedFlatEntityMapsForSideEffect } from 'src/engine/metadata-modules/flat-entity/types/metadata-flat-entity-and-related-flat-entity-maps-for-side-effect.type'; +import { type MetadataUniversalFlatEntity } from 'src/engine/metadata-modules/flat-entity/types/metadata-universal-flat-entity.type'; +import { getMetadataManyToOneRelatedNames } from 'src/engine/metadata-modules/flat-entity/utils/get-metadata-many-to-one-related-names.util'; +import { getMetadataSideEffectCompanionNames } from 'src/engine/metadata-modules/flat-entity/utils/get-metadata-side-effect-companion-names.util'; +import { isSystemSideEffectFlatEntity } from 'src/engine/metadata-modules/flat-entity/utils/is-system-side-effect-flat-entity.util'; +import { MetadataSideEffectHandlerRegistryService } from 'src/engine/metadata-modules/metadata-side-effect/registry/metadata-side-effect-handler-registry.service'; +import { type MetadataSideEffectContext } from 'src/engine/metadata-modules/metadata-side-effect/types/metadata-side-effect-context.type'; +import { type MetadataSideEffectExpansionResult } from 'src/engine/metadata-modules/metadata-side-effect/types/metadata-side-effect-expansion-result.type'; +import { + METADATA_SIDE_EFFECT_OPERATIONS, + type MetadataSideEffectOperation, +} from 'src/engine/metadata-modules/metadata-side-effect/types/metadata-side-effect-operation.type'; +import { type MetadataSideEffectFailure } from 'src/engine/metadata-modules/metadata-side-effect/types/metadata-side-effect-result.type'; +import { type SystemSideEffectUniversalIdentifierCollision } from 'src/engine/metadata-modules/metadata-side-effect/types/system-side-effect-universal-identifier-collision.type'; +import { mapSystemSideEffectCollisionToFailure } from 'src/engine/metadata-modules/metadata-side-effect/utils/map-system-side-effect-collision-to-failure.util'; +import { EMPTY_ORCHESTRATOR_FAILURE_REPORT } from 'src/engine/workspace-manager/workspace-migration/constant/empty-orchestrator-failure-report.constant'; +import { pushToOrchestratorFailureReport } from 'src/engine/workspace-manager/workspace-migration/utils/merge-orchestrator-failure-reports.util'; + +type GenericUniversalFlatEntity = { universalIdentifier: string }; +type GenericFlatEntityOperationRecord = { + flatEntityToCreate: Record; + flatEntityToUpdate: Record; + flatEntityToDelete: Record; +}; +type GenericAllFlatEntityOperationRecordByMetadataName = Record< + string, + GenericFlatEntityOperationRecord | undefined +>; + +type GenericPartialFlatEntityOperationRecord = { + flatEntityToCreate?: Record; + flatEntityToUpdate?: Record; + flatEntityToDelete?: Record; +}; +type GenericMetadataSideEffectOperationsByMetadataName = Record< + string, + GenericPartialFlatEntityOperationRecord | undefined +>; + +const OPERATION_TO_FLAT_ENTITY_RECORD_KEY = { + create: 'flatEntityToCreate', + update: 'flatEntityToUpdate', + delete: 'flatEntityToDelete', +} as const satisfies Record< + MetadataSideEffectOperation, + keyof GenericFlatEntityOperationRecord +>; + +@Injectable() +export class MetadataSideEffectEngineService { + constructor( + private readonly metadataSideEffectHandlerRegistryService: MetadataSideEffectHandlerRegistryService, + ) {} + + getSideEffectRelatedMetadataNames( + triggerMetadataNames: AllMetadataName[], + ): AllMetadataName[] { + const relatedMetadataNames = new Set(); + + for (const { + metadataName, + } of this.metadataSideEffectHandlerRegistryService.getRegisteredHandlerKeys()) { + if (!triggerMetadataNames.includes(metadataName)) { + continue; + } + + for (const relatedMetadataName of [ + metadataName, + ...getMetadataManyToOneRelatedNames(metadataName), + ...getMetadataSideEffectCompanionNames(metadataName), + ]) { + relatedMetadataNames.add(relatedMetadataName); + } + } + + return [...relatedMetadataNames]; + } + + expandWithSideEffects({ + allFlatEntityOperationRecordByMetadataName, + sideEffectRelatedFlatEntityMaps, + context, + }: { + allFlatEntityOperationRecordByMetadataName: AllFlatEntityOperationRecordByMetadataName; + sideEffectRelatedFlatEntityMaps: Partial; + context: MetadataSideEffectContext; + }): MetadataSideEffectExpansionResult { + const expandedMatrix = this.cloneMatrix( + allFlatEntityOperationRecordByMetadataName, + ); + const systemSideEffectUniversalIdentifierCollisions: SystemSideEffectUniversalIdentifierCollision[] = + []; + const sideEffectFailures: MetadataSideEffectFailure[] = []; + + const triggerMatrix = + allFlatEntityOperationRecordByMetadataName as unknown as GenericAllFlatEntityOperationRecordByMetadataName; + + for (const { + operation, + metadataName, + } of this.metadataSideEffectHandlerRegistryService.getRegisteredHandlerKeys()) { + const handlers = + this.metadataSideEffectHandlerRegistryService.getHandlers( + operation, + metadataName, + ); + + if (handlers.length === 0) { + continue; + } + + const triggerFlatEntities = Object.values( + triggerMatrix[metadataName]?.[ + OPERATION_TO_FLAT_ENTITY_RECORD_KEY[operation] + ] ?? {}, + ); + + for (const triggerFlatEntity of triggerFlatEntities) { + for (const handler of handlers) { + const sideEffectResult = handler.buildSideEffects({ + flatEntity: + triggerFlatEntity as unknown as MetadataUniversalFlatEntity, + allFlatEntityOperationRecordByMetadataName: + expandedMatrix as unknown as AllFlatEntityOperationRecordByMetadataName, + relatedFlatEntityMaps: + sideEffectRelatedFlatEntityMaps as unknown as MetadataFlatEntityAndRelatedFlatEntityMapsForSideEffect, + context, + }); + + if (sideEffectResult.status === 'fail') { + sideEffectFailures.push(sideEffectResult); + continue; + } + + if (sideEffectResult.status === 'noop') { + continue; + } + + this.mergeSideEffectsIntoMatrix({ + expandedMatrix, + sideEffectOperations: + sideEffectResult.operations as unknown as GenericMetadataSideEffectOperationsByMetadataName, + systemSideEffectUniversalIdentifierCollisions, + }); + } + } + } + + const allSideEffectFailures: MetadataSideEffectFailure[] = [ + ...sideEffectFailures, + ...systemSideEffectUniversalIdentifierCollisions.map( + mapSystemSideEffectCollisionToFailure, + ), + ]; + + if (allSideEffectFailures.length > 0) { + const report = EMPTY_ORCHESTRATOR_FAILURE_REPORT(); + + for (const sideEffectFailure of allSideEffectFailures) { + pushToOrchestratorFailureReport({ + report, + metadataName: sideEffectFailure.metadataName, + items: [sideEffectFailure], + }); + } + + return { + status: 'fail', + report, + }; + } + + return { + status: 'success', + allFlatEntityOperationRecordByMetadataName: + expandedMatrix as unknown as AllFlatEntityOperationRecordByMetadataName, + }; + } + + private mergeSideEffectsIntoMatrix({ + expandedMatrix, + sideEffectOperations, + systemSideEffectUniversalIdentifierCollisions, + }: { + expandedMatrix: GenericAllFlatEntityOperationRecordByMetadataName; + sideEffectOperations: GenericMetadataSideEffectOperationsByMetadataName; + systemSideEffectUniversalIdentifierCollisions: SystemSideEffectUniversalIdentifierCollision[]; + }): void { + for (const metadataName of Object.keys(sideEffectOperations)) { + const operationBuckets = sideEffectOperations[metadataName]; + + if (!isDefined(operationBuckets)) { + continue; + } + + for (const operation of METADATA_SIDE_EFFECT_OPERATIONS) { + const sideEffectFlatEntities = Object.values( + operationBuckets[OPERATION_TO_FLAT_ENTITY_RECORD_KEY[operation]] ?? + {}, + ); + + for (const sideEffectFlatEntity of sideEffectFlatEntities) { + this.addToOperationIfAbsent({ + expandedMatrix, + operation, + metadataName, + flatEntity: sideEffectFlatEntity, + systemSideEffectUniversalIdentifierCollisions, + }); + } + } + } + } + + private cloneMatrix( + allFlatEntityOperationRecordByMetadataName: AllFlatEntityOperationRecordByMetadataName, + ): GenericAllFlatEntityOperationRecordByMetadataName { + const genericMatrix = + allFlatEntityOperationRecordByMetadataName as unknown as GenericAllFlatEntityOperationRecordByMetadataName; + const clonedMatrix: GenericAllFlatEntityOperationRecordByMetadataName = {}; + + for (const metadataName of Object.keys(genericMatrix)) { + const operations = genericMatrix[metadataName]; + + if (!isDefined(operations)) { + continue; + } + + clonedMatrix[metadataName] = { + flatEntityToCreate: { ...operations.flatEntityToCreate }, + flatEntityToUpdate: { ...operations.flatEntityToUpdate }, + flatEntityToDelete: { ...operations.flatEntityToDelete }, + }; + } + + return clonedMatrix; + } + + private addToOperationIfAbsent({ + expandedMatrix, + operation, + metadataName, + flatEntity, + systemSideEffectUniversalIdentifierCollisions, + }: { + expandedMatrix: GenericAllFlatEntityOperationRecordByMetadataName; + operation: MetadataSideEffectOperation; + metadataName: string; + flatEntity: GenericUniversalFlatEntity; + systemSideEffectUniversalIdentifierCollisions: SystemSideEffectUniversalIdentifierCollision[]; + }): void { + const operations = (expandedMatrix[metadataName] ??= { + flatEntityToCreate: {}, + flatEntityToUpdate: {}, + flatEntityToDelete: {}, + }); + + const flatEntityRecordKey = OPERATION_TO_FLAT_ENTITY_RECORD_KEY[operation]; + const flatEntityRecord = operations[flatEntityRecordKey]; + + const existingFlatEntity = flatEntityRecord[flatEntity.universalIdentifier]; + + if (isDefined(existingFlatEntity)) { + this.recordUniversalIdentifierCollisionIfNeeded({ + existingFlatEntity, + operation, + metadataName, + flatEntity, + systemSideEffectUniversalIdentifierCollisions, + }); + + return; + } + + flatEntityRecord[flatEntity.universalIdentifier] = flatEntity; + } + + private recordUniversalIdentifierCollisionIfNeeded({ + existingFlatEntity, + operation, + metadataName, + flatEntity, + systemSideEffectUniversalIdentifierCollisions, + }: { + existingFlatEntity: GenericUniversalFlatEntity; + operation: MetadataSideEffectOperation; + metadataName: string; + flatEntity: GenericUniversalFlatEntity; + systemSideEffectUniversalIdentifierCollisions: SystemSideEffectUniversalIdentifierCollision[]; + }): void { + const isIncomingSystemSideEffect = isSystemSideEffectFlatEntity( + flatEntity as unknown as MetadataUniversalFlatEntity, + ); + + if (!isIncomingSystemSideEffect) { + return; + } + + if ( + isSystemSideEffectFlatEntity( + existingFlatEntity as unknown as MetadataUniversalFlatEntity, + ) + ) { + return; + } + + systemSideEffectUniversalIdentifierCollisions.push({ + metadataName: metadataName as AllMetadataName, + operation, + universalIdentifier: flatEntity.universalIdentifier, + name: this.extractFlatEntityName(flatEntity), + }); + } + + private extractFlatEntityName( + flatEntity: GenericUniversalFlatEntity, + ): string | undefined { + const { name } = flatEntity as { name?: unknown }; + + return typeof name === 'string' ? name : undefined; + } +} diff --git a/packages/twenty-server/src/engine/metadata-modules/metadata-side-effect/types/metadata-side-effect-context.type.ts b/packages/twenty-server/src/engine/metadata-modules/metadata-side-effect/types/metadata-side-effect-context.type.ts new file mode 100644 index 0000000000..277db4cd8d --- /dev/null +++ b/packages/twenty-server/src/engine/metadata-modules/metadata-side-effect/types/metadata-side-effect-context.type.ts @@ -0,0 +1,5 @@ +import { type WorkspaceMigrationBuilderOptions } from 'src/engine/workspace-manager/workspace-migration/workspace-migration-builder/types/workspace-migration-builder-options.type'; + +export type MetadataSideEffectContext = { + buildOptions: WorkspaceMigrationBuilderOptions; +}; diff --git a/packages/twenty-server/src/engine/metadata-modules/metadata-side-effect/types/metadata-side-effect-expansion-result.type.ts b/packages/twenty-server/src/engine/metadata-modules/metadata-side-effect/types/metadata-side-effect-expansion-result.type.ts new file mode 100644 index 0000000000..a26b6621bd --- /dev/null +++ b/packages/twenty-server/src/engine/metadata-modules/metadata-side-effect/types/metadata-side-effect-expansion-result.type.ts @@ -0,0 +1,15 @@ +import { type AllFlatEntityOperationRecordByMetadataName } from 'src/engine/metadata-modules/flat-entity/types/all-flat-entity-operation-record-by-metadata-name.type'; +import { type OrchestratorFailureReport } from 'src/engine/workspace-manager/workspace-migration/types/workspace-migration-orchestrator.type'; + +// Same contract as the builder (WorkspaceMigrationOrchestratorFailedResult): +// collisions and handler failures are merged into a single OrchestratorFailureReport +// so callers handle side-effect and builder failures through one uniform channel. +export type MetadataSideEffectExpansionResult = + | { + status: 'success'; + allFlatEntityOperationRecordByMetadataName: AllFlatEntityOperationRecordByMetadataName; + } + | { + status: 'fail'; + report: OrchestratorFailureReport; + }; diff --git a/packages/twenty-server/src/engine/metadata-modules/metadata-side-effect/types/metadata-side-effect-operation.type.ts b/packages/twenty-server/src/engine/metadata-modules/metadata-side-effect/types/metadata-side-effect-operation.type.ts new file mode 100644 index 0000000000..5960464049 --- /dev/null +++ b/packages/twenty-server/src/engine/metadata-modules/metadata-side-effect/types/metadata-side-effect-operation.type.ts @@ -0,0 +1,24 @@ +import { type AllMetadataName } from 'twenty-shared/metadata'; + +export type MetadataSideEffectOperation = 'create' | 'update' | 'delete'; + +export const METADATA_SIDE_EFFECT_OPERATIONS = [ + 'create', + 'update', + 'delete', +] as const satisfies readonly MetadataSideEffectOperation[]; + +export type MetadataSideEffectHandlerKey = + `${MetadataSideEffectOperation}:${AllMetadataName}`; + +export const buildMetadataSideEffectHandlerKey = ( + operation: MetadataSideEffectOperation, + metadataName: AllMetadataName, +): MetadataSideEffectHandlerKey => `${operation}:${metadataName}`; + +export type MetadataSideEffectHandlerDescriptor = { + operation: MetadataSideEffectOperation; + metadataName: AllMetadataName; + name: string; + description: string; +}; diff --git a/packages/twenty-server/src/engine/metadata-modules/metadata-side-effect/types/metadata-side-effect-operations-by-metadata-name.type.ts b/packages/twenty-server/src/engine/metadata-modules/metadata-side-effect/types/metadata-side-effect-operations-by-metadata-name.type.ts new file mode 100644 index 0000000000..fed82b3de7 --- /dev/null +++ b/packages/twenty-server/src/engine/metadata-modules/metadata-side-effect/types/metadata-side-effect-operations-by-metadata-name.type.ts @@ -0,0 +1,17 @@ +import { type AllMetadataName } from 'twenty-shared/metadata'; + +import { type MetadataUniversalFlatEntity } from 'src/engine/metadata-modules/flat-entity/types/metadata-universal-flat-entity.type'; + +// Record-native, partial companion operations a handler emits. Each bucket is keyed by +// universalIdentifier so the engine merges/deduplicates against the matrix in O(1), +// matching the shape it consumes (AllFlatEntityOperationRecordByMetadataName). +export type MetadataSideEffectOperationsByMetadataName = { + [P in AllMetadataName]?: { + flatEntityToCreate?: Record< + string, + MetadataUniversalFlatEntity

& { id?: string } + >; + flatEntityToUpdate?: Record>; + flatEntityToDelete?: Record>; + }; +}; diff --git a/packages/twenty-server/src/engine/metadata-modules/metadata-side-effect/types/metadata-side-effect-result.type.ts b/packages/twenty-server/src/engine/metadata-modules/metadata-side-effect/types/metadata-side-effect-result.type.ts new file mode 100644 index 0000000000..e51e7a7d5d --- /dev/null +++ b/packages/twenty-server/src/engine/metadata-modules/metadata-side-effect/types/metadata-side-effect-result.type.ts @@ -0,0 +1,27 @@ +import { type AllMetadataName } from 'twenty-shared/metadata'; + +import { type WorkspaceMigrationActionType } from 'src/engine/metadata-modules/flat-entity/types/metadata-workspace-migration-action.type'; +import { type MetadataSideEffectOperationsByMetadataName } from 'src/engine/metadata-modules/metadata-side-effect/types/metadata-side-effect-operations-by-metadata-name.type'; +import { type FailedFlatEntityValidation } from 'src/engine/workspace-manager/workspace-migration/workspace-migration-builder/builders/types/failed-flat-entity-validation.type'; + +// A single failure emitted by a side-effect handler, using the exact same shape +// as the builder's per-entity validation failure (a `status: 'fail'` tag spread +// over a FailedFlatEntityValidation), so both sources merge into one report. +export type MetadataSideEffectFailure = { + status: 'fail'; +} & FailedFlatEntityValidation; + +// Discriminated union of the three handler outcomes: `noop` when there is +// nothing to do, `success` when it produced companion operations to merge, or +// `fail` (fail-slow) with a validation failure the engine aggregates instead of +// throwing. `success` therefore always carries operations, keeping the empty +// case explicit as `noop` rather than a success with `operations: {}`. +export type MetadataSideEffectResult = + | { + status: 'noop'; + } + | { + status: 'success'; + operations: MetadataSideEffectOperationsByMetadataName; + } + | MetadataSideEffectFailure; diff --git a/packages/twenty-server/src/engine/metadata-modules/metadata-side-effect/types/system-side-effect-universal-identifier-collision.type.ts b/packages/twenty-server/src/engine/metadata-modules/metadata-side-effect/types/system-side-effect-universal-identifier-collision.type.ts new file mode 100644 index 0000000000..b40abe7ad0 --- /dev/null +++ b/packages/twenty-server/src/engine/metadata-modules/metadata-side-effect/types/system-side-effect-universal-identifier-collision.type.ts @@ -0,0 +1,10 @@ +import { type AllMetadataName } from 'twenty-shared/metadata'; + +import { type MetadataSideEffectOperation } from 'src/engine/metadata-modules/metadata-side-effect/types/metadata-side-effect-operation.type'; + +export type SystemSideEffectUniversalIdentifierCollision = { + metadataName: AllMetadataName; + operation: MetadataSideEffectOperation; + universalIdentifier: string; + name?: string; +}; diff --git a/packages/twenty-server/src/engine/metadata-modules/metadata-side-effect/utils/map-system-side-effect-collision-to-failure.util.ts b/packages/twenty-server/src/engine/metadata-modules/metadata-side-effect/utils/map-system-side-effect-collision-to-failure.util.ts new file mode 100644 index 0000000000..247c57b9fb --- /dev/null +++ b/packages/twenty-server/src/engine/metadata-modules/metadata-side-effect/utils/map-system-side-effect-collision-to-failure.util.ts @@ -0,0 +1,29 @@ +import { msg, t } from '@lingui/core/macro'; +import { type AllMetadataName } from 'twenty-shared/metadata'; + +import { type MetadataFlatEntity } from 'src/engine/metadata-modules/flat-entity/types/metadata-flat-entity.type'; +import { MetadataSideEffectExceptionCode } from 'src/engine/metadata-modules/metadata-side-effect/exceptions/metadata-side-effect-exception-code'; +import { type MetadataSideEffectFailure } from 'src/engine/metadata-modules/metadata-side-effect/types/metadata-side-effect-result.type'; +import { type SystemSideEffectUniversalIdentifierCollision } from 'src/engine/metadata-modules/metadata-side-effect/types/system-side-effect-universal-identifier-collision.type'; + +// A reserved-identifier collision is just another side-effect failure: mapping it +// to the shared MetadataSideEffectFailure shape lets the engine merge collisions +// and handler failures into a single report. +export const mapSystemSideEffectCollisionToFailure = ( + collision: SystemSideEffectUniversalIdentifierCollision, +): MetadataSideEffectFailure => ({ + status: 'fail', + type: collision.operation, + metadataName: collision.metadataName, + flatEntityMinimalInformation: { + universalIdentifier: collision.universalIdentifier, + ...(collision.name !== undefined ? { name: collision.name } : {}), + } as Partial>, + errors: [ + { + code: MetadataSideEffectExceptionCode.RESERVED_SYSTEM_UNIVERSAL_IDENTIFIER, + message: t`Universal identifier is reserved for system-managed metadata`, + userFriendlyMessage: msg`This identifier is reserved by the system`, + }, + ], +}); diff --git a/packages/twenty-server/src/engine/workspace-manager/twenty-standard-application/utils/index/compute-company-standard-flat-index-metadata.util.ts b/packages/twenty-server/src/engine/workspace-manager/twenty-standard-application/utils/index/compute-company-standard-flat-index-metadata.util.ts index 82d744b624..d8cd47ee7c 100644 --- a/packages/twenty-server/src/engine/workspace-manager/twenty-standard-application/utils/index/compute-company-standard-flat-index-metadata.util.ts +++ b/packages/twenty-server/src/engine/workspace-manager/twenty-standard-application/utils/index/compute-company-standard-flat-index-metadata.util.ts @@ -36,6 +36,7 @@ export const buildCompanyStandardFlatIndexMetadatas = ({ indexName: 'domainNameUniqueIndex', relatedFieldNames: ['domainName'], isUnique: true, + hasDeterministicUniversalIdentifier: true, }, standardObjectMetadataRelatedEntityIds, dependencyFlatEntityMaps, diff --git a/packages/twenty-server/src/engine/workspace-manager/twenty-standard-application/utils/index/compute-person-standard-flat-index-metadata.util.ts b/packages/twenty-server/src/engine/workspace-manager/twenty-standard-application/utils/index/compute-person-standard-flat-index-metadata.util.ts index e931b27e8f..3e2422e1d2 100644 --- a/packages/twenty-server/src/engine/workspace-manager/twenty-standard-application/utils/index/compute-person-standard-flat-index-metadata.util.ts +++ b/packages/twenty-server/src/engine/workspace-manager/twenty-standard-application/utils/index/compute-person-standard-flat-index-metadata.util.ts @@ -36,6 +36,7 @@ export const buildPersonStandardFlatIndexMetadatas = ({ indexName: 'emailsUniqueIndex', relatedFieldNames: ['emails'], isUnique: true, + hasDeterministicUniversalIdentifier: true, }, standardObjectMetadataRelatedEntityIds, dependencyFlatEntityMaps, diff --git a/packages/twenty-server/src/engine/workspace-manager/twenty-standard-application/utils/index/compute-workspace-member-standard-flat-index-metadata.util.ts b/packages/twenty-server/src/engine/workspace-manager/twenty-standard-application/utils/index/compute-workspace-member-standard-flat-index-metadata.util.ts index a8809eb5d1..a1a24ec05f 100644 --- a/packages/twenty-server/src/engine/workspace-manager/twenty-standard-application/utils/index/compute-workspace-member-standard-flat-index-metadata.util.ts +++ b/packages/twenty-server/src/engine/workspace-manager/twenty-standard-application/utils/index/compute-workspace-member-standard-flat-index-metadata.util.ts @@ -24,6 +24,7 @@ export const buildWorkspaceMemberStandardFlatIndexMetadatas = ({ indexName: 'userEmailUniqueIndex', relatedFieldNames: ['userEmail'], isUnique: true, + hasDeterministicUniversalIdentifier: true, }, standardObjectMetadataRelatedEntityIds, dependencyFlatEntityMaps, diff --git a/packages/twenty-server/src/engine/workspace-manager/twenty-standard-application/utils/index/create-standard-index-flat-metadata.util.ts b/packages/twenty-server/src/engine/workspace-manager/twenty-standard-application/utils/index/create-standard-index-flat-metadata.util.ts index 1380d0d8e9..eb9906b571 100644 --- a/packages/twenty-server/src/engine/workspace-manager/twenty-standard-application/utils/index/create-standard-index-flat-metadata.util.ts +++ b/packages/twenty-server/src/engine/workspace-manager/twenty-standard-application/utils/index/create-standard-index-flat-metadata.util.ts @@ -1,3 +1,4 @@ +import { getIndexUniversalIdentifier } from 'twenty-shared/application'; import { STANDARD_OBJECTS } from 'twenty-shared/metadata'; import { isDefined } from 'twenty-shared/utils'; import { v4 } from 'uuid'; @@ -9,7 +10,7 @@ import { type FlatIndexMetadata, } from 'src/engine/metadata-modules/flat-index-metadata/types/flat-index-metadata.type'; import { IndexType } from 'src/engine/metadata-modules/index-metadata/types/indexType.types'; -import { generateFlatIndexMetadataWithNameOrThrow } from 'src/engine/metadata-modules/index-metadata/utils/generate-flat-index.util'; +import { computeFlatIndexNameOrThrow } from 'src/engine/metadata-modules/index-metadata/utils/compute-flat-index-name.util'; import { TWENTY_STANDARD_APPLICATION } from 'src/engine/workspace-manager/twenty-standard-application/constants/twenty-standard-applications'; import { type AllStandardObjectFieldName } from 'src/engine/workspace-manager/twenty-standard-application/types/all-standard-object-field-name.type'; import { type AllStandardObjectIndexName } from 'src/engine/workspace-manager/twenty-standard-application/types/all-standard-object-index-name.type'; @@ -20,6 +21,7 @@ import { type UniversalFlatIndexFieldMetadata } from 'src/engine/workspace-manag export type CreateStandardIndexOptions = { indexName: AllStandardObjectIndexName; relatedFieldNames: AllStandardObjectFieldName[]; + hasDeterministicUniversalIdentifier?: boolean; } & Partial< Pick >; @@ -42,6 +44,7 @@ export const createStandardIndexFlatMetadata = < indexType = IndexType.BTREE, indexWhereClause = null, isUnique = false, + hasDeterministicUniversalIdentifier = false, }, standardObjectMetadataRelatedEntityIds, dependencyFlatEntityMaps: { flatFieldMetadataMaps, flatObjectMetadataMaps }, @@ -88,41 +91,55 @@ export const createStandardIndexFlatMetadata = < const indexId = v4(); - const unviersalFlatIndex = generateFlatIndexMetadataWithNameOrThrow({ - flatIndex: { - createdAt: now, - applicationUniversalIdentifier: - TWENTY_STANDARD_APPLICATION.universalIdentifier, - indexType, - indexWhereClause, - isCustom: false, - isUnique, - isSystemSideEffect: true, - objectMetadataUniversalIdentifier: flatObjectMetadata.universalIdentifier, - universalIdentifier: indexDefinition.universalIdentifier, - updatedAt: now, - universalFlatIndexFieldMetadatas: - flatFieldMetadatas.map( - ( - { universalIdentifier: fieldMetadataUniversalIdentifier }, - index, - ) => ({ - createdAt: now, - order: index, - subFieldName: null, - updatedAt: now, - fieldMetadataUniversalIdentifier, - indexMetadataUniversalIdentifier: - indexDefinition.universalIdentifier, - }), - ), - }, + const computedIndexName = computeFlatIndexNameOrThrow({ flatObjectMetadata, objectFlatFieldMetadatas: flatFieldMetadatas, + indexFields: flatFieldMetadatas.map((flatFieldMetadata, index) => ({ + order: index, + fieldMetadataUniversalIdentifier: flatFieldMetadata.universalIdentifier, + subFieldName: null, + })), + isUnique, + indexWhereClause, }); + const universalIdentifier = hasDeterministicUniversalIdentifier + ? getIndexUniversalIdentifier({ + applicationUniversalIdentifier: + TWENTY_STANDARD_APPLICATION.universalIdentifier, + objectUniversalIdentifier: flatObjectMetadata.universalIdentifier, + name: computedIndexName, + }) + : indexDefinition.universalIdentifier; + + const universalFlatIndex = { + createdAt: now, + applicationUniversalIdentifier: + TWENTY_STANDARD_APPLICATION.universalIdentifier, + indexType, + indexWhereClause, + isCustom: false, + isUnique, + isSystemSideEffect: true, + name: computedIndexName, + objectMetadataUniversalIdentifier: flatObjectMetadata.universalIdentifier, + universalIdentifier, + updatedAt: now, + universalFlatIndexFieldMetadatas: + flatFieldMetadatas.map( + ({ universalIdentifier: fieldMetadataUniversalIdentifier }, index) => ({ + createdAt: now, + order: index, + subFieldName: null, + updatedAt: now, + fieldMetadataUniversalIdentifier, + indexMetadataUniversalIdentifier: universalIdentifier, + }), + ), + }; + return { - ...unviersalFlatIndex, + ...universalFlatIndex, applicationId: twentyStandardApplicationId, id: v4(), flatIndexFieldMetadatas: relatedFieldIds.map( diff --git a/packages/twenty-server/src/engine/workspace-manager/workspace-migration/services/workspace-migration-flat-entity-maps.service.ts b/packages/twenty-server/src/engine/workspace-manager/workspace-migration/services/workspace-migration-flat-entity-maps.service.ts new file mode 100644 index 0000000000..855cce0ec0 --- /dev/null +++ b/packages/twenty-server/src/engine/workspace-manager/workspace-migration/services/workspace-migration-flat-entity-maps.service.ts @@ -0,0 +1,341 @@ +import { Injectable } from '@nestjs/common'; + +import { AllMetadataName } from 'twenty-shared/metadata'; +import { isDefined } from 'twenty-shared/utils'; + +import { FlatApplicationCacheMaps } from 'src/engine/core-modules/application/types/flat-application-cache-maps.type'; +import { ALL_MANY_TO_ONE_METADATA_RELATIONS } from 'src/engine/metadata-modules/flat-entity/constant/all-many-to-one-metadata-relations.constant'; +import { + FlatEntityMapsException, + FlatEntityMapsExceptionCode, +} from 'src/engine/metadata-modules/flat-entity/exceptions/flat-entity-maps.exception'; +import { AllFlatEntityMaps } from 'src/engine/metadata-modules/flat-entity/types/all-flat-entity-maps.type'; +import { AllFlatEntityOperationRecordByMetadataName } from 'src/engine/metadata-modules/flat-entity/types/all-flat-entity-operation-record-by-metadata-name.type'; +import { MetadataUniversalFlatEntity } from 'src/engine/metadata-modules/flat-entity/types/metadata-universal-flat-entity.type'; +import { getMetadataFlatEntityMapsKey } from 'src/engine/metadata-modules/flat-entity/utils/get-metadata-flat-entity-maps-key.util'; +import { getMetadataRelatedMetadataNamesForValidation } from 'src/engine/metadata-modules/flat-entity/utils/get-metadata-related-metadata-names-for-validation.util'; +import { getSubAllFlatEntityMapsByApplicationIdsOrThrow } from 'src/engine/metadata-modules/flat-entity/utils/get-sub-all-flat-entity-maps-by-application-ids-or-throw.util'; +import { MetadataSideEffectEngineService } from 'src/engine/metadata-modules/metadata-side-effect/services/metadata-side-effect-engine.service'; +import { WorkspaceCacheService } from 'src/engine/workspace-cache/services/workspace-cache.service'; +import { TWENTY_STANDARD_APPLICATION } from 'src/engine/workspace-manager/twenty-standard-application/constants/twenty-standard-applications'; +import { WORKSPACE_MIGRATION_ADDITIONAL_CACHE_DATA_MAPS_KEY } from 'src/engine/workspace-manager/workspace-migration/constant/workspace-migration-additional-cache-data-maps-key.constant'; +import { IdByUniversalIdentifierByMetadataName } from 'src/engine/workspace-manager/workspace-migration/services/utils/enrich-create-workspace-migration-action-with-ids.util'; +import { WorkspaceMigrationBuilderAdditionalCacheDataMaps } from 'src/engine/workspace-manager/workspace-migration/types/workspace-migration-builder-additional-cache-data-maps.type'; +import { FromToAllUniversalFlatEntityMaps } from 'src/engine/workspace-manager/workspace-migration/types/workspace-migration-orchestrator.type'; +import { computeUniversalFlatEntityMapsFromToThroughMutation } from 'src/engine/workspace-manager/workspace-migration/utils/compute-universal-flat-entity-maps-from-to-through-mutation.util'; +import { InferDeletionFromMissingEntities } from 'src/engine/workspace-manager/workspace-migration/workspace-migration-builder/types/infer-deletion-from-missing-entities.type'; + +export type WorkspaceMigrationRelatedFlatEntityMaps = + Partial & WorkspaceMigrationBuilderAdditionalCacheDataMaps; + +type FlatEntityMapsBundle = { + flatApplicationMaps: FlatApplicationCacheMaps; + allRelatedFlatEntityMaps: WorkspaceMigrationRelatedFlatEntityMaps; + allMetadataNameCacheToCompute: AllMetadataName[]; +}; + +type ComputeFromToAllFlatEntityMapsAndBuildOptionsArgs = { + allFlatEntityOperationRecordByMetadataName: AllFlatEntityOperationRecordByMetadataName; + applicationUniversalIdentifier: string; +} & FlatEntityMapsBundle; + +@Injectable() +export class WorkspaceMigrationFlatEntityMapsService { + constructor( + private readonly workspaceCacheService: WorkspaceCacheService, + private readonly metadataSideEffectEngineService: MetadataSideEffectEngineService, + ) {} + + async getOrRecomputeAllRelatedFlatEntityMaps({ + workspaceId, + callerMetadataNames, + }: { + workspaceId: string; + callerMetadataNames: AllMetadataName[]; + }): Promise { + const allMetadataNameCacheToCompute = [ + ...new Set( + [ + ...callerMetadataNames, + ...this.metadataSideEffectEngineService.getSideEffectRelatedMetadataNames( + callerMetadataNames, + ), + ].flatMap((metadataName) => [ + metadataName, + ...getMetadataRelatedMetadataNamesForValidation(metadataName), + ]), + ), + ]; + + const { flatApplicationMaps, ...allRelatedFlatEntityMaps } = + await this.workspaceCacheService.getOrRecompute(workspaceId, [ + ...allMetadataNameCacheToCompute.map(getMetadataFlatEntityMapsKey), + ...WORKSPACE_MIGRATION_ADDITIONAL_CACHE_DATA_MAPS_KEY, + 'flatApplicationMaps', + ]); + + return { + flatApplicationMaps, + allRelatedFlatEntityMaps, + allMetadataNameCacheToCompute, + }; + } + + computeFromToAllFlatEntityMapsAndBuildOptions({ + allFlatEntityOperationRecordByMetadataName, + applicationUniversalIdentifier, + flatApplicationMaps, + allRelatedFlatEntityMaps, + allMetadataNameCacheToCompute, + }: ComputeFromToAllFlatEntityMapsAndBuildOptionsArgs): { + fromToAllFlatEntityMaps: FromToAllUniversalFlatEntityMaps; + inferDeletionFromMissingEntities: InferDeletionFromMissingEntities; + dependencyAllFlatEntityMaps: Partial; + additionalCacheDataMaps: WorkspaceMigrationBuilderAdditionalCacheDataMaps; + idByUniversalIdentifierByMetadataName: IdByUniversalIdentifierByMetadataName; + } { + const { dependencyAllFlatEntityMaps, additionalCacheDataMaps } = + this.computeDependencyAndAdditionalCacheDataMaps({ + allFlatEntityOperationRecordByMetadataName, + applicationUniversalIdentifier, + flatApplicationMaps, + allRelatedFlatEntityMaps, + allMetadataNameCacheToCompute, + }); + + const fromToAllFlatEntityMaps: FromToAllUniversalFlatEntityMaps = {}; + const idByUniversalIdentifierByMetadataName: IdByUniversalIdentifierByMetadataName = + {}; + const inferDeletionFromMissingEntities: InferDeletionFromMissingEntities = + {}; + const allMetadataNameToCompare = Object.keys( + allFlatEntityOperationRecordByMetadataName, + ) as AllMetadataName[]; + + for (const metadataName of allMetadataNameToCompare) { + const flatEntityOperations = + allFlatEntityOperationRecordByMetadataName[metadataName]; + + if (!isDefined(flatEntityOperations)) { + throw new FlatEntityMapsException( + `Could not load flat entity maps to compare for ${metadataName}, should never occur`, + FlatEntityMapsExceptionCode.INTERNAL_SERVER_ERROR, + ); + } + + // The record matrix is the canonical form; the from/to mutation helper still + // consumes arrays, so we flatten each bucket at this boundary only. + const flatEntityToCreate = Object.values( + flatEntityOperations.flatEntityToCreate, + ); + const flatEntityToUpdate = Object.values( + flatEntityOperations.flatEntityToUpdate, + ); + const flatEntityToDelete = Object.values( + flatEntityOperations.flatEntityToDelete, + ); + + const idByUniversalIdentifier = Object.fromEntries( + flatEntityToCreate + .filter( + ( + flatEntity, + ): flatEntity is MetadataUniversalFlatEntity< + typeof metadataName + > & { id: string } => isDefined(flatEntity.id), + ) + .map((flatEntity) => [flatEntity.universalIdentifier, flatEntity.id]), + ); + + if (Object.keys(idByUniversalIdentifier).length > 0) { + idByUniversalIdentifierByMetadataName[metadataName] = + idByUniversalIdentifier; + } + + const flatEntityMapsKey = getMetadataFlatEntityMapsKey(metadataName); + const flatEntityMaps = allRelatedFlatEntityMaps[flatEntityMapsKey]; + + if (!isDefined(flatEntityMaps)) { + throw new FlatEntityMapsException( + `Flat entity maps for ${metadataName} were not pre-fetched; the up-front cache key union must cover every metadata name the side-effect expansion can add`, + FlatEntityMapsExceptionCode.INTERNAL_SERVER_ERROR, + ); + } + + // @ts-expect-error Metadata flat entity maps cache key and metadataName colliding + fromToAllFlatEntityMaps[flatEntityMapsKey] = + computeUniversalFlatEntityMapsFromToThroughMutation({ + flatEntityMaps: structuredClone(flatEntityMaps), + flatEntityToCreate, + flatEntityToDelete, + flatEntityToUpdate, + }); + + if (flatEntityToDelete.length > 0) { + inferDeletionFromMissingEntities[metadataName] = true; + } + } + + return { + fromToAllFlatEntityMaps, + inferDeletionFromMissingEntities, + dependencyAllFlatEntityMaps, + additionalCacheDataMaps, + idByUniversalIdentifierByMetadataName, + }; + } + + private computeDependencyAndAdditionalCacheDataMaps({ + allFlatEntityOperationRecordByMetadataName, + applicationUniversalIdentifier, + flatApplicationMaps, + allRelatedFlatEntityMaps, + allMetadataNameCacheToCompute, + }: ComputeFromToAllFlatEntityMapsAndBuildOptionsArgs) { + const applicationIds = this.computeAllInvolvedApplicationIds({ + allFlatEntityOperationRecordByMetadataName, + flatApplicationMaps, + applicationUniversalIdentifier, + allRelatedFlatEntityMaps, + }); + + const dependencyAllFlatEntityMaps = + getSubAllFlatEntityMapsByApplicationIdsOrThrow({ + applicationIds, + metadataNames: allMetadataNameCacheToCompute, + fromAllFlatEntityMaps: allRelatedFlatEntityMaps, + }); + + const additionalCacheDataMaps = + WORKSPACE_MIGRATION_ADDITIONAL_CACHE_DATA_MAPS_KEY.reduce( + (acc, additionalCacheDataMapsKey) => { + return { + ...acc, + [additionalCacheDataMapsKey]: + allRelatedFlatEntityMaps[additionalCacheDataMapsKey], + }; + }, + {} as WorkspaceMigrationBuilderAdditionalCacheDataMaps, + ); + + return { + dependencyAllFlatEntityMaps, + additionalCacheDataMaps, + }; + } + + private computeAllInvolvedApplicationIds({ + allFlatEntityOperationRecordByMetadataName, + flatApplicationMaps, + applicationUniversalIdentifier, + allRelatedFlatEntityMaps, + }: { + allFlatEntityOperationRecordByMetadataName: AllFlatEntityOperationRecordByMetadataName; + flatApplicationMaps: FlatApplicationCacheMaps; + applicationUniversalIdentifier: string; + allRelatedFlatEntityMaps: Partial; + }): string[] { + const applicationIds = new Set(); + + const applicationId = + flatApplicationMaps.idByUniversalIdentifier[ + applicationUniversalIdentifier + ]; + + const twentyStandardApplicationId = + flatApplicationMaps.idByUniversalIdentifier[ + TWENTY_STANDARD_APPLICATION.universalIdentifier + ]; + + if (!isDefined(twentyStandardApplicationId) || !isDefined(applicationId)) { + throw new FlatEntityMapsException( + 'Application to build and its dependent application not found', + FlatEntityMapsExceptionCode.ENTITY_NOT_FOUND, + ); + } + + applicationIds.add(applicationId); + + const isBuildingTwentyStandardApplication = + applicationUniversalIdentifier === + TWENTY_STANDARD_APPLICATION.universalIdentifier; + + if (!isBuildingTwentyStandardApplication) { + applicationIds.add(twentyStandardApplicationId); + } + + for (const metadataName of Object.keys( + allFlatEntityOperationRecordByMetadataName, + ) as AllMetadataName[]) { + const flatEntityOperations = + allFlatEntityOperationRecordByMetadataName[metadataName]; + + if (!isDefined(flatEntityOperations)) { + continue; + } + + const { flatEntityToCreate, flatEntityToUpdate, flatEntityToDelete } = + flatEntityOperations; + + const relations = ALL_MANY_TO_ONE_METADATA_RELATIONS[metadataName]; + + for (const flatEntity of [ + ...Object.values(flatEntityToCreate), + ...Object.values(flatEntityToUpdate), + ...Object.values(flatEntityToDelete), + ]) { + const entityApplicationId = + flatApplicationMaps.idByUniversalIdentifier[ + flatEntity.applicationUniversalIdentifier + ]; + + if (isDefined(entityApplicationId)) { + applicationIds.add(entityApplicationId); + } + + for (const relation of Object.values(relations) as ({ + foreignKey: string; + metadataName: AllMetadataName; + isNullable: boolean; + universalForeignKey: string; + } | null)[]) { + if (!isDefined(relation)) { + continue; + } + + const { universalForeignKey, metadataName: targetMetadataName } = + relation; + + const referencedUniversalIdentifier = + flatEntity[universalForeignKey as keyof typeof flatEntity]; + + if (!isDefined(referencedUniversalIdentifier)) { + continue; + } + + const targetFlatEntityMaps = + allRelatedFlatEntityMaps[ + getMetadataFlatEntityMapsKey( + targetMetadataName as AllMetadataName, + ) + ]; + + if (!isDefined(targetFlatEntityMaps)) { + continue; + } + + const referencedEntity = + targetFlatEntityMaps.byUniversalIdentifier[ + referencedUniversalIdentifier + ]; + + if (isDefined(referencedEntity)) { + applicationIds.add(referencedEntity.applicationId); + } + } + } + } + + return [...applicationIds]; + } +} diff --git a/packages/twenty-server/src/engine/workspace-manager/workspace-migration/services/workspace-migration-validate-build-and-run-service.ts b/packages/twenty-server/src/engine/workspace-manager/workspace-migration/services/workspace-migration-validate-build-and-run-service.ts index f34bebeb10..571a5f0b32 100644 --- a/packages/twenty-server/src/engine/workspace-manager/workspace-migration/services/workspace-migration-validate-build-and-run-service.ts +++ b/packages/twenty-server/src/engine/workspace-manager/workspace-migration/services/workspace-migration-validate-build-and-run-service.ts @@ -4,53 +4,43 @@ import { AllMetadataName, WorkspaceMigrationV2ExceptionCode, } from 'twenty-shared/metadata'; -import { isDefined } from 'twenty-shared/utils'; -import { FlatApplicationCacheMaps } from 'src/engine/core-modules/application/types/flat-application-cache-maps.type'; import { LoggerService } from 'src/engine/core-modules/logger/logger.service'; import { TwentyConfigService } from 'src/engine/core-modules/twenty-config/twenty-config.service'; -import { ALL_MANY_TO_ONE_METADATA_RELATIONS } from 'src/engine/metadata-modules/flat-entity/constant/all-many-to-one-metadata-relations.constant'; -import { createEmptyFlatEntityMaps } from 'src/engine/metadata-modules/flat-entity/constant/create-empty-flat-entity-maps.constant'; -import { - FlatEntityMapsException, - FlatEntityMapsExceptionCode, -} from 'src/engine/metadata-modules/flat-entity/exceptions/flat-entity-maps.exception'; -import { AllFlatEntityMaps } from 'src/engine/metadata-modules/flat-entity/types/all-flat-entity-maps.type'; -import { FlatEntityToCreateDeleteUpdate } from 'src/engine/metadata-modules/flat-entity/types/flat-entity-to-create-delete-update.type'; -import { MetadataFlatEntity } from 'src/engine/metadata-modules/flat-entity/types/metadata-flat-entity.type'; -import { MetadataUniversalFlatEntity } from 'src/engine/metadata-modules/flat-entity/types/metadata-universal-flat-entity.type'; -import { getMetadataFlatEntityMapsKey } from 'src/engine/metadata-modules/flat-entity/utils/get-metadata-flat-entity-maps-key.util'; +import { AllFlatEntityOperationRecordByMetadataName } from 'src/engine/metadata-modules/flat-entity/types/all-flat-entity-operation-record-by-metadata-name.type'; +import { AllFlatEntityOperationByMetadataName } from 'src/engine/metadata-modules/flat-entity/types/flat-entity-to-create-delete-update.type'; import { getFlatEntityMapsExceptionContext } from 'src/engine/metadata-modules/flat-entity/utils/get-flat-entity-maps-exception-context.util'; -import { getMetadataRelatedMetadataNamesForValidation } from 'src/engine/metadata-modules/flat-entity/utils/get-metadata-related-metadata-names-for-validation.util'; -import { getSubFlatEntityMapsByApplicationIdsOrThrow } from 'src/engine/metadata-modules/flat-entity/utils/get-sub-flat-entity-maps-by-application-ids-or-throw.util'; +import { transpileFlatEntityOperationArrayToRecord } from 'src/engine/metadata-modules/flat-entity/utils/transpile-flat-entity-operation-array-to-record.util'; +import { MetadataSideEffectEngineService } from 'src/engine/metadata-modules/metadata-side-effect/services/metadata-side-effect-engine.service'; import { MetadataEventEmitter } from 'src/engine/subscriptions/metadata-event/metadata-event-emitter'; -import { WorkspaceCacheService } from 'src/engine/workspace-cache/services/workspace-cache.service'; -import { TWENTY_STANDARD_APPLICATION } from 'src/engine/workspace-manager/twenty-standard-application/constants/twenty-standard-applications'; import { WorkspaceMigrationV2Exception } from 'src/engine/workspace-manager/workspace-migration.exception'; -import { WORKSPACE_MIGRATION_ADDITIONAL_CACHE_DATA_MAPS_KEY } from 'src/engine/workspace-manager/workspace-migration/constant/workspace-migration-additional-cache-data-maps-key.constant'; import { enrichCreateWorkspaceMigrationActionsWithIds, IdByUniversalIdentifierByMetadataName, } from 'src/engine/workspace-manager/workspace-migration/services/utils/enrich-create-workspace-migration-action-with-ids.util'; import { WorkspaceMigrationBuildOrchestratorService } from 'src/engine/workspace-manager/workspace-migration/services/workspace-migration-build-orchestrator.service'; -import { WorkspaceMigrationBuilderAdditionalCacheDataMaps } from 'src/engine/workspace-manager/workspace-migration/types/workspace-migration-builder-additional-cache-data-maps.type'; +import { WorkspaceMigrationFlatEntityMapsService } from 'src/engine/workspace-manager/workspace-migration/services/workspace-migration-flat-entity-maps.service'; import { - FromToAllUniversalFlatEntityMaps, WorkspaceMigrationOrchestratorBuildArgs, WorkspaceMigrationOrchestratorFailedResult, WorkspaceMigrationOrchestratorSuccessfulResult, } from 'src/engine/workspace-manager/workspace-migration/types/workspace-migration-orchestrator.type'; -import { computeUniversalFlatEntityMapsFromToThroughMutation } from 'src/engine/workspace-manager/workspace-migration/utils/compute-universal-flat-entity-maps-from-to-through-mutation.util'; -import { InferDeletionFromMissingEntities } from 'src/engine/workspace-manager/workspace-migration/workspace-migration-builder/types/infer-deletion-from-missing-entities.type'; import { WorkspaceMigrationRunnerService } from 'src/engine/workspace-manager/workspace-migration/workspace-migration-runner/services/workspace-migration-runner.service'; type ValidateBuildAndRunWorkspaceMigrationFromMatriceArgs = { workspaceId: string; - allFlatEntityOperationByMetadataName: { - [P in AllMetadataName]?: FlatEntityToCreateDeleteUpdate

; - }; + allFlatEntityOperationByMetadataName: AllFlatEntityOperationByMetadataName; isSystemBuild?: boolean; applicationUniversalIdentifier: string; + dryRun?: boolean; +}; + +type ValidateBuildAndRunWorkspaceMigrationFromRecordArgs = { + workspaceId: string; + allFlatEntityOperationRecordByMetadataName: AllFlatEntityOperationRecordByMetadataName; + isSystemBuild?: boolean; + applicationUniversalIdentifier: string; + dryRun?: boolean; }; @Injectable() @@ -60,8 +50,9 @@ export class WorkspaceMigrationValidateBuildAndRunService { constructor( private readonly workspaceMigrationRunnerService: WorkspaceMigrationRunnerService, private readonly workspaceMigrationBuildOrchestratorService: WorkspaceMigrationBuildOrchestratorService, - private readonly workspaceCacheService: WorkspaceCacheService, + private readonly workspaceMigrationFlatEntityMapsService: WorkspaceMigrationFlatEntityMapsService, private readonly metadataEventEmitter: MetadataEventEmitter, + private readonly metadataSideEffectEngineService: MetadataSideEffectEngineService, private readonly logger: LoggerService, twentyConfigService: TwentyConfigService, ) { @@ -70,291 +61,6 @@ export class WorkspaceMigrationValidateBuildAndRunService { this.isDebugEnabled = logLevels.includes('debug'); } - private computeAllInvolvedApplicationIds({ - allFlatEntityOperationByMetadataName, - flatApplicationMaps, - applicationUniversalIdentifier, - allRelatedFlatEntityMaps, - }: Pick< - ValidateBuildAndRunWorkspaceMigrationFromMatriceArgs, - 'allFlatEntityOperationByMetadataName' | 'applicationUniversalIdentifier' - > & { - flatApplicationMaps: FlatApplicationCacheMaps; - allRelatedFlatEntityMaps: Partial; - }): string[] { - const applicationIds = new Set(); - - const applicationId = - flatApplicationMaps.idByUniversalIdentifier[ - applicationUniversalIdentifier - ]; - - const twentyStandardApplicationId = - flatApplicationMaps.idByUniversalIdentifier[ - TWENTY_STANDARD_APPLICATION.universalIdentifier - ]; - - if (!isDefined(twentyStandardApplicationId) || !isDefined(applicationId)) { - throw new FlatEntityMapsException( - 'Application to build and its dependent application not found', - FlatEntityMapsExceptionCode.ENTITY_NOT_FOUND, - ); - } - - applicationIds.add(applicationId); - - const isBuildingTwentyStandardApplication = - applicationUniversalIdentifier === - TWENTY_STANDARD_APPLICATION.universalIdentifier; - - if (!isBuildingTwentyStandardApplication) { - applicationIds.add(twentyStandardApplicationId); - } - - for (const metadataName of Object.keys( - allFlatEntityOperationByMetadataName, - ) as AllMetadataName[]) { - const flatEntityOperations = - allFlatEntityOperationByMetadataName[metadataName]; - - if (!isDefined(flatEntityOperations)) { - continue; - } - - const { flatEntityToCreate, flatEntityToUpdate, flatEntityToDelete } = - flatEntityOperations; - - const relations = ALL_MANY_TO_ONE_METADATA_RELATIONS[metadataName]; - - for (const flatEntity of [ - ...flatEntityToCreate, - ...flatEntityToUpdate, - ...flatEntityToDelete, - ]) { - const entityApplicationId = - flatApplicationMaps.idByUniversalIdentifier[ - flatEntity.applicationUniversalIdentifier - ]; - - if (isDefined(entityApplicationId)) { - applicationIds.add(entityApplicationId); - } - - for (const relation of Object.values(relations) as ({ - foreignKey: string; - metadataName: AllMetadataName; - isNullable: boolean; - universalForeignKey: string; - } | null)[]) { - if (!isDefined(relation)) { - continue; - } - - const { universalForeignKey, metadataName: targetMetadataName } = - relation; - - const referencedUniversalIdentifier = - flatEntity[universalForeignKey as keyof typeof flatEntity]; - - if (!isDefined(referencedUniversalIdentifier)) { - continue; - } - - const targetFlatEntityMaps = - allRelatedFlatEntityMaps[ - getMetadataFlatEntityMapsKey( - targetMetadataName as AllMetadataName, - ) - ]; - - if (!isDefined(targetFlatEntityMaps)) { - continue; - } - - const referencedEntity = - targetFlatEntityMaps.byUniversalIdentifier[ - referencedUniversalIdentifier - ]; - - if (isDefined(referencedEntity)) { - applicationIds.add(referencedEntity.applicationId); - } - } - } - } - - return [...applicationIds]; - } - - private async computeAllRelatedFlatEntityMaps({ - allFlatEntityOperationByMetadataName, - workspaceId, - applicationUniversalIdentifier, - }: ValidateBuildAndRunWorkspaceMigrationFromMatriceArgs) { - const allMetadataNameToCompare = Object.keys( - allFlatEntityOperationByMetadataName, - ) as AllMetadataName[]; - const allMetadataNameCacheToCompute = [ - ...new Set([ - ...allMetadataNameToCompare, - ...allMetadataNameToCompare.flatMap( - getMetadataRelatedMetadataNamesForValidation, - ), - ]), - ]; - const allFlatEntityMapsCacheKeysToCompute = - allMetadataNameCacheToCompute.map(getMetadataFlatEntityMapsKey); - - const { flatApplicationMaps, ...allRelatedFlatEntityMaps } = - await this.workspaceCacheService.getOrRecompute(workspaceId, [ - ...allFlatEntityMapsCacheKeysToCompute, - ...WORKSPACE_MIGRATION_ADDITIONAL_CACHE_DATA_MAPS_KEY, - 'flatApplicationMaps', - ]); - - const initialAccumulator = allMetadataNameCacheToCompute.reduce< - Partial - >( - (allFlatEntityMaps, metadataName) => ({ - ...allFlatEntityMaps, - [getMetadataFlatEntityMapsKey(metadataName)]: - createEmptyFlatEntityMaps(), - }), - {}, - ); - - const applicationIds = this.computeAllInvolvedApplicationIds({ - allFlatEntityOperationByMetadataName, - flatApplicationMaps, - applicationUniversalIdentifier, - allRelatedFlatEntityMaps, - }); - - const dependencyAllFlatEntityMaps = allMetadataNameCacheToCompute.reduce( - (allFlatEntityMaps, metadataName) => { - const metadataFlatEntityMapsKey = - getMetadataFlatEntityMapsKey(metadataName); - - return { - ...allFlatEntityMaps, - [metadataFlatEntityMapsKey]: - getSubFlatEntityMapsByApplicationIdsOrThrow< - MetadataFlatEntity - >({ - applicationIds, - flatEntityMaps: - allRelatedFlatEntityMaps[metadataFlatEntityMapsKey], - }), - }; - }, - initialAccumulator, - ); - - const additionalCacheDataMaps = - WORKSPACE_MIGRATION_ADDITIONAL_CACHE_DATA_MAPS_KEY.reduce( - (acc, additionalCacheDataMapsKey) => { - return { - ...acc, - [additionalCacheDataMapsKey]: - allRelatedFlatEntityMaps[additionalCacheDataMapsKey], - }; - }, - {} as WorkspaceMigrationBuilderAdditionalCacheDataMaps, - ); - - return { - allRelatedFlatEntityMaps, - dependencyAllFlatEntityMaps, - additionalCacheDataMaps, - }; - } - - private async computeFromToAllFlatEntityMapsAndBuildOptions({ - allFlatEntityOperationByMetadataName, - workspaceId, - applicationUniversalIdentifier, - }: ValidateBuildAndRunWorkspaceMigrationFromMatriceArgs): Promise<{ - fromToAllFlatEntityMaps: FromToAllUniversalFlatEntityMaps; - inferDeletionFromMissingEntities: InferDeletionFromMissingEntities; - dependencyAllFlatEntityMaps: Partial; - additionalCacheDataMaps: WorkspaceMigrationBuilderAdditionalCacheDataMaps; - idByUniversalIdentifierByMetadataName: IdByUniversalIdentifierByMetadataName; - }> { - const { - allRelatedFlatEntityMaps, - dependencyAllFlatEntityMaps, - additionalCacheDataMaps, - } = await this.computeAllRelatedFlatEntityMaps({ - allFlatEntityOperationByMetadataName, - workspaceId, - applicationUniversalIdentifier, - }); - - const fromToAllFlatEntityMaps: FromToAllUniversalFlatEntityMaps = {}; - const idByUniversalIdentifierByMetadataName: IdByUniversalIdentifierByMetadataName = - {}; - const inferDeletionFromMissingEntities: InferDeletionFromMissingEntities = - {}; - const allMetadataNameToCompare = Object.keys( - allFlatEntityOperationByMetadataName, - ) as AllMetadataName[]; - - for (const metadataName of allMetadataNameToCompare) { - const flatEntityOperations = - allFlatEntityOperationByMetadataName[metadataName]; - - if (!isDefined(flatEntityOperations)) { - throw new FlatEntityMapsException( - `Could not load flat entity maps to compare for ${metadataName}, should never occur`, - FlatEntityMapsExceptionCode.INTERNAL_SERVER_ERROR, - ); - } - const { flatEntityToCreate, flatEntityToDelete, flatEntityToUpdate } = - flatEntityOperations; - - const idByUniversalIdentifier = Object.fromEntries( - flatEntityToCreate - .filter( - ( - flatEntity, - ): flatEntity is MetadataUniversalFlatEntity< - typeof metadataName - > & { id: string } => isDefined(flatEntity.id), - ) - .map((flatEntity) => [flatEntity.universalIdentifier, flatEntity.id]), - ); - - if (Object.keys(idByUniversalIdentifier).length > 0) { - idByUniversalIdentifierByMetadataName[metadataName] = - idByUniversalIdentifier; - } - - const flatEntityMapsKey = getMetadataFlatEntityMapsKey(metadataName); - const flatEntityMaps = allRelatedFlatEntityMaps[flatEntityMapsKey]; - - // @ts-expect-error Metadata flat entity maps cache key and metadataName colliding - fromToAllFlatEntityMaps[flatEntityMapsKey] = - computeUniversalFlatEntityMapsFromToThroughMutation({ - flatEntityMaps: structuredClone(flatEntityMaps), - flatEntityToCreate, - flatEntityToDelete, - flatEntityToUpdate, - }); - - if (flatEntityToDelete.length > 0) { - inferDeletionFromMissingEntities[metadataName] = true; - } - } - - return { - fromToAllFlatEntityMaps, - inferDeletionFromMissingEntities, - dependencyAllFlatEntityMaps, - additionalCacheDataMaps, - idByUniversalIdentifierByMetadataName, - }; - } - public async validateBuildAndRunWorkspaceMigrationFromTo( args: WorkspaceMigrationOrchestratorBuildArgs & { idByUniversalIdentifierByMetadataName?: IdByUniversalIdentifierByMetadataName; @@ -456,25 +162,87 @@ export class WorkspaceMigrationValidateBuildAndRunService { } public async validateBuildAndRunWorkspaceMigration({ - allFlatEntityOperationByMetadataName: allFlatEntities, + allFlatEntityOperationByMetadataName, workspaceId, isSystemBuild = false, applicationUniversalIdentifier, + dryRun, }: ValidateBuildAndRunWorkspaceMigrationFromMatriceArgs): Promise< | WorkspaceMigrationOrchestratorFailedResult - | WorkspaceMigrationOrchestratorSuccessfulResult + | (WorkspaceMigrationOrchestratorSuccessfulResult & { + hasSchemaMetadataChanged: boolean; + }) > { + return await this.validateBuildAndRunWorkspaceMigrationFromRecord({ + allFlatEntityOperationRecordByMetadataName: + transpileFlatEntityOperationArrayToRecord( + allFlatEntityOperationByMetadataName, + ), + workspaceId, + isSystemBuild, + applicationUniversalIdentifier, + dryRun, + }); + } + + public async validateBuildAndRunWorkspaceMigrationFromRecord({ + allFlatEntityOperationRecordByMetadataName, + workspaceId, + isSystemBuild = false, + applicationUniversalIdentifier, + dryRun, + }: ValidateBuildAndRunWorkspaceMigrationFromRecordArgs): Promise< + | WorkspaceMigrationOrchestratorFailedResult + | (WorkspaceMigrationOrchestratorSuccessfulResult & { + hasSchemaMetadataChanged: boolean; + }) + > { + const callerMetadataNames = Object.keys( + allFlatEntityOperationRecordByMetadataName, + ) as AllMetadataName[]; + + const { + flatApplicationMaps, + allRelatedFlatEntityMaps, + allMetadataNameCacheToCompute, + } = + await this.workspaceMigrationFlatEntityMapsService.getOrRecomputeAllRelatedFlatEntityMaps( + { + workspaceId, + callerMetadataNames, + }, + ); + + const sideEffectExpansionResult = + this.metadataSideEffectEngineService.expandWithSideEffects({ + allFlatEntityOperationRecordByMetadataName, + sideEffectRelatedFlatEntityMaps: allRelatedFlatEntityMaps, + context: { + buildOptions: { isSystemBuild, applicationUniversalIdentifier }, + }, + }); + + if (sideEffectExpansionResult.status === 'fail') { + return sideEffectExpansionResult; + } + const { fromToAllFlatEntityMaps, inferDeletionFromMissingEntities, dependencyAllFlatEntityMaps, additionalCacheDataMaps, idByUniversalIdentifierByMetadataName, - } = await this.computeFromToAllFlatEntityMapsAndBuildOptions({ - allFlatEntityOperationByMetadataName: allFlatEntities, - workspaceId, - applicationUniversalIdentifier, - }); + } = + this.workspaceMigrationFlatEntityMapsService.computeFromToAllFlatEntityMapsAndBuildOptions( + { + allFlatEntityOperationRecordByMetadataName: + sideEffectExpansionResult.allFlatEntityOperationRecordByMetadataName, + applicationUniversalIdentifier, + flatApplicationMaps, + allRelatedFlatEntityMaps, + allMetadataNameCacheToCompute, + }, + ); return await this.validateBuildAndRunWorkspaceMigrationFromTo({ buildOptions: { @@ -487,6 +255,7 @@ export class WorkspaceMigrationValidateBuildAndRunService { dependencyAllFlatEntityMaps, additionalCacheDataMaps, idByUniversalIdentifierByMetadataName, + dryRun, }); } } diff --git a/packages/twenty-server/src/engine/workspace-manager/workspace-migration/workspace-migration-runner/action-handlers/field/services/update-field-action-handler.service.ts b/packages/twenty-server/src/engine/workspace-manager/workspace-migration/workspace-migration-runner/action-handlers/field/services/update-field-action-handler.service.ts index f9ff3c6657..92f1d3ee74 100644 --- a/packages/twenty-server/src/engine/workspace-manager/workspace-migration/workspace-migration-runner/action-handlers/field/services/update-field-action-handler.service.ts +++ b/packages/twenty-server/src/engine/workspace-manager/workspace-migration/workspace-migration-runner/action-handlers/field/services/update-field-action-handler.service.ts @@ -154,8 +154,8 @@ export class UpdateFieldActionHandlerService extends WorkspaceMigrationRunnerAct // isUnique is derived from IndexMetadata at cache build time and has // no underlying column on fieldMetadata. It travels in the update // payload only so per-type validators (e.g. FILES rejection) can run - // — the actual state change is handled by the side-effect index - // create/delete in handleIndexChangesDuringFieldUpdate. + // — the actual state change is handled by the metadata side-effect + // engine, which owns the backing unique index lifecycle. const { isUnique: _droppedIsUnique, ...persistedUpdate } = update; if (Object.keys(persistedUpdate).length === 0) { diff --git a/packages/twenty-server/src/engine/workspace-manager/workspace-migration/workspace-migration.module.ts b/packages/twenty-server/src/engine/workspace-manager/workspace-migration/workspace-migration.module.ts index 7ff8a4720d..feb4325fbe 100644 --- a/packages/twenty-server/src/engine/workspace-manager/workspace-migration/workspace-migration.module.ts +++ b/packages/twenty-server/src/engine/workspace-manager/workspace-migration/workspace-migration.module.ts @@ -1,7 +1,9 @@ import { Module } from '@nestjs/common'; +import { MetadataSideEffectModule } from 'src/engine/metadata-modules/metadata-side-effect/metadata-side-effect.module'; import { WorkspaceCacheModule } from 'src/engine/workspace-cache/workspace-cache.module'; import { WorkspaceMigrationBuildOrchestratorService } from 'src/engine/workspace-manager/workspace-migration/services/workspace-migration-build-orchestrator.service'; +import { WorkspaceMigrationFlatEntityMapsService } from 'src/engine/workspace-manager/workspace-migration/services/workspace-migration-flat-entity-maps.service'; import { WorkspaceMigrationValidateBuildAndRunService } from 'src/engine/workspace-manager/workspace-migration/services/workspace-migration-validate-build-and-run-service'; import { WorkspaceMigrationBuilderModule } from 'src/engine/workspace-manager/workspace-migration/workspace-migration-builder/workspace-migration-builder.module'; import { WorkspaceMigrationRunnerModule } from 'src/engine/workspace-manager/workspace-migration/workspace-migration-runner/workspace-migration-runner.module'; @@ -11,10 +13,12 @@ import { WorkspaceMigrationRunnerModule } from 'src/engine/workspace-manager/wor WorkspaceMigrationBuilderModule, WorkspaceMigrationRunnerModule, WorkspaceCacheModule, + MetadataSideEffectModule, ], providers: [ WorkspaceMigrationValidateBuildAndRunService, WorkspaceMigrationBuildOrchestratorService, + WorkspaceMigrationFlatEntityMapsService, ], exports: [ WorkspaceMigrationValidateBuildAndRunService, diff --git a/packages/twenty-server/test/integration/metadata/suites/application/__snapshots__/failing-sync-application-unique-index-side-effect-collision.integration-spec.ts.snap b/packages/twenty-server/test/integration/metadata/suites/application/__snapshots__/failing-sync-application-unique-index-side-effect-collision.integration-spec.ts.snap new file mode 100644 index 0000000000..052fee4657 --- /dev/null +++ b/packages/twenty-server/test/integration/metadata/suites/application/__snapshots__/failing-sync-application-unique-index-side-effect-collision.integration-spec.ts.snap @@ -0,0 +1,37 @@ +// Jest Snapshot v1, https://goo.gl/fbAQLP + +exports[`Sync application should fail on unique index side-effect collision should fail when a declared index reuses the reserved deterministic identifier of the engine-owned unique backing index 1`] = ` +{ + "extensions": { + "code": "METADATA_VALIDATION_FAILED", + "errors": { + "index": [ + { + "errors": [ + { + "code": "RESERVED_SYSTEM_UNIVERSAL_IDENTIFIER", + "message": "Universal identifier is reserved for system-managed metadata", + "userFriendlyMessage": "This identifier is reserved by the system", + }, + ], + "flatEntityMinimalInformation": { + "name": "IDX_UNIQUE_790b15e5a08e02973ecff530bce", + "universalIdentifier": Any, + }, + "metadataName": "index", + "status": "fail", + "type": "create", + }, + ], + }, + "message": "Validation failed for 1 index", + "summary": { + "index": 1, + "totalErrors": 1, + }, + "userFriendlyMessage": "This identifier is reserved by the system", + }, + "message": "Validation errors occurred while syncing application manifest metadata", + "name": "GraphQLError", +} +`; diff --git a/packages/twenty-server/test/integration/metadata/suites/application/failing-sync-application-unique-index-side-effect-collision.integration-spec.ts b/packages/twenty-server/test/integration/metadata/suites/application/failing-sync-application-unique-index-side-effect-collision.integration-spec.ts new file mode 100644 index 0000000000..5f310dd6e1 --- /dev/null +++ b/packages/twenty-server/test/integration/metadata/suites/application/failing-sync-application-unique-index-side-effect-collision.integration-spec.ts @@ -0,0 +1,119 @@ +import { expectOneNotInternalServerErrorSnapshot } from 'test/integration/graphql/utils/expect-one-not-internal-server-error-snapshot.util'; +import { buildBaseManifest } from 'test/integration/metadata/suites/application/utils/build-base-manifest.util'; +import { buildDefaultObjectManifest } from 'test/integration/metadata/suites/application/utils/build-default-object-manifest.util'; +import { cleanupApplicationAndAppRegistration } from 'test/integration/metadata/suites/application/utils/cleanup-application-and-app-registration.util'; +import { setupApplicationForSync } from 'test/integration/metadata/suites/application/utils/setup-application-for-sync.util'; +import { syncApplication } from 'test/integration/metadata/suites/application/utils/sync-application.util'; +import { + getIndexUniversalIdentifier, + type Manifest, +} from 'twenty-shared/application'; +import { FieldMetadataType } from 'twenty-shared/types'; +import { v4 as uuidv4 } from 'uuid'; + +import { generateDeterministicIndexName } from 'src/engine/metadata-modules/index-metadata/utils/generate-deterministic-index-name'; +import { type UniversalFlatObjectMetadata } from 'src/engine/workspace-manager/workspace-migration/universal-flat-entity/types/universal-flat-object-metadata.type'; + +const TEST_APP_ID = uuidv4(); +const TEST_ROLE_ID = uuidv4(); + +const OBJECT_NAME_SINGULAR = 'collidingIndexObject'; +const UNIQUE_FIELD_NAME = 'externalId'; + +const computeSystemUniqueIndexUniversalIdentifier = ( + objectUniversalIdentifier: string, +): string => { + const name = generateDeterministicIndexName({ + flatObjectMetadata: { + nameSingular: OBJECT_NAME_SINGULAR, + applicationUniversalIdentifier: TEST_APP_ID, + } as UniversalFlatObjectMetadata, + orderedIndexColumnNames: [UNIQUE_FIELD_NAME], + isUnique: true, + indexWhereClause: null, + }); + + return getIndexUniversalIdentifier({ + applicationUniversalIdentifier: TEST_APP_ID, + objectUniversalIdentifier, + name, + }); +}; + +// The object is otherwise fully valid (all system fields present) so the reserved-identifier +// violation is the only validation error surfaced. +const buildCollidingManifest = (): Manifest => { + const objectUniversalIdentifier = uuidv4(); + const uniqueFieldUniversalIdentifier = uuidv4(); + + const object = buildDefaultObjectManifest({ + nameSingular: OBJECT_NAME_SINGULAR, + namePlural: 'collidingIndexObjects', + labelSingular: 'Colliding Index Object', + labelPlural: 'Colliding Index Objects', + description: 'Object used to test unique index side-effect collision', + universalIdentifier: objectUniversalIdentifier, + additionalFields: [ + { + universalIdentifier: uniqueFieldUniversalIdentifier, + type: FieldMetadataType.TEXT, + name: UNIQUE_FIELD_NAME, + label: 'External ID', + description: 'Unique external identifier', + icon: 'IconId', + isUnique: true, + isNullable: true, + }, + ], + }); + + return buildBaseManifest({ + appId: TEST_APP_ID, + roleId: TEST_ROLE_ID, + overrides: { + objects: [object], + fields: [], + indexes: [ + { + universalIdentifier: computeSystemUniqueIndexUniversalIdentifier( + objectUniversalIdentifier, + ), + objectUniversalIdentifier, + isUnique: true, + fields: [ + { + universalIdentifier: uuidv4(), + fieldUniversalIdentifier: uniqueFieldUniversalIdentifier, + }, + ], + }, + ], + }, + }); +}; + +describe('Sync application should fail on unique index side-effect collision', () => { + beforeAll(async () => { + await setupApplicationForSync({ + applicationUniversalIdentifier: TEST_APP_ID, + name: 'Test Unique Index Collision App', + description: 'App for testing unique index side-effect collision', + sourcePath: 'test-unique-index-collision', + }); + }, 60000); + + afterAll(async () => { + await cleanupApplicationAndAppRegistration({ + applicationUniversalIdentifier: TEST_APP_ID, + }); + }); + + it('should fail when a declared index reuses the reserved deterministic identifier of the engine-owned unique backing index', async () => { + const { errors } = await syncApplication({ + manifest: buildCollidingManifest(), + expectToFail: true, + }); + + expectOneNotInternalServerErrorSnapshot({ errors }); + }, 60000); +}); diff --git a/packages/twenty-server/test/integration/metadata/suites/application/failing-sync-application-cross-app-view-field.integration-spec.ts b/packages/twenty-server/test/integration/metadata/suites/application/successful-sync-application-cross-app-view-field.integration-spec.ts similarity index 57% rename from packages/twenty-server/test/integration/metadata/suites/application/failing-sync-application-cross-app-view-field.integration-spec.ts rename to packages/twenty-server/test/integration/metadata/suites/application/successful-sync-application-cross-app-view-field.integration-spec.ts index 757fc86eaf..5d0ad5db54 100644 --- a/packages/twenty-server/test/integration/metadata/suites/application/failing-sync-application-cross-app-view-field.integration-spec.ts +++ b/packages/twenty-server/test/integration/metadata/suites/application/successful-sync-application-cross-app-view-field.integration-spec.ts @@ -2,9 +2,13 @@ import { buildBaseManifest } from 'test/integration/metadata/suites/application/ import { cleanupApplicationAndAppRegistration } from 'test/integration/metadata/suites/application/utils/cleanup-application-and-app-registration.util'; import { setupApplicationForSync } from 'test/integration/metadata/suites/application/utils/setup-application-for-sync.util'; import { syncApplication } from 'test/integration/metadata/suites/application/utils/sync-application.util'; +import { findManyObjectMetadataWithIndexes } from 'test/integration/metadata/suites/object-metadata/utils/find-many-object-metadata-with-indexes.util'; +import { findViewFields } from 'test/integration/metadata/suites/view-field/utils/find-view-fields.util'; +import { findViews } from 'test/integration/metadata/suites/view/utils/find-views.util'; import type { FieldManifest } from 'twenty-shared/application'; import { STANDARD_OBJECTS } from 'twenty-shared/metadata'; import { FieldMetadataType, ViewType } from 'twenty-shared/types'; +import { isDefined } from 'twenty-shared/utils'; import { v4 as uuidv4 } from 'uuid'; const APP_A_ID = uuidv4(); @@ -19,6 +23,8 @@ const APP_B_VIEW_FIELD_ID = uuidv4(); const PERSON_OBJECT_UNIVERSAL_IDENTIFIER = STANDARD_OBJECTS.person.universalIdentifier; +const APP_A_VIEW_NAME = 'App A View'; + const appBPersonField: FieldManifest = { universalIdentifier: APP_B_FIELD_ID, type: FieldMetadataType.TEXT, @@ -29,7 +35,7 @@ const appBPersonField: FieldManifest = { objectUniversalIdentifier: PERSON_OBJECT_UNIVERSAL_IDENTIFIER, }; -describe('Sync application should fail when creating a view field on a view owned by another app', () => { +describe('Sync application should succeed when extending another app view with a view field', () => { beforeAll(async () => { await setupApplicationForSync({ applicationUniversalIdentifier: APP_A_ID, @@ -41,7 +47,7 @@ describe('Sync application should fail when creating a view field on a view owne await setupApplicationForSync({ applicationUniversalIdentifier: APP_B_ID, name: 'App B', - description: 'App attempting to add a view field on App A view', + description: 'App extending App A view with a view field', sourcePath: 'test-cross-app-view-field-app-b', }); @@ -50,10 +56,19 @@ describe('Sync application should fail when creating a view field on a view owne appId: APP_A_ID, roleId: APP_A_ROLE_ID, overrides: { + // Role labels are unique workspace-wide (PG constraint), so each app + // must ship a distinct label + roles: [ + { + universalIdentifier: APP_A_ROLE_ID, + label: 'App A Role', + description: 'Role owned by App A', + }, + ], views: [ { universalIdentifier: APP_A_VIEW_ID, - name: 'App A View', + name: APP_A_VIEW_NAME, objectUniversalIdentifier: PERSON_OBJECT_UNIVERSAL_IDENTIFIER, type: ViewType.TABLE, icon: 'IconList', @@ -74,12 +89,19 @@ describe('Sync application should fail when creating a view field on a view owne }); }); - it('rejects a standalone view field from App B targeting an App A view', async () => { + it('accepts a standalone view field from App B targeting an App A view', async () => { const { errors } = await syncApplication({ manifest: buildBaseManifest({ appId: APP_B_ID, roleId: APP_B_ROLE_ID, overrides: { + roles: [ + { + universalIdentifier: APP_B_ROLE_ID, + label: 'App B Role', + description: 'Role owned by App B', + }, + ], fields: [appBPersonField], viewFields: [ { @@ -93,17 +115,49 @@ describe('Sync application should fail when creating a view field on a view owne ], }, }), - expectToFail: true, + expectToFail: false, }); - expect(errors).toBeDefined(); - expect(errors.length).toBeGreaterThan(0); + expect(isDefined(errors)).toBe(false); - const [error] = errors; + const objects = await findManyObjectMetadataWithIndexes({ + expectToFail: false, + }); - expect(error.extensions.code).toBe('METADATA_VALIDATION_FAILED'); - expect(error.extensions.summary.totalErrors).toBe(1); - expect(error.extensions.summary.viewField).toBe(1); - expect(error.extensions.message).toMatch(/viewField/); + const personObject = objects.find( + (objectMetadata) => + objectMetadata.universalIdentifier === + PERSON_OBJECT_UNIVERSAL_IDENTIFIER, + ); + + expect(personObject).toBeDefined(); + + const appBField = personObject?.fieldsList.find( + (field) => field.name === appBPersonField.name, + ); + + expect(appBField).toBeDefined(); + + const { data: viewsData } = await findViews({ + objectMetadataId: personObject?.id, + expectToFail: false, + }); + + const appAView = viewsData?.getViews.find( + (view) => view.name === APP_A_VIEW_NAME, + ); + + expect(appAView).toBeDefined(); + + const { data: viewFieldsData } = await findViewFields({ + viewId: appAView?.id ?? '', + expectToFail: false, + }); + + const appBViewField = (viewFieldsData?.getViewFields ?? []).find( + (viewField) => viewField.fieldMetadataId === appBField?.id, + ); + + expect(appBViewField).toBeDefined(); }, 60000); }); diff --git a/packages/twenty-server/test/integration/metadata/suites/application/successful-sync-application-unique-field-rename.integration-spec.ts b/packages/twenty-server/test/integration/metadata/suites/application/successful-sync-application-unique-field-rename.integration-spec.ts new file mode 100644 index 0000000000..c58ea0ae19 --- /dev/null +++ b/packages/twenty-server/test/integration/metadata/suites/application/successful-sync-application-unique-field-rename.integration-spec.ts @@ -0,0 +1,245 @@ +import { buildBaseManifest } from 'test/integration/metadata/suites/application/utils/build-base-manifest.util'; +import { buildDefaultObjectManifest } from 'test/integration/metadata/suites/application/utils/build-default-object-manifest.util'; +import { cleanupApplicationAndAppRegistration } from 'test/integration/metadata/suites/application/utils/cleanup-application-and-app-registration.util'; +import { setupApplicationForSync } from 'test/integration/metadata/suites/application/utils/setup-application-for-sync.util'; +import { syncApplication } from 'test/integration/metadata/suites/application/utils/sync-application.util'; +import { findManyObjectMetadataWithIndexes } from 'test/integration/metadata/suites/object-metadata/utils/find-many-object-metadata-with-indexes.util'; +import { + getFieldUniversalIdentifier, + type FieldManifest, +} from 'twenty-shared/application'; +import { FieldMetadataType } from 'twenty-shared/types'; +import { v4 as uuidv4 } from 'uuid'; + +import { generateDeterministicIndexName } from 'src/engine/metadata-modules/index-metadata/utils/generate-deterministic-index-name'; +import { type UniversalFlatObjectMetadata } from 'src/engine/workspace-manager/workspace-migration/universal-flat-entity/types/universal-flat-object-metadata.type'; + +const TEST_APP_ID = uuidv4(); +const TEST_ROLE_ID = uuidv4(); + +const TEST_OBJECT = buildDefaultObjectManifest({ + nameSingular: 'uniqueRenameObject', + namePlural: 'uniqueRenameObjects', + labelSingular: 'Unique Rename Object', + labelPlural: 'Unique Rename Objects', + description: 'Object used to test unique field renames', + icon: 'IconTag', +}); + +const buildUniqueFieldManifest = ({ + universalIdentifier, + name, +}: { + universalIdentifier: string; + name: string; +}): FieldManifest => ({ + universalIdentifier, + type: FieldMetadataType.TEXT, + name, + label: 'Unique Reference', + description: 'Unique reference field', + icon: 'IconId', + isUnique: true, + isNullable: true, + objectUniversalIdentifier: TEST_OBJECT.universalIdentifier, +}); + +const buildManifest = (fields: FieldManifest[]) => + buildBaseManifest({ + appId: TEST_APP_ID, + roleId: TEST_ROLE_ID, + overrides: { objects: [TEST_OBJECT], fields }, + }); + +// Mirrors the engine's derivation so the test asserts the exact backing index +// the side-effect engine is expected to own for a given field name. +const computeExpectedUniqueIndexName = (fieldName: string): string => + generateDeterministicIndexName({ + flatObjectMetadata: { + nameSingular: TEST_OBJECT.nameSingular, + applicationUniversalIdentifier: TEST_APP_ID, + } as UniversalFlatObjectMetadata, + orderedIndexColumnNames: [fieldName], + isUnique: true, + indexWhereClause: null, + }); + +const findTestObject = async () => { + const objects = await findManyObjectMetadataWithIndexes({ + expectToFail: false, + }); + + const object = objects.find( + (objectMetadata) => + objectMetadata.universalIdentifier === TEST_OBJECT.universalIdentifier, + ); + + expect(object).toBeDefined(); + + return object!; +}; + +describe('Sync application unique field rename', () => { + beforeEach(async () => { + await setupApplicationForSync({ + applicationUniversalIdentifier: TEST_APP_ID, + name: 'Unique Field Rename App', + description: 'App for testing unique field rename behaviors', + sourcePath: 'test-unique-field-rename', + }); + }, 60000); + + afterEach(async () => { + await cleanupApplicationAndAppRegistration({ + applicationUniversalIdentifier: TEST_APP_ID, + }); + }); + + it('should drop and recreate the field and its backing unique index when the rename changes the field universal identifier', async () => { + const computeNameDerivedFieldUniversalIdentifier = (name: string) => + getFieldUniversalIdentifier({ + applicationUniversalIdentifier: TEST_APP_ID, + objectUniversalIdentifier: TEST_OBJECT.universalIdentifier, + name, + }); + + await syncApplication({ + manifest: buildManifest([ + buildUniqueFieldManifest({ + universalIdentifier: + computeNameDerivedFieldUniversalIdentifier('tata'), + name: 'tata', + }), + ]), + expectToFail: false, + }); + + const objectAfterFirstSync = await findTestObject(); + const tataField = objectAfterFirstSync.fieldsList.find( + (field) => field.name === 'tata', + ); + + expect(tataField).toBeDefined(); + expect( + objectAfterFirstSync.indexMetadataList.find( + (index) => index.name === computeExpectedUniqueIndexName('tata'), + ), + ).toBeDefined(); + + await syncApplication({ + manifest: buildManifest([ + buildUniqueFieldManifest({ + universalIdentifier: + computeNameDerivedFieldUniversalIdentifier('toto'), + name: 'toto', + }), + ]), + expectToFail: false, + }); + + const objectAfterSecondSync = await findTestObject(); + + const totoField = objectAfterSecondSync.fieldsList.find( + (field) => field.name === 'toto', + ); + + expect(totoField).toBeDefined(); + expect(totoField?.universalIdentifier).toBe( + computeNameDerivedFieldUniversalIdentifier('toto'), + ); + // A different metadata id proves the field was dropped and recreated, + // not updated in place. + expect(totoField?.id).not.toBe(tataField?.id); + expect( + objectAfterSecondSync.fieldsList.find((field) => field.name === 'tata'), + ).toBeUndefined(); + + const totoUniqueIndex = objectAfterSecondSync.indexMetadataList.find( + (index) => index.name === computeExpectedUniqueIndexName('toto'), + ); + + expect(totoUniqueIndex).toBeDefined(); + expect(totoUniqueIndex?.isUnique).toBe(true); + expect( + totoUniqueIndex?.indexFieldMetadataList.some( + (indexField) => indexField.fieldMetadataId === totoField?.id, + ), + ).toBe(true); + + expect( + objectAfterSecondSync.indexMetadataList.find( + (index) => index.name === computeExpectedUniqueIndexName('tata'), + ), + ).toBeUndefined(); + }, 60000); + + it('should update the field in place and swap its backing unique index when the rename keeps a static universal identifier', async () => { + const staticFieldUniversalIdentifier = uuidv4(); + + await syncApplication({ + manifest: buildManifest([ + buildUniqueFieldManifest({ + universalIdentifier: staticFieldUniversalIdentifier, + name: 'tata', + }), + ]), + expectToFail: false, + }); + + const objectAfterFirstSync = await findTestObject(); + const tataField = objectAfterFirstSync.fieldsList.find( + (field) => field.name === 'tata', + ); + + expect(tataField).toBeDefined(); + expect(tataField?.universalIdentifier).toBe(staticFieldUniversalIdentifier); + expect( + objectAfterFirstSync.indexMetadataList.find( + (index) => index.name === computeExpectedUniqueIndexName('tata'), + ), + ).toBeDefined(); + + await syncApplication({ + manifest: buildManifest([ + buildUniqueFieldManifest({ + universalIdentifier: staticFieldUniversalIdentifier, + name: 'toto', + }), + ]), + expectToFail: false, + }); + + const objectAfterSecondSync = await findTestObject(); + + const renamedField = objectAfterSecondSync.fieldsList.find( + (field) => field.universalIdentifier === staticFieldUniversalIdentifier, + ); + + expect(renamedField).toBeDefined(); + expect(renamedField?.name).toBe('toto'); + // The same metadata id proves the field was updated in place, not dropped + // and recreated. + expect(renamedField?.id).toBe(tataField?.id); + expect( + objectAfterSecondSync.fieldsList.find((field) => field.name === 'tata'), + ).toBeUndefined(); + + const totoUniqueIndex = objectAfterSecondSync.indexMetadataList.find( + (index) => index.name === computeExpectedUniqueIndexName('toto'), + ); + + expect(totoUniqueIndex).toBeDefined(); + expect(totoUniqueIndex?.isUnique).toBe(true); + expect( + totoUniqueIndex?.indexFieldMetadataList.some( + (indexField) => indexField.fieldMetadataId === renamedField?.id, + ), + ).toBe(true); + + expect( + objectAfterSecondSync.indexMetadataList.find( + (index) => index.name === computeExpectedUniqueIndexName('tata'), + ), + ).toBeUndefined(); + }, 60000); +});