Migrate Views-xxx Index Field Object Skill to be fully universal ( all actions and metadata runner and builder ) + all metadata update actions runner (#17687)
# What this PR does Overall naming `universal` versus `flat` is not always the most updated and so on Will make a big cleaning tour after I've finished the whole migration Migrating all `view` and ( filter fields etc ) `field` `object` `index` to the universal pattern on all `services`, `builder` and `runner` levels ## Universal and flat optimistic tooling `addUniversalFlatEntityToUniversalFlatEntityAndRelatedEntityMapsThroughMutationOrThrow` and its delete counterpart maintain the consistency of `UniversalFlatEntityMaps` when an entity is created or removed. Beyond inserting/removing the entity from its own maps, they walk through `ALL_UNIVERSAL_METADATA_RELATIONS` to update the **aggregator arrays** on related parent entities — the add appends the new entity's `universalIdentifier` to the parent's aggregator (e.g. a new viewField's identifier gets appended to its parent view's `viewFieldUniversalIdentifiers`), and the delete filters it out. This keeps the maps in sync so that diff computations and relation lookups remain accurate throughout the migration building process. ## ALL_UNIVERSAL_METADATA_RELATIONS `ALL_UNIVERSAL_METADATA_RELATIONS` is the universal counterpart of `ALL_METADATA_RELATIONS`. It maps each metadata entity to its many-to-one and one-to-many relations using universal foreign keys (`*UniversalIdentifier`) instead of database IDs (`*Id`). This allows migration actions to reference related entities in a workspace-agnostic way. Relations that are workspace-specific (e.g. `workspace`, `dataSource`, `userWorkspace`) are set to `null` and skipped during resolution. ## `workspaceMigrationCreateIdEnrichment` Reserved to API metadata ( will be able to validate at app installation lvl ) - Workspace migration `create` actions now carry an optional `id` (and `fieldIdByUniversalIdentifier` for object/field actions) so that caller-provided IDs flow through the entire build-validate-run pipeline. - New `enrichCreateWorkspaceMigrationActionsWithIds` utility resolves `universalIdentifier → id` mappings after the builder runs and injects them into the migration actions before the runner persists entities. - Runner action handlers use the provided IDs instead of generating new UUIDs, enabling deterministic entity creation for synchronization workflows. ## `resolveUniversalUpdateRelationIdentifiersToIds` `resolveUniversalUpdateRelationIdentifiersToIds` converts universal identifiers (workspace-agnostic, stable keys) in a migration update payload into concrete database UUIDs, so the update can be applied to a specific workspace. It iterates over the many-to-one relations defined in `ALL_UNIVERSAL_METADATA_RELATIONS` for the given entity type, replaces each `*UniversalIdentifier` property with its corresponding `*Id` by looking up the target entity in `allFlatEntityMaps`, and throws if a non-null identifier can't be resolved. Used by all `update` action handlers in `transpileUniversalActionToFlatAction`, avoiding duplicated resolution logic across handlers. ## What this PR does not - Migrating twenty-standard declaration to universal - Migrating all the inputs transpilers to universal - Migrating all metadata to be fully universal ( we still need to de-scope the type of all of them and refactor their validator very close ) --------- Co-authored-by: Charles Bochet <charles@twenty.com>
This commit is contained in:
Paul Rastoin
+140
@@ -0,0 +1,140 @@
|
||||
import { type AllMetadataName } from 'twenty-shared/metadata';
|
||||
import { assertUnreachable, isDefined } from 'twenty-shared/utils';
|
||||
|
||||
import { type UniversalCreateFieldAction } from 'src/engine/workspace-manager/workspace-migration/workspace-migration-builder/builders/field/types/workspace-migration-field-action';
|
||||
import { type UniversalCreateObjectAction } from 'src/engine/workspace-manager/workspace-migration/workspace-migration-builder/builders/object/types/workspace-migration-object-action';
|
||||
import { type WorkspaceMigration } from 'src/engine/workspace-manager/workspace-migration/workspace-migration-builder/types/workspace-migration';
|
||||
|
||||
export type IdByUniversalIdentifierByMetadataName = {
|
||||
[P in AllMetadataName]?: Record<string, string>;
|
||||
};
|
||||
|
||||
const buildFieldIdByUniversalIdentifier = ({
|
||||
action,
|
||||
fieldMetadataIdByUniversalIdentifier,
|
||||
}: {
|
||||
action: UniversalCreateObjectAction | UniversalCreateFieldAction;
|
||||
fieldMetadataIdByUniversalIdentifier: Record<string, string>;
|
||||
}): Record<string, string> | undefined => {
|
||||
const fieldIdByUniversalIdentifier = {
|
||||
...action.fieldIdByUniversalIdentifier,
|
||||
};
|
||||
|
||||
for (const universalFlatFieldMetadata of action.universalFlatFieldMetadatas) {
|
||||
const providedFieldId =
|
||||
fieldMetadataIdByUniversalIdentifier[
|
||||
universalFlatFieldMetadata.universalIdentifier
|
||||
];
|
||||
|
||||
if (isDefined(providedFieldId)) {
|
||||
fieldIdByUniversalIdentifier[
|
||||
universalFlatFieldMetadata.universalIdentifier
|
||||
] = providedFieldId;
|
||||
}
|
||||
}
|
||||
|
||||
if (Object.keys(fieldIdByUniversalIdentifier).length === 0) {
|
||||
return undefined;
|
||||
}
|
||||
|
||||
return fieldIdByUniversalIdentifier;
|
||||
};
|
||||
|
||||
export const enrichCreateWorkspaceMigrationActionsWithIds = ({
|
||||
workspaceMigration,
|
||||
idByUniversalIdentifierByMetadataName,
|
||||
}: {
|
||||
workspaceMigration: WorkspaceMigration;
|
||||
idByUniversalIdentifierByMetadataName: IdByUniversalIdentifierByMetadataName;
|
||||
}): WorkspaceMigration => {
|
||||
const fieldMetadataIdByUniversalIdentifier =
|
||||
idByUniversalIdentifierByMetadataName.fieldMetadata;
|
||||
|
||||
const enrichedActions = workspaceMigration.actions.map((action) => {
|
||||
if (action.type !== 'create') {
|
||||
return action;
|
||||
}
|
||||
|
||||
const idByUniversalIdentifier =
|
||||
idByUniversalIdentifierByMetadataName[action.metadataName];
|
||||
|
||||
if (
|
||||
!isDefined(idByUniversalIdentifier) &&
|
||||
!isDefined(fieldMetadataIdByUniversalIdentifier)
|
||||
) {
|
||||
return action;
|
||||
}
|
||||
|
||||
switch (action.metadataName) {
|
||||
case 'objectMetadata': {
|
||||
const id = isDefined(idByUniversalIdentifier)
|
||||
? idByUniversalIdentifier[action.flatEntity.universalIdentifier]
|
||||
: undefined;
|
||||
const fieldIdByUniversalIdentifier = isDefined(
|
||||
fieldMetadataIdByUniversalIdentifier,
|
||||
)
|
||||
? buildFieldIdByUniversalIdentifier({
|
||||
action,
|
||||
fieldMetadataIdByUniversalIdentifier,
|
||||
})
|
||||
: undefined;
|
||||
|
||||
return {
|
||||
...action,
|
||||
id,
|
||||
fieldIdByUniversalIdentifier,
|
||||
};
|
||||
}
|
||||
case 'fieldMetadata': {
|
||||
if (!isDefined(fieldMetadataIdByUniversalIdentifier)) {
|
||||
return action;
|
||||
}
|
||||
|
||||
return {
|
||||
...action,
|
||||
fieldIdByUniversalIdentifier: buildFieldIdByUniversalIdentifier({
|
||||
action,
|
||||
fieldMetadataIdByUniversalIdentifier,
|
||||
}),
|
||||
};
|
||||
}
|
||||
case 'view':
|
||||
case 'viewField':
|
||||
case 'viewGroup':
|
||||
case 'rowLevelPermissionPredicate':
|
||||
case 'rowLevelPermissionPredicateGroup':
|
||||
case 'viewFilterGroup':
|
||||
case 'index':
|
||||
case 'logicFunction':
|
||||
case 'viewFilter':
|
||||
case 'role':
|
||||
case 'roleTarget':
|
||||
case 'agent':
|
||||
case 'skill':
|
||||
case 'pageLayout':
|
||||
case 'pageLayoutWidget':
|
||||
case 'pageLayoutTab':
|
||||
case 'commandMenuItem':
|
||||
case 'navigationMenuItem':
|
||||
case 'frontComponent':
|
||||
case 'webhook': {
|
||||
if (!isDefined(idByUniversalIdentifier)) {
|
||||
return action;
|
||||
}
|
||||
|
||||
return {
|
||||
...action,
|
||||
id: idByUniversalIdentifier[action.flatEntity.universalIdentifier],
|
||||
};
|
||||
}
|
||||
default: {
|
||||
assertUnreachable(action);
|
||||
}
|
||||
}
|
||||
});
|
||||
|
||||
return {
|
||||
...workspaceMigration,
|
||||
actions: enrichedActions,
|
||||
};
|
||||
};
|
||||
+8
-22
@@ -4,8 +4,7 @@ import { type AllMetadataName } from 'twenty-shared/metadata';
|
||||
import { isDefined } from 'twenty-shared/utils';
|
||||
|
||||
import { createEmptyAllFlatEntityMaps } from 'src/engine/metadata-modules/flat-entity/constant/create-empty-all-flat-entity-maps.constant';
|
||||
import { type AllFlatEntityMaps } from 'src/engine/metadata-modules/flat-entity/types/all-flat-entity-maps.type';
|
||||
import { type MetadataFlatEntityAndRelatedFlatEntityMaps } from 'src/engine/metadata-modules/flat-entity/types/metadata-related-types.type';
|
||||
import { type MetadataUniversalFlatEntityAndRelatedUniversalFlatEntityMaps } from 'src/engine/metadata-modules/flat-entity/types/metadata-related-types.type';
|
||||
import { createEmptyOrchestratorActionsReport } from 'src/engine/workspace-manager/workspace-migration/constant/empty-orchestrator-actions-report.constant';
|
||||
import { EMPTY_ORCHESTRATOR_FAILURE_REPORT } from 'src/engine/workspace-manager/workspace-migration/constant/empty-orchestrator-failure-report.constant';
|
||||
import {
|
||||
@@ -13,6 +12,7 @@ import {
|
||||
type WorkspaceMigrationOrchestratorFailedResult,
|
||||
type WorkspaceMigrationOrchestratorSuccessfulResult,
|
||||
} from 'src/engine/workspace-manager/workspace-migration/types/workspace-migration-orchestrator.type';
|
||||
import { AllUniversalFlatEntityMaps } from 'src/engine/workspace-manager/workspace-migration/universal-flat-entity/types/all-universal-flat-entity-maps.type';
|
||||
import { aggregateOrchestratorActionsReport } from 'src/engine/workspace-manager/workspace-migration/utils/aggregate-orchestrator-actions-report.util';
|
||||
import { WorkspaceMigrationAgentActionsBuilderService } from 'src/engine/workspace-manager/workspace-migration/workspace-migration-builder/builders/agent/workspace-migration-agent-actions-builder.service';
|
||||
import { WorkspaceMigrationCommandMenuItemActionsBuilderService } from 'src/engine/workspace-manager/workspace-migration/workspace-migration-builder/builders/command-menu-item/workspace-migration-command-menu-item-actions-builder.service';
|
||||
@@ -70,12 +70,12 @@ export class WorkspaceMigrationBuildOrchestratorService {
|
||||
}: Pick<
|
||||
WorkspaceMigrationOrchestratorBuildArgs,
|
||||
'fromToAllFlatEntityMaps' | 'dependencyAllFlatEntityMaps'
|
||||
>): AllFlatEntityMaps {
|
||||
>): AllUniversalFlatEntityMaps {
|
||||
const allFromToFlatEntityMapsKeys = Object.keys(
|
||||
fromToAllFlatEntityMaps,
|
||||
) as (keyof AllFlatEntityMaps)[];
|
||||
) as (keyof AllUniversalFlatEntityMaps)[];
|
||||
|
||||
return allFromToFlatEntityMapsKeys.reduce<AllFlatEntityMaps>(
|
||||
return allFromToFlatEntityMapsKeys.reduce<AllUniversalFlatEntityMaps>(
|
||||
(allFlatEntityMaps, currFlatMaps) => {
|
||||
const fromToOccurence = fromToAllFlatEntityMaps[currFlatMaps];
|
||||
|
||||
@@ -104,12 +104,12 @@ export class WorkspaceMigrationBuildOrchestratorService {
|
||||
allFlatEntityMaps,
|
||||
flatEntityMapsAndRelatedFlatEntityMaps,
|
||||
}: {
|
||||
flatEntityMapsAndRelatedFlatEntityMaps: MetadataFlatEntityAndRelatedFlatEntityMaps<T>;
|
||||
allFlatEntityMaps: AllFlatEntityMaps;
|
||||
flatEntityMapsAndRelatedFlatEntityMaps: MetadataUniversalFlatEntityAndRelatedUniversalFlatEntityMaps<T>;
|
||||
allFlatEntityMaps: AllUniversalFlatEntityMaps;
|
||||
}) {
|
||||
const flatEntityMapsKeys = Object.keys(
|
||||
flatEntityMapsAndRelatedFlatEntityMaps,
|
||||
) as (keyof MetadataFlatEntityAndRelatedFlatEntityMaps<T>)[];
|
||||
) as (keyof MetadataUniversalFlatEntityAndRelatedUniversalFlatEntityMaps<T>)[];
|
||||
|
||||
for (const flatEntityMapsKey of flatEntityMapsKeys) {
|
||||
// @ts-expect-error TODO improve
|
||||
@@ -183,20 +183,6 @@ export class WorkspaceMigrationBuildOrchestratorService {
|
||||
...flatFieldMetadataMaps?.from.byUniversalIdentifier,
|
||||
...flatFieldMetadataMaps?.to.byUniversalIdentifier,
|
||||
},
|
||||
universalIdentifierById: {
|
||||
...dependencyAllFlatEntityMaps?.flatFieldMetadataMaps
|
||||
?.universalIdentifierById,
|
||||
...flatFieldMetadataMaps?.from.universalIdentifierById,
|
||||
...flatFieldMetadataMaps?.to.universalIdentifierById,
|
||||
},
|
||||
universalIdentifiersByApplicationId: {
|
||||
...dependencyAllFlatEntityMaps?.flatFieldMetadataMaps
|
||||
?.universalIdentifiersByApplicationId,
|
||||
...flatFieldMetadataMaps?.from
|
||||
.universalIdentifiersByApplicationId,
|
||||
...flatFieldMetadataMaps?.to
|
||||
.universalIdentifiersByApplicationId,
|
||||
},
|
||||
},
|
||||
},
|
||||
///
|
||||
|
||||
+52
-16
@@ -11,11 +11,15 @@ import { ALL_METADATA_REQUIRED_METADATA_FOR_VALIDATION } from 'src/engine/metada
|
||||
import { createEmptyFlatEntityMaps } from 'src/engine/metadata-modules/flat-entity/constant/create-empty-flat-entity-maps.constant';
|
||||
import { AllFlatEntityMaps } from 'src/engine/metadata-modules/flat-entity/types/all-flat-entity-maps.type';
|
||||
import { FlatEntityToCreateDeleteUpdate } from 'src/engine/metadata-modules/flat-entity/types/flat-entity-to-create-delete-update.type';
|
||||
import { computeFlatEntityMapsFromTo } from 'src/engine/metadata-modules/flat-entity/utils/compute-flat-entity-maps-from-to.util';
|
||||
import { MetadataUniversalFlatEntity } from 'src/engine/metadata-modules/flat-entity/types/metadata-universal-flat-entity.type';
|
||||
import { getMetadataFlatEntityMapsKey } from 'src/engine/metadata-modules/flat-entity/utils/get-metadata-flat-entity-maps-key.util';
|
||||
import { WorkspaceCacheService } from 'src/engine/workspace-cache/services/workspace-cache.service';
|
||||
import { WorkspaceMigrationV2Exception } from 'src/engine/workspace-manager/workspace-migration.exception';
|
||||
import { WORKSPACE_MIGRATION_ADDITIONAL_CACHE_DATA_MAPS_KEY } from 'src/engine/workspace-manager/workspace-migration/constant/workspace-migration-additional-cache-data-maps-key.constant';
|
||||
import {
|
||||
enrichCreateWorkspaceMigrationActionsWithIds,
|
||||
IdByUniversalIdentifierByMetadataName,
|
||||
} from 'src/engine/workspace-manager/workspace-migration/services/utils/enrich-create-workspace-migration-action-with-ids.util';
|
||||
import { WorkspaceMigrationBuildOrchestratorService } from 'src/engine/workspace-manager/workspace-migration/services/workspace-migration-build-orchestrator.service';
|
||||
import { WorkspaceMigrationBuilderAdditionalCacheDataMaps } from 'src/engine/workspace-manager/workspace-migration/types/workspace-migration-builder-additional-cache-data-maps.type';
|
||||
import {
|
||||
@@ -23,6 +27,7 @@ import {
|
||||
WorkspaceMigrationOrchestratorBuildArgs,
|
||||
WorkspaceMigrationOrchestratorFailedResult,
|
||||
} from 'src/engine/workspace-manager/workspace-migration/types/workspace-migration-orchestrator.type';
|
||||
import { computeUniversalFlatEntityMapsFromTo } from 'src/engine/workspace-manager/workspace-migration/utils/compute-universal-flat-entity-maps-from-to.util';
|
||||
import { InferDeletionFromMissingEntities } from 'src/engine/workspace-manager/workspace-migration/workspace-migration-builder/types/infer-deletion-from-missing-entities.type';
|
||||
import { WorkspaceMigrationRunnerService } from 'src/engine/workspace-manager/workspace-migration/workspace-migration-runner/services/workspace-migration-runner.service';
|
||||
|
||||
@@ -132,6 +137,7 @@ export class WorkspaceMigrationValidateBuildAndRunService {
|
||||
inferDeletionFromMissingEntities: InferDeletionFromMissingEntities;
|
||||
dependencyAllFlatEntityMaps: Partial<AllFlatEntityMaps>;
|
||||
additionalCacheDataMaps: WorkspaceMigrationBuilderAdditionalCacheDataMaps;
|
||||
idByUniversalIdentifierByMetadataName: IdByUniversalIdentifierByMetadataName;
|
||||
}> {
|
||||
const {
|
||||
allRelatedFlatEntityMaps,
|
||||
@@ -146,6 +152,8 @@ export class WorkspaceMigrationValidateBuildAndRunService {
|
||||
const fromToAllFlatEntityMaps: FromToAllFlatEntityMaps = {};
|
||||
const inferDeletionFromMissingEntities: InferDeletionFromMissingEntities =
|
||||
{};
|
||||
const idByUniversalIdentifierByMetadataName: IdByUniversalIdentifierByMetadataName =
|
||||
{};
|
||||
const allMetadataNameToCompare = Object.keys(
|
||||
allFlatEntityOperationByMetadataName,
|
||||
) as AllMetadataName[];
|
||||
@@ -159,16 +167,35 @@ export class WorkspaceMigrationValidateBuildAndRunService {
|
||||
}
|
||||
const { flatEntityToCreate, flatEntityToDelete, flatEntityToUpdate } =
|
||||
flatEntityOperations;
|
||||
|
||||
const idByUniversalIdentifier = Object.fromEntries(
|
||||
flatEntityToCreate
|
||||
.filter(
|
||||
(
|
||||
flatEntity,
|
||||
): flatEntity is MetadataUniversalFlatEntity<
|
||||
typeof metadataName
|
||||
> & { id: string } => isDefined(flatEntity.id),
|
||||
)
|
||||
.map((flatEntity) => [flatEntity.universalIdentifier, flatEntity.id]),
|
||||
);
|
||||
|
||||
if (Object.keys(idByUniversalIdentifier).length > 0) {
|
||||
idByUniversalIdentifierByMetadataName[metadataName] =
|
||||
idByUniversalIdentifier;
|
||||
}
|
||||
|
||||
const flatEntityMapsKey = getMetadataFlatEntityMapsKey(metadataName);
|
||||
const flatEntityMaps = allRelatedFlatEntityMaps[flatEntityMapsKey];
|
||||
|
||||
// @ts-expect-error Metadata flat entity maps cache key and metadataName colliding
|
||||
fromToAllFlatEntityMaps[flatEntityMapsKey] = computeFlatEntityMapsFromTo({
|
||||
flatEntityMaps,
|
||||
flatEntityToCreate,
|
||||
flatEntityToDelete,
|
||||
flatEntityToUpdate,
|
||||
});
|
||||
fromToAllFlatEntityMaps[flatEntityMapsKey] =
|
||||
computeUniversalFlatEntityMapsFromTo({
|
||||
flatEntityMaps,
|
||||
flatEntityToCreate,
|
||||
flatEntityToDelete,
|
||||
flatEntityToUpdate,
|
||||
});
|
||||
|
||||
if (flatEntityToDelete.length > 0) {
|
||||
inferDeletionFromMissingEntities[metadataName] = true;
|
||||
@@ -180,15 +207,20 @@ export class WorkspaceMigrationValidateBuildAndRunService {
|
||||
inferDeletionFromMissingEntities,
|
||||
dependencyAllFlatEntityMaps,
|
||||
additionalCacheDataMaps,
|
||||
idByUniversalIdentifierByMetadataName,
|
||||
};
|
||||
}
|
||||
|
||||
public async validateBuildAndRunWorkspaceMigrationFromTo(
|
||||
args: WorkspaceMigrationOrchestratorBuildArgs,
|
||||
args: WorkspaceMigrationOrchestratorBuildArgs & {
|
||||
idByUniversalIdentifierByMetadataName?: IdByUniversalIdentifierByMetadataName;
|
||||
},
|
||||
) {
|
||||
const { idByUniversalIdentifierByMetadataName, ...buildArgs } = args;
|
||||
|
||||
const validateAndBuildResult =
|
||||
await this.workspaceMigrationBuildOrchestratorService
|
||||
.buildWorkspaceMigration(args)
|
||||
.buildWorkspaceMigration(buildArgs)
|
||||
.catch((error) => {
|
||||
this.logger.error(error);
|
||||
throw new WorkspaceMigrationV2Exception(
|
||||
@@ -205,16 +237,18 @@ export class WorkspaceMigrationValidateBuildAndRunService {
|
||||
return validateAndBuildResult;
|
||||
}
|
||||
|
||||
// Note: This should be removed once we've refactored the runner optimistic rendering
|
||||
// As with the current implementation passing an empty workspace migration might result
|
||||
// in dependency flat entity maps invalidation
|
||||
if (validateAndBuildResult.workspaceMigration.actions.length === 0) {
|
||||
return;
|
||||
return undefined;
|
||||
}
|
||||
|
||||
await this.workspaceMigrationRunnerService.run(
|
||||
validateAndBuildResult.workspaceMigration,
|
||||
);
|
||||
const workspaceMigration = isDefined(idByUniversalIdentifierByMetadataName)
|
||||
? enrichCreateWorkspaceMigrationActionsWithIds({
|
||||
idByUniversalIdentifierByMetadataName,
|
||||
workspaceMigration: validateAndBuildResult.workspaceMigration,
|
||||
})
|
||||
: validateAndBuildResult.workspaceMigration;
|
||||
|
||||
await this.workspaceMigrationRunnerService.run(workspaceMigration);
|
||||
}
|
||||
|
||||
public async validateBuildAndRunWorkspaceMigration({
|
||||
@@ -230,6 +264,7 @@ export class WorkspaceMigrationValidateBuildAndRunService {
|
||||
inferDeletionFromMissingEntities,
|
||||
dependencyAllFlatEntityMaps,
|
||||
additionalCacheDataMaps,
|
||||
idByUniversalIdentifierByMetadataName,
|
||||
} = await this.computeFromToAllFlatEntityMapsAndBuildOptions({
|
||||
allFlatEntityOperationByMetadataName: allFlatEntities,
|
||||
workspaceId,
|
||||
@@ -246,6 +281,7 @@ export class WorkspaceMigrationValidateBuildAndRunService {
|
||||
workspaceId,
|
||||
dependencyAllFlatEntityMaps,
|
||||
additionalCacheDataMaps,
|
||||
idByUniversalIdentifierByMetadataName,
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user