Migrate Views-xxx Index Field Object Skill to be fully universal ( all actions and metadata runner and builder ) + all metadata update actions runner (#17687)

# What this PR does
Overall naming `universal` versus `flat` is not always the most updated
and so on
Will make a big cleaning tour after I've finished the whole migration
Migrating all `view` and ( filter fields etc ) `field` `object` `index`
to the universal pattern on all `services`, `builder` and `runner`
levels

## Universal and flat optimistic tooling

`addUniversalFlatEntityToUniversalFlatEntityAndRelatedEntityMapsThroughMutationOrThrow`
and its delete counterpart maintain the consistency of
`UniversalFlatEntityMaps` when an entity is created or removed. Beyond
inserting/removing the entity from its own maps, they walk through
`ALL_UNIVERSAL_METADATA_RELATIONS` to update the **aggregator arrays**
on related parent entities — the add appends the new entity's
`universalIdentifier` to the parent's aggregator (e.g. a new viewField's
identifier gets appended to its parent view's
`viewFieldUniversalIdentifiers`), and the delete filters it out. This
keeps the maps in sync so that diff computations and relation lookups
remain accurate throughout the migration building process.

## ALL_UNIVERSAL_METADATA_RELATIONS
`ALL_UNIVERSAL_METADATA_RELATIONS` is the universal counterpart of
`ALL_METADATA_RELATIONS`. It maps each metadata entity to its
many-to-one and one-to-many relations using universal foreign keys
(`*UniversalIdentifier`) instead of database IDs (`*Id`). This allows
migration actions to reference related entities in a workspace-agnostic
way. Relations that are workspace-specific (e.g. `workspace`,
`dataSource`, `userWorkspace`) are set to `null` and skipped during
resolution.

## `workspaceMigrationCreateIdEnrichment`
Reserved to API metadata ( will be able to validate at app installation
lvl )
- Workspace migration `create` actions now carry an optional `id` (and
`fieldIdByUniversalIdentifier` for object/field actions) so that
caller-provided IDs flow through the entire build-validate-run pipeline.
- New `enrichCreateWorkspaceMigrationActionsWithIds` utility resolves
`universalIdentifier → id` mappings after the builder runs and injects
them into the migration actions before the runner persists entities.
- Runner action handlers use the provided IDs instead of generating new
UUIDs, enabling deterministic entity creation for synchronization
workflows.


## `resolveUniversalUpdateRelationIdentifiersToIds`
`resolveUniversalUpdateRelationIdentifiersToIds` converts universal
identifiers (workspace-agnostic, stable keys) in a migration update
payload into concrete database UUIDs, so the update can be applied to a
specific workspace.

It iterates over the many-to-one relations defined in
`ALL_UNIVERSAL_METADATA_RELATIONS` for the given entity type, replaces
each `*UniversalIdentifier` property with its corresponding `*Id` by
looking up the target entity in `allFlatEntityMaps`, and throws if a
non-null identifier can't be resolved.

Used by all `update` action handlers in
`transpileUniversalActionToFlatAction`, avoiding duplicated resolution
logic across handlers.

## What this PR does not
- Migrating twenty-standard declaration to universal
- Migrating all the inputs transpilers to universal
- Migrating all metadata to be fully universal ( we still need to
de-scope the type of all of them and refactor their validator very close
)

---------

Co-authored-by: Charles Bochet <charles@twenty.com>

packages/twenty-server/src/engine/workspace-manager/workspace-migration/universal-flat-entity/constants/all-jsonb-properties-with-serialized-relation-by-metadata-name.constant.ts

@@ -3,6 +3,10 @@ import { type AllMetadataName } from 'twenty-shared/metadata';
 import { type MetadataEntity } from 'src/engine/metadata-modules/flat-entity/types/metadata-entity.type';
 import { type ExtractJsonbProperties } from 'src/engine/workspace-manager/workspace-migration/universal-flat-entity/types/extract-jsonb-properties.type';
 
+/**
+ * Important notice: This file is not strictly typed and needs manual referencement
+ * of jsonb properties that may contain a serialized relation
+ */
 export const ALL_JSONB_PROPERTIES_WITH_SERIALIZED_RELATION_BY_METADATA_NAME = {
   fieldMetadata: {
     settings: 'settings',
@@ -32,6 +36,7 @@ export const ALL_JSONB_PROPERTIES_WITH_SERIALIZED_RELATION_BY_METADATA_NAME = {
   frontComponent: {},
 } as const satisfies {
   [P in AllMetadataName]: Partial<{
+    // TODO prastoin: improve strict typing to recursively serach for nested SerializedRelation
     [K in ExtractJsonbProperties<MetadataEntity<P>>]: K;
   }>;
 };

packages/twenty-server/src/engine/workspace-manager/workspace-migration/universal-flat-entity/constants/all-universal-metadata-relations.constant.ts

@@ -0,0 +1,517 @@
+import { type AllMetadataName } from 'twenty-shared/metadata';
+
+import { type ALL_METADATA_RELATIONS } from 'src/engine/metadata-modules/flat-entity/constant/all-metadata-relations.constant';
+
+export type ToUniversalForeignKey<T extends string> =
+  T extends `${infer Prefix}Id` ? `${Prefix}UniversalIdentifier` : never;
+
+type ToUniversalAggregator<T extends string> = T extends `${infer Prefix}Ids`
+  ? `${Prefix}UniversalIdentifiers`
+  : never;
+
+export type ToUniversalMetadataManyToOneRelationConfiguration<T> = T extends {
+  metadataName: infer M extends AllMetadataName;
+  foreignKey: infer FK extends string;
+  flatEntityForeignKeyAggregator: infer Agg;
+  isNullable: infer N extends boolean;
+}
+  ? {
+      metadataName: M;
+      foreignKey: FK;
+      universalForeignKey: ToUniversalForeignKey<FK>;
+      universalFlatEntityForeignKeyAggregator: Agg extends string
+        ? ToUniversalAggregator<Agg>
+        : null;
+      isNullable: N;
+    }
+  : null;
+
+type ToUniversalManyToOneRelations<T> = {
+  [K in keyof T]: ToUniversalMetadataManyToOneRelationConfiguration<T[K]>;
+};
+
+type ToUniversalOneToManyRelations<T> = T;
+
+export type UniversalMetadataRelationsProperties = {
+  [M in AllMetadataName]: {
+    manyToOne: ToUniversalManyToOneRelations<
+      (typeof ALL_METADATA_RELATIONS)[M]['manyToOne']
+    >;
+    oneToMany: ToUniversalOneToManyRelations<
+      (typeof ALL_METADATA_RELATIONS)[M]['oneToMany']
+    >;
+  };
+};
+
+export const ALL_UNIVERSAL_METADATA_RELATIONS = {
+  agent: {
+    manyToOne: {
+      workspace: null,
+      application: null,
+    },
+    oneToMany: {},
+  },
+  skill: {
+    manyToOne: {
+      workspace: null,
+      application: null,
+    },
+    oneToMany: {},
+  },
+  commandMenuItem: {
+    manyToOne: {
+      workspace: null,
+      application: null,
+      availabilityObjectMetadata: {
+        metadataName: 'objectMetadata',
+        foreignKey: 'availabilityObjectMetadataId',
+        universalFlatEntityForeignKeyAggregator: null,
+        universalForeignKey: 'availabilityObjectMetadataUniversalIdentifier',
+        isNullable: true,
+      },
+      frontComponent: {
+        metadataName: 'frontComponent',
+        foreignKey: 'frontComponentId',
+        universalFlatEntityForeignKeyAggregator: null,
+        universalForeignKey: 'frontComponentUniversalIdentifier',
+        isNullable: true,
+      },
+    },
+    oneToMany: {},
+  },
+  navigationMenuItem: {
+    manyToOne: {
+      workspace: null,
+      userWorkspace: null,
+      application: null,
+      targetObjectMetadata: {
+        metadataName: 'objectMetadata',
+        foreignKey: 'targetObjectMetadataId',
+        universalFlatEntityForeignKeyAggregator: null,
+        universalForeignKey: 'targetObjectMetadataUniversalIdentifier',
+        isNullable: true,
+      },
+      folder: {
+        metadataName: 'navigationMenuItem',
+        foreignKey: 'folderId',
+        universalFlatEntityForeignKeyAggregator: null,
+        universalForeignKey: 'folderUniversalIdentifier',
+        isNullable: true,
+      },
+      view: {
+        metadataName: 'view',
+        foreignKey: 'viewId',
+        universalFlatEntityForeignKeyAggregator: null,
+        universalForeignKey: 'viewUniversalIdentifier',
+        isNullable: true,
+      },
+    },
+    oneToMany: {},
+  },
+  fieldMetadata: {
+    manyToOne: {
+      object: {
+        metadataName: 'objectMetadata',
+        foreignKey: 'objectMetadataId',
+        universalFlatEntityForeignKeyAggregator: 'fieldUniversalIdentifiers',
+        universalForeignKey: 'objectMetadataUniversalIdentifier',
+        isNullable: false,
+      },
+      workspace: null,
+      application: null,
+      relationTargetFieldMetadata: {
+        metadataName: 'fieldMetadata',
+        foreignKey: 'relationTargetFieldMetadataId',
+        universalFlatEntityForeignKeyAggregator: null,
+        universalForeignKey: 'relationTargetFieldMetadataUniversalIdentifier',
+        isNullable: true,
+      },
+      relationTargetObjectMetadata: {
+        metadataName: 'objectMetadata',
+        foreignKey: 'relationTargetObjectMetadataId',
+        universalFlatEntityForeignKeyAggregator: null,
+        universalForeignKey: 'relationTargetObjectMetadataUniversalIdentifier',
+        isNullable: true,
+      },
+    },
+    oneToMany: {
+      fieldPermissions: null,
+      indexFieldMetadatas: null,
+      viewFields: { metadataName: 'viewField' },
+      viewFilters: { metadataName: 'viewFilter' },
+      kanbanAggregateOperationViews: { metadataName: 'view' },
+      calendarViews: { metadataName: 'view' },
+      mainGroupByFieldMetadataViews: { metadataName: 'view' },
+    },
+  },
+  objectMetadata: {
+    manyToOne: {
+      dataSource: null,
+      workspace: null,
+      application: null,
+    },
+    oneToMany: {
+      fields: { metadataName: 'fieldMetadata' },
+      indexMetadatas: { metadataName: 'index' },
+      targetRelationFields: { metadataName: 'fieldMetadata' },
+      objectPermissions: null,
+      fieldPermissions: null,
+      views: { metadataName: 'view' },
+    },
+  },
+  view: {
+    manyToOne: {
+      objectMetadata: {
+        metadataName: 'objectMetadata',
+        foreignKey: 'objectMetadataId',
+        universalFlatEntityForeignKeyAggregator: 'viewUniversalIdentifiers',
+        universalForeignKey: 'objectMetadataUniversalIdentifier',
+        isNullable: false,
+      },
+      workspace: null,
+      createdBy: null,
+      application: null,
+      calendarFieldMetadata: {
+        metadataName: 'fieldMetadata',
+        foreignKey: 'calendarFieldMetadataId',
+        universalFlatEntityForeignKeyAggregator:
+          'calendarViewUniversalIdentifiers',
+        universalForeignKey: 'calendarFieldMetadataUniversalIdentifier',
+        isNullable: true,
+      },
+      kanbanAggregateOperationFieldMetadata: {
+        metadataName: 'fieldMetadata',
+        foreignKey: 'kanbanAggregateOperationFieldMetadataId',
+        universalFlatEntityForeignKeyAggregator:
+          'kanbanAggregateOperationViewUniversalIdentifiers',
+        universalForeignKey:
+          'kanbanAggregateOperationFieldMetadataUniversalIdentifier',
+        isNullable: true,
+      },
+      mainGroupByFieldMetadata: {
+        metadataName: 'fieldMetadata',
+        foreignKey: 'mainGroupByFieldMetadataId',
+        universalFlatEntityForeignKeyAggregator:
+          'mainGroupByFieldMetadataViewUniversalIdentifiers',
+        universalForeignKey: 'mainGroupByFieldMetadataUniversalIdentifier',
+        isNullable: true,
+      },
+    },
+    oneToMany: {
+      viewFields: { metadataName: 'viewField' },
+      viewFilters: { metadataName: 'viewFilter' },
+      viewFilterGroups: { metadataName: 'viewFilterGroup' },
+      viewGroups: { metadataName: 'viewGroup' },
+      // TODO migrate viewSort to v2
+      viewSorts: null,
+    },
+  },
+  viewField: {
+    manyToOne: {
+      fieldMetadata: {
+        metadataName: 'fieldMetadata',
+        foreignKey: 'fieldMetadataId',
+        universalFlatEntityForeignKeyAggregator:
+          'viewFieldUniversalIdentifiers',
+        universalForeignKey: 'fieldMetadataUniversalIdentifier',
+        isNullable: false,
+      },
+      view: {
+        metadataName: 'view',
+        foreignKey: 'viewId',
+        universalFlatEntityForeignKeyAggregator:
+          'viewFieldUniversalIdentifiers',
+        universalForeignKey: 'viewUniversalIdentifier',
+        isNullable: false,
+      },
+      workspace: null,
+      application: null,
+    },
+    oneToMany: {},
+  },
+  viewFilter: {
+    manyToOne: {
+      fieldMetadata: {
+        metadataName: 'fieldMetadata',
+        foreignKey: 'fieldMetadataId',
+        universalFlatEntityForeignKeyAggregator:
+          'viewFilterUniversalIdentifiers',
+        universalForeignKey: 'fieldMetadataUniversalIdentifier',
+        isNullable: false,
+      },
+      view: {
+        metadataName: 'view',
+        foreignKey: 'viewId',
+        universalFlatEntityForeignKeyAggregator:
+          'viewFilterUniversalIdentifiers',
+        universalForeignKey: 'viewUniversalIdentifier',
+        isNullable: false,
+      },
+      viewFilterGroup: {
+        metadataName: 'viewFilterGroup',
+        foreignKey: 'viewFilterGroupId',
+        universalFlatEntityForeignKeyAggregator:
+          'viewFilterUniversalIdentifiers',
+        universalForeignKey: 'viewFilterGroupUniversalIdentifier',
+        isNullable: true,
+      },
+      workspace: null,
+      application: null,
+    },
+    oneToMany: {},
+  },
+  viewGroup: {
+    manyToOne: {
+      view: {
+        metadataName: 'view',
+        foreignKey: 'viewId',
+        universalFlatEntityForeignKeyAggregator:
+          'viewGroupUniversalIdentifiers',
+        universalForeignKey: 'viewUniversalIdentifier',
+        isNullable: false,
+      },
+      workspace: null,
+      application: null,
+    },
+    oneToMany: {},
+  },
+  index: {
+    manyToOne: {
+      objectMetadata: {
+        metadataName: 'objectMetadata',
+        foreignKey: 'objectMetadataId',
+        universalFlatEntityForeignKeyAggregator:
+          'indexMetadataUniversalIdentifiers',
+        universalForeignKey: 'objectMetadataUniversalIdentifier',
+        isNullable: false,
+      },
+      workspace: null,
+      application: null,
+    },
+    oneToMany: {
+      indexFieldMetadatas: null,
+    },
+  },
+  logicFunction: {
+    manyToOne: {
+      workspace: null,
+      application: null,
+    },
+    oneToMany: {},
+  },
+  role: {
+    manyToOne: {
+      workspace: null,
+      application: null,
+    },
+    oneToMany: {
+      roleTargets: { metadataName: 'roleTarget' },
+      objectPermissions: null,
+      permissionFlags: null,
+      fieldPermissions: null,
+      rowLevelPermissionPredicates: {
+        metadataName: 'rowLevelPermissionPredicate',
+      },
+      rowLevelPermissionPredicateGroups: {
+        metadataName: 'rowLevelPermissionPredicateGroup',
+      },
+    },
+  },
+  roleTarget: {
+    manyToOne: {
+      role: {
+        metadataName: 'role',
+        foreignKey: 'roleId',
+        universalFlatEntityForeignKeyAggregator:
+          'roleTargetUniversalIdentifiers',
+        universalForeignKey: 'roleUniversalIdentifier',
+        isNullable: false,
+      },
+      apiKey: null,
+      workspace: null,
+      application: null,
+    },
+    oneToMany: {},
+  },
+  pageLayout: {
+    manyToOne: {
+      workspace: null,
+      objectMetadata: {
+        metadataName: 'objectMetadata',
+        foreignKey: 'objectMetadataId',
+        universalFlatEntityForeignKeyAggregator: null,
+        universalForeignKey: 'objectMetadataUniversalIdentifier',
+        isNullable: true,
+      },
+      application: null,
+      defaultTabToFocusOnMobileAndSidePanel: {
+        metadataName: 'pageLayoutTab',
+        foreignKey: 'defaultTabToFocusOnMobileAndSidePanelId',
+        universalFlatEntityForeignKeyAggregator: null,
+        universalForeignKey:
+          'defaultTabToFocusOnMobileAndSidePanelUniversalIdentifier',
+        isNullable: true,
+      },
+    },
+    oneToMany: {
+      tabs: { metadataName: 'pageLayoutTab' },
+    },
+  },
+  pageLayoutTab: {
+    manyToOne: {
+      workspace: null,
+      pageLayout: {
+        metadataName: 'pageLayout',
+        foreignKey: 'pageLayoutId',
+        universalFlatEntityForeignKeyAggregator: 'tabUniversalIdentifiers',
+        universalForeignKey: 'pageLayoutUniversalIdentifier',
+        isNullable: false,
+      },
+      application: null,
+    },
+    oneToMany: {
+      widgets: { metadataName: 'pageLayoutWidget' },
+    },
+  },
+  pageLayoutWidget: {
+    manyToOne: {
+      workspace: null,
+      pageLayoutTab: {
+        metadataName: 'pageLayoutTab',
+        foreignKey: 'pageLayoutTabId',
+        universalFlatEntityForeignKeyAggregator: 'widgetUniversalIdentifiers',
+        universalForeignKey: 'pageLayoutTabUniversalIdentifier',
+        isNullable: false,
+      },
+      objectMetadata: {
+        metadataName: 'objectMetadata',
+        foreignKey: 'objectMetadataId',
+        universalFlatEntityForeignKeyAggregator: null,
+        universalForeignKey: 'objectMetadataUniversalIdentifier',
+        isNullable: true,
+      },
+      application: null,
+    },
+    oneToMany: {},
+  },
+  rowLevelPermissionPredicate: {
+    manyToOne: {
+      workspace: null,
+      role: {
+        metadataName: 'role',
+        foreignKey: 'roleId',
+        universalFlatEntityForeignKeyAggregator: null,
+        universalForeignKey: 'roleUniversalIdentifier',
+        isNullable: false,
+      },
+      fieldMetadata: {
+        metadataName: 'fieldMetadata',
+        foreignKey: 'fieldMetadataId',
+        universalFlatEntityForeignKeyAggregator: null,
+        universalForeignKey: 'fieldMetadataUniversalIdentifier',
+        isNullable: false,
+      },
+      workspaceMemberFieldMetadata: {
+        metadataName: 'fieldMetadata',
+        foreignKey: 'workspaceMemberFieldMetadataId',
+        universalFlatEntityForeignKeyAggregator: null,
+        universalForeignKey: 'workspaceMemberFieldMetadataUniversalIdentifier',
+        isNullable: true,
+      },
+      objectMetadata: {
+        metadataName: 'objectMetadata',
+        foreignKey: 'objectMetadataId',
+        universalFlatEntityForeignKeyAggregator: null,
+        universalForeignKey: 'objectMetadataUniversalIdentifier',
+        isNullable: false,
+      },
+      rowLevelPermissionPredicateGroup: {
+        metadataName: 'rowLevelPermissionPredicateGroup',
+        foreignKey: 'rowLevelPermissionPredicateGroupId',
+        universalFlatEntityForeignKeyAggregator: null,
+        universalForeignKey:
+          'rowLevelPermissionPredicateGroupUniversalIdentifier',
+        isNullable: true,
+      },
+      application: null,
+    },
+    oneToMany: {},
+  },
+  rowLevelPermissionPredicateGroup: {
+    manyToOne: {
+      objectMetadata: {
+        metadataName: 'objectMetadata',
+        foreignKey: 'objectMetadataId',
+        universalFlatEntityForeignKeyAggregator: null,
+        universalForeignKey: 'objectMetadataUniversalIdentifier',
+        isNullable: false,
+      },
+      role: {
+        metadataName: 'role',
+        foreignKey: 'roleId',
+        universalFlatEntityForeignKeyAggregator: null,
+        universalForeignKey: 'roleUniversalIdentifier',
+        isNullable: false,
+      },
+      parentRowLevelPermissionPredicateGroup: {
+        metadataName: 'rowLevelPermissionPredicateGroup',
+        foreignKey: 'parentRowLevelPermissionPredicateGroupId',
+        universalFlatEntityForeignKeyAggregator:
+          'childRowLevelPermissionPredicateGroupUniversalIdentifiers',
+        universalForeignKey:
+          'parentRowLevelPermissionPredicateGroupUniversalIdentifier',
+        isNullable: true,
+      },
+      workspace: null,
+      application: null,
+    },
+    oneToMany: {
+      childRowLevelPermissionPredicateGroups: {
+        metadataName: 'rowLevelPermissionPredicateGroup',
+      },
+      rowLevelPermissionPredicates: {
+        metadataName: 'rowLevelPermissionPredicate',
+      },
+    },
+  },
+  viewFilterGroup: {
+    manyToOne: {
+      application: null,
+      parentViewFilterGroup: {
+        metadataName: 'viewFilterGroup',
+        foreignKey: 'parentViewFilterGroupId',
+        universalFlatEntityForeignKeyAggregator:
+          'childViewFilterGroupUniversalIdentifiers',
+        universalForeignKey: 'parentViewFilterGroupUniversalIdentifier',
+        isNullable: true,
+      },
+      view: {
+        metadataName: 'view',
+        foreignKey: 'viewId',
+        universalFlatEntityForeignKeyAggregator:
+          'viewFilterGroupUniversalIdentifiers',
+        universalForeignKey: 'viewUniversalIdentifier',
+        isNullable: false,
+      },
+      workspace: null,
+    },
+    oneToMany: {
+      childViewFilterGroups: { metadataName: 'viewFilterGroup' },
+      viewFilters: { metadataName: 'viewFilter' },
+    },
+  },
+  frontComponent: {
+    manyToOne: {
+      workspace: null,
+      application: null,
+    },
+    oneToMany: {},
+  },
+  webhook: {
+    manyToOne: {
+      workspace: null,
+      application: null,
+    },
+    oneToMany: {},
+  },
+} as const satisfies UniversalMetadataRelationsProperties;

packages/twenty-server/src/engine/workspace-manager/workspace-migration/universal-flat-entity/types/__tests__/metadata-universal-flat-entity-maps.type-test.ts

@@ -0,0 +1,21 @@
+import { type Equal, type Expect } from 'twenty-shared/testing';
+
+import { type MetadataUniversalFlatEntityMaps } from 'src/engine/workspace-manager/workspace-migration/universal-flat-entity/types/metadata-universal-flat-entity-maps.type';
+import { type UniversalFlatFieldMetadata } from 'src/engine/workspace-manager/workspace-migration/universal-flat-entity/types/universal-flat-field-metadata.type';
+import { type UniversalFlatViewField } from 'src/engine/workspace-manager/workspace-migration/universal-flat-entity/types/universal-flat-view-field.type';
+
+// eslint-disable-next-line unused-imports/no-unused-vars
+type Assertions = [
+  Expect<
+    Equal<
+      MetadataUniversalFlatEntityMaps<'fieldMetadata'>['byUniversalIdentifier'][string],
+      UniversalFlatFieldMetadata | undefined
+    >
+  >,
+  Expect<
+    Equal<
+      MetadataUniversalFlatEntityMaps<'viewField'>['byUniversalIdentifier'][string],
+      UniversalFlatViewField | undefined
+    >
+  >,
+];

packages/twenty-server/src/engine/workspace-manager/workspace-migration/universal-flat-entity/types/__tests__/universal-flat-entity-update.test-type.ts

@@ -0,0 +1,39 @@
+import { type Equal, type Expect } from 'twenty-shared/testing';
+
+import { type UniversalFlatEntityUpdate } from 'src/engine/workspace-manager/workspace-migration/universal-flat-entity/types/universal-flat-entity-update.type';
+
+// eslint-disable-next-line unused-imports/no-unused-vars
+type Assertions = [
+  Expect<
+    Equal<
+      keyof UniversalFlatEntityUpdate<'fieldMetadata'>,
+      | 'name'
+      | 'label'
+      | 'icon'
+      | 'description'
+      | 'isActive'
+      | 'defaultValue'
+      | 'standardOverrides'
+      | 'options'
+      | 'isUnique'
+      | 'isLabelSyncedWithName'
+      | 'universalSettings'
+    >
+  >,
+
+  Expect<
+    Equal<
+      keyof UniversalFlatEntityUpdate<'objectMetadata'>,
+      | 'icon'
+      | 'description'
+      | 'isActive'
+      | 'standardOverrides'
+      | 'isLabelSyncedWithName'
+      | 'nameSingular'
+      | 'namePlural'
+      | 'labelSingular'
+      | 'labelPlural'
+      | 'labelIdentifierFieldMetadataUniversalIdentifier'
+    >
+  >,
+];

packages/twenty-server/src/engine/workspace-manager/workspace-migration/universal-flat-entity/types/all-universal-flat-entity-maps.type.ts

@@ -0,0 +1,8 @@
+import { type AllMetadataName } from 'twenty-shared/metadata';
+
+import { type MetadataToFlatEntityMapsKey } from 'src/engine/metadata-modules/flat-entity/types/metadata-to-flat-entity-maps-key';
+import { type MetadataUniversalFlatEntityMaps } from 'src/engine/workspace-manager/workspace-migration/universal-flat-entity/types/metadata-universal-flat-entity-maps.type';
+
+export type AllUniversalFlatEntityMaps = {
+  [P in AllMetadataName as MetadataToFlatEntityMapsKey<P>]: MetadataUniversalFlatEntityMaps<P>;
+};

packages/twenty-server/src/engine/workspace-manager/workspace-migration/universal-flat-entity/types/metadata-universal-flat-entity-maps.type.ts

@@ -0,0 +1,9 @@
+import { type AllMetadataName } from 'twenty-shared/metadata';
+
+import { type AllFlatEntityTypesByMetadataName } from 'src/engine/metadata-modules/flat-entity/types/all-flat-entity-types-by-metadata-name';
+import { type UniversalFlatEntityMaps } from 'src/engine/workspace-manager/workspace-migration/universal-flat-entity/types/universal-flat-entity-maps.type';
+
+export type MetadataUniversalFlatEntityMaps<T extends AllMetadataName> =
+  UniversalFlatEntityMaps<
+    AllFlatEntityTypesByMetadataName[T]['universalFlatEntity']
+  >;

packages/twenty-server/src/engine/workspace-manager/workspace-migration/universal-flat-entity/types/metadata-universal-flat-entity-properties-to-compare.type.ts

@@ -0,0 +1,63 @@
+import { type AllMetadataName } from 'twenty-shared/metadata';
+import { type Equal, type Expect } from 'twenty-shared/testing';
+
+import { type ALL_ENTITY_PROPERTIES_CONFIGURATION_BY_METADATA_NAME } from 'src/engine/metadata-modules/flat-entity/constant/all-entity-properties-configuration-by-metadata-name.constant';
+import { type MetadataUniversalFlatEntity } from 'src/engine/metadata-modules/flat-entity/types/metadata-universal-flat-entity.type';
+
+type ExtractPropertyToCompare<
+  MetadataConfig,
+  P extends keyof MetadataConfig,
+> = MetadataConfig[P] extends { universalProperty: string }
+  ? MetadataConfig[P]['universalProperty']
+  : P;
+
+export type MetadataUniversalFlatEntityPropertiesToCompare<
+  T extends AllMetadataName,
+  MetadataConfig = (typeof ALL_ENTITY_PROPERTIES_CONFIGURATION_BY_METADATA_NAME)[T],
+> = {
+  [P in keyof MetadataConfig]: ExtractPropertyToCompare<MetadataConfig, P>;
+}[keyof MetadataConfig] &
+  keyof MetadataUniversalFlatEntity<T>;
+
+// eslint-disable-next-line unused-imports/no-unused-vars
+type Assertions = [
+  Expect<
+    Equal<
+      MetadataUniversalFlatEntityPropertiesToCompare<'fieldMetadata'>,
+      | 'name'
+      | 'label'
+      | 'icon'
+      | 'description'
+      | 'isActive'
+      | 'defaultValue'
+      | 'standardOverrides'
+      | 'options'
+      | 'isUnique'
+      | 'isLabelSyncedWithName'
+      | 'universalSettings'
+    >
+  >,
+
+  Expect<
+    Equal<
+      MetadataUniversalFlatEntityPropertiesToCompare<'view'>,
+      | 'name'
+      | 'icon'
+      | 'deletedAt'
+      | 'type'
+      | 'position'
+      | 'key'
+      | 'isCompact'
+      | 'openRecordIn'
+      | 'kanbanAggregateOperation'
+      | 'kanbanAggregateOperationFieldMetadataUniversalIdentifier'
+      | 'calendarLayout'
+      | 'calendarFieldMetadataUniversalIdentifier'
+      | 'mainGroupByFieldMetadataUniversalIdentifier'
+      | 'shouldHideEmptyGroups'
+      | 'anyFieldFilterValue'
+      | 'visibility'
+      | 'createdByUserWorkspaceId'
+    >
+  >,
+];

packages/twenty-server/src/engine/workspace-manager/workspace-migration/universal-flat-entity/types/metadata-universal-flat-entity-properties-to-stringify.type.ts

@@ -0,0 +1,13 @@
+import { type AllMetadataName } from 'twenty-shared/metadata';
+
+import { type ALL_ENTITY_PROPERTIES_CONFIGURATION_BY_METADATA_NAME } from 'src/engine/metadata-modules/flat-entity/constant/all-entity-properties-configuration-by-metadata-name.constant';
+
+type ExtractKeysWithToStringify<T> = {
+  [K in keyof T]: T[K] extends { toStringify: true } ? K : never;
+}[keyof T];
+
+export type MetadataUniversalFlatEntityPropertiesToStringify<
+  T extends AllMetadataName,
+> = ExtractKeysWithToStringify<
+  (typeof ALL_ENTITY_PROPERTIES_CONFIGURATION_BY_METADATA_NAME)[T]
+>;

packages/twenty-server/src/engine/workspace-manager/workspace-migration/universal-flat-entity/types/universal-flat-entity-from.type.ts

@@ -1,9 +1,9 @@
 import { type AllMetadataName } from 'twenty-shared/metadata';
 
+import { type ALL_METADATA_RELATIONS } from 'src/engine/metadata-modules/flat-entity/constant/all-metadata-relations.constant';
 import { type AddSuffixToEntityManyToOneProperties } from 'src/engine/metadata-modules/flat-entity/types/add-suffix-to-entity-many-to-one-properties.type';
 import { type AddSuffixToEntityOneToManyProperties } from 'src/engine/metadata-modules/flat-entity/types/add-suffix-to-entity-one-to-many-properties.type';
 import { type CastRecordTypeOrmDatePropertiesToString } from 'src/engine/metadata-modules/flat-entity/types/cast-record-typeorm-date-properties-to-string.type';
-import { type ExtractEntityRelatedEntityProperties } from 'src/engine/metadata-modules/flat-entity/types/extract-entity-related-entity-properties.type';
 import { type FromMetadataEntityToMetadataName } from 'src/engine/metadata-modules/flat-entity/types/from-metadata-entity-to-metadata-name.type';
 import { type MetadataManyToOneJoinColumn } from 'src/engine/metadata-modules/flat-entity/types/metadata-many-to-one-join-column.type';
 import { type SyncableEntity } from 'src/engine/workspace-manager/types/syncable-entity.interface';
@@ -46,7 +46,8 @@ export type UniversalFlatEntityFrom<
   | 'applicationId'
   | 'workspaceId'
   | 'id'
-  | ExtractEntityRelatedEntityProperties<TEntity>
+  | keyof (typeof ALL_METADATA_RELATIONS)[TMetadataName]['manyToOne']
+  | keyof (typeof ALL_METADATA_RELATIONS)[TMetadataName]['oneToMany']
   | Extract<MetadataManyToOneJoinColumn<TMetadataName>, keyof TEntity>
   | keyof CastRecordTypeOrmDatePropertiesToString<TEntity>
   | AllJsonbPropertiesWithSerializedPropertiesForMetadataName<TMetadataName>

packages/twenty-server/src/engine/workspace-manager/workspace-migration/universal-flat-entity/types/universal-flat-entity-maps.type.ts

@@ -0,0 +1,8 @@
+import { type SyncableFlatEntity } from 'src/engine/metadata-modules/flat-entity/types/flat-entity-from.type';
+import { type UniversalSyncableFlatEntity } from 'src/engine/workspace-manager/workspace-migration/universal-flat-entity/types/universal-flat-entity-from.type';
+
+export type UniversalFlatEntityMaps<
+  T extends SyncableFlatEntity | UniversalSyncableFlatEntity,
+> = {
+  byUniversalIdentifier: Partial<Record<string, T>>;
+};

packages/twenty-server/src/engine/workspace-manager/workspace-migration/universal-flat-entity/types/universal-flat-entity-update.type.ts

@@ -0,0 +1,11 @@
+import { type AllMetadataName } from 'twenty-shared/metadata';
+
+import { type MetadataUniversalFlatEntity } from 'src/engine/metadata-modules/flat-entity/types/metadata-universal-flat-entity.type';
+import { type MetadataUniversalFlatEntityPropertiesToCompare } from 'src/engine/workspace-manager/workspace-migration/universal-flat-entity/types/metadata-universal-flat-entity-properties-to-compare.type';
+
+export type UniversalFlatEntityUpdate<T extends AllMetadataName> = Partial<
+  Pick<
+    MetadataUniversalFlatEntity<T>,
+    MetadataUniversalFlatEntityPropertiesToCompare<T>
+  >
+>;

packages/twenty-server/src/engine/workspace-manager/workspace-migration/universal-flat-entity/types/universal-flat-index-metadata.type.ts

@@ -0,0 +1,23 @@
+import { type CastRecordTypeOrmDatePropertiesToString } from 'src/engine/metadata-modules/flat-entity/types/cast-record-typeorm-date-properties-to-string.type';
+import { type IndexFieldMetadataEntity } from 'src/engine/metadata-modules/index-metadata/index-field-metadata.entity';
+import { type IndexMetadataEntity } from 'src/engine/metadata-modules/index-metadata/index-metadata.entity';
+import { type UniversalFlatEntityFrom } from 'src/engine/workspace-manager/workspace-migration/universal-flat-entity/types/universal-flat-entity-from.type';
+
+// Note: IndexFieldMetadataEntity is just in between a SyncableEntity and a jsonb we should decide one of both
+export type UniversalFlatIndexFieldMetadata = Omit<
+  IndexFieldMetadataEntity,
+  | 'indexMetadataId'
+  | 'indexMetadata'
+  | 'fieldMetadataId'
+  | 'fieldMetadata'
+  | 'id'
+  | keyof CastRecordTypeOrmDatePropertiesToString<IndexFieldMetadataEntity>
+> & {
+  indexMetadataUniversalIdentifier: string;
+  fieldMetadataUniversalIdentifier: string;
+} & CastRecordTypeOrmDatePropertiesToString<IndexFieldMetadataEntity>;
+
+export type UniversalFlatIndexMetadata =
+  UniversalFlatEntityFrom<IndexMetadataEntity> & {
+    universalFlatIndexFieldMetadatas: UniversalFlatIndexFieldMetadata[];
+  };

packages/twenty-server/src/engine/workspace-manager/workspace-migration/universal-flat-entity/types/universal-flat-object-metadata.type.ts

@@ -2,7 +2,13 @@ import { type ObjectMetadataEntity } from 'src/engine/metadata-modules/object-me
 import { type UniversalFlatEntityFrom } from 'src/engine/workspace-manager/workspace-migration/universal-flat-entity/types/universal-flat-entity-from.type';
 
 export type UniversalFlatObjectMetadata = UniversalFlatEntityFrom<
-  Omit<ObjectMetadataEntity, 'targetRelationFields' | 'dataSourceId'>,
+  Omit<
+    ObjectMetadataEntity,
+    | 'targetRelationFields'
+    | 'dataSourceId'
+    | 'labelIdentifierFieldMetadataId'
+    | 'imageIdentifierFieldMetadataId'
+  >,
   'objectMetadata'
 > & {
   // NOTE: below fields are not reflected on the final UniversalFlatEntity either they should we should define a common source

packages/twenty-server/src/engine/workspace-manager/workspace-migration/universal-flat-entity/types/universal-flat-skill.type.ts

@@ -0,0 +1,4 @@
+import { type SkillEntity } from 'src/engine/metadata-modules/skill/entities/skill.entity';
+import { type UniversalFlatEntityFrom } from 'src/engine/workspace-manager/workspace-migration/universal-flat-entity/types/universal-flat-entity-from.type';
+
+export type UniversalFlatSkill = UniversalFlatEntityFrom<SkillEntity, 'skill'>;

packages/twenty-server/src/engine/workspace-manager/workspace-migration/universal-flat-entity/types/universal-flat-view-field.type.ts

@@ -0,0 +1,7 @@
+import { type ViewFieldEntity } from 'src/engine/metadata-modules/view-field/entities/view-field.entity';
+import { type UniversalFlatEntityFrom } from 'src/engine/workspace-manager/workspace-migration/universal-flat-entity/types/universal-flat-entity-from.type';
+
+export type UniversalFlatViewField = UniversalFlatEntityFrom<
+  ViewFieldEntity,
+  'viewField'
+>;

packages/twenty-server/src/engine/workspace-manager/workspace-migration/universal-flat-entity/types/universal-flat-view-filter-group.type.ts

@@ -0,0 +1,7 @@
+import { type ViewFilterGroupEntity } from 'src/engine/metadata-modules/view-filter-group/entities/view-filter-group.entity';
+import { type UniversalFlatEntityFrom } from 'src/engine/workspace-manager/workspace-migration/universal-flat-entity/types/universal-flat-entity-from.type';
+
+export type UniversalFlatViewFilterGroup = UniversalFlatEntityFrom<
+  ViewFilterGroupEntity,
+  'viewFilterGroup'
+>;

packages/twenty-server/src/engine/workspace-manager/workspace-migration/universal-flat-entity/types/universal-flat-view-filter.type.ts

@@ -0,0 +1,7 @@
+import { type ViewFilterEntity } from 'src/engine/metadata-modules/view-filter/entities/view-filter.entity';
+import { type UniversalFlatEntityFrom } from 'src/engine/workspace-manager/workspace-migration/universal-flat-entity/types/universal-flat-entity-from.type';
+
+export type UniversalFlatViewFilter = UniversalFlatEntityFrom<
+  ViewFilterEntity,
+  'viewFilter'
+>;

packages/twenty-server/src/engine/workspace-manager/workspace-migration/universal-flat-entity/types/universal-flat-view-group.type.ts

@@ -0,0 +1,7 @@
+import { type ViewGroupEntity } from 'src/engine/metadata-modules/view-group/entities/view-group.entity';
+import { type UniversalFlatEntityFrom } from 'src/engine/workspace-manager/workspace-migration/universal-flat-entity/types/universal-flat-entity-from.type';
+
+export type UniversalFlatViewGroup = UniversalFlatEntityFrom<
+  ViewGroupEntity,
+  'viewGroup'
+>;

packages/twenty-server/src/engine/workspace-manager/workspace-migration/universal-flat-entity/types/universal-flat-view.type.ts

@@ -0,0 +1,8 @@
+import { type ViewEntity } from 'src/engine/metadata-modules/view/entities/view.entity';
+import { type UniversalFlatEntityFrom } from 'src/engine/workspace-manager/workspace-migration/universal-flat-entity/types/universal-flat-entity-from.type';
+
+// TODO remove once viewSorts has been migrated to v2
+export type UniversalFlatView = UniversalFlatEntityFrom<
+  Omit<ViewEntity, 'viewSorts'>,
+  'view'
+>;

packages/twenty-server/src/engine/workspace-manager/workspace-migration/universal-flat-entity/utils/__tests__/__snapshots__/compare-two-universal-flat-entity.util.spec.ts.snap

@@ -0,0 +1,13 @@
+// Jest Snapshot v1, https://jestjs.io/docs/snapshot-testing
+
+exports[`compareTwoFlatEntity It should detect flat field metadata isActive diff from true to false 1`] = `
+{
+  "isActive": false,
+}
+`;
+
+exports[`compareTwoFlatEntity It should detect flat field metadata isActive diff from true to false 2`] = `
+{
+  "isActive": true,
+}
+`;

packages/twenty-server/src/engine/workspace-manager/workspace-migration/universal-flat-entity/utils/__tests__/__snapshots__/sanitize-universal-flat-entity-update.util.spec.ts.snap

@@ -0,0 +1,64 @@
+// Jest Snapshot v1, https://jestjs.io/docs/snapshot-testing
+
+exports[`sanitizeFlatEntityUpdate should handle agent metadata updates 1`] = `
+{
+  "description": "Agent description",
+  "name": "My Agent",
+}
+`;
+
+exports[`sanitizeFlatEntityUpdate should handle role metadata updates 1`] = `
+{
+  "description": "Administrator role",
+  "label": "Admin Role",
+}
+`;
+
+exports[`sanitizeFlatEntityUpdate should handle viewField metadata updates 1`] = `
+{
+  "isVisible": true,
+  "position": 1,
+  "size": 100,
+}
+`;
+
+exports[`sanitizeFlatEntityUpdate should handle webhook metadata updates 1`] = `
+{
+  "description": "My webhook",
+  "targetUrl": "https://example.com/webhook",
+}
+`;
+
+exports[`sanitizeFlatEntityUpdate should preserve null values as they are valid updates for fieldMetadata 1`] = `
+{
+  "description": "Updated description",
+  "isActive": false,
+  "label": null,
+}
+`;
+
+exports[`sanitizeFlatEntityUpdate should return empty object when all properties are undefined for fieldMetadata 1`] = `{}`;
+
+exports[`sanitizeFlatEntityUpdate should return empty object when update is empty for fieldMetadata 1`] = `{}`;
+
+exports[`sanitizeFlatEntityUpdate should return only valid properties for fieldMetadata when update contains valid and invalid properties 1`] = `
+{
+  "isActive": true,
+  "label": "New Label",
+}
+`;
+
+exports[`sanitizeFlatEntityUpdate should return only valid properties for objectMetadata 1`] = `
+{
+  "isActive": true,
+  "labelPlural": "People",
+  "labelSingular": "Person",
+}
+`;
+
+exports[`sanitizeFlatEntityUpdate should return only valid properties for view metadata 1`] = `
+{
+  "icon": "IconUser",
+  "name": "My View",
+}
+`;

packages/twenty-server/src/engine/workspace-manager/workspace-migration/universal-flat-entity/utils/__tests__/__snapshots__/universal-flat-entity-deleted-created-updated-matrix-dispatcher.util.spec.ts.snap

@@ -0,0 +1,259 @@
+// Jest Snapshot v1, https://jestjs.io/docs/snapshot-testing
+
+exports[`flatEntityDeletedCreatedUpdatedMatrixDispatcher It should detect a created entity 1`] = `
+{
+  "createdFlatEntityMaps": {
+    "byUniversalIdentifier": {
+      "universal-identifier-1": {
+        "applicationId": "application-id-1",
+        "applicationUniversalIdentifier": "application-universal-identifier-1",
+        "calendarViewIds": [],
+        "calendarViewUniversalIdentifiers": [],
+        "createdAt": "2024-01-01T00:00:00.000Z",
+        "defaultValue": null,
+        "description": "default flat field metadata description",
+        "icon": "icon",
+        "id": "field-id-1",
+        "isActive": true,
+        "isCustom": true,
+        "isLabelSyncedWithName": false,
+        "isNullable": true,
+        "isSystem": false,
+        "isUIReadOnly": false,
+        "isUnique": false,
+        "kanbanAggregateOperationViewIds": [],
+        "kanbanAggregateOperationViewUniversalIdentifiers": [],
+        "label": "flat field metadata label",
+        "mainGroupByFieldMetadataViewIds": [],
+        "mainGroupByFieldMetadataViewUniversalIdentifiers": [],
+        "morphId": null,
+        "name": "flatFieldMetadataName",
+        "objectMetadataId": "object-metadata-id-1",
+        "objectMetadataUniversalIdentifier": "object-metadata-universal-identifier-1",
+        "options": null,
+        "relationTargetFieldMetadataId": null,
+        "relationTargetFieldMetadataUniversalIdentifier": null,
+        "relationTargetObjectMetadataId": null,
+        "relationTargetObjectMetadataUniversalIdentifier": null,
+        "settings": null,
+        "standardOverrides": null,
+        "type": "TEXT",
+        "universalIdentifier": "universal-identifier-1",
+        "universalSettings": null,
+        "updatedAt": "2024-01-01T00:00:00.000Z",
+        "viewFieldIds": [],
+        "viewFieldUniversalIdentifiers": [],
+        "viewFilterIds": [],
+        "viewFilterUniversalIdentifiers": [],
+        "workspaceId": "workspace-id-1",
+      },
+    },
+  },
+  "deletedFlatEntityMaps": {
+    "byUniversalIdentifier": {},
+  },
+  "updatedFlatEntityMaps": {
+    "byUniversalIdentifier": {},
+  },
+}
+`;
+
+exports[`flatEntityDeletedCreatedUpdatedMatrixDispatcher It should detect a deleted entity 1`] = `
+{
+  "createdFlatEntityMaps": {
+    "byUniversalIdentifier": {},
+  },
+  "deletedFlatEntityMaps": {
+    "byUniversalIdentifier": {
+      "universal-identifier-1": {
+        "applicationId": "application-id-1",
+        "applicationUniversalIdentifier": "application-universal-identifier-1",
+        "calendarViewIds": [],
+        "calendarViewUniversalIdentifiers": [],
+        "createdAt": "2024-01-01T00:00:00.000Z",
+        "defaultValue": null,
+        "description": "default flat field metadata description",
+        "icon": "icon",
+        "id": "field-id-1",
+        "isActive": true,
+        "isCustom": true,
+        "isLabelSyncedWithName": false,
+        "isNullable": true,
+        "isSystem": false,
+        "isUIReadOnly": false,
+        "isUnique": false,
+        "kanbanAggregateOperationViewIds": [],
+        "kanbanAggregateOperationViewUniversalIdentifiers": [],
+        "label": "flat field metadata label",
+        "mainGroupByFieldMetadataViewIds": [],
+        "mainGroupByFieldMetadataViewUniversalIdentifiers": [],
+        "morphId": null,
+        "name": "flatFieldMetadataName",
+        "objectMetadataId": "object-metadata-id-1",
+        "objectMetadataUniversalIdentifier": "object-metadata-universal-identifier-1",
+        "options": null,
+        "relationTargetFieldMetadataId": null,
+        "relationTargetFieldMetadataUniversalIdentifier": null,
+        "relationTargetObjectMetadataId": null,
+        "relationTargetObjectMetadataUniversalIdentifier": null,
+        "settings": null,
+        "standardOverrides": null,
+        "type": "TEXT",
+        "universalIdentifier": "universal-identifier-1",
+        "universalSettings": null,
+        "updatedAt": "2024-01-01T00:00:00.000Z",
+        "viewFieldIds": [],
+        "viewFieldUniversalIdentifiers": [],
+        "viewFilterIds": [],
+        "viewFilterUniversalIdentifiers": [],
+        "workspaceId": "workspace-id-1",
+      },
+    },
+  },
+  "updatedFlatEntityMaps": {
+    "byUniversalIdentifier": {},
+  },
+}
+`;
+
+exports[`flatEntityDeletedCreatedUpdatedMatrixDispatcher It should detect an updated entity 1`] = `
+{
+  "createdFlatEntityMaps": {
+    "byUniversalIdentifier": {},
+  },
+  "deletedFlatEntityMaps": {
+    "byUniversalIdentifier": {},
+  },
+  "updatedFlatEntityMaps": {
+    "byUniversalIdentifier": {
+      "universal-identifier-1": {
+        "update": {
+          "isActive": true,
+        },
+      },
+    },
+  },
+}
+`;
+
+exports[`flatEntityDeletedCreatedUpdatedMatrixDispatcher It should detect created, deleted and updated entities 1`] = `
+{
+  "createdFlatEntityMaps": {
+    "byUniversalIdentifier": {
+      "universal-identifier-3": {
+        "applicationId": "application-id-1",
+        "applicationUniversalIdentifier": "application-universal-identifier-1",
+        "calendarViewIds": [],
+        "calendarViewUniversalIdentifiers": [],
+        "createdAt": "2024-01-01T00:00:00.000Z",
+        "defaultValue": null,
+        "description": "default flat field metadata description",
+        "icon": "icon",
+        "id": "field-id-3",
+        "isActive": true,
+        "isCustom": true,
+        "isLabelSyncedWithName": false,
+        "isNullable": true,
+        "isSystem": false,
+        "isUIReadOnly": false,
+        "isUnique": false,
+        "kanbanAggregateOperationViewIds": [],
+        "kanbanAggregateOperationViewUniversalIdentifiers": [],
+        "label": "flat field metadata label",
+        "mainGroupByFieldMetadataViewIds": [],
+        "mainGroupByFieldMetadataViewUniversalIdentifiers": [],
+        "morphId": null,
+        "name": "flatFieldMetadataName",
+        "objectMetadataId": "object-metadata-id-1",
+        "objectMetadataUniversalIdentifier": "object-metadata-universal-identifier-1",
+        "options": null,
+        "relationTargetFieldMetadataId": null,
+        "relationTargetFieldMetadataUniversalIdentifier": null,
+        "relationTargetObjectMetadataId": null,
+        "relationTargetObjectMetadataUniversalIdentifier": null,
+        "settings": null,
+        "standardOverrides": null,
+        "type": "TEXT",
+        "universalIdentifier": "universal-identifier-3",
+        "universalSettings": null,
+        "updatedAt": "2024-01-01T00:00:00.000Z",
+        "viewFieldIds": [],
+        "viewFieldUniversalIdentifiers": [],
+        "viewFilterIds": [],
+        "viewFilterUniversalIdentifiers": [],
+        "workspaceId": "workspace-id-1",
+      },
+    },
+  },
+  "deletedFlatEntityMaps": {
+    "byUniversalIdentifier": {
+      "universal-identifier-2": {
+        "applicationId": "application-id-1",
+        "applicationUniversalIdentifier": "application-universal-identifier-1",
+        "calendarViewIds": [],
+        "calendarViewUniversalIdentifiers": [],
+        "createdAt": "2024-01-01T00:00:00.000Z",
+        "defaultValue": null,
+        "description": "default flat field metadata description",
+        "icon": "icon",
+        "id": "field-id-2",
+        "isActive": true,
+        "isCustom": true,
+        "isLabelSyncedWithName": false,
+        "isNullable": true,
+        "isSystem": false,
+        "isUIReadOnly": false,
+        "isUnique": false,
+        "kanbanAggregateOperationViewIds": [],
+        "kanbanAggregateOperationViewUniversalIdentifiers": [],
+        "label": "flat field metadata label",
+        "mainGroupByFieldMetadataViewIds": [],
+        "mainGroupByFieldMetadataViewUniversalIdentifiers": [],
+        "morphId": null,
+        "name": "flatFieldMetadataName",
+        "objectMetadataId": "object-metadata-id-1",
+        "objectMetadataUniversalIdentifier": "object-metadata-universal-identifier-1",
+        "options": null,
+        "relationTargetFieldMetadataId": null,
+        "relationTargetFieldMetadataUniversalIdentifier": null,
+        "relationTargetObjectMetadataId": null,
+        "relationTargetObjectMetadataUniversalIdentifier": null,
+        "settings": null,
+        "standardOverrides": null,
+        "type": "TEXT",
+        "universalIdentifier": "universal-identifier-2",
+        "universalSettings": null,
+        "updatedAt": "2024-01-01T00:00:00.000Z",
+        "viewFieldIds": [],
+        "viewFieldUniversalIdentifiers": [],
+        "viewFilterIds": [],
+        "viewFilterUniversalIdentifiers": [],
+        "workspaceId": "workspace-id-1",
+      },
+    },
+  },
+  "updatedFlatEntityMaps": {
+    "byUniversalIdentifier": {
+      "universal-identifier-1": {
+        "update": {
+          "isActive": false,
+        },
+      },
+    },
+  },
+}
+`;
+
+exports[`flatEntityDeletedCreatedUpdatedMatrixDispatcher It should not detect deleted entities when inferDeletionFromMissingEntities is false 1`] = `
+{
+  "createdFlatEntityMaps": {
+    "byUniversalIdentifier": {},
+  },
+  "deletedFlatEntityMaps": {
+    "byUniversalIdentifier": {},
+  },
+  "updatedFlatEntityMaps": {
+    "byUniversalIdentifier": {},
+  },
+}
+`;

packages/twenty-server/src/engine/workspace-manager/workspace-migration/universal-flat-entity/utils/__tests__/add-universal-flat-entity-to-universal-flat-entity-and-related-entity-maps-through-mutation-or-throw.util.spec.ts

@@ -0,0 +1,239 @@
+import { FieldMetadataType } from 'twenty-shared/types';
+
+import {
+  FlatEntityMapsException,
+  FlatEntityMapsExceptionCode,
+} from 'src/engine/metadata-modules/flat-entity/exceptions/flat-entity-maps.exception';
+import { type MetadataUniversalFlatEntityAndRelatedFlatEntityMapsForValidation } from 'src/engine/metadata-modules/flat-entity/types/metadata-flat-entity-and-related-flat-entity-maps-for-validation.type';
+import { type MetadataUniversalFlatEntity } from 'src/engine/metadata-modules/flat-entity/types/metadata-universal-flat-entity.type';
+import { findFlatEntityByUniversalIdentifier } from 'src/engine/metadata-modules/flat-entity/utils/find-flat-entity-by-universal-identifier.util';
+import { type FlatFieldMetadata } from 'src/engine/metadata-modules/flat-field-metadata/types/flat-field-metadata.type';
+import { type FlatObjectMetadata } from 'src/engine/metadata-modules/flat-object-metadata/types/flat-object-metadata.type';
+import { type FlatView } from 'src/engine/metadata-modules/flat-view/types/flat-view.type';
+import { type UniversalFlatEntityMaps } from 'src/engine/workspace-manager/workspace-migration/universal-flat-entity/types/universal-flat-entity-maps.type';
+import { addUniversalFlatEntityToUniversalFlatEntityAndRelatedEntityMapsThroughMutationOrThrow } from 'src/engine/workspace-manager/workspace-migration/universal-flat-entity/utils/add-universal-flat-entity-to-universal-flat-entity-and-related-entity-maps-through-mutation-or-throw.util';
+import { addUniversalFlatEntityToUniversalFlatEntityMapsThroughMutationOrThrow } from 'src/engine/workspace-manager/workspace-migration/universal-flat-entity/utils/add-universal-flat-entity-to-universal-flat-entity-maps-through-mutation-or-throw.util';
+
+const createEmptyUniversalFlatEntityMaps = <T>(): UniversalFlatEntityMaps<
+  T & { universalIdentifier: string; applicationUniversalIdentifier: string }
+> => ({
+  byUniversalIdentifier: {},
+});
+
+describe('addUniversalFlatEntityToUniversalFlatEntityAndRelatedEntityMapsThroughMutationOrThrow', () => {
+  it('should add a view and update related objectMetadata with viewUniversalIdentifier', () => {
+    const objectMetadataUniversalIdentifier = 'object-universal-1';
+    const fieldMetadataUniversalIdentifier = 'field-universal-1';
+    const viewUniversalIdentifier = 'view-universal-1';
+    const applicationUniversalIdentifier =
+      '20202020-f3ad-452e-b5b6-2d49d3ea88b1';
+
+    const mockUniversalObjectMetadata = {
+      universalIdentifier: objectMetadataUniversalIdentifier,
+      viewUniversalIdentifiers: [],
+      fieldUniversalIdentifiers: [],
+      indexMetadataUniversalIdentifiers: [],
+      description: 'default flat object metadata description',
+      icon: 'icon',
+      isActive: true,
+      isAuditLogged: true,
+      isCustom: true,
+      isLabelSyncedWithName: false,
+      isRemote: false,
+      isSearchable: true,
+      isSystem: false,
+      isUIReadOnly: false,
+      labelPlural: 'default flat object metadata label plural',
+      labelSingular: 'default flat object metadata label singular',
+      namePlural: 'defaultflatObjectMetadataNamePlural',
+      nameSingular: 'defaultflatObjectMetadataNameSingular',
+      shortcut: 'shortcut',
+      standardOverrides: null,
+      targetTableName: '',
+      createdAt: '2024-01-01T00:00:00.000Z',
+      updatedAt: '2024-01-01T00:00:00.000Z',
+      duplicateCriteria: null,
+      applicationUniversalIdentifier,
+      labelIdentifierFieldMetadataUniversalIdentifier:
+        '20202020-1a62-405c-87fa-4d4fd215851b',
+      imageIdentifierFieldMetadataUniversalIdentifier:
+        '20202020-9d65-415f-b0e1-216a2e257ea4',
+    };
+
+    const mockUniversalFieldMetadata = {
+      universalIdentifier: fieldMetadataUniversalIdentifier,
+      objectMetadataUniversalIdentifier,
+      type: FieldMetadataType.DATE,
+      applicationUniversalIdentifier,
+      viewFieldUniversalIdentifiers: [],
+      viewFilterUniversalIdentifiers: [],
+      calendarViewUniversalIdentifiers: [],
+      mainGroupByFieldMetadataViewUniversalIdentifiers: [],
+      kanbanAggregateOperationViewUniversalIdentifiers: [],
+      createdAt: '2024-01-01T00:00:00.000Z',
+      updatedAt: '2024-01-01T00:00:00.000Z',
+      defaultValue: null,
+      options: null,
+      morphId: null,
+      settings: null,
+      universalSettings: null,
+      description: 'default flat field metadata description',
+      icon: 'icon',
+      isActive: true,
+      isCustom: true,
+      name: 'flatFieldMetadataName',
+      label: 'flat field metadata label',
+      isNullable: true,
+      isUnique: false,
+      isUIReadOnly: false,
+      isLabelSyncedWithName: false,
+      isSystem: false,
+      standardOverrides: null,
+      relationTargetObjectMetadataUniversalIdentifier: null,
+      relationTargetFieldMetadataUniversalIdentifier: null,
+    };
+
+    const mockUniversalView = {
+      universalIdentifier: viewUniversalIdentifier,
+      objectMetadataUniversalIdentifier,
+      viewFieldUniversalIdentifiers: [],
+      viewFilterUniversalIdentifiers: [],
+      viewGroupUniversalIdentifiers: [],
+      applicationUniversalIdentifier,
+      calendarFieldMetadataUniversalIdentifier:
+        fieldMetadataUniversalIdentifier,
+      createdAt: '2024-01-01T00:00:00.000Z',
+      updatedAt: '2024-01-01T00:00:00.000Z',
+      name: 'Test View',
+      type: 'table',
+      icon: 'IconList',
+      isCompact: false,
+      position: 0,
+      key: null,
+      kanbanFieldMetadataUniversalIdentifier: null,
+      kanbanAggregateOperationFieldMetadataUniversalIdentifier: null,
+      kanbanAggregateOperation: null,
+      mainGroupByFieldMetadataUniversalIdentifier: null,
+      viewFilterGroupUniversalIdentifiers: [],
+      rowLevelPermissionPredicateGroupUniversalIdentifiers: [],
+    };
+
+    const flatObjectMetadataMaps =
+      createEmptyUniversalFlatEntityMaps<typeof mockUniversalObjectMetadata>();
+
+    addUniversalFlatEntityToUniversalFlatEntityMapsThroughMutationOrThrow({
+      universalFlatEntity: mockUniversalObjectMetadata,
+      universalFlatEntityMapsToMutate: flatObjectMetadataMaps,
+    });
+
+    const flatFieldMetadataMaps =
+      createEmptyUniversalFlatEntityMaps<typeof mockUniversalFieldMetadata>();
+
+    addUniversalFlatEntityToUniversalFlatEntityMapsThroughMutationOrThrow({
+      universalFlatEntity: mockUniversalFieldMetadata,
+      universalFlatEntityMapsToMutate: flatFieldMetadataMaps,
+    });
+
+    const universalFlatEntityAndRelatedMapsToMutate = {
+      flatFieldMetadataMaps,
+      flatObjectMetadataMaps,
+      flatViewMaps:
+        createEmptyUniversalFlatEntityMaps<typeof mockUniversalView>(),
+    } as unknown as MetadataUniversalFlatEntityAndRelatedFlatEntityMapsForValidation<'view'>;
+
+    addUniversalFlatEntityToUniversalFlatEntityAndRelatedEntityMapsThroughMutationOrThrow(
+      {
+        metadataName: 'view',
+        universalFlatEntity:
+          mockUniversalView as unknown as MetadataUniversalFlatEntity<'view'>,
+        universalFlatEntityAndRelatedMapsToMutate,
+      },
+    );
+
+    expect(
+      findFlatEntityByUniversalIdentifier({
+        universalIdentifier: viewUniversalIdentifier,
+        flatEntityMaps:
+          universalFlatEntityAndRelatedMapsToMutate.flatViewMaps as UniversalFlatEntityMaps<FlatView>,
+      }),
+    ).toMatchObject({
+      universalIdentifier: viewUniversalIdentifier,
+    });
+
+    const updatedObjectMetadata = findFlatEntityByUniversalIdentifier({
+      universalIdentifier: objectMetadataUniversalIdentifier,
+      flatEntityMaps:
+        universalFlatEntityAndRelatedMapsToMutate.flatObjectMetadataMaps as UniversalFlatEntityMaps<FlatObjectMetadata>,
+    });
+
+    expect(updatedObjectMetadata).toMatchObject({
+      viewUniversalIdentifiers: [viewUniversalIdentifier],
+    });
+
+    const updatedFieldMetadata = findFlatEntityByUniversalIdentifier({
+      universalIdentifier: fieldMetadataUniversalIdentifier,
+      flatEntityMaps:
+        universalFlatEntityAndRelatedMapsToMutate.flatFieldMetadataMaps as UniversalFlatEntityMaps<FlatFieldMetadata>,
+    });
+
+    expect(updatedFieldMetadata).toMatchObject({
+      calendarViewUniversalIdentifiers: [viewUniversalIdentifier],
+    });
+  });
+
+  it('should throw when related entity is not found', () => {
+    const viewUniversalIdentifier = 'view-universal-1';
+    const nonExistentObjectMetadataUniversalIdentifier =
+      'non-existent-object-universal';
+    const applicationUniversalIdentifier =
+      '20202020-f3ad-452e-b5b6-2d49d3ea88b1';
+
+    const mockUniversalView = {
+      universalIdentifier: viewUniversalIdentifier,
+      objectMetadataUniversalIdentifier:
+        nonExistentObjectMetadataUniversalIdentifier,
+      viewFieldUniversalIdentifiers: [],
+      viewFilterUniversalIdentifiers: [],
+      viewGroupUniversalIdentifiers: [],
+      applicationUniversalIdentifier,
+      calendarFieldMetadataUniversalIdentifier: null,
+      createdAt: '2024-01-01T00:00:00.000Z',
+      updatedAt: '2024-01-01T00:00:00.000Z',
+      name: 'Test View',
+      type: 'table',
+      icon: 'IconList',
+      isCompact: false,
+      position: 0,
+      key: null,
+      kanbanFieldMetadataUniversalIdentifier: null,
+      kanbanAggregateOperationFieldMetadataUniversalIdentifier: null,
+      kanbanAggregateOperation: null,
+      mainGroupByFieldMetadataUniversalIdentifier: null,
+      viewFilterGroupUniversalIdentifiers: [],
+      rowLevelPermissionPredicateGroupUniversalIdentifiers: [],
+    };
+
+    const universalFlatEntityAndRelatedMapsToMutate = {
+      flatFieldMetadataMaps: createEmptyUniversalFlatEntityMaps(),
+      flatObjectMetadataMaps: createEmptyUniversalFlatEntityMaps(),
+      flatViewMaps:
+        createEmptyUniversalFlatEntityMaps<typeof mockUniversalView>(),
+    } as unknown as MetadataUniversalFlatEntityAndRelatedFlatEntityMapsForValidation<'view'>;
+
+    expect(() =>
+      addUniversalFlatEntityToUniversalFlatEntityAndRelatedEntityMapsThroughMutationOrThrow(
+        {
+          metadataName: 'view',
+          universalFlatEntity:
+            mockUniversalView as unknown as MetadataUniversalFlatEntity<'view'>,
+          universalFlatEntityAndRelatedMapsToMutate,
+        },
+      ),
+    ).toThrow(
+      new FlatEntityMapsException(
+        `Could not find flat entity with universal identifier ${nonExistentObjectMetadataUniversalIdentifier}`,
+        FlatEntityMapsExceptionCode.ENTITY_NOT_FOUND,
+      ),
+    );
+  });
+});

packages/twenty-server/src/engine/workspace-manager/workspace-migration/universal-flat-entity/utils/__tests__/compare-two-universal-flat-entity.util.spec.ts

@@ -0,0 +1,76 @@
+import { extractRecordIdsAndDatesAsExpectAny } from 'test/utils/extract-record-ids-and-dates-as-expect-any';
+import { type AllMetadataName } from 'twenty-shared/metadata';
+import {
+  eachTestingContextFilter,
+  type EachTestingContext,
+} from 'twenty-shared/testing';
+import { FieldMetadataType, type FromTo } from 'twenty-shared/types';
+
+import { type MetadataUniversalFlatEntity } from 'src/engine/metadata-modules/flat-entity/types/metadata-universal-flat-entity.type';
+import { getFlatFieldMetadataMock } from 'src/engine/metadata-modules/flat-field-metadata/__mocks__/get-flat-field-metadata.mock';
+import { compareTwoFlatEntity } from 'src/engine/workspace-manager/workspace-migration/universal-flat-entity/utils/compare-two-universal-flat-entity.util';
+
+type TestContext<T extends AllMetadataName = AllMetadataName> = FromTo<
+  MetadataUniversalFlatEntity<T>,
+  'universalFlatEntity'
+> & { metadataName: T };
+
+describe('compareTwoFlatEntity', () => {
+  const testCases = [
+    {
+      title:
+        'It should detect flat field metadata isActive diff from true to false',
+      context: {
+        fromUniversalFlatEntity: getFlatFieldMetadataMock({
+          objectMetadataId: 'object-metadata-id',
+          type: FieldMetadataType.TEXT,
+          universalIdentifier: 'universal-identifier',
+          isActive: true,
+        }),
+        metadataName: 'fieldMetadata',
+        toUniversalFlatEntity: getFlatFieldMetadataMock({
+          objectMetadataId: 'object-metadata-id',
+          type: FieldMetadataType.TEXT,
+          universalIdentifier: 'universal-identifier',
+          isActive: false,
+        }),
+      },
+    },
+    {
+      title:
+        'It should detect flat field metadata isActive diff from true to false',
+      context: {
+        fromUniversalFlatEntity: getFlatFieldMetadataMock({
+          objectMetadataId: 'object-metadata-id',
+          type: FieldMetadataType.TEXT,
+          universalIdentifier: 'universal-identifier',
+          isActive: false,
+        }),
+        metadataName: 'fieldMetadata',
+        toUniversalFlatEntity: getFlatFieldMetadataMock({
+          objectMetadataId: 'object-metadata-id',
+          type: FieldMetadataType.TEXT,
+          universalIdentifier: 'universal-identifier',
+          isActive: true,
+        }),
+      },
+    },
+  ] as const satisfies EachTestingContext<TestContext>[];
+
+  test.each(eachTestingContextFilter(testCases))(
+    '$title',
+    ({
+      context: { metadataName, fromUniversalFlatEntity, toUniversalFlatEntity },
+    }) => {
+      const result = compareTwoFlatEntity({
+        fromUniversalFlatEntity,
+        toUniversalFlatEntity,
+        metadataName,
+      });
+
+      expect(result).toMatchSnapshot(
+        extractRecordIdsAndDatesAsExpectAny({ ...result }),
+      );
+    },
+  );
+});

packages/twenty-server/src/engine/workspace-manager/workspace-migration/universal-flat-entity/utils/__tests__/delete-universal-flat-entity-from-universal-flat-entity-and-related-entity-maps-through-mutation-or-throw.util.spec.ts

@@ -0,0 +1,192 @@
+import { FieldMetadataType } from 'twenty-shared/types';
+
+import { type MetadataUniversalFlatEntityAndRelatedFlatEntityMapsForValidation } from 'src/engine/metadata-modules/flat-entity/types/metadata-flat-entity-and-related-flat-entity-maps-for-validation.type';
+import { type MetadataUniversalFlatEntity } from 'src/engine/metadata-modules/flat-entity/types/metadata-universal-flat-entity.type';
+import { findFlatEntityByUniversalIdentifier } from 'src/engine/metadata-modules/flat-entity/utils/find-flat-entity-by-universal-identifier.util';
+import { type FlatFieldMetadata } from 'src/engine/metadata-modules/flat-field-metadata/types/flat-field-metadata.type';
+import { type FlatObjectMetadata } from 'src/engine/metadata-modules/flat-object-metadata/types/flat-object-metadata.type';
+import { type FlatView } from 'src/engine/metadata-modules/flat-view/types/flat-view.type';
+import { type UniversalFlatEntityMaps } from 'src/engine/workspace-manager/workspace-migration/universal-flat-entity/types/universal-flat-entity-maps.type';
+import { addUniversalFlatEntityToUniversalFlatEntityMapsThroughMutationOrThrow } from 'src/engine/workspace-manager/workspace-migration/universal-flat-entity/utils/add-universal-flat-entity-to-universal-flat-entity-maps-through-mutation-or-throw.util';
+import { deleteUniversalFlatEntityFromUniversalFlatEntityAndRelatedEntityMapsThroughMutationOrThrow } from 'src/engine/workspace-manager/workspace-migration/universal-flat-entity/utils/delete-universal-flat-entity-from-universal-flat-entity-and-related-entity-maps-through-mutation-or-throw.util';
+
+const createEmptyUniversalFlatEntityMaps = <T>(): UniversalFlatEntityMaps<
+  T & { universalIdentifier: string; applicationUniversalIdentifier: string }
+> => ({
+  byUniversalIdentifier: {},
+});
+
+describe('deleteUniversalFlatEntityFromUniversalFlatEntityAndRelatedEntityMapsThroughMutationOrThrow', () => {
+  it('should delete a view and update related objectMetadata by removing viewUniversalIdentifier', () => {
+    const objectMetadataUniversalIdentifier = 'object-universal-1';
+    const fieldMetadataUniversalIdentifier = 'field-universal-1';
+    const viewUniversalIdentifier = 'view-universal-1';
+    const applicationUniversalIdentifier =
+      '20202020-f3ad-452e-b5b6-2d49d3ea88b1';
+
+    const mockUniversalObjectMetadata = {
+      universalIdentifier: objectMetadataUniversalIdentifier,
+      viewUniversalIdentifiers: [viewUniversalIdentifier],
+      fieldUniversalIdentifiers: [],
+      indexMetadataUniversalIdentifiers: [],
+      description: 'default flat object metadata description',
+      icon: 'icon',
+      isActive: true,
+      isAuditLogged: true,
+      isCustom: true,
+      isLabelSyncedWithName: false,
+      isRemote: false,
+      isSearchable: true,
+      isSystem: false,
+      isUIReadOnly: false,
+      labelPlural: 'default flat object metadata label plural',
+      labelSingular: 'default flat object metadata label singular',
+      namePlural: 'defaultflatObjectMetadataNamePlural',
+      nameSingular: 'defaultflatObjectMetadataNameSingular',
+      shortcut: 'shortcut',
+      standardOverrides: null,
+      targetTableName: '',
+      createdAt: '2024-01-01T00:00:00.000Z',
+      updatedAt: '2024-01-01T00:00:00.000Z',
+      duplicateCriteria: null,
+      applicationUniversalIdentifier,
+      labelIdentifierFieldMetadataUniversalIdentifier:
+        '20202020-1a62-405c-87fa-4d4fd215851b',
+      imageIdentifierFieldMetadataUniversalIdentifier:
+        '20202020-9d65-415f-b0e1-216a2e257ea4',
+    };
+
+    const mockUniversalFieldMetadata = {
+      universalIdentifier: fieldMetadataUniversalIdentifier,
+      objectMetadataUniversalIdentifier,
+      type: FieldMetadataType.DATE,
+      applicationUniversalIdentifier,
+      viewFieldUniversalIdentifiers: [],
+      viewFilterUniversalIdentifiers: [],
+      calendarViewUniversalIdentifiers: [viewUniversalIdentifier],
+      mainGroupByFieldMetadataViewUniversalIdentifiers: [],
+      kanbanAggregateOperationViewUniversalIdentifiers: [],
+      createdAt: '2024-01-01T00:00:00.000Z',
+      updatedAt: '2024-01-01T00:00:00.000Z',
+      defaultValue: null,
+      options: null,
+      morphId: null,
+      settings: null,
+      universalSettings: null,
+      description: 'default flat field metadata description',
+      icon: 'icon',
+      isActive: true,
+      isCustom: true,
+      name: 'flatFieldMetadataName',
+      label: 'flat field metadata label',
+      isNullable: true,
+      isUnique: false,
+      isUIReadOnly: false,
+      isLabelSyncedWithName: false,
+      isSystem: false,
+      standardOverrides: null,
+      relationTargetObjectMetadataUniversalIdentifier: null,
+      relationTargetFieldMetadataUniversalIdentifier: null,
+    };
+
+    const mockUniversalView = {
+      universalIdentifier: viewUniversalIdentifier,
+      objectMetadataUniversalIdentifier,
+      viewFieldUniversalIdentifiers: [],
+      viewFilterUniversalIdentifiers: [],
+      viewGroupUniversalIdentifiers: [],
+      applicationUniversalIdentifier,
+      calendarFieldMetadataUniversalIdentifier:
+        fieldMetadataUniversalIdentifier,
+      createdAt: '2024-01-01T00:00:00.000Z',
+      updatedAt: '2024-01-01T00:00:00.000Z',
+      name: 'Test View',
+      type: 'table',
+      icon: 'IconList',
+      isCompact: false,
+      position: 0,
+      key: null,
+      kanbanFieldMetadataUniversalIdentifier: null,
+      kanbanAggregateOperationFieldMetadataUniversalIdentifier: null,
+      kanbanAggregateOperation: null,
+      mainGroupByFieldMetadataUniversalIdentifier: null,
+      viewFilterGroupUniversalIdentifiers: [],
+      rowLevelPermissionPredicateGroupUniversalIdentifiers: [],
+    };
+
+    const flatObjectMetadataMaps =
+      createEmptyUniversalFlatEntityMaps<typeof mockUniversalObjectMetadata>();
+
+    addUniversalFlatEntityToUniversalFlatEntityMapsThroughMutationOrThrow({
+      universalFlatEntity: mockUniversalObjectMetadata,
+      universalFlatEntityMapsToMutate: flatObjectMetadataMaps,
+    });
+
+    const flatFieldMetadataMaps =
+      createEmptyUniversalFlatEntityMaps<typeof mockUniversalFieldMetadata>();
+
+    addUniversalFlatEntityToUniversalFlatEntityMapsThroughMutationOrThrow({
+      universalFlatEntity: mockUniversalFieldMetadata,
+      universalFlatEntityMapsToMutate: flatFieldMetadataMaps,
+    });
+
+    const flatViewMaps =
+      createEmptyUniversalFlatEntityMaps<typeof mockUniversalView>();
+
+    addUniversalFlatEntityToUniversalFlatEntityMapsThroughMutationOrThrow({
+      universalFlatEntity: mockUniversalView,
+      universalFlatEntityMapsToMutate: flatViewMaps,
+    });
+
+    const universalFlatEntityAndRelatedMapsToMutate = {
+      flatFieldMetadataMaps,
+      flatObjectMetadataMaps,
+      flatViewMaps,
+    } as unknown as MetadataUniversalFlatEntityAndRelatedFlatEntityMapsForValidation<'view'>;
+
+    expect(
+      findFlatEntityByUniversalIdentifier({
+        universalIdentifier: viewUniversalIdentifier,
+        flatEntityMaps:
+          universalFlatEntityAndRelatedMapsToMutate.flatViewMaps as UniversalFlatEntityMaps<FlatView>,
+      }),
+    ).toBeDefined();
+
+    deleteUniversalFlatEntityFromUniversalFlatEntityAndRelatedEntityMapsThroughMutationOrThrow(
+      {
+        metadataName: 'view',
+        universalFlatEntity:
+          mockUniversalView as unknown as MetadataUniversalFlatEntity<'view'>,
+        universalFlatEntityAndRelatedMapsToMutate,
+      },
+    );
+
+    expect(
+      findFlatEntityByUniversalIdentifier({
+        universalIdentifier: viewUniversalIdentifier,
+        flatEntityMaps:
+          universalFlatEntityAndRelatedMapsToMutate.flatViewMaps as UniversalFlatEntityMaps<FlatView>,
+      }),
+    ).toBeUndefined();
+
+    const updatedObjectMetadata = findFlatEntityByUniversalIdentifier({
+      universalIdentifier: objectMetadataUniversalIdentifier,
+      flatEntityMaps:
+        universalFlatEntityAndRelatedMapsToMutate.flatObjectMetadataMaps as UniversalFlatEntityMaps<FlatObjectMetadata>,
+    });
+
+    expect(updatedObjectMetadata).toMatchObject({
+      viewUniversalIdentifiers: [],
+    });
+
+    const updatedFieldMetadata = findFlatEntityByUniversalIdentifier({
+      universalIdentifier: fieldMetadataUniversalIdentifier,
+      flatEntityMaps:
+        universalFlatEntityAndRelatedMapsToMutate.flatFieldMetadataMaps as UniversalFlatEntityMaps<FlatFieldMetadata>,
+    });
+
+    expect(updatedFieldMetadata).toMatchObject({
+      calendarViewUniversalIdentifiers: [],
+    });
+  });
+});

packages/twenty-server/src/engine/workspace-manager/workspace-migration/universal-flat-entity/utils/__tests__/sanitize-universal-flat-entity-update.util.spec.ts

@@ -0,0 +1,132 @@
+import { type AllMetadataName } from 'twenty-shared/metadata';
+
+import { sanitizeUniversalFlatEntityUpdate } from 'src/engine/workspace-manager/workspace-migration/universal-flat-entity/utils/sanitize-universal-flat-entity-update.util';
+
+describe('sanitizeFlatEntityUpdate', () => {
+  it('should return only valid properties for fieldMetadata when update contains valid and invalid properties', () => {
+    const result = sanitizeUniversalFlatEntityUpdate({
+      flatEntityUpdate: {
+        isActive: true,
+        label: 'New Label',
+        invalidProperty: 'should be removed',
+      } as any,
+      metadataName: 'fieldMetadata' as AllMetadataName,
+    });
+
+    expect(result).toMatchSnapshot();
+  });
+
+  it('should return empty object when all properties are undefined for fieldMetadata', () => {
+    const result = sanitizeUniversalFlatEntityUpdate({
+      flatEntityUpdate: {
+        isActive: undefined,
+        label: undefined,
+      } as any,
+      metadataName: 'fieldMetadata' as AllMetadataName,
+    });
+
+    expect(result).toMatchSnapshot();
+  });
+
+  it('should return empty object when update is empty for fieldMetadata', () => {
+    const result = sanitizeUniversalFlatEntityUpdate({
+      flatEntityUpdate: {} as any,
+      metadataName: 'fieldMetadata' as AllMetadataName,
+    });
+
+    expect(result).toMatchSnapshot();
+  });
+
+  it('should preserve null values as they are valid updates for fieldMetadata', () => {
+    const result = sanitizeUniversalFlatEntityUpdate({
+      flatEntityUpdate: {
+        isActive: false,
+        label: null,
+        description: 'Updated description',
+      } as any,
+      metadataName: 'fieldMetadata' as AllMetadataName,
+    });
+
+    expect(result).toMatchSnapshot();
+  });
+
+  it('should return only valid properties for view metadata', () => {
+    const result = sanitizeUniversalFlatEntityUpdate({
+      flatEntityUpdate: {
+        name: 'My View',
+        icon: 'IconUser',
+        invalidProperty: 'should be removed',
+      } as any,
+      metadataName: 'view' as AllMetadataName,
+    });
+
+    expect(result).toMatchSnapshot();
+  });
+
+  it('should return only valid properties for objectMetadata', () => {
+    const result = sanitizeUniversalFlatEntityUpdate({
+      flatEntityUpdate: {
+        isActive: true,
+        labelSingular: 'Person',
+        labelPlural: 'People',
+        invalidProperty: 'should be removed',
+      } as any,
+      metadataName: 'objectMetadata' as AllMetadataName,
+    });
+
+    expect(result).toMatchSnapshot();
+  });
+
+  it('should handle viewField metadata updates', () => {
+    const result = sanitizeUniversalFlatEntityUpdate({
+      flatEntityUpdate: {
+        position: 1,
+        size: 100,
+        isVisible: true,
+        invalidProperty: 'should be removed',
+      } as any,
+      metadataName: 'viewField' as AllMetadataName,
+    });
+
+    expect(result).toMatchSnapshot();
+  });
+
+  it('should handle role metadata updates', () => {
+    const result = sanitizeUniversalFlatEntityUpdate({
+      flatEntityUpdate: {
+        label: 'Admin Role',
+        description: 'Administrator role',
+        invalidProperty: 'should be removed',
+      } as any,
+      metadataName: 'role' as AllMetadataName,
+    });
+
+    expect(result).toMatchSnapshot();
+  });
+
+  it('should handle agent metadata updates', () => {
+    const result = sanitizeUniversalFlatEntityUpdate({
+      flatEntityUpdate: {
+        name: 'My Agent',
+        description: 'Agent description',
+        invalidProperty: 'should be removed',
+      } as any,
+      metadataName: 'agent' as AllMetadataName,
+    });
+
+    expect(result).toMatchSnapshot();
+  });
+
+  it('should handle webhook metadata updates', () => {
+    const result = sanitizeUniversalFlatEntityUpdate({
+      flatEntityUpdate: {
+        targetUrl: 'https://example.com/webhook',
+        description: 'My webhook',
+        invalidProperty: 'should be removed',
+      } as any,
+      metadataName: 'webhook' as AllMetadataName,
+    });
+
+    expect(result).toMatchSnapshot();
+  });
+});

packages/twenty-server/src/engine/workspace-manager/workspace-migration/universal-flat-entity/utils/__tests__/universal-flat-entity-deleted-created-updated-matrix-dispatcher.util.spec.ts

@@ -0,0 +1,232 @@
+import {
+  eachTestingContextFilter,
+  type EachTestingContext,
+} from 'twenty-shared/testing';
+import { FieldMetadataType, type FromTo } from 'twenty-shared/types';
+
+import { type MetadataFlatEntity } from 'src/engine/metadata-modules/flat-entity/types/metadata-flat-entity.type';
+import { getFlatFieldMetadataMock } from 'src/engine/metadata-modules/flat-field-metadata/__mocks__/get-flat-field-metadata.mock';
+import { flatEntityDeletedCreatedUpdatedMatrixDispatcher } from 'src/engine/workspace-manager/workspace-migration/universal-flat-entity/utils/universal-flat-entity-deleted-created-updated-matrix-dispatcher.util';
+import { type WorkspaceMigrationBuilderOptions } from 'src/engine/workspace-manager/workspace-migration/workspace-migration-builder/types/workspace-migration-builder-options.type';
+
+type TestContext = FromTo<MetadataFlatEntity<'fieldMetadata'>[]> & {
+  metadataName: 'fieldMetadata';
+  buildOptions: WorkspaceMigrationBuilderOptions;
+};
+
+describe('flatEntityDeletedCreatedUpdatedMatrixDispatcher', () => {
+  const testCases = [
+    {
+      title: 'It should detect a created entity',
+      context: {
+        from: [],
+        to: [
+          getFlatFieldMetadataMock({
+            objectMetadataId: 'object-metadata-id-1',
+            type: FieldMetadataType.TEXT,
+            universalIdentifier: 'universal-identifier-1',
+            id: 'field-id-1',
+            workspaceId: 'workspace-id-1',
+            applicationId: 'application-id-1',
+            createdAt: '2024-01-01T00:00:00.000Z',
+            updatedAt: '2024-01-01T00:00:00.000Z',
+            applicationUniversalIdentifier:
+              'application-universal-identifier-1',
+            objectMetadataUniversalIdentifier:
+              'object-metadata-universal-identifier-1',
+          }),
+        ],
+        metadataName: 'fieldMetadata',
+        buildOptions: {
+          inferDeletionFromMissingEntities: { fieldMetadata: false },
+          isSystemBuild: false,
+        },
+      },
+    },
+    {
+      title: 'It should detect a deleted entity',
+      context: {
+        from: [
+          getFlatFieldMetadataMock({
+            objectMetadataId: 'object-metadata-id-1',
+            type: FieldMetadataType.TEXT,
+            universalIdentifier: 'universal-identifier-1',
+            id: 'field-id-1',
+            workspaceId: 'workspace-id-1',
+            applicationId: 'application-id-1',
+            createdAt: '2024-01-01T00:00:00.000Z',
+            updatedAt: '2024-01-01T00:00:00.000Z',
+            applicationUniversalIdentifier:
+              'application-universal-identifier-1',
+            objectMetadataUniversalIdentifier:
+              'object-metadata-universal-identifier-1',
+          }),
+        ],
+        to: [],
+        metadataName: 'fieldMetadata',
+        buildOptions: {
+          inferDeletionFromMissingEntities: { fieldMetadata: true },
+          isSystemBuild: false,
+        },
+      },
+    },
+    {
+      title: 'It should detect an updated entity',
+      context: {
+        from: [
+          getFlatFieldMetadataMock({
+            objectMetadataId: 'object-metadata-id-1',
+            type: FieldMetadataType.TEXT,
+            universalIdentifier: 'universal-identifier-1',
+            id: 'field-id-1',
+            workspaceId: 'workspace-id-1',
+            applicationId: 'application-id-1',
+            createdAt: '2024-01-01T00:00:00.000Z',
+            updatedAt: '2024-01-01T00:00:00.000Z',
+            isActive: false,
+            applicationUniversalIdentifier:
+              'application-universal-identifier-1',
+            objectMetadataUniversalIdentifier:
+              'object-metadata-universal-identifier-1',
+          }),
+        ],
+        to: [
+          getFlatFieldMetadataMock({
+            objectMetadataId: 'object-metadata-id-1',
+            type: FieldMetadataType.TEXT,
+            universalIdentifier: 'universal-identifier-1',
+            id: 'field-id-1',
+            workspaceId: 'workspace-id-1',
+            applicationId: 'application-id-1',
+            createdAt: '2024-01-01T00:00:00.000Z',
+            updatedAt: '2024-01-01T00:00:00.000Z',
+            isActive: true,
+            applicationUniversalIdentifier:
+              'application-universal-identifier-1',
+            objectMetadataUniversalIdentifier:
+              'object-metadata-universal-identifier-1',
+          }),
+        ],
+        metadataName: 'fieldMetadata',
+        buildOptions: {
+          inferDeletionFromMissingEntities: { fieldMetadata: false },
+          isSystemBuild: false,
+        },
+      },
+    },
+    {
+      title: 'It should detect created, deleted and updated entities',
+      context: {
+        from: [
+          getFlatFieldMetadataMock({
+            objectMetadataId: 'object-metadata-id-1',
+            type: FieldMetadataType.TEXT,
+            universalIdentifier: 'universal-identifier-1',
+            id: 'field-id-1',
+            workspaceId: 'workspace-id-1',
+            applicationId: 'application-id-1',
+            createdAt: '2024-01-01T00:00:00.000Z',
+            updatedAt: '2024-01-01T00:00:00.000Z',
+            applicationUniversalIdentifier:
+              'application-universal-identifier-1',
+            objectMetadataUniversalIdentifier:
+              'object-metadata-universal-identifier-1',
+            isActive: true,
+          }),
+          getFlatFieldMetadataMock({
+            objectMetadataId: 'object-metadata-id-1',
+            type: FieldMetadataType.TEXT,
+            universalIdentifier: 'universal-identifier-2',
+            id: 'field-id-2',
+            workspaceId: 'workspace-id-1',
+            applicationId: 'application-id-1',
+            createdAt: '2024-01-01T00:00:00.000Z',
+            updatedAt: '2024-01-01T00:00:00.000Z',
+            applicationUniversalIdentifier:
+              'application-universal-identifier-1',
+            objectMetadataUniversalIdentifier:
+              'object-metadata-universal-identifier-1',
+          }),
+        ],
+        to: [
+          getFlatFieldMetadataMock({
+            objectMetadataId: 'object-metadata-id-1',
+            type: FieldMetadataType.TEXT,
+            universalIdentifier: 'universal-identifier-1',
+            id: 'field-id-1',
+            workspaceId: 'workspace-id-1',
+            applicationId: 'application-id-1',
+            createdAt: '2024-01-01T00:00:00.000Z',
+            applicationUniversalIdentifier:
+              'application-universal-identifier-1',
+            objectMetadataUniversalIdentifier:
+              'object-metadata-universal-identifier-1',
+            updatedAt: '2024-01-01T00:00:00.000Z',
+            isActive: false,
+          }),
+          getFlatFieldMetadataMock({
+            objectMetadataId: 'object-metadata-id-1',
+            type: FieldMetadataType.TEXT,
+            universalIdentifier: 'universal-identifier-3',
+            id: 'field-id-3',
+            workspaceId: 'workspace-id-1',
+            applicationId: 'application-id-1',
+            createdAt: '2024-01-01T00:00:00.000Z',
+            applicationUniversalIdentifier:
+              'application-universal-identifier-1',
+            objectMetadataUniversalIdentifier:
+              'object-metadata-universal-identifier-1',
+            updatedAt: '2024-01-01T00:00:00.000Z',
+          }),
+        ],
+        metadataName: 'fieldMetadata',
+        buildOptions: {
+          inferDeletionFromMissingEntities: { fieldMetadata: true },
+          isSystemBuild: false,
+        },
+      },
+    },
+    {
+      title:
+        'It should not detect deleted entities when inferDeletionFromMissingEntities is false',
+      context: {
+        from: [
+          getFlatFieldMetadataMock({
+            objectMetadataId: 'object-metadata-id-1',
+            type: FieldMetadataType.TEXT,
+            universalIdentifier: 'universal-identifier-1',
+            id: 'field-id-1',
+            workspaceId: 'workspace-id-1',
+            applicationId: 'application-id-1',
+            createdAt: '2024-01-01T00:00:00.000Z',
+            applicationUniversalIdentifier:
+              'application-universal-identifier-1',
+            objectMetadataUniversalIdentifier:
+              'object-metadata-universal-identifier-1',
+            updatedAt: '2024-01-01T00:00:00.000Z',
+          }),
+        ],
+        to: [],
+        metadataName: 'fieldMetadata',
+        buildOptions: {
+          inferDeletionFromMissingEntities: { fieldMetadata: false },
+          isSystemBuild: false,
+        },
+      },
+    },
+  ] as EachTestingContext<TestContext>[];
+
+  test.each(eachTestingContextFilter(testCases))(
+    '$title',
+    ({ context: { from, to, metadataName, buildOptions } }) => {
+      const result = flatEntityDeletedCreatedUpdatedMatrixDispatcher({
+        from: from as any,
+        to: to as any,
+        metadataName,
+        buildOptions,
+      });
+
+      expect(result).toMatchSnapshot();
+    },
+  );
+});

packages/twenty-server/src/engine/workspace-manager/workspace-migration/universal-flat-entity/utils/add-universal-flat-entity-to-universal-flat-entity-and-related-entity-maps-through-mutation-or-throw.util.ts

@@ -0,0 +1,116 @@
+import { type AllMetadataName } from 'twenty-shared/metadata';
+import { isDefined } from 'twenty-shared/utils';
+
+import {
+  FlatEntityMapsException,
+  FlatEntityMapsExceptionCode,
+} from 'src/engine/metadata-modules/flat-entity/exceptions/flat-entity-maps.exception';
+import { type MetadataUniversalFlatEntityAndRelatedUniversalFlatEntityMaps } from 'src/engine/metadata-modules/flat-entity/types/metadata-related-types.type';
+import { type MetadataRelatedFlatEntityMapsKeys } from 'src/engine/metadata-modules/flat-entity/types/metadata-related-flat-entity-maps-keys.type';
+import { type MetadataUniversalFlatEntity } from 'src/engine/metadata-modules/flat-entity/types/metadata-universal-flat-entity.type';
+import { findFlatEntityByUniversalIdentifierOrThrow } from 'src/engine/metadata-modules/flat-entity/utils/find-flat-entity-by-universal-identifier-or-throw.util';
+import { getMetadataFlatEntityMapsKey } from 'src/engine/metadata-modules/flat-entity/utils/get-metadata-flat-entity-maps-key.util';
+import { ALL_UNIVERSAL_METADATA_RELATIONS } from 'src/engine/workspace-manager/workspace-migration/universal-flat-entity/constants/all-universal-metadata-relations.constant';
+import { type UniversalFlatEntityMaps } from 'src/engine/workspace-manager/workspace-migration/universal-flat-entity/types/universal-flat-entity-maps.type';
+import { addUniversalFlatEntityToUniversalFlatEntityMapsThroughMutationOrThrow } from 'src/engine/workspace-manager/workspace-migration/universal-flat-entity/utils/add-universal-flat-entity-to-universal-flat-entity-maps-through-mutation-or-throw.util';
+import { replaceUniversalFlatEntityInUniversalFlatEntityMapsThroughMutationOrThrow } from 'src/engine/workspace-manager/workspace-migration/universal-flat-entity/utils/replace-universal-flat-entity-in-universal-flat-entity-maps-through-mutation-or-throw.util';
+
+type AddUniversalFlatEntityToUniversalFlatEntityAndRelatedEntityMapsThroughMutationOrThrowArgs<
+  T extends AllMetadataName,
+> = {
+  metadataName: T;
+  universalFlatEntity: MetadataUniversalFlatEntity<T>;
+  universalFlatEntityAndRelatedMapsToMutate: MetadataUniversalFlatEntityAndRelatedUniversalFlatEntityMaps<T>;
+};
+
+export const addUniversalFlatEntityToUniversalFlatEntityAndRelatedEntityMapsThroughMutationOrThrow =
+  <T extends AllMetadataName>({
+    metadataName,
+    universalFlatEntity,
+    universalFlatEntityAndRelatedMapsToMutate,
+  }: AddUniversalFlatEntityToUniversalFlatEntityAndRelatedEntityMapsThroughMutationOrThrowArgs<T>) => {
+    const flatEntityMapsKey = getMetadataFlatEntityMapsKey(metadataName);
+
+    addUniversalFlatEntityToUniversalFlatEntityMapsThroughMutationOrThrow({
+      universalFlatEntity,
+      universalFlatEntityMapsToMutate:
+        universalFlatEntityAndRelatedMapsToMutate[flatEntityMapsKey],
+    });
+
+    const universalManyToOneRelations = Object.values(
+      ALL_UNIVERSAL_METADATA_RELATIONS[metadataName].manyToOne,
+    ) as Array<{
+      metadataName: AllMetadataName;
+      universalFlatEntityForeignKeyAggregator: string | null;
+      universalForeignKey: string;
+    } | null>;
+
+    for (const universalRelation of universalManyToOneRelations) {
+      if (!isDefined(universalRelation)) {
+        continue;
+      }
+
+      const {
+        metadataName: relatedMetadataName,
+        universalFlatEntityForeignKeyAggregator,
+        universalForeignKey,
+      } = universalRelation;
+
+      if (!isDefined(universalFlatEntityForeignKeyAggregator)) {
+        continue;
+      }
+
+      const relatedFlatEntityMapsKey =
+        getMetadataFlatEntityMapsKey(relatedMetadataName);
+
+      const relatedUniversalFlatEntityMaps =
+        universalFlatEntityAndRelatedMapsToMutate[
+          relatedFlatEntityMapsKey as MetadataRelatedFlatEntityMapsKeys<T>
+        ] as UniversalFlatEntityMaps<
+          MetadataUniversalFlatEntity<typeof relatedMetadataName>
+        >;
+
+      const universalForeignKeyValue = (
+        universalFlatEntity as unknown as Record<string, string | undefined>
+      )[universalForeignKey];
+
+      if (!isDefined(universalForeignKeyValue)) {
+        continue;
+      }
+
+      const relatedUniversalFlatEntity =
+        findFlatEntityByUniversalIdentifierOrThrow({
+          universalIdentifier: universalForeignKeyValue,
+          flatEntityMaps: relatedUniversalFlatEntityMaps,
+        });
+
+      if (
+        !Object.prototype.hasOwnProperty.call(
+          relatedUniversalFlatEntity,
+          universalFlatEntityForeignKeyAggregator,
+        )
+      ) {
+        throw new FlatEntityMapsException(
+          `Should never occur, invalid flat entity typing. flat ${relatedMetadataName} should contain ${universalFlatEntityForeignKeyAggregator}`,
+          FlatEntityMapsExceptionCode.ENTITY_MALFORMED,
+        );
+      }
+
+      const updatedRelatedEntity = {
+        ...relatedUniversalFlatEntity,
+        [universalFlatEntityForeignKeyAggregator]: [
+          ...((
+            relatedUniversalFlatEntity as unknown as Record<string, string[]>
+          )[universalFlatEntityForeignKeyAggregator] ?? []),
+          universalFlatEntity.universalIdentifier,
+        ],
+      };
+
+      replaceUniversalFlatEntityInUniversalFlatEntityMapsThroughMutationOrThrow(
+        {
+          universalFlatEntity: updatedRelatedEntity,
+          universalFlatEntityMapsToMutate: relatedUniversalFlatEntityMaps,
+        },
+      );
+    }
+  };

packages/twenty-server/src/engine/workspace-manager/workspace-migration/universal-flat-entity/utils/add-universal-flat-entity-to-universal-flat-entity-maps-or-throw.util.ts

@@ -0,0 +1,43 @@
+import { isDefined } from 'twenty-shared/utils';
+
+import {
+  FlatEntityMapsException,
+  FlatEntityMapsExceptionCode,
+} from 'src/engine/metadata-modules/flat-entity/exceptions/flat-entity-maps.exception';
+import { type SyncableFlatEntity } from 'src/engine/metadata-modules/flat-entity/types/flat-entity-from.type';
+import { type UniversalSyncableFlatEntity } from 'src/engine/workspace-manager/workspace-migration/universal-flat-entity/types/universal-flat-entity-from.type';
+import { type UniversalFlatEntityMaps } from 'src/engine/workspace-manager/workspace-migration/universal-flat-entity/types/universal-flat-entity-maps.type';
+
+type AddUniversalFlatEntityToUniversalFlatEntityMapsOrThrowArgs<
+  T extends SyncableFlatEntity | UniversalSyncableFlatEntity,
+> = {
+  universalFlatEntity: T;
+  universalFlatEntityMaps: UniversalFlatEntityMaps<T>;
+};
+
+export const addUniversalFlatEntityToUniversalFlatEntityMapsOrThrow = <
+  T extends SyncableFlatEntity | UniversalSyncableFlatEntity,
+>({
+  universalFlatEntity,
+  universalFlatEntityMaps,
+}: AddUniversalFlatEntityToUniversalFlatEntityMapsOrThrowArgs<T>): UniversalFlatEntityMaps<T> => {
+  if (
+    isDefined(
+      universalFlatEntityMaps.byUniversalIdentifier[
+        universalFlatEntity.universalIdentifier
+      ],
+    )
+  ) {
+    throw new FlatEntityMapsException(
+      'addUniversalFlatEntityToUniversalFlatEntityMapsOrThrow: flat entity to add already exists',
+      FlatEntityMapsExceptionCode.ENTITY_ALREADY_EXISTS,
+    );
+  }
+
+  return {
+    byUniversalIdentifier: {
+      ...universalFlatEntityMaps.byUniversalIdentifier,
+      [universalFlatEntity.universalIdentifier]: universalFlatEntity,
+    },
+  };
+};

packages/twenty-server/src/engine/workspace-manager/workspace-migration/universal-flat-entity/utils/add-universal-flat-entity-to-universal-flat-entity-maps-through-mutation-or-throw.util.ts

@@ -0,0 +1,39 @@
+import { isDefined } from 'twenty-shared/utils';
+
+import {
+  FlatEntityMapsException,
+  FlatEntityMapsExceptionCode,
+} from 'src/engine/metadata-modules/flat-entity/exceptions/flat-entity-maps.exception';
+import { type SyncableFlatEntity } from 'src/engine/metadata-modules/flat-entity/types/flat-entity-from.type';
+import { type UniversalSyncableFlatEntity } from 'src/engine/workspace-manager/workspace-migration/universal-flat-entity/types/universal-flat-entity-from.type';
+import { type UniversalFlatEntityMaps } from 'src/engine/workspace-manager/workspace-migration/universal-flat-entity/types/universal-flat-entity-maps.type';
+
+type AddUniversalFlatEntityToUniversalFlatEntityMapsThroughMutationOrThrowArgs<
+  T extends SyncableFlatEntity | UniversalSyncableFlatEntity,
+> = {
+  universalFlatEntity: T;
+  universalFlatEntityMapsToMutate: UniversalFlatEntityMaps<T>;
+};
+
+export const addUniversalFlatEntityToUniversalFlatEntityMapsThroughMutationOrThrow =
+  <T extends SyncableFlatEntity | UniversalSyncableFlatEntity>({
+    universalFlatEntity,
+    universalFlatEntityMapsToMutate,
+  }: AddUniversalFlatEntityToUniversalFlatEntityMapsThroughMutationOrThrowArgs<T>): void => {
+    if (
+      isDefined(
+        universalFlatEntityMapsToMutate.byUniversalIdentifier[
+          universalFlatEntity.universalIdentifier
+        ],
+      )
+    ) {
+      throw new FlatEntityMapsException(
+        'addUniversalFlatEntityToUniversalFlatEntityMapsThroughMutationOrThrow: flat entity to add already exists',
+        FlatEntityMapsExceptionCode.ENTITY_ALREADY_EXISTS,
+      );
+    }
+
+    universalFlatEntityMapsToMutate.byUniversalIdentifier[
+      universalFlatEntity.universalIdentifier
+    ] = universalFlatEntity;
+  };

packages/twenty-server/src/engine/workspace-manager/workspace-migration/universal-flat-entity/utils/compare-two-universal-flat-entity.util.ts

@@ -0,0 +1,77 @@
+import diff from 'microdiff';
+import { type AllMetadataName } from 'twenty-shared/metadata';
+import { type FromTo } from 'twenty-shared/types';
+import { parseJson } from 'twenty-shared/utils';
+
+import { ALL_UNIVERSAL_FLAT_ENTITY_PROPERTIES_TO_COMPARE_AND_STRINGIFY } from 'src/engine/metadata-modules/flat-entity/constant/all-universal-flat-entity-properties-to-compare-and-stringify.constant';
+import { type MetadataUniversalFlatEntity } from 'src/engine/metadata-modules/flat-entity/types/metadata-universal-flat-entity.type';
+import { type MetadataUniversalFlatEntityPropertiesToStringify } from 'src/engine/workspace-manager/workspace-migration/universal-flat-entity/types/metadata-universal-flat-entity-properties-to-stringify.type';
+import { type UniversalFlatEntityUpdate } from 'src/engine/workspace-manager/workspace-migration/universal-flat-entity/types/universal-flat-entity-update.type';
+import { transformUniversalFlatEntityForComparison } from 'src/engine/workspace-manager/workspace-migration/universal-flat-entity/utils/transform-universal-flat-entity-for-comparison.util';
+
+type CompareTwoUniversalFlatEntityArgs<T extends AllMetadataName> = FromTo<
+  MetadataUniversalFlatEntity<T>,
+  'universalFlatEntity'
+> & { metadataName: T };
+
+export const compareTwoFlatEntity = <T extends AllMetadataName>({
+  fromUniversalFlatEntity,
+  toUniversalFlatEntity,
+  metadataName,
+}: CompareTwoUniversalFlatEntityArgs<T>):
+  | UniversalFlatEntityUpdate<T>
+  | undefined => {
+  const { propertiesToStringify, propertiesToCompare } =
+    ALL_UNIVERSAL_FLAT_ENTITY_PROPERTIES_TO_COMPARE_AND_STRINGIFY[metadataName];
+
+  const [transformedFromUniversalFlatEntity, transformedToUniversalFlatEntity] =
+    [fromUniversalFlatEntity, toUniversalFlatEntity].map(
+      (universalFlatEntity) =>
+        transformUniversalFlatEntityForComparison({
+          universalFlatEntity,
+          propertiesToCompare,
+          propertiesToStringify,
+        }),
+    );
+
+  const flatEntityDifferences = diff(
+    transformedFromUniversalFlatEntity,
+    transformedToUniversalFlatEntity,
+  );
+
+  if (flatEntityDifferences.length === 0) {
+    return undefined;
+  }
+
+  const initialAccumulator: UniversalFlatEntityUpdate<T> = {};
+
+  return flatEntityDifferences.reduce((accumulator, difference) => {
+    switch (difference.type) {
+      case 'CHANGE': {
+        const { path, value } = difference;
+        const property = path[0];
+        const isJsonb = propertiesToStringify.includes(
+          property as MetadataUniversalFlatEntityPropertiesToStringify<T>,
+        );
+
+        if (isJsonb) {
+          return {
+            ...accumulator,
+            [property]: parseJson(value),
+          };
+        }
+
+        return {
+          ...accumulator,
+          [property]: value,
+        };
+      }
+      case 'CREATE':
+      case 'REMOVE':
+      default: {
+        // Should never occur, we should only provide null never undefined and so on
+        return accumulator;
+      }
+    }
+  }, initialAccumulator);
+};

packages/twenty-server/src/engine/workspace-manager/workspace-migration/universal-flat-entity/utils/delete-universal-flat-entity-from-universal-flat-entity-and-related-entity-maps-through-mutation-or-throw.util.ts

@@ -0,0 +1,125 @@
+import { type AllMetadataName } from 'twenty-shared/metadata';
+import { isDefined } from 'twenty-shared/utils';
+
+import {
+  FlatEntityMapsException,
+  FlatEntityMapsExceptionCode,
+} from 'src/engine/metadata-modules/flat-entity/exceptions/flat-entity-maps.exception';
+import { type MetadataRelatedFlatEntityMapsKeys } from 'src/engine/metadata-modules/flat-entity/types/metadata-related-flat-entity-maps-keys.type';
+import { type MetadataUniversalFlatEntityAndRelatedUniversalFlatEntityMaps } from 'src/engine/metadata-modules/flat-entity/types/metadata-related-types.type';
+import { type MetadataUniversalFlatEntity } from 'src/engine/metadata-modules/flat-entity/types/metadata-universal-flat-entity.type';
+import { findFlatEntityByUniversalIdentifier } from 'src/engine/metadata-modules/flat-entity/utils/find-flat-entity-by-universal-identifier.util';
+import { getMetadataFlatEntityMapsKey } from 'src/engine/metadata-modules/flat-entity/utils/get-metadata-flat-entity-maps-key.util';
+import { ALL_UNIVERSAL_METADATA_RELATIONS } from 'src/engine/workspace-manager/workspace-migration/universal-flat-entity/constants/all-universal-metadata-relations.constant';
+import { type UniversalFlatEntityMaps } from 'src/engine/workspace-manager/workspace-migration/universal-flat-entity/types/universal-flat-entity-maps.type';
+import { deleteUniversalFlatEntityFromUniversalFlatEntityMapsThroughMutationOrThrow } from 'src/engine/workspace-manager/workspace-migration/universal-flat-entity/utils/delete-universal-flat-entity-from-universal-flat-entity-maps-through-mutation-or-throw.util';
+import { replaceUniversalFlatEntityInUniversalFlatEntityMapsThroughMutationOrThrow } from 'src/engine/workspace-manager/workspace-migration/universal-flat-entity/utils/replace-universal-flat-entity-in-universal-flat-entity-maps-through-mutation-or-throw.util';
+
+type DeleteUniversalFlatEntityFromUniversalFlatEntityAndRelatedEntityMapsThroughMutationOrThrowArgs<
+  T extends AllMetadataName,
+> = {
+  metadataName: T;
+  universalFlatEntity: MetadataUniversalFlatEntity<T>;
+  universalFlatEntityAndRelatedMapsToMutate: MetadataUniversalFlatEntityAndRelatedUniversalFlatEntityMaps<T>;
+};
+
+export const deleteUniversalFlatEntityFromUniversalFlatEntityAndRelatedEntityMapsThroughMutationOrThrow =
+  <T extends AllMetadataName>({
+    metadataName,
+    universalFlatEntity,
+    universalFlatEntityAndRelatedMapsToMutate,
+  }: DeleteUniversalFlatEntityFromUniversalFlatEntityAndRelatedEntityMapsThroughMutationOrThrowArgs<T>) => {
+    const flatEntityMapsKey = getMetadataFlatEntityMapsKey(metadataName);
+
+    deleteUniversalFlatEntityFromUniversalFlatEntityMapsThroughMutationOrThrow({
+      universalIdentifierToDelete: universalFlatEntity.universalIdentifier,
+      universalFlatEntityMapsToMutate:
+        universalFlatEntityAndRelatedMapsToMutate[
+          flatEntityMapsKey
+        ] as UniversalFlatEntityMaps<
+          MetadataUniversalFlatEntity<typeof metadataName>
+        >,
+    });
+
+    const universalManyToOneRelations = Object.values(
+      ALL_UNIVERSAL_METADATA_RELATIONS[metadataName].manyToOne,
+    ) as Array<{
+      metadataName: AllMetadataName;
+      universalFlatEntityForeignKeyAggregator: string | null;
+      universalForeignKey: string;
+    } | null>;
+
+    for (const universalRelation of universalManyToOneRelations) {
+      if (!isDefined(universalRelation)) {
+        continue;
+      }
+
+      const {
+        metadataName: relatedMetadataName,
+        universalFlatEntityForeignKeyAggregator,
+        universalForeignKey,
+      } = universalRelation;
+
+      if (!isDefined(universalFlatEntityForeignKeyAggregator)) {
+        continue;
+      }
+
+      const relatedFlatEntityMapsKey =
+        getMetadataFlatEntityMapsKey(relatedMetadataName);
+
+      const relatedUniversalFlatEntityMaps =
+        universalFlatEntityAndRelatedMapsToMutate[
+          relatedFlatEntityMapsKey as MetadataRelatedFlatEntityMapsKeys<T>
+        ] as UniversalFlatEntityMaps<
+          MetadataUniversalFlatEntity<typeof relatedMetadataName>
+        >;
+
+      const universalForeignKeyValue = (
+        universalFlatEntity as unknown as Record<string, string | undefined>
+      )[universalForeignKey];
+
+      if (!isDefined(universalForeignKeyValue)) {
+        continue;
+      }
+
+      const relatedUniversalFlatEntity = findFlatEntityByUniversalIdentifier({
+        universalIdentifier: universalForeignKeyValue,
+        flatEntityMaps: relatedUniversalFlatEntityMaps,
+      });
+
+      if (!isDefined(relatedUniversalFlatEntity)) {
+        continue;
+      }
+
+      if (
+        !Object.prototype.hasOwnProperty.call(
+          relatedUniversalFlatEntity,
+          universalFlatEntityForeignKeyAggregator,
+        )
+      ) {
+        throw new FlatEntityMapsException(
+          `Should never occur, invalid flat entity typing. flat ${relatedMetadataName} should contain ${universalFlatEntityForeignKeyAggregator}`,
+          FlatEntityMapsExceptionCode.ENTITY_MALFORMED,
+        );
+      }
+
+      const updatedRelatedEntity = {
+        ...relatedUniversalFlatEntity,
+        [universalFlatEntityForeignKeyAggregator]: (
+          (relatedUniversalFlatEntity as unknown as Record<string, string[]>)[
+            universalFlatEntityForeignKeyAggregator
+          ] ?? []
+        ).filter(
+          (universalIdentifier) =>
+            universalIdentifier !== universalFlatEntity.universalIdentifier,
+        ),
+      };
+
+      replaceUniversalFlatEntityInUniversalFlatEntityMapsThroughMutationOrThrow(
+        {
+          universalFlatEntity: updatedRelatedEntity,
+          universalFlatEntityMapsToMutate: relatedUniversalFlatEntityMaps,
+        },
+      );
+    }
+  };

packages/twenty-server/src/engine/workspace-manager/workspace-migration/universal-flat-entity/utils/delete-universal-flat-entity-from-universal-flat-entity-maps-through-mutation-or-throw.util.ts

@@ -0,0 +1,38 @@
+import { isDefined } from 'twenty-shared/utils';
+
+import {
+  FlatEntityMapsException,
+  FlatEntityMapsExceptionCode,
+} from 'src/engine/metadata-modules/flat-entity/exceptions/flat-entity-maps.exception';
+import { type SyncableFlatEntity } from 'src/engine/metadata-modules/flat-entity/types/flat-entity-from.type';
+import { type UniversalSyncableFlatEntity } from 'src/engine/workspace-manager/workspace-migration/universal-flat-entity/types/universal-flat-entity-from.type';
+import { type UniversalFlatEntityMaps } from 'src/engine/workspace-manager/workspace-migration/universal-flat-entity/types/universal-flat-entity-maps.type';
+
+export type DeleteUniversalFlatEntityFromUniversalFlatEntityMapsThroughMutationOrThrowArgs<
+  T extends SyncableFlatEntity | UniversalSyncableFlatEntity,
+> = {
+  universalIdentifierToDelete: string;
+  universalFlatEntityMapsToMutate: UniversalFlatEntityMaps<T>;
+};
+
+export const deleteUniversalFlatEntityFromUniversalFlatEntityMapsThroughMutationOrThrow =
+  <T extends SyncableFlatEntity | UniversalSyncableFlatEntity>({
+    universalFlatEntityMapsToMutate,
+    universalIdentifierToDelete,
+  }: DeleteUniversalFlatEntityFromUniversalFlatEntityMapsThroughMutationOrThrowArgs<T>): void => {
+    const entityToDelete =
+      universalFlatEntityMapsToMutate.byUniversalIdentifier[
+        universalIdentifierToDelete
+      ];
+
+    if (!isDefined(entityToDelete)) {
+      throw new FlatEntityMapsException(
+        'deleteUniversalFlatEntityFromUniversalFlatEntityMapsThroughMutationOrThrow: entity to delete not found',
+        FlatEntityMapsExceptionCode.ENTITY_NOT_FOUND,
+      );
+    }
+
+    delete universalFlatEntityMapsToMutate.byUniversalIdentifier[
+      universalIdentifierToDelete
+    ];
+  };

packages/twenty-server/src/engine/workspace-manager/workspace-migration/universal-flat-entity/utils/replace-universal-flat-entity-in-universal-flat-entity-maps-through-mutation-or-throw.util.ts

@@ -0,0 +1,28 @@
+import { type SyncableFlatEntity } from 'src/engine/metadata-modules/flat-entity/types/flat-entity-from.type';
+import { type UniversalSyncableFlatEntity } from 'src/engine/workspace-manager/workspace-migration/universal-flat-entity/types/universal-flat-entity-from.type';
+import { type UniversalFlatEntityMaps } from 'src/engine/workspace-manager/workspace-migration/universal-flat-entity/types/universal-flat-entity-maps.type';
+import { addUniversalFlatEntityToUniversalFlatEntityMapsThroughMutationOrThrow } from 'src/engine/workspace-manager/workspace-migration/universal-flat-entity/utils/add-universal-flat-entity-to-universal-flat-entity-maps-through-mutation-or-throw.util';
+import { deleteUniversalFlatEntityFromUniversalFlatEntityMapsThroughMutationOrThrow } from 'src/engine/workspace-manager/workspace-migration/universal-flat-entity/utils/delete-universal-flat-entity-from-universal-flat-entity-maps-through-mutation-or-throw.util';
+
+export type ReplaceUniversalFlatEntityInUniversalFlatEntityMapsThroughMutationOrThrowArgs<
+  T extends SyncableFlatEntity | UniversalSyncableFlatEntity,
+> = {
+  universalFlatEntity: T;
+  universalFlatEntityMapsToMutate: UniversalFlatEntityMaps<T>;
+};
+
+export const replaceUniversalFlatEntityInUniversalFlatEntityMapsThroughMutationOrThrow =
+  <T extends SyncableFlatEntity | UniversalSyncableFlatEntity>({
+    universalFlatEntity,
+    universalFlatEntityMapsToMutate,
+  }: ReplaceUniversalFlatEntityInUniversalFlatEntityMapsThroughMutationOrThrowArgs<T>): void => {
+    deleteUniversalFlatEntityFromUniversalFlatEntityMapsThroughMutationOrThrow({
+      universalFlatEntityMapsToMutate,
+      universalIdentifierToDelete: universalFlatEntity.universalIdentifier,
+    });
+
+    addUniversalFlatEntityToUniversalFlatEntityMapsThroughMutationOrThrow({
+      universalFlatEntity,
+      universalFlatEntityMapsToMutate,
+    });
+  };

packages/twenty-server/src/engine/workspace-manager/workspace-migration/universal-flat-entity/utils/resolve-universal-relation-identifiers-to-ids.util.ts

@@ -0,0 +1,156 @@
+import { t } from '@lingui/core/macro';
+import { type AllMetadataName } from 'twenty-shared/metadata';
+import { isDefined } from 'twenty-shared/utils';
+
+import { type MetadataManyToOneRelationConfiguration } from 'src/engine/metadata-modules/flat-entity/constant/all-metadata-relations.constant';
+import {
+  FlatEntityMapsException,
+  FlatEntityMapsExceptionCode,
+} from 'src/engine/metadata-modules/flat-entity/exceptions/flat-entity-maps.exception';
+import { type AllFlatEntityMaps } from 'src/engine/metadata-modules/flat-entity/types/all-flat-entity-maps.type';
+import { type ExtractEntityManyToOneEntityRelationProperties } from 'src/engine/metadata-modules/flat-entity/types/extract-entity-many-to-one-entity-relation-properties.type';
+import { type MetadataEntity } from 'src/engine/metadata-modules/flat-entity/types/metadata-entity.type';
+import { type MetadataFlatEntity } from 'src/engine/metadata-modules/flat-entity/types/metadata-flat-entity.type';
+import { type MetadataToFlatEntityMapsKey } from 'src/engine/metadata-modules/flat-entity/types/metadata-to-flat-entity-maps-key';
+import { findFlatEntityByUniversalIdentifier } from 'src/engine/metadata-modules/flat-entity/utils/find-flat-entity-by-universal-identifier.util';
+import { getMetadataFlatEntityMapsKey } from 'src/engine/metadata-modules/flat-entity/utils/get-metadata-flat-entity-maps-key.util';
+import {
+  ALL_UNIVERSAL_METADATA_RELATIONS,
+  type ToUniversalMetadataManyToOneRelationConfiguration,
+} from 'src/engine/workspace-manager/workspace-migration/universal-flat-entity/constants/all-universal-metadata-relations.constant';
+
+type UniversalManyToOneConfig<T extends AllMetadataName> =
+  (typeof ALL_UNIVERSAL_METADATA_RELATIONS)[T]['manyToOne'];
+
+type ExtractUniversalForeignKeys<T> = {
+  [K in keyof T]: T[K] extends { universalForeignKey: infer UFK } ? UFK : never;
+}[keyof T];
+
+type MetadataUniversalManyToOneJoinColumn<T extends AllMetadataName> =
+  ExtractUniversalForeignKeys<UniversalManyToOneConfig<T>>;
+
+type TargetMetadataNamesForUniversalForeignKeys<
+  T extends AllMetadataName,
+  TProvidedKeys extends string,
+> = {
+  [K in keyof UniversalManyToOneConfig<T>]: UniversalManyToOneConfig<T>[K] extends {
+    universalForeignKey: infer _UFK extends TProvidedKeys;
+    metadataName: infer MN extends AllMetadataName;
+  }
+    ? MN
+    : never;
+}[keyof UniversalManyToOneConfig<T>];
+
+type RequiredFlatEntityMapsForUniversalForeignKeys<
+  T extends AllMetadataName,
+  TProvidedKeys extends string,
+> = Pick<
+  AllFlatEntityMaps,
+  MetadataToFlatEntityMapsKey<
+    TargetMetadataNamesForUniversalForeignKeys<T, TProvidedKeys>
+  >
+>;
+
+type ResolvedForeignKeyIds<
+  T extends AllMetadataName,
+  TProvidedKeys extends Extract<
+    MetadataUniversalManyToOneJoinColumn<T>,
+    string
+  >,
+> = {
+  [K in keyof UniversalManyToOneConfig<T> as UniversalManyToOneConfig<T>[K] extends {
+    universalForeignKey: infer UFK extends string;
+    foreignKey: infer FK extends string;
+  }
+    ? UFK extends TProvidedKeys
+      ? FK
+      : never
+    : never]: UniversalManyToOneConfig<T>[K] extends { isNullable: true }
+    ? string | null
+    : string;
+};
+
+export const resolveUniversalRelationIdentifiersToIds = <
+  T extends AllMetadataName,
+  TProvidedKeys extends Extract<
+    MetadataUniversalManyToOneJoinColumn<T>,
+    string
+  > = Extract<MetadataUniversalManyToOneJoinColumn<T>, string>,
+>({
+  metadataName,
+  universalForeignKeyValues,
+  flatEntityMaps,
+}: {
+  metadataName: T;
+  universalForeignKeyValues: Record<TProvidedKeys, string | null | undefined>;
+  flatEntityMaps: RequiredFlatEntityMapsForUniversalForeignKeys<
+    T,
+    TProvidedKeys
+  >;
+}): ResolvedForeignKeyIds<T, TProvidedKeys> => {
+  const relations = ALL_UNIVERSAL_METADATA_RELATIONS[metadataName].manyToOne;
+  const result: Record<string, string | null> = {};
+
+  for (const relation of Object.values(
+    relations,
+  ) as ToUniversalMetadataManyToOneRelationConfiguration<
+    MetadataManyToOneRelationConfiguration<
+      T,
+      ExtractEntityManyToOneEntityRelationProperties<MetadataEntity<T>>
+    >
+  >[]) {
+    if (!isDefined(relation)) {
+      continue;
+    }
+
+    const {
+      foreignKey,
+      universalForeignKey,
+      metadataName: targetMetadataName,
+      isNullable,
+    } = relation;
+
+    if (
+      !Object.prototype.hasOwnProperty.call(
+        universalForeignKeyValues,
+        universalForeignKey,
+      )
+    ) {
+      continue;
+    }
+
+    const universalIdentifierValue =
+      universalForeignKeyValues[
+        universalForeignKey as keyof typeof universalForeignKeyValues
+      ];
+
+    const mapsKey = getMetadataFlatEntityMapsKey(
+      targetMetadataName,
+    ) as keyof RequiredFlatEntityMapsForUniversalForeignKeys<T, TProvidedKeys>;
+    const targetFlatEntityMaps = flatEntityMaps[mapsKey];
+
+    if (isNullable && !isDefined(universalIdentifierValue)) {
+      result[foreignKey] = null;
+
+      continue;
+    }
+
+    const targetEntity = findFlatEntityByUniversalIdentifier<
+      MetadataFlatEntity<typeof targetMetadataName>
+    >({
+      flatEntityMaps: targetFlatEntityMaps,
+      universalIdentifier: universalIdentifierValue as string,
+    });
+
+    if (!isDefined(targetEntity)) {
+      throw new FlatEntityMapsException(
+        t`Could not find ${targetMetadataName} for given ${universalForeignKey}`,
+        FlatEntityMapsExceptionCode.ENTITY_NOT_FOUND,
+      );
+    }
+
+    result[foreignKey] = targetEntity.id;
+  }
+
+  return result as ResolvedForeignKeyIds<T, TProvidedKeys>;
+};

packages/twenty-server/src/engine/workspace-manager/workspace-migration/universal-flat-entity/utils/resolve-universal-update-relation-identifiers-to-ids.util.ts

@@ -0,0 +1,95 @@
+import { type AllMetadataName } from 'twenty-shared/metadata';
+import { isDefined } from 'twenty-shared/utils';
+
+import { type MetadataManyToOneRelationConfiguration } from 'src/engine/metadata-modules/flat-entity/constant/all-metadata-relations.constant';
+import {
+  FlatEntityMapsException,
+  FlatEntityMapsExceptionCode,
+} from 'src/engine/metadata-modules/flat-entity/exceptions/flat-entity-maps.exception';
+import { type AllFlatEntityMaps } from 'src/engine/metadata-modules/flat-entity/types/all-flat-entity-maps.type';
+import { type ExtractEntityManyToOneEntityRelationProperties } from 'src/engine/metadata-modules/flat-entity/types/extract-entity-many-to-one-entity-relation-properties.type';
+import { type FlatEntityUpdate } from 'src/engine/metadata-modules/flat-entity/types/flat-entity-update.type';
+import { type MetadataEntity } from 'src/engine/metadata-modules/flat-entity/types/metadata-entity.type';
+import { type MetadataFlatEntity } from 'src/engine/metadata-modules/flat-entity/types/metadata-flat-entity.type';
+import { findFlatEntityByUniversalIdentifier } from 'src/engine/metadata-modules/flat-entity/utils/find-flat-entity-by-universal-identifier.util';
+import { getMetadataFlatEntityMapsKey } from 'src/engine/metadata-modules/flat-entity/utils/get-metadata-flat-entity-maps-key.util';
+import {
+  ALL_UNIVERSAL_METADATA_RELATIONS,
+  type ToUniversalMetadataManyToOneRelationConfiguration,
+} from 'src/engine/workspace-manager/workspace-migration/universal-flat-entity/constants/all-universal-metadata-relations.constant';
+import { type UniversalFlatEntityUpdate } from 'src/engine/workspace-manager/workspace-migration/universal-flat-entity/types/universal-flat-entity-update.type';
+
+export const resolveUniversalUpdateRelationIdentifiersToIds = <
+  T extends AllMetadataName,
+>({
+  metadataName,
+  universalUpdate,
+  allFlatEntityMaps,
+}: {
+  metadataName: T;
+  universalUpdate: UniversalFlatEntityUpdate<T>;
+  allFlatEntityMaps: AllFlatEntityMaps;
+}): FlatEntityUpdate<T> => {
+  const relations = ALL_UNIVERSAL_METADATA_RELATIONS[metadataName].manyToOne;
+  const result: Record<string, unknown> = { ...universalUpdate };
+
+  for (const relation of Object.values(
+    relations,
+  ) as ToUniversalMetadataManyToOneRelationConfiguration<
+    MetadataManyToOneRelationConfiguration<
+      T,
+      ExtractEntityManyToOneEntityRelationProperties<MetadataEntity<T>>
+    >
+  >[]) {
+    if (!isDefined(relation)) {
+      continue;
+    }
+
+    const {
+      foreignKey,
+      universalForeignKey,
+      metadataName: targetMetadataName,
+    } = relation;
+
+    if (!Object.prototype.hasOwnProperty.call(result, universalForeignKey)) {
+      continue;
+    }
+
+    const universalIdentifierValue = result[universalForeignKey] as
+      | string
+      | null
+      | undefined;
+
+    delete result[universalForeignKey];
+
+    if (!isDefined(universalIdentifierValue)) {
+      if (universalIdentifierValue === null) {
+        result[foreignKey] = null;
+      }
+
+      continue;
+    }
+
+    const mapsKey = getMetadataFlatEntityMapsKey(targetMetadataName);
+
+    const targetFlatEntityMaps = allFlatEntityMaps[mapsKey];
+
+    const targetEntity = findFlatEntityByUniversalIdentifier<
+      MetadataFlatEntity<typeof targetMetadataName>
+    >({
+      flatEntityMaps: targetFlatEntityMaps,
+      universalIdentifier: universalIdentifierValue,
+    });
+
+    if (!isDefined(targetEntity)) {
+      throw new FlatEntityMapsException(
+        `Could not resolve ${universalForeignKey} to ${foreignKey}: no ${targetMetadataName} found for universal identifier ${universalIdentifierValue}`,
+        FlatEntityMapsExceptionCode.ENTITY_NOT_FOUND,
+      );
+    }
+
+    result[foreignKey] = targetEntity.id;
+  }
+
+  return result as FlatEntityUpdate<T>;
+};

packages/twenty-server/src/engine/workspace-manager/workspace-migration/universal-flat-entity/utils/sanitize-universal-flat-entity-update.util.ts

@@ -0,0 +1,30 @@
+import { type AllMetadataName } from 'twenty-shared/metadata';
+
+import { ALL_UNIVERSAL_FLAT_ENTITY_PROPERTIES_TO_COMPARE_AND_STRINGIFY } from 'src/engine/metadata-modules/flat-entity/constant/all-universal-flat-entity-properties-to-compare-and-stringify.constant';
+import { type UniversalFlatEntityUpdate } from 'src/engine/workspace-manager/workspace-migration/universal-flat-entity/types/universal-flat-entity-update.type';
+
+export const sanitizeUniversalFlatEntityUpdate = <T extends AllMetadataName>({
+  flatEntityUpdate,
+  metadataName,
+}: {
+  flatEntityUpdate: UniversalFlatEntityUpdate<T>;
+  metadataName: T;
+}): UniversalFlatEntityUpdate<T> => {
+  const { propertiesToCompare } =
+    ALL_UNIVERSAL_FLAT_ENTITY_PROPERTIES_TO_COMPARE_AND_STRINGIFY[metadataName];
+
+  const initialAccumulator: UniversalFlatEntityUpdate<T> = {};
+
+  return propertiesToCompare.reduce((accumulator, property) => {
+    const updatedValue = flatEntityUpdate[property];
+
+    if (updatedValue === undefined) {
+      return accumulator;
+    }
+
+    return {
+      ...accumulator,
+      [property]: updatedValue,
+    };
+  }, initialAccumulator);
+};

packages/twenty-server/src/engine/workspace-manager/workspace-migration/universal-flat-entity/utils/transform-universal-flat-entity-for-comparison.util.ts

@@ -0,0 +1,43 @@
+import { type AllMetadataName } from 'twenty-shared/metadata';
+
+import { type MetadataUniversalFlatEntity } from 'src/engine/metadata-modules/flat-entity/types/metadata-universal-flat-entity.type';
+import { orderObjectProperties } from 'src/engine/metadata-modules/flat-entity/utils/order-object-properties.util';
+import { type MetadataUniversalFlatEntityPropertiesToCompare } from 'src/engine/workspace-manager/workspace-migration/universal-flat-entity/types/metadata-universal-flat-entity-properties-to-compare.type';
+import { type MetadataUniversalFlatEntityPropertiesToStringify } from 'src/engine/workspace-manager/workspace-migration/universal-flat-entity/types/metadata-universal-flat-entity-properties-to-stringify.type';
+
+export function transformUniversalFlatEntityForComparison<
+  T extends AllMetadataName,
+>({
+  universalFlatEntity,
+  propertiesToCompare,
+  propertiesToStringify,
+}: {
+  universalFlatEntity: MetadataUniversalFlatEntity<T>;
+  propertiesToCompare: MetadataUniversalFlatEntityPropertiesToCompare<T>[];
+  propertiesToStringify: MetadataUniversalFlatEntityPropertiesToStringify<T>[];
+}) {
+  return propertiesToCompare.reduce(
+    (flatEntityAccumulator, propertyToCompare) => {
+      const currentValue = universalFlatEntity[propertyToCompare];
+
+      if (
+        propertiesToStringify.includes(
+          propertyToCompare as MetadataUniversalFlatEntityPropertiesToStringify<T>,
+        )
+      ) {
+        const orderedValue = orderObjectProperties(currentValue);
+
+        return {
+          ...flatEntityAccumulator,
+          [propertyToCompare]: JSON.stringify(orderedValue),
+        };
+      }
+
+      return {
+        ...flatEntityAccumulator,
+        [propertyToCompare]: currentValue,
+      };
+    },
+    {},
+  );
+}

packages/twenty-server/src/engine/workspace-manager/workspace-migration/universal-flat-entity/utils/universal-flat-entity-deleted-created-updated-matrix-dispatcher.util.ts

@@ -0,0 +1,102 @@
+import { type AllMetadataName } from 'twenty-shared/metadata';
+import { type FromTo } from 'twenty-shared/types';
+import { isDefined } from 'twenty-shared/utils';
+
+import { type MetadataUniversalFlatEntity } from 'src/engine/metadata-modules/flat-entity/types/metadata-universal-flat-entity.type';
+import { type MetadataUniversalFlatEntityMaps } from 'src/engine/workspace-manager/workspace-migration/universal-flat-entity/types/metadata-universal-flat-entity-maps.type';
+import { type UniversalFlatEntityUpdate } from 'src/engine/workspace-manager/workspace-migration/universal-flat-entity/types/universal-flat-entity-update.type';
+import { compareTwoFlatEntity } from 'src/engine/workspace-manager/workspace-migration/universal-flat-entity/utils/compare-two-universal-flat-entity.util';
+import { addUniversalFlatEntityToUniversalFlatEntityMapsThroughMutationOrThrow } from 'src/engine/workspace-manager/workspace-migration/utils/add-universal-flat-entity-to-universal-flat-entity-maps-through-mutation-or-throw.util';
+import { shouldInferDeletionFromMissingEntities } from 'src/engine/workspace-manager/workspace-migration/utils/should-infer-deletion-from-missing-entities.util';
+import { type WorkspaceMigrationBuilderOptions } from 'src/engine/workspace-manager/workspace-migration/workspace-migration-builder/types/workspace-migration-builder-options.type';
+
+export type DeletedCreatedUpdatedMatrix<T extends AllMetadataName> = {
+  createdFlatEntityMaps: MetadataUniversalFlatEntityMaps<T>;
+  deletedFlatEntityMaps: MetadataUniversalFlatEntityMaps<T>;
+  updatedFlatEntityMaps: {
+    byUniversalIdentifier: Record<
+      string,
+      {
+        update: UniversalFlatEntityUpdate<T>;
+      }
+    >;
+  };
+};
+
+export type UniversalIdentifierItem = {
+  universalIdentifier: string;
+};
+
+type FlatEntityDeletedCreatedUpdatedMatrixDispatcherArgs<
+  T extends AllMetadataName,
+> = FromTo<MetadataUniversalFlatEntity<T>[]> & {
+  metadataName: T;
+  buildOptions: WorkspaceMigrationBuilderOptions;
+};
+
+export const flatEntityDeletedCreatedUpdatedMatrixDispatcher = <
+  T extends AllMetadataName,
+>({
+  from,
+  to,
+  metadataName,
+  buildOptions,
+}: FlatEntityDeletedCreatedUpdatedMatrixDispatcherArgs<T>): DeletedCreatedUpdatedMatrix<T> => {
+  const initialDispatcher: DeletedCreatedUpdatedMatrix<T> = {
+    createdFlatEntityMaps: { byUniversalIdentifier: {} },
+    deletedFlatEntityMaps: { byUniversalIdentifier: {} },
+    updatedFlatEntityMaps: { byUniversalIdentifier: {} },
+  };
+
+  const fromMap = new Map(from.map((obj) => [obj.universalIdentifier, obj]));
+  const toMap = new Map(to.map((obj) => [obj.universalIdentifier, obj]));
+
+  if (shouldInferDeletionFromMissingEntities({ buildOptions, metadataName })) {
+    for (const [universalIdentifier, fromEntity] of fromMap) {
+      if (toMap.has(universalIdentifier)) {
+        continue;
+      }
+
+      addUniversalFlatEntityToUniversalFlatEntityMapsThroughMutationOrThrow({
+        universalFlatEntity: fromEntity,
+        universalFlatEntityMapsToMutate:
+          initialDispatcher.deletedFlatEntityMaps,
+      });
+    }
+  }
+
+  for (const [universalIdentifier, toFlatEntity] of toMap) {
+    if (fromMap.has(universalIdentifier)) {
+      continue;
+    }
+    addUniversalFlatEntityToUniversalFlatEntityMapsThroughMutationOrThrow({
+      universalFlatEntity: toFlatEntity,
+      universalFlatEntityMapsToMutate: initialDispatcher.createdFlatEntityMaps,
+    });
+  }
+
+  for (const [universalIdentifier, fromUniversalFlatEntity] of fromMap) {
+    const toUniversalFlatEntity = toMap.get(universalIdentifier);
+
+    if (!isDefined(toUniversalFlatEntity)) {
+      continue;
+    }
+    const update = compareTwoFlatEntity({
+      fromUniversalFlatEntity,
+      toUniversalFlatEntity,
+      metadataName,
+    });
+
+    if (!isDefined(update)) {
+      continue;
+    }
+
+    initialDispatcher.updatedFlatEntityMaps.byUniversalIdentifier[
+      fromUniversalFlatEntity.universalIdentifier
+    ] = {
+      update,
+    };
+  }
+
+  return initialDispatcher;
+};