houseassassin/twenty
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

ObjectMetadata and FieldMetadata agnostic workspace migration runner (#17572)

Browse Source 
# Introduction
Important note: This PR officially deprecates the `standardId`, about to
drop col and entity property after this has been merged

Important note2: Haven't updated the optimistic tool to also update the
universal identifier aggregators only the ids one, they should not be
consumed in the runner context -> need to improve typing or either the
optimistic tooling

In this PR we're introducing all the devxp allowing future metadata
incremental universal migration -> this has an impact on all existing
metadata actions handler ( explaining its size )
This PR also introduce workspace agnostic create update actions runner
for both field and object metadata in order to battle test the described
above devxp

Noting that these two metadata are the most complex to handle

Notes:
- A workspace migration is now highly bind to a
`applicationUniversalIdentifier`. Though we don't strictly validate
application scope for the moment

## Next
Migrate both object and field builder to universal comparison

## Universal Actions vs Flat Actions Architecture

### Concept

The migration system uses a two-phase action model:

1. **Universal Actions** - Actions defined using `universalIdentifier`
(stable, portable identifiers like `standardId` + `applicationId`)
2. **Flat Actions** - Actions defined using database `entityId` (UUIDs
specific to a workspace)

### Why This Separation?

- **Universal actions are portable**: They can be serialized, stored,
and replayed across different workspaces
- **Flat actions are executable**: They contain the actual database IDs
needed to perform operations
- **Decoupling**: The builder produces universal actions; the runner
transpiles them to flat actions at execution time

### Transpiler Pattern

Each action handler must implement
`transpileUniversalActionToFlatAction()`:

```typescript
@Injectable()
export class CreateFieldActionHandlerService extends WorkspaceMigrationRunnerActionHandler(
  'create',
  'fieldMetadata',
) {
  override async transpileUniversalActionToFlatAction(
    context: WorkspaceMigrationActionRunnerArgs<UniversalCreateFieldAction>,
  ): Promise<FlatCreateFieldAction> {
    // Resolve universal identifiers to database IDs
    const flatObjectMetadata = findFlatEntityByUniversalIdentifierOrThrow({
      flatEntityMaps: allFlatEntityMaps.flatObjectMetadataMaps,
      universalIdentifier: action.objectMetadataUniversalIdentifier,
    });
    
    return {
      type: action.type,
      metadataName: action.metadataName,
      objectMetadataId: flatObjectMetadata.id, // Resolved ID
      flatFieldMetadatas: /* ... transpiled entities ... */,
    };
  }
}
```

### Action Handler Base Class

`BaseWorkspaceMigrationRunnerActionHandlerService<TActionType,
TMetadataName>` provides:

- **`transpileUniversalActionToFlatAction()`** - Abstract method each
handler must implement
- **`transpileUniversalDeleteActionToFlatDeleteAction()`** - Shared
helper for delete actions

## FlatEntityMaps custom properties
Introduced a `TWithCustomMapsProperties` generic parameter to control
whether custom indexing structures are included:

- **`false` (default)**: Returns `FlatEntityMaps<MetadataFlatEntity<T>>`
- used in builder/runner contexts
- **`true`**: Returns the full maps type with custom properties (e.g.,
`byUserWorkspaceIdAndFolderId`) - used in cache contexts

## Create Field Actions Refactor

Refactored create-field actions to support relation field pairs
bundling.

**Problem:** Relation fields (e.g., `Attachment.targetTask` ↔
`Task.attachments`) couldn't resolve each other's IDs during
transpilation because they were in separate actions with independent
`fieldIdByUniversalIdentifier` maps.

**Solution:** 
- Removed `objectMetadataUniversalIdentifier` from
`UniversalCreateFieldAction` and `objectMetadataId` from
`FlatCreateFieldAction` - each field now carries its own
- Runner groups fields by object internally and processes each table
separately
- Split aggregator into two focused utilities:
- `aggregateNonRelationFieldsIntoObjectActions` - merges non-relation
fields into object actions
- `aggregateRelationFieldPairs` - bundles relation pairs with shared
`fieldIdByUniversalIdentifier`
This commit is contained in:
Paul Rastoin
2026-02-02 13:22:38 +01:00
committed by GitHub
parent f0bc9fcb43
commit bd9688421f
235 changed files with 4536 additions and 2739 deletions
Download Patch File Download Diff File Expand all files Collapse all files
packages/twenty-server/src/engine/workspace-manager/workspace-migration/universal-flat-entity/types/__tests__/contains-serialized-relation.type-test.ts
-146
View File
@@ -1,146 +0,0 @@
import { type Equal, type Expect } from 'twenty-shared/testing';
import { type SerializedRelation } from 'twenty-shared/types';
import { type ContainsSerializedRelation } from 'src/engine/workspace-manager/workspace-migration/universal-flat-entity/types/contains-serialized-relation.type';
// eslint-disable-next-line @typescript-eslint/no-empty-object-type
type EmptyObject = {};
// ContainsSerializedRelation checks for SerializedRelation in object properties
// It recurses into nested objects and arrays, but stops at primitives
// Direct property tests
// eslint-disable-next-line unused-imports/no-unused-vars
type DirectPropertyAssertions = [
// Direct SerializedRelation property
Expect<
Equal<ContainsSerializedRelation<{ targetId: SerializedRelation }>, true>
>,
// Nullable SerializedRelation property
Expect<
Equal<
ContainsSerializedRelation<{ targetId: SerializedRelation | null }>,
true
>
>,
// Optional SerializedRelation property
Expect<
Equal<ContainsSerializedRelation<{ targetId?: SerializedRelation }>, true>
>,
// Array of SerializedRelation
Expect<
Equal<ContainsSerializedRelation<{ ids: SerializedRelation[] }>, true>
>,
// Plain properties only - no SerializedRelation
Expect<
Equal<ContainsSerializedRelation<{ name: string; count: number }>, false>
>,
// Empty object
Expect<Equal<ContainsSerializedRelation<EmptyObject>, false>>,
];
// Nested object tests - should recurse
// eslint-disable-next-line unused-imports/no-unused-vars
type NestedObjectAssertions = [
// Nested object with SerializedRelation
Expect<
Equal<
ContainsSerializedRelation<{
nested: { targetId: SerializedRelation };
}>,
true
>
>,
// Deeply nested SerializedRelation
Expect<
Equal<
ContainsSerializedRelation<{
level1: { level2: { level3: { id: SerializedRelation } } };
}>,
true
>
>,
// Nested object without SerializedRelation
Expect<
Equal<
ContainsSerializedRelation<{
nested: { name: string; count: number };
}>,
false
>
>,
];
// Nested array tests - should recurse
// eslint-disable-next-line unused-imports/no-unused-vars
type NestedArrayAssertions = [
// Array of objects with SerializedRelation
Expect<
Equal<
ContainsSerializedRelation<{
items: { targetId: SerializedRelation }[];
}>,
true
>
>,
// 2D array of SerializedRelation
Expect<
Equal<ContainsSerializedRelation<{ matrix: SerializedRelation[][] }>, true>
>,
// Array of plain objects
Expect<
Equal<
ContainsSerializedRelation<{
items: { name: string }[];
}>,
false
>
>,
// 2D array of strings
Expect<Equal<ContainsSerializedRelation<{ matrix: string[][] }>, false>>,
];
// Primitive types - should return false (not objects)
// eslint-disable-next-line unused-imports/no-unused-vars
type PrimitiveAssertions = [
Expect<Equal<ContainsSerializedRelation<string>, false>>,
Expect<Equal<ContainsSerializedRelation<number>, false>>,
Expect<Equal<ContainsSerializedRelation<boolean>, false>>,
Expect<Equal<ContainsSerializedRelation<null>, false>>,
Expect<Equal<ContainsSerializedRelation<undefined>, false>>,
];
// Real-world tests
// eslint-disable-next-line unused-imports/no-unused-vars
type RealWorldAssertions = [
// Settings with SerializedRelation
Expect<
Equal<
ContainsSerializedRelation<{
relationType?: string;
junctionTargetFieldId?: SerializedRelation;
}>,
true
>
>,
// Workflow config with nested SerializedRelation in array
Expect<
Equal<
ContainsSerializedRelation<{
steps: { assigneeId: SerializedRelation; action: string }[];
}>,
true
>
>,
];
packages/twenty-server/src/engine/workspace-manager/workspace-migration/universal-flat-entity/types/contains-serialized-relation.type.ts
-24
View File
@@ -1,24 +0,0 @@
import {
type IsEmptyObject,
type IsNever,
type IsSerializedRelation,
} from 'twenty-shared/types';
type ContainsSerializedRelationInner<T> = T extends unknown
? IsNever<T> extends true
? false
: unknown extends T
? false
: IsSerializedRelation<T> extends true
? true
: T extends readonly (infer U)[]
? ContainsSerializedRelationInner<U>
: T extends object
? IsEmptyObject<T> extends true
? false
: ContainsSerializedRelationInner<T[keyof T]>
: false
: never;
export type ContainsSerializedRelation<T> =
true extends ContainsSerializedRelationInner<T> ? true : false;
packages/twenty-server/src/engine/workspace-manager/workspace-migration/universal-flat-entity/types/universal-flat-entity-from.type.ts
+3 -6
View File
@@ -8,7 +8,6 @@ import { type FromMetadataEntityToMetadataName } from 'src/engine/metadata-modul
import { type MetadataManyToOneJoinColumn } from 'src/engine/metadata-modules/flat-entity/types/metadata-many-to-one-join-column.type';
import { type SyncableEntity } from 'src/engine/workspace-manager/types/syncable-entity.interface';
import { type AllJsonbPropertiesWithSerializedPropertiesForMetadataName } from 'src/engine/workspace-manager/workspace-migration/universal-flat-entity/constants/all-jsonb-properties-with-serialized-relation-by-metadata-name.constant';
import { type ContainsSerializedRelation } from 'src/engine/workspace-manager/workspace-migration/universal-flat-entity/types/contains-serialized-relation.type';
import { type FormatRecordSerializedRelationProperties } from 'src/engine/workspace-manager/workspace-migration/universal-flat-entity/types/format-record-serialized-relation-properties.type';
export type UniversalSyncableFlatEntity = Omit<
@@ -33,11 +32,9 @@ export type UniversalFlatEntityExtraProperties<
} & {
[P in AllJsonbPropertiesWithSerializedPropertiesForMetadataName<TMetadataName> &
keyof TEntity &
string as `universal${Capitalize<P>}`]: true extends ContainsSerializedRelation<
NonNullable<TEntity[P]>
>
? FormatRecordSerializedRelationProperties<TEntity[P]>
: null;
string as `universal${Capitalize<P>}`]: FormatRecordSerializedRelationProperties<
TEntity[P]
>;
};
export type UniversalFlatEntityFrom<
packages/twenty-server/src/engine/workspace-manager/workspace-migration/universal-flat-entity/types/universal-flat-object-metadata.type.ts
+12
View File
@@ -0,0 +1,12 @@
import { type ObjectMetadataEntity } from 'src/engine/metadata-modules/object-metadata/object-metadata.entity';
import { type UniversalFlatEntityFrom } from 'src/engine/workspace-manager/workspace-migration/universal-flat-entity/types/universal-flat-entity-from.type';
export type UniversalFlatObjectMetadata = UniversalFlatEntityFrom<
Omit<ObjectMetadataEntity, 'targetRelationFields' | 'dataSourceId'>,
'objectMetadata'
> & {
// NOTE: below fields are not reflected on the final UniversalFlatEntity either they should we should define a common source
// TODO remove once https://github.com/twentyhq/core-team-issues/issues/2172 has been resolved
labelIdentifierFieldMetadataUniversalIdentifier: string | null;
imageIdentifierFieldMetadataUniversalIdentifier: string | null;
};