Refactor and standardize isSystem field and object (#17992)

# Introduction ## Centralize system field definitions - Extract a single `PARTIAL_SYSTEM_FLAT_FIELD_METADATAS` constant as the source of truth for all 8 system fields (`id`, `createdAt`, `updatedAt`, `deletedAt`, `createdBy`, `updatedBy`, `position`, `searchVector`), eliminating duplication across custom object and standard app field builders - Refactor `buildDefaultFlatFieldMetadatasForCustomObject` to use the shared constant via a new `buildObjectSystemFlatFieldMetadatas` helper ## Mark system fields as `isSystem: true` - Fields `id`, `createdAt`, `updatedAt`, `deletedAt`, `createdBy`, `updatedBy`, `position`, `searchVector` are now properly flagged as system fields across all standard objects and custom object creation - Standard app field builders for all ~30 standard objects updated to set `isSystem: true` on `createdAt`, `updatedAt`, `deletedAt`, `createdBy`, `updatedBy` - System-only standard objects (blocklist, calendar channels, message threads, etc.) now also include `createdBy`, `updatedBy`, `position`, `searchVector` field definitions that were previously missing ## Validate system fields on object creation - New transversal validation (`crossEntityTransversalValidation`) runs after all atomic entity validations in the build orchestrator, ensuring all 8 system fields are present with correct `type` and `isSystem: true` when an object is created - New `buildUniversalFlatObjectFieldByNameAndJoinColumnMaps` utility to resolve field names to universal identifiers for a given object - New exception codes: `MISSING_SYSTEM_FIELD` and `INVALID_SYSTEM_FIELD` on `ObjectMetadataExceptionCode` ## Protect system fields and objects from mutation - Field validators now block update/delete of `isSystem` fields by non-system callers (`FIELD_MUTATION_NOT_ALLOWED`) - Object validators now block update/delete of `isSystem` objects by non-system callers - `POSITION` and `TS_VECTOR` field type validators replaced: instead of rejecting creation outright, they now validate that the field is named correctly (`position` / `searchVector`) and has `isSystem: true` ## Distinguish `isSystemBuild` from `isCallerTwentyStandardApp` - New `isCallerTwentyStandardApp` utility checks whether the caller's `applicationUniversalIdentifier` matches the twenty standard app - Name-sync logic (`isFlatFieldMetadataNameSyncedWithLabel`, `areFlatObjectMetadataNamesSyncedWithLabels`) refactored to use `isCallerTwentyStandardApp` for custom suffix decisions, keeping `isSystemBuild` for mutation permission checks - `WorkspaceMigrationBuilderOptions` type updated to include `applicationUniversalIdentifier` ## Adapt frontend filtering - New `HIDDEN_SYSTEM_FIELD_NAMES` constant (`id`, `position`, `searchVector`) and `isHiddenSystemField` utility to only hide truly internal fields while keeping user-facing system fields (`createdAt`, `updatedAt`, `deletedAt`, `createdBy`, `updatedBy`) visible in the UI - ~20 frontend files updated to replace `!field.isSystem` checks with `!isHiddenSystemField(field)` across record index, settings, data model, charts, workflows, spreadsheet import, aggregations, and role permissions ## Add 1.19 upgrade commands - **`backfill-system-fields-is-system`**: Raw SQL command to set `isSystem = true` on existing workspace fields matching system field names, and fix `position` field type from `NUMBER` to `POSITION` for `favorite`/`favoriteFolder` objects. Includes proper cache invalidation. - **`add-missing-system-fields-to-standard-objects`**: Codegen'd workspace migration to create missing `position`, `searchVector`, `createdBy`, `updatedBy` fields on standard objects that didn't previously have them. Runs via `WorkspaceMigrationRunnerService` in a single transaction with idempotency check. **Known limitation**: assumes all standard objects exist and are valid in the target workspace. ## Add `universalIdentifier` for system fields in standard object constants - `standard-object.constant.ts` updated to include `universalIdentifier` for `createdBy`, `updatedBy`, `position`, and `searchVector` across all standard objects - `fieldManifestType.ts` updated to support the new field manifest shape ## System relation Completely removed and backfilled all `isSystem` relation to be false false As we won't require an object to have any relation system fields ## Add integration tests - New test suite `failing-sync-application-object-system-fields` covering: missing system fields, wrong field types (`id` as TEXT, `createdAt` as TEXT, `position` as TEXT), system field deletion attempts, and system field update attempts - New test utilities: `buildDefaultObjectManifest` (builds an object manifest with all 8 system fields) and `setupApplicationForSync` (centralizes application setup) - Existing successful sync test updated to verify system fields are created with correct properties ## Next step Make the builder scope the compared entity to be the currently built app + nor twenty standard app
2026-02-19 11:13:50 +01:00
parent 082400f751
commit 88424611ec
137 changed files with 8850 additions and 2056 deletions
@@ -0,0 +1,89 @@
+import { InjectRepository } from '@nestjs/typeorm';
+
+import { Command } from 'nest-commander';
+import { isDefined } from 'twenty-shared/utils';
+import { Repository } from 'typeorm';
+
+import { ActiveOrSuspendedWorkspacesMigrationCommandRunner } from 'src/database/commands/command-runners/active-or-suspended-workspaces-migration.command-runner';
+import { RunOnWorkspaceArgs } from 'src/database/commands/command-runners/workspaces-migration.command-runner';
+import { ADD_MISSING_SYSTEM_FIELDS_TO_STANDARD_OBJECTS_1771420702241 } from 'src/database/commands/upgrade-version-command/workspace-migrations/1771420702241-add-missing-system-fields-to-standard-objects';
+import { WorkspaceEntity } from 'src/engine/core-modules/workspace/workspace.entity';
+import { DataSourceService } from 'src/engine/metadata-modules/data-source/data-source.service';
+import { GlobalWorkspaceOrmManager } from 'src/engine/twenty-orm/global-workspace-datasource/global-workspace-orm.manager';
+import { WorkspaceCacheService } from 'src/engine/workspace-cache/services/workspace-cache.service';
+import { WorkspaceMigrationRunnerService } from 'src/engine/workspace-manager/workspace-migration/workspace-migration-runner/services/workspace-migration-runner.service';
+
+const FIRST_FIELD_UNIVERSAL_IDENTIFIER =
+  ADD_MISSING_SYSTEM_FIELDS_TO_STANDARD_OBJECTS_1771420702241.actions[0]
+    .flatEntity.universalIdentifier;
+
+@Command({
+  name: 'upgrade:1-19:add-missing-system-fields-to-standard-objects',
+  description:
+    'Add missing system fields (position, searchVector, createdBy, updatedBy) to standard objects',
+})
+export class AddMissingSystemFieldsToStandardObjectsCommand extends ActiveOrSuspendedWorkspacesMigrationCommandRunner {
+  constructor(
+    @InjectRepository(WorkspaceEntity)
+    protected readonly workspaceRepository: Repository<WorkspaceEntity>,
+    protected readonly twentyORMGlobalManager: GlobalWorkspaceOrmManager,
+    protected readonly dataSourceService: DataSourceService,
+    private readonly workspaceCacheService: WorkspaceCacheService,
+    private readonly workspaceMigrationRunnerService: WorkspaceMigrationRunnerService,
+  ) {
+    super(workspaceRepository, twentyORMGlobalManager, dataSourceService);
+  }
+
+  // The entire migration runs in a single transaction, so checking the first
+  // field is enough to know whether the migration has already been applied.
+  // In the future we will maintain a list of passed migrations
+  private async hasAlreadyRun(workspaceId: string): Promise<boolean> {
+    const { flatFieldMetadataMaps } =
+      await this.workspaceCacheService.getOrRecompute(workspaceId, [
+        'flatFieldMetadataMaps',
+      ]);
+
+    return isDefined(
+      flatFieldMetadataMaps.byUniversalIdentifier[
+        FIRST_FIELD_UNIVERSAL_IDENTIFIER
+      ],
+    );
+  }
+
+  override async runOnWorkspace({
+    workspaceId,
+    options,
+  }: RunOnWorkspaceArgs): Promise<void> {
+    const dryRun = options?.dryRun ?? false;
+
+    this.logger.log(
+      `${dryRun ? '[DRY RUN] ' : ''}Adding missing system fields to standard objects in workspace ${workspaceId}`,
+    );
+
+    if (dryRun) {
+      this.logger.log(
+        `[DRY RUN] Would add ${ADD_MISSING_SYSTEM_FIELDS_TO_STANDARD_OBJECTS_1771420702241.actions.length} missing system fields to standard objects in workspace ${workspaceId}. Skipping.`,
+      );
+
+      return;
+    }
+
+    if (await this.hasAlreadyRun(workspaceId)) {
+      this.logger.log(
+        `Migration already applied for workspace ${workspaceId}, skipping.`,
+      );
+
+      return;
+    }
+
+    await this.workspaceMigrationRunnerService.run({
+      workspaceId,
+      workspaceMigration:
+        ADD_MISSING_SYSTEM_FIELDS_TO_STANDARD_OBJECTS_1771420702241,
+    });
+
+    this.logger.log(
+      `Successfully added missing system fields to standard objects in workspace ${workspaceId}`,
+    );
+  }
+}
@@ -151,6 +151,8 @@ export class BackfillMessageChannelMessageAssociationMessageFolderCommand extend
        {
          buildOptions: {
            isSystemBuild: true,
+            applicationUniversalIdentifier:
+              twentyStandardFlatApplication.universalIdentifier,
          },
          fromToAllFlatEntityMaps: {
            flatObjectMetadataMaps: {
@@ -166,8 +168,7 @@ export class BackfillMessageChannelMessageAssociationMessageFolderCommand extend
          additionalCacheDataMaps: {
            featureFlagsMap,
          },
-          applicationUniversalIdentifier:
-            twentyStandardFlatApplication.universalIdentifier,
+
          idByUniversalIdentifierByMetadataName,
        },
      );
@@ -0,0 +1,176 @@
+import { InjectDataSource, InjectRepository } from '@nestjs/typeorm';
+
+import { Command } from 'nest-commander';
+import { STANDARD_OBJECTS } from 'twenty-shared/metadata';
+import { FieldMetadataType } from 'twenty-shared/types';
+import { DataSource, Repository } from 'typeorm';
+
+import { ActiveOrSuspendedWorkspacesMigrationCommandRunner } from 'src/database/commands/command-runners/active-or-suspended-workspaces-migration.command-runner';
+import { RunOnWorkspaceArgs } from 'src/database/commands/command-runners/workspaces-migration.command-runner';
+import { WorkspaceEntity } from 'src/engine/core-modules/workspace/workspace.entity';
+import { DataSourceService } from 'src/engine/metadata-modules/data-source/data-source.service';
+import { getMetadataFlatEntityMapsKey } from 'src/engine/metadata-modules/flat-entity/utils/get-metadata-flat-entity-maps-key.util';
+import { getMetadataRelatedMetadataNames } from 'src/engine/metadata-modules/flat-entity/utils/get-metadata-related-metadata-names.util';
+import { PARTIAL_SYSTEM_FLAT_FIELD_METADATAS } from 'src/engine/metadata-modules/object-metadata/constants/partial-system-flat-field-metadatas.constant';
+import { WorkspaceMetadataVersionService } from 'src/engine/metadata-modules/workspace-metadata-version/services/workspace-metadata-version.service';
+import { GlobalWorkspaceOrmManager } from 'src/engine/twenty-orm/global-workspace-datasource/global-workspace-orm.manager';
+import { WorkspaceCacheStorageService } from 'src/engine/workspace-cache-storage/workspace-cache-storage.service';
+import { WorkspaceCacheService } from 'src/engine/workspace-cache/services/workspace-cache.service';
+import { type WorkspaceCacheKeyName } from 'src/engine/workspace-cache/types/workspace-cache-key.type';
+
+const SYSTEM_FIELD_NAMES = Object.keys(PARTIAL_SYSTEM_FLAT_FIELD_METADATAS);
+
+const POSITION_FIELDS_TO_FIX_TYPE = [
+  STANDARD_OBJECTS.favorite.fields.position.universalIdentifier,
+  STANDARD_OBJECTS.favoriteFolder.fields.position.universalIdentifier,
+];
+
+const RELATION_FIELD_TYPES = [
+  FieldMetadataType.RELATION,
+  FieldMetadataType.MORPH_RELATION,
+];
+
+@Command({
+  name: 'upgrade:1-19:backfill-system-fields-is-system',
+  description:
+    'Set isSystem to true for system field names, set isSystem to false for relation/morph_relation fields, and fix position field type for favorite/favoriteFolder',
+})
+export class BackfillSystemFieldsIsSystemCommand extends ActiveOrSuspendedWorkspacesMigrationCommandRunner {
+  constructor(
+    @InjectRepository(WorkspaceEntity)
+    protected readonly workspaceRepository: Repository<WorkspaceEntity>,
+    @InjectDataSource()
+    private readonly coreDataSource: DataSource,
+    protected readonly twentyORMGlobalManager: GlobalWorkspaceOrmManager,
+    protected readonly dataSourceService: DataSourceService,
+    private readonly workspaceCacheService: WorkspaceCacheService,
+    private readonly workspaceCacheStorageService: WorkspaceCacheStorageService,
+    private readonly workspaceMetadataVersionService: WorkspaceMetadataVersionService,
+  ) {
+    super(workspaceRepository, twentyORMGlobalManager, dataSourceService);
+  }
+
+  override async runOnWorkspace({
+    workspaceId,
+    options,
+  }: RunOnWorkspaceArgs): Promise<void> {
+    const dryRun = options?.dryRun ?? false;
+
+    this.logger.log(
+      `${dryRun ? '[DRY RUN] ' : ''}Backfilling isSystem for system fields in workspace ${workspaceId}`,
+    );
+
+    if (dryRun) {
+      this.logger.log(
+        `[DRY RUN] Would set isSystem=true for fields named [${SYSTEM_FIELD_NAMES.join(', ')}], set isSystem=false for relation fields, and fix position field types in workspace ${workspaceId}. Skipping.`,
+      );
+
+      return;
+    }
+
+    const queryRunner = this.coreDataSource.createQueryRunner();
+
+    await queryRunner.connect();
+
+    try {
+      let needsCacheInvalidation = false;
+
+      const isSystemResult = await queryRunner.query(
+        `UPDATE core."fieldMetadata"
+         SET "isSystem" = true
+         WHERE "workspaceId" = $1
+           AND "name" = ANY($2)
+           AND "isSystem" = false`,
+        [workspaceId, SYSTEM_FIELD_NAMES],
+      );
+
+      const isSystemUpdatedCount = isSystemResult?.[1] ?? 0;
+
+      if (isSystemUpdatedCount > 0) {
+        this.logger.log(
+          `Set isSystem=true for ${isSystemUpdatedCount} field(s) in workspace ${workspaceId}`,
+        );
+        needsCacheInvalidation = true;
+      }
+
+      const relationIsSystemResult = await queryRunner.query(
+        `UPDATE core."fieldMetadata"
+         SET "isSystem" = false
+         WHERE "workspaceId" = $1
+           AND "type" = ANY($2)
+           AND "isSystem" = true`,
+        [workspaceId, RELATION_FIELD_TYPES],
+      );
+
+      const relationIsSystemUpdatedCount = relationIsSystemResult?.[1] ?? 0;
+
+      if (relationIsSystemUpdatedCount > 0) {
+        this.logger.log(
+          `Set isSystem=false for ${relationIsSystemUpdatedCount} relation field(s) in workspace ${workspaceId}`,
+        );
+        needsCacheInvalidation = true;
+      }
+
+      const positionTypeResult = await queryRunner.query(
+        `UPDATE core."fieldMetadata"
+         SET "type" = $1
+         WHERE "workspaceId" = $2
+           AND "universalIdentifier" = ANY($3)
+           AND "type" = $4`,
+        [
+          FieldMetadataType.POSITION,
+          workspaceId,
+          POSITION_FIELDS_TO_FIX_TYPE,
+          FieldMetadataType.NUMBER,
+        ],
+      );
+
+      const positionTypeUpdatedCount = positionTypeResult?.[1] ?? 0;
+
+      if (positionTypeUpdatedCount > 0) {
+        this.logger.log(
+          `Fixed type from NUMBER to POSITION for ${positionTypeUpdatedCount} field(s) in workspace ${workspaceId}`,
+        );
+        needsCacheInvalidation = true;
+      }
+
+      if (needsCacheInvalidation) {
+        await this.invalidateCaches(workspaceId);
+      } else {
+        this.logger.log(
+          `No fields needed updating in workspace ${workspaceId}`,
+        );
+      }
+    } finally {
+      await queryRunner.release();
+    }
+  }
+
+  private async invalidateCaches(workspaceId: string): Promise<void> {
+    const modifiedMetadataNames = ['fieldMetadata'] as const;
+
+    const cacheKeysToInvalidate: WorkspaceCacheKeyName[] = [
+      ...new Set(
+        modifiedMetadataNames
+          .flatMap((name) => [name, ...getMetadataRelatedMetadataNames(name)])
+          .map(getMetadataFlatEntityMapsKey),
+      ),
+      'ORMEntityMetadatas',
+    ];
+
+    await this.workspaceCacheService.invalidateAndRecompute(
+      workspaceId,
+      cacheKeysToInvalidate,
+    );
+
+    await this.workspaceMetadataVersionService.incrementMetadataVersion(
+      workspaceId,
+    );
+
+    await this.workspaceCacheStorageService.flush(workspaceId);
+
+    this.logger.log(
+      `Cache invalidated and metadata version incremented for workspace ${workspaceId}`,
+    );
+  }
+}
@@ -1,11 +1,16 @@
 import { Module } from '@nestjs/common';
 import { TypeOrmModule } from '@nestjs/typeorm';

+import { AddMissingSystemFieldsToStandardObjectsCommand } from 'src/database/commands/upgrade-version-command/1-19/1-19-add-missing-system-fields-to-standard-objects.command';
 import { BackfillMessageChannelMessageAssociationMessageFolderCommand } from 'src/database/commands/upgrade-version-command/1-19/1-19-backfill-message-channel-message-association-message-folder.command';
+import { BackfillSystemFieldsIsSystemCommand } from 'src/database/commands/upgrade-version-command/1-19/1-19-backfill-system-fields-is-system.command';
 import { ApplicationModule } from 'src/engine/core-modules/application/application.module';
 import { WorkspaceEntity } from 'src/engine/core-modules/workspace/workspace.entity';
 import { DataSourceModule } from 'src/engine/metadata-modules/data-source/data-source.module';
+import { WorkspaceMetadataVersionModule } from 'src/engine/metadata-modules/workspace-metadata-version/workspace-metadata-version.module';
+import { WorkspaceCacheStorageModule } from 'src/engine/workspace-cache-storage/workspace-cache-storage.module';
 import { WorkspaceCacheModule } from 'src/engine/workspace-cache/workspace-cache.module';
+import { WorkspaceMigrationRunnerModule } from 'src/engine/workspace-manager/workspace-migration/workspace-migration-runner/workspace-migration-runner.module';
 import { WorkspaceMigrationModule } from 'src/engine/workspace-manager/workspace-migration/workspace-migration.module';

@Module({
@@ -13,10 +18,21 @@ import { WorkspaceMigrationModule } from 'src/engine/workspace-manager/workspace
    TypeOrmModule.forFeature([WorkspaceEntity]),
    DataSourceModule,
    WorkspaceCacheModule,
+    WorkspaceCacheStorageModule,
+    WorkspaceMetadataVersionModule,
+    WorkspaceMigrationRunnerModule,
    ApplicationModule,
    WorkspaceMigrationModule,
  ],
-  providers: [BackfillMessageChannelMessageAssociationMessageFolderCommand],
-  exports: [BackfillMessageChannelMessageAssociationMessageFolderCommand],
+  providers: [
+    BackfillSystemFieldsIsSystemCommand,
+    AddMissingSystemFieldsToStandardObjectsCommand,
+    BackfillMessageChannelMessageAssociationMessageFolderCommand,
+  ],
+  exports: [
+    BackfillSystemFieldsIsSystemCommand,
+    AddMissingSystemFieldsToStandardObjectsCommand,
+    BackfillMessageChannelMessageAssociationMessageFolderCommand,
+  ],
 })
 export class V1_19_UpgradeVersionCommandModule {}
@@ -27,7 +27,9 @@ import { MigrateFavoritesToNavigationMenuItemsCommand } from 'src/database/comma
 import { MigratePersonAvatarFilesCommand } from 'src/database/commands/upgrade-version-command/1-18/1-18-migrate-person-avatar-files.command';
 import { MigrateWorkflowSendEmailAttachmentsCommand } from 'src/database/commands/upgrade-version-command/1-18/1-18-migrate-workflow-send-email-attachments.command';
 import { MigrateWorkspacePicturesCommand } from 'src/database/commands/upgrade-version-command/1-18/1-18-migrate-workspace-pictures.command';
+import { AddMissingSystemFieldsToStandardObjectsCommand } from 'src/database/commands/upgrade-version-command/1-19/1-19-add-missing-system-fields-to-standard-objects.command';
 import { BackfillMessageChannelMessageAssociationMessageFolderCommand } from 'src/database/commands/upgrade-version-command/1-19/1-19-backfill-message-channel-message-association-message-folder.command';
+import { BackfillSystemFieldsIsSystemCommand } from 'src/database/commands/upgrade-version-command/1-19/1-19-backfill-system-fields-is-system.command';
 import { TwentyConfigService } from 'src/engine/core-modules/twenty-config/twenty-config.service';
 import { WorkspaceEntity } from 'src/engine/core-modules/workspace/workspace.entity';
 import { DataSourceService } from 'src/engine/metadata-modules/data-source/data-source.service';
@@ -70,6 +72,8 @@ export class UpgradeCommand extends UpgradeCommandRunner {
    protected readonly migrateWorkflowSendEmailAttachmentsCommand: MigrateWorkflowSendEmailAttachmentsCommand,

    // 1.19 Commands
+    protected readonly backfillSystemFieldsIsSystemCommand: BackfillSystemFieldsIsSystemCommand,
+    protected readonly addMissingSystemFieldsToStandardObjectsCommand: AddMissingSystemFieldsToStandardObjectsCommand,
    protected readonly backfillMessageChannelMessageAssociationMessageFolderCommand: BackfillMessageChannelMessageAssociationMessageFolderCommand,
  ) {
    super(
@@ -108,6 +112,8 @@ export class UpgradeCommand extends UpgradeCommandRunner {
    ];

    const commands_1190: VersionCommands = [
+      this.backfillSystemFieldsIsSystemCommand,
+      this.addMissingSystemFieldsToStandardObjectsCommand,
      this.backfillMessageChannelMessageAssociationMessageFolderCommand,
    ];