fix(server): auto-index target<X>Id join columns on polymorphic standard objects (#20820)

Closes #20726 ## The bug `timelineActivity` (and the three other polymorphic standard objects — `attachment`, `noteTarget`, `taskTarget`) store relations as N nullable `target<X>Id` columns, one per related object. Each one is a join key queried as `WHERE target<X>Id IN (...) AND deletedAt IS NULL`. For **built-in** related objects (Person, Company, Opportunity, …), each `target<X>Id` column gets a BTREE index, declared statically in `compute-{timelineActivity,attachment,noteTarget,taskTarget}-standard-flat-index-metadata.util.ts`. For **custom** related objects, the same `target<CustomObject>Id` column was added — **without an index**. On a `timelineActivity` table at issue-reporter scale (~21.9M rows, 7.1 GB), this turned record loads into 20–40s sequential scans and produced `QueryFailedError: Query read timeout` for end users. ## Diagnosis The morph/relation field generator (`generateMorphOrRelationFlatFieldMetadataPair`) already creates a BTREE index for the field that owns the join column and returns it alongside the field metadata pair. The two user-driven entry points (`fromRelationCreateFieldInput…`, `fromMorphRelationCreateFieldInput…`) correctly destructure and propagate that index. But the **custom-object creation path** — `buildDefaultRelationFlatFieldMetadatasForCustomObject`, called when a user creates a new custom object — destructured only `{ flatFieldMetadatas }` and threw away `indexMetadatas`. So every `target<CustomObject>Id` column added to the four polymorphic standard objects has been shipping unindexed since custom morph relations went in. ## The fix Three commits. ### 1. `fix(server): index target<CustomObject>Id columns on standard polymorphic objects` 13 lines across 2 files. - `build-default-relation-flat-field-metadatas-for-custom-object.util.ts` — also destructure `indexMetadatas` from the pair generator and accumulate them into the returned record (new field `standardTargetFlatIndexMetadatas`). - `from-create-object-input-to-flat-object-metadata-and-flat-field-metadatas-to-create.util.ts` — append the accumulated indexes to `flatIndexMetadataToCreate`. The migration pipeline at `object-metadata.service.ts:559–562` already passes `flatIndexMetadataToCreate` to the migration runner, so no further wiring is needed. From now on, creating a custom object also creates the four BTREE indexes — one per polymorphic standard object's new `target<CustomObject>Id` column — atomically with the rest of the migration. ### 2. `feat(server): backfill workspace command for relation join column indexes` For existing workspaces whose custom objects were created before the forward-fix. `upgrade:2-8:backfill-relation-join-column-indexes` is a `@RegisteredWorkspaceCommand('2.8.0', 1798100000000)` matching the pattern from `2-7-workspace-command-…-drop-connected-account-standard-object.command.ts`. Per workspace: 1. Load `flatObjectMetadataMaps`, `flatFieldMetadataMaps`, `flatIndexMaps` from the workspace cache. 2. Resolve the four polymorphic standard object IDs by `nameSingular` against `DEFAULT_RELATIONS_OBJECTS_STANDARD_IDS`. 3. Collect every field ID that's already covered by any existing index. 4. Filter `flatFieldMetadataMaps` to MORPH_RELATION fields on those four objects whose `settings.relationType === MANY_TO_ONE` (i.e. owns a join column) and whose ID isn't in the indexed set. 5. Generate a BTREE `UniversalFlatIndexMetadata` for each via `generateIndexForFlatFieldMetadata` (same helper the forward-fix uses). 6. Create the indexes in the workspace schema with **CONCURRENTLY** (see commit 3). 7. Submit the metadata through `WorkspaceMigrationValidateBuildAndRunService` so it lands in `indexMetadata` and the cache — same pipeline as a normal metadata change. The pipeline's own `CREATE INDEX IF NOT EXISTS` no-ops because the index already exists. Properties: - **Idempotent.** Re-running is a no-op once indexes exist. - **Scoped.** Only the four polymorphic standard objects, only their MANY_TO_ONE morph relation fields, only those with no covering index. - **Same code path as the forward-fix.** The backfill produces exactly the indexes the forward-fix would have created at custom-object creation time. - **`--dry-run` supported** via the base `ActiveOrSuspendedWorkspaceCommandRunner`. ### 3. `feat(server): create index CONCURRENTLY in relation join column backfill` Adds an opt-in `concurrently` flag to `WorkspaceSchemaIndexManagerService.createIndex` (threaded through `createIndexInWorkspaceSchema`). When `true`, emits `CREATE INDEX CONCURRENTLY IF NOT EXISTS …`. Defaults to `false` — every existing caller keeps the current transactional `CREATE INDEX` behavior. The backfill command opts in. It creates a QueryRunner **without** `startTransaction()`, issues the CONCURRENTLY indexes one-by-one (each waits for the previous to finish), then submits the metadata through the normal migration pipeline whose own `CREATE INDEX IF NOT EXISTS` is now a no-op. Why not flip the default for the helper: - `CREATE INDEX CONCURRENTLY` cannot run inside a transaction — Postgres errors out. The migration pipeline calls `createIndex` from inside a transactional schema migration. - CONCURRENTLY doesn't roll back with the transaction. If the surrounding migration fails, the index remains and you end up with metadata/schema drift. - Failed CONCURRENTLY builds leave an INVALID index behind that needs manual `DROP`. - UNIQUE indexes have different failure semantics under CONCURRENTLY (deferred, not immediate). So CONCURRENTLY is opt-in, used only where it's the right tool (post-hoc backfills on populated tables). ## Decisions / tradeoffs - **Single-column BTREE vs partial `WHERE deletedAt IS NULL` vs composite.** Twenty's queries always include `deletedAt IS NULL`. A partial index would be slightly better than a plain BTREE (smaller, no wasted seeks on soft-deleted rows). This PR ships single-column to match the existing built-in target index pattern, which already covers >95% of the available speedup (the 20s→4ms drop the reporter saw comes from having any index — composite/partial is a second-order effect). Switching all relation indexes to partial is a separate, broader change. - **CONCURRENTLY operator caveat.** If a CONCURRENTLY build is interrupted (kill, connection drop, OOM), Postgres leaves the index as INVALID. We deliberately don't probe `pg_index` for invalid leftovers on every create — catalog-table queries can be slow at multi-tenant scale and the failure mode is rare. Recovery is manual: `DROP INDEX <name>` and re-run the backfill. - **Forward-fix is not gated** behind a feature flag. The change is metadata-pipeline-internal; before, custom-object creation silently produced a degraded state. After, it produces the correct state. No new public API, no behavioural change for end users besides the indexes existing. ## Risk - Forward-fix: changes only the metadata produced during custom-object creation. New objects get four extra `FlatIndexMetadata` rows and four extra `CREATE INDEX` statements during their creation migration. Tables are empty at that point so the index builds in microseconds. - Helper change: API-compatible, default behavior unchanged. The new `concurrently` parameter is optional. - Backfill: read-only state probe → CONCURRENTLY index creation (no write blocking) → metadata insert via the normal migration pipeline. Idempotent. Reverting is `DROP INDEX`. ## Test plan - [ ] Verify forward-fix: create a custom object, confirm four new BTREE indexes appear on `timelineActivity`, `attachment`, `noteTarget`, `taskTarget` for the new `target<CustomObject>Id` columns, and that `flatIndexMaps` has matching entries. - [ ] Verify backfill on a workspace that had custom objects created before the fix: run `--dry-run` first, confirm the expected indexes are listed; then run for real, confirm the indexes appear in pg (and as `indisvalid = true` in `pg_index`) and in `flatIndexMaps`. Re-run; confirm no-op. - [ ] Verify backfill on a clean workspace: should log "no missing indexes" and exit. - [ ] Verify CONCURRENTLY behavior under load: run backfill against a workspace with active writes on `timelineActivity`; confirm inserts/updates keep working during index build (no `ShareLock` waits in `pg_stat_activity`). - [ ] On the affected reporter-scale workspace, confirm `EXPLAIN ANALYZE` switches from sequential scan to index scan and timeline activity timeouts go away.
2026-05-22 11:56:33 +02:00
parent 788d120b71
commit 084fa8eaba
14 changed files with 1310 additions and 62 deletions
@@ -0,0 +1,20 @@
+import { Module } from '@nestjs/common';
+
+import { WorkspaceIteratorModule } from 'src/database/commands/command-runners/workspace-iterator.module';
+import { BackfillRelationJoinColumnIndexesCommand } from 'src/database/commands/upgrade-version-command/2-8/2-8-workspace-command-1798100000000-backfill-relation-join-column-indexes.command';
+import { ApplicationModule } from 'src/engine/core-modules/application/application.module';
+import { WorkspaceSchemaManagerModule } from 'src/engine/twenty-orm/workspace-schema-manager/workspace-schema-manager.module';
+import { WorkspaceCacheModule } from 'src/engine/workspace-cache/workspace-cache.module';
+import { WorkspaceMigrationModule } from 'src/engine/workspace-manager/workspace-migration/workspace-migration.module';
+
+@Module({
+  imports: [
+    ApplicationModule,
+    WorkspaceCacheModule,
+    WorkspaceIteratorModule,
+    WorkspaceMigrationModule,
+    WorkspaceSchemaManagerModule,
+  ],
+  providers: [BackfillRelationJoinColumnIndexesCommand],
+})
+export class V2_8_UpgradeVersionCommandModule {}
@@ -0,0 +1,225 @@
+import { Command } from 'nest-commander';
+import { DEFAULT_RELATIONS_OBJECTS_STANDARD_IDS } from 'twenty-shared/metadata';
+import { RelationType } from 'twenty-shared/types';
+import { isDefined } from 'twenty-shared/utils';
+
+import { ActiveOrSuspendedWorkspaceCommandRunner } from 'src/database/commands/command-runners/active-or-suspended-workspace.command-runner';
+import { WorkspaceIteratorService } from 'src/database/commands/command-runners/workspace-iterator.service';
+import { type RunOnWorkspaceArgs } from 'src/database/commands/command-runners/workspace.command-runner';
+import { ApplicationService } from 'src/engine/core-modules/application/application.service';
+import { RegisteredWorkspaceCommand } from 'src/engine/core-modules/upgrade/decorators/registered-workspace-command.decorator';
+import { computeMorphOrRelationFieldJoinColumnName } from 'src/engine/metadata-modules/field-metadata/utils/compute-morph-or-relation-field-join-column-name.util';
+import { findFlatEntityByIdInFlatEntityMapsOrThrow } from 'src/engine/metadata-modules/flat-entity/utils/find-flat-entity-by-id-in-flat-entity-maps-or-throw.util';
+import { generateIndexForFlatFieldMetadata } from 'src/engine/metadata-modules/flat-field-metadata/utils/generate-index-for-flat-field-metadata.util';
+import { isMorphOrRelationFlatFieldMetadata } from 'src/engine/metadata-modules/flat-field-metadata/utils/is-morph-or-relation-flat-field-metadata.util';
+import { type FlatObjectMetadata } from 'src/engine/metadata-modules/flat-object-metadata/types/flat-object-metadata.type';
+import { WorkspaceSchemaManagerService } from 'src/engine/twenty-orm/workspace-schema-manager/workspace-schema-manager.service';
+import { WorkspaceCacheService } from 'src/engine/workspace-cache/services/workspace-cache.service';
+import { getWorkspaceSchemaContextForMigration } from 'src/engine/workspace-manager/workspace-migration/workspace-migration-runner/utils/get-workspace-schema-context-for-migration.util';
+import { WorkspaceMigrationValidateBuildAndRunService } from 'src/engine/workspace-manager/workspace-migration/services/workspace-migration-validate-build-and-run-service';
+
+const POLYMORPHIC_STANDARD_OBJECT_NAMES_SINGULAR: ReadonlySet<string> = new Set(
+  DEFAULT_RELATIONS_OBJECTS_STANDARD_IDS,
+);
+
+@RegisteredWorkspaceCommand('2.8.0', 1798100000000)
+@Command({
+  name: 'upgrade:2-8:backfill-relation-join-column-indexes',
+  description:
+    'Backfill missing BTREE indexes on target<X>Id join columns added to polymorphic standard objects (timelineActivity, attachment, noteTarget, taskTarget) when custom objects were created before the auto-index fix. Indexes are created with CONCURRENTLY so writes are not blocked.',
+})
+export class BackfillRelationJoinColumnIndexesCommand extends ActiveOrSuspendedWorkspaceCommandRunner {
+  constructor(
+    protected readonly workspaceIteratorService: WorkspaceIteratorService,
+    private readonly applicationService: ApplicationService,
+    private readonly workspaceMigrationValidateBuildAndRunService: WorkspaceMigrationValidateBuildAndRunService,
+    private readonly workspaceCacheService: WorkspaceCacheService,
+    private readonly workspaceSchemaManagerService: WorkspaceSchemaManagerService,
+  ) {
+    super(workspaceIteratorService);
+  }
+
+  override async runOnWorkspace({
+    workspaceId,
+    dataSource,
+    options,
+  }: RunOnWorkspaceArgs): Promise<void> {
+    const isDryRun = options.dryRun ?? false;
+
+    if (!dataSource) {
+      this.logger.log(`No data source for workspace ${workspaceId}, skipping`);
+
+      return;
+    }
+
+    const { flatObjectMetadataMaps, flatFieldMetadataMaps, flatIndexMaps } =
+      await this.workspaceCacheService.getOrRecompute(workspaceId, [
+        'flatObjectMetadataMaps',
+        'flatFieldMetadataMaps',
+        'flatIndexMaps',
+      ]);
+
+    const polymorphicStandardObjectIds = new Set(
+      Object.values(flatObjectMetadataMaps.byUniversalIdentifier)
+        .filter(isDefined)
+        .filter((flatObject) =>
+          POLYMORPHIC_STANDARD_OBJECT_NAMES_SINGULAR.has(
+            flatObject.nameSingular,
+          ),
+        )
+        .map((flatObject) => flatObject.id),
+    );
+
+    if (polymorphicStandardObjectIds.size === 0) {
+      this.logger.log(
+        `No polymorphic standard objects found for workspace ${workspaceId}, skipping`,
+      );
+
+      return;
+    }
+
+    const indexedFieldIds = new Set<string>();
+
+    for (const flatIndex of Object.values(flatIndexMaps.byUniversalIdentifier)) {
+      if (!isDefined(flatIndex)) {
+        continue;
+      }
+
+      for (const indexField of flatIndex.flatIndexFieldMetadatas) {
+        indexedFieldIds.add(indexField.fieldMetadataId);
+      }
+    }
+
+    const fieldsNeedingIndex = Object.values(
+      flatFieldMetadataMaps.byUniversalIdentifier,
+    )
+      .filter(isDefined)
+      .filter(isMorphOrRelationFlatFieldMetadata)
+      .filter((flatField) =>
+        polymorphicStandardObjectIds.has(flatField.objectMetadataId),
+      )
+      .filter(
+        (flatField) =>
+          flatField.settings?.relationType === RelationType.MANY_TO_ONE,
+      )
+      .filter((flatField) => !indexedFieldIds.has(flatField.id));
+
+    if (fieldsNeedingIndex.length === 0) {
+      this.logger.log(
+        `No missing relation join column indexes for workspace ${workspaceId}, skipping`,
+      );
+
+      return;
+    }
+
+    const flatIndexBuildPlans = fieldsNeedingIndex.map((flatField) => {
+      const flatObjectMetadata =
+        findFlatEntityByIdInFlatEntityMapsOrThrow<FlatObjectMetadata>({
+          flatEntityId: flatField.objectMetadataId,
+          flatEntityMaps: flatObjectMetadataMaps,
+        });
+
+      const universalFlatIndexMetadata = generateIndexForFlatFieldMetadata({
+        flatFieldMetadata: flatField,
+        flatObjectMetadata,
+      });
+
+      const joinColumnName = computeMorphOrRelationFieldJoinColumnName({
+        name: flatField.name,
+      });
+
+      return {
+        flatObjectMetadata,
+        universalFlatIndexMetadata,
+        joinColumnName,
+      };
+    });
+
+    this.logger.log(
+      `${isDryRun ? '[DRY RUN] ' : ''}Found ${flatIndexBuildPlans.length} missing relation join column index(es) for workspace ${workspaceId}: ${flatIndexBuildPlans.map(({ universalFlatIndexMetadata }) => universalFlatIndexMetadata.name).join(', ')}`,
+    );
+
+    if (isDryRun) {
+      return;
+    }
+
+    const queryRunner = dataSource.createQueryRunner();
+    let isQueryRunnerConnected = false;
+
+    try {
+      await queryRunner.connect();
+      isQueryRunnerConnected = true;
+
+      for (const {
+        flatObjectMetadata,
+        universalFlatIndexMetadata,
+        joinColumnName,
+      } of flatIndexBuildPlans) {
+        const { schemaName, tableName } = getWorkspaceSchemaContextForMigration({
+          workspaceId,
+          objectMetadata: flatObjectMetadata,
+        });
+
+        await this.workspaceSchemaManagerService.indexManager.createIndex({
+          queryRunner,
+          schemaName,
+          tableName,
+          index: {
+            name: universalFlatIndexMetadata.name,
+            columns: [joinColumnName],
+            isUnique: universalFlatIndexMetadata.isUnique,
+            type: universalFlatIndexMetadata.indexType,
+            where: universalFlatIndexMetadata.indexWhereClause ?? undefined,
+          },
+          concurrently: true,
+        });
+
+        this.logger.log(
+          `Created index ${universalFlatIndexMetadata.name} on workspace ${workspaceId}`,
+        );
+      }
+    } finally {
+      if (isQueryRunnerConnected) {
+        await queryRunner.release();
+      }
+    }
+
+    const { twentyStandardFlatApplication } =
+      await this.applicationService.findWorkspaceTwentyStandardAndCustomApplicationOrThrow(
+        { workspaceId },
+      );
+
+    const validateAndBuildResult =
+      await this.workspaceMigrationValidateBuildAndRunService.validateBuildAndRunWorkspaceMigration(
+        {
+          isSystemBuild: true,
+          allFlatEntityOperationByMetadataName: {
+            index: {
+              flatEntityToCreate: flatIndexBuildPlans.map(
+                ({ universalFlatIndexMetadata }) => universalFlatIndexMetadata,
+              ),
+              flatEntityToDelete: [],
+              flatEntityToUpdate: [],
+            },
+          },
+          workspaceId,
+          applicationUniversalIdentifier:
+            twentyStandardFlatApplication.universalIdentifier,
+        },
+      );
+
+    if (validateAndBuildResult.status === 'fail') {
+      this.logger.error(
+        `Failed to persist relation join column index metadata:\n${JSON.stringify(validateAndBuildResult, null, 2)}`,
+      );
+
+      throw new Error(
+        `Failed to persist relation join column index metadata for workspace ${workspaceId}`,
+      );
+    }
+
+    this.logger.log(
+      `Successfully backfilled ${flatIndexBuildPlans.length} relation join column index(es) for workspace ${workspaceId}`,
+    );
+  }
+}
@@ -10,6 +10,7 @@ import { V2_3_UpgradeVersionCommandModule } from 'src/database/commands/upgrade-
 import { V2_4_UpgradeVersionCommandModule } from 'src/database/commands/upgrade-version-command/2-4/2-4-upgrade-version-command.module';
 import { V2_5_UpgradeVersionCommandModule } from 'src/database/commands/upgrade-version-command/2-5/2-5-upgrade-version-command.module';
 import { V2_7_UpgradeVersionCommandModule } from 'src/database/commands/upgrade-version-command/2-7/2-7-upgrade-version-command.module';
+import { V2_8_UpgradeVersionCommandModule } from 'src/database/commands/upgrade-version-command/2-8/2-8-upgrade-version-command.module';

@Module({
  imports: [
@@ -23,6 +24,7 @@ import { V2_7_UpgradeVersionCommandModule } from 'src/database/commands/upgrade-
    V2_4_UpgradeVersionCommandModule,
    V2_5_UpgradeVersionCommandModule,
    V2_7_UpgradeVersionCommandModule,
+    V2_8_UpgradeVersionCommandModule,
  ],
 })
 export class WorkspaceCommandProviderModule {}