Centralize system field side effects + search field metadata (#22594)
## Introduction Closes twentyhq/core-team-issues#2635 and twentyhq/core-team-issues#2642 and twentyhq/core-team-issues#2589 Object system fields (`searchVector` + its GIN index + `searchFieldMetadata`, the reserved system fields, default relations) were provisioned through several scattered, path-specific code paths. As a result the **app-manifest sync path** authored objects with an empty/`NULL` `searchVector` and **zero `searchFieldMetadata`**, so app-owned objects shipped a broken generated search column (see #22657). The generation logic also lived partly in imperative services rather than in the metadata side-effect engine, and relied on non-deterministic (`v4`) universal identifiers that `twenty apply` could not converge, destroying manually backfilled rows. This PR centralizes every object-creation system side effect into the **metadata side-effect engine**, extends the engine to keep search metadata consistent on field delete and object relabel, makes the standard app's search identifiers deterministic, and ships upgrade commands to reconcile existing workspaces. ## What changed ### Side effects moved into the metadata side-effect engine New dedicated, self-contained handlers — so every write path (API and app manifest) gets identical results, and side effects never trigger other side effects. **Object create / delete** (`handlers/object-metadata`) * **`objectSystemFieldsOnCreate`** — generates the 7 reserved system fields (`id`, `createdAt`, `updatedAt`, `deletedAt`, `createdBy`, `updatedBy`, `position`). * **`objectSearchVectorOnCreate`** — provisions the full-text search surface as one unit: the `searchVector` `TS_VECTOR` field, its backing GIN index, and the `searchFieldMetadata` row (for searchable objects whose label identifier is a searchable field) that keeps `searchVector` populated instead of `NULL`. * **`objectSystemSideEffectsOnDelete`** — tears the above down on object deletion. **Search-metadata consistency on relabel / field delete** (new — these are what close the manifest-path gaps) * **`objectSearchVectorOnUpdate`** (`handlers/object-metadata`) — when a searchable object is relabeled onto a new searchable field, provisions the `searchFieldMetadata` row that indexes it. Relabeling is **additive**: existing rows (e.g. the provisioned `name` row) are preserved, so the previous label identifier stays searchable. Mirrors the API update path so a manifest re-sync that changes the label identifier reaches search parity. No-ops for junction objects (`id` label identifier) and non-searchable field types. * **`fieldSearchFieldMetadataOnDelete`** (`handlers/field-metadata`) — when a field is deleted, cascade-deletes every `searchFieldMetadata` row that indexes it. `searchFieldMetadata` is excluded from manifest deletion inference, so this explicit cascade is what covers **both the API and manifest paths** (the object-scoped DB cascade only fires on object deletion). Uses the `searchFieldMetadataUniversalIdentifiers` aggregator on the flat field for an O(k) lookup instead of scanning all rows. The **default `name` field and default relations are now caller-provided default fields** (SDK autocomplete on the manifest path, input transpiler on the API path) rather than system side effects — removing duplicate name generation, the imperative `build-default-*-for-custom-object` utilities, and the ad-hoc system-field integrity validator. ### Deterministic identifiers for the standard app The twenty-standard search GIN index and `searchFieldMetadata` now derive deterministic universal identifiers (`getIndexUniversalIdentifier` / `getSearchFieldUniversalIdentifier`) instead of `v4`, so `twenty apply` converges instead of recreating. ### Upgrade commands (`2-20`) to reconcile existing workspaces **Instance commands** (run once per instance; ordered fast → slow → workspace): 1. **`AddIsSystemSideEffectToSearchFieldMetadata`** (fast) — adds the `isSystemSideEffect` column to `core.searchFieldMetadata`. Defaults to `true`, which also correctly backfills every existing row since `searchFieldMetadata` is always system-derived (never user-authored). 2. **`BackfillNameFieldIsSystemSideEffect`** (slow) — re-flags existing `name` fields from `isSystemSideEffect: true` → `false`, since the default `name` field is now a caller-provided default like any other user-owned field (it was provisioned as `true` in 2.15 → 2.19). This is a pure data backfill, so the bulk `UPDATE` lives in `runDataMigration()` rather than `up()` — keeping it out of the fast schema transaction avoids holding an `ACCESS EXCLUSIVE` lock that could stall reads during the deploy. Slow instance commands still run before every workspace command of the version, so the fresh value is in place before the search-reconcile workspace commands recompute the `fieldMetadata` flat-entity cache. Scoping by name alone is safe (no engine-owned field is named `name`); `down()` is best-effort (pre-2.15 `false` rows are indistinguishable from flipped ones). **Workspace commands** (idempotent, dry-run supported): 1. **`reconcile-search-vector-gin-index-universal-identifier`** — re-owns every searchVector GIN index UID to its deterministic value (all applications), then backfills the missing GIN index for installed-app objects. 2. **`reconcile-search-field-metadata`** — re-owns every `searchFieldMetadata` UID (all applications), then backfills the missing rows for installed-app searchable objects. 3. **`rebuild-installed-app-search-vectors`** — rebuilds the `searchVector` column of every installed-app `TS_VECTOR` field, once the index and rows exist. Design notes: * **Re-own is global** (twenty-standard, workspace-custom, installed) — a UID convergence keyed on each row's own application. * **Backfill is installed-app only** — standard/custom objects already have these rows via the manifest funnel. * Re-own runs **before** backfill and is transaction-guarded; a failure aborts that workspace to avoid a unique-identifier collision. ## Tests * Integration: app manifest sync now asserts system fields + searchable objects (searchVector, GIN index, searchFieldMetadata) are created; a new relabel suite drives three manifest syncs and asserts records stay searchable through the old + new label identifiers and lose searchability when a field is removed; removed the obsolete system-fields-integrity suite/snapshots. * Unit: per-handler side-effect specs (including the new `objectSearchVectorOnUpdate` and `fieldSearchFieldMetadataOnDelete` handlers), and per-util specs for the re-own / backfill operation builders and the GIN-index classifier. ## Upgrade / migration notes * Existing workspaces converge on the next upgrade run via the `2-20` instance + workspace commands (idempotent, dry-run supported). * Backfill and rebuild go through the workspace-migration runner (automatic cache invalidation); the re-own step invalidates only the affected flat-entity maps directly. * The cross-version upgrade CI now flushes the cache before running the upgrade, so the new version recomputes every flat-entity map from the database instead of reading blobs the old version serialized in an older shape. ## Follow-up * `object-metadata.service.ts` still carries a `TODO: remove once default view fields move to the metadata side effect engine` — default view fields are the next candidate to move into the engine. * A single manifest sync cannot yet both create a field and relabel the object onto it, because `objectMetadata.update` is ordered before `fieldMetadata.create` in the migration runner. Tracked in twentyhq/core-team-issues#2655; to be fixed in a follow-up.
This commit is contained in:
+23
-16
@@ -5,15 +5,16 @@ import { buildSearchFieldMetadataBackfillOperations } from 'src/database/command
|
||||
import { type FlatEntityMaps } from 'src/engine/metadata-modules/flat-entity/types/flat-entity-maps.type';
|
||||
import { getFlatFieldMetadataMock } from 'src/engine/metadata-modules/flat-field-metadata/__mocks__/get-flat-field-metadata.mock';
|
||||
import { type FlatFieldMetadata } from 'src/engine/metadata-modules/flat-field-metadata/types/flat-field-metadata.type';
|
||||
import { SEARCH_VECTOR_FIELD } from 'src/engine/metadata-modules/search-field-metadata/constants/search-vector-field.constants';
|
||||
import {
|
||||
getFlatObjectMetadataMock,
|
||||
getStandardFlatObjectMetadataMock,
|
||||
} from 'src/engine/metadata-modules/flat-object-metadata/__mocks__/get-flat-object-metadata.mock';
|
||||
import { type FlatObjectMetadata } from 'src/engine/metadata-modules/flat-object-metadata/types/flat-object-metadata.type';
|
||||
import { type FlatSearchFieldMetadata } from 'src/engine/metadata-modules/flat-search-field-metadata/types/flat-search-field-metadata.type';
|
||||
import { SEARCH_VECTOR_FIELD } from 'src/engine/metadata-modules/search-field-metadata/constants/search-vector-field.constants';
|
||||
|
||||
const CUSTOM_APPLICATION_UNIVERSAL_IDENTIFIER = 'custom-application-uid';
|
||||
const CUSTOM_APPLICATION_UNIVERSAL_IDENTIFIER =
|
||||
'c0c1c2c3-c4c5-4000-8000-000000000001';
|
||||
const CUSTOM_APPLICATION_ID = 'custom-application-id';
|
||||
|
||||
const buildUniversalIdentifiersByApplicationId = (
|
||||
@@ -129,6 +130,7 @@ const buildSearchFieldMetadata = ({
|
||||
applicationId: 'unused-application-id',
|
||||
applicationUniversalIdentifier,
|
||||
position,
|
||||
isSystemSideEffect: true,
|
||||
workspaceId: 'workspace-id',
|
||||
createdAt,
|
||||
updatedAt: createdAt,
|
||||
@@ -221,8 +223,9 @@ describe('buildSearchFieldMetadataBackfillOperations', () => {
|
||||
searchVectorField,
|
||||
]),
|
||||
flatSearchFieldMetadataMaps: buildFlatSearchFieldMetadataMaps([]),
|
||||
standardFlatSearchFieldMetadataMaps:
|
||||
buildFlatSearchFieldMetadataMaps([]),
|
||||
standardFlatSearchFieldMetadataMaps: buildFlatSearchFieldMetadataMaps(
|
||||
[],
|
||||
),
|
||||
customApplicationId: CUSTOM_APPLICATION_ID,
|
||||
});
|
||||
|
||||
@@ -288,8 +291,9 @@ describe('buildSearchFieldMetadataBackfillOperations', () => {
|
||||
flatObjectMetadataMaps: buildFlatObjectMetadataMaps([junctionObject]),
|
||||
flatFieldMetadataMaps: buildFlatFieldMetadataMaps([idField]),
|
||||
flatSearchFieldMetadataMaps: buildFlatSearchFieldMetadataMaps([]),
|
||||
standardFlatSearchFieldMetadataMaps:
|
||||
buildFlatSearchFieldMetadataMaps([]),
|
||||
standardFlatSearchFieldMetadataMaps: buildFlatSearchFieldMetadataMaps(
|
||||
[],
|
||||
),
|
||||
customApplicationId: CUSTOM_APPLICATION_ID,
|
||||
});
|
||||
|
||||
@@ -592,8 +596,7 @@ describe('buildSearchFieldMetadataBackfillOperations', () => {
|
||||
fieldMetadataId: nameField.id,
|
||||
objectMetadataUniversalIdentifier: customObject.universalIdentifier,
|
||||
fieldMetadataUniversalIdentifier: nameField.universalIdentifier,
|
||||
applicationUniversalIdentifier:
|
||||
CUSTOM_APPLICATION_UNIVERSAL_IDENTIFIER,
|
||||
applicationUniversalIdentifier: CUSTOM_APPLICATION_UNIVERSAL_IDENTIFIER,
|
||||
});
|
||||
|
||||
const { flatSearchFieldMetadatasToCreateByApplicationUniversalIdentifier } =
|
||||
@@ -606,8 +609,9 @@ describe('buildSearchFieldMetadataBackfillOperations', () => {
|
||||
flatSearchFieldMetadataMaps: buildFlatSearchFieldMetadataMaps([
|
||||
existingSearchFieldMetadata,
|
||||
]),
|
||||
standardFlatSearchFieldMetadataMaps:
|
||||
buildFlatSearchFieldMetadataMaps([]),
|
||||
standardFlatSearchFieldMetadataMaps: buildFlatSearchFieldMetadataMaps(
|
||||
[],
|
||||
),
|
||||
customApplicationId: CUSTOM_APPLICATION_ID,
|
||||
});
|
||||
|
||||
@@ -650,8 +654,9 @@ describe('buildSearchFieldMetadataBackfillOperations', () => {
|
||||
flatObjectMetadataMaps: buildFlatObjectMetadataMaps([customObject]),
|
||||
flatFieldMetadataMaps: buildFlatFieldMetadataMaps([relationNameField]),
|
||||
flatSearchFieldMetadataMaps: buildFlatSearchFieldMetadataMaps([]),
|
||||
standardFlatSearchFieldMetadataMaps:
|
||||
buildFlatSearchFieldMetadataMaps([]),
|
||||
standardFlatSearchFieldMetadataMaps: buildFlatSearchFieldMetadataMaps(
|
||||
[],
|
||||
),
|
||||
customApplicationId: CUSTOM_APPLICATION_ID,
|
||||
});
|
||||
|
||||
@@ -685,8 +690,9 @@ describe('buildSearchFieldMetadataBackfillOperations', () => {
|
||||
nameDescriptionField,
|
||||
]),
|
||||
flatSearchFieldMetadataMaps: buildFlatSearchFieldMetadataMaps([]),
|
||||
standardFlatSearchFieldMetadataMaps:
|
||||
buildFlatSearchFieldMetadataMaps([]),
|
||||
standardFlatSearchFieldMetadataMaps: buildFlatSearchFieldMetadataMaps(
|
||||
[],
|
||||
),
|
||||
customApplicationId: CUSTOM_APPLICATION_ID,
|
||||
});
|
||||
|
||||
@@ -716,8 +722,9 @@ describe('buildSearchFieldMetadataBackfillOperations', () => {
|
||||
nameDescriptionField,
|
||||
]),
|
||||
flatSearchFieldMetadataMaps: buildFlatSearchFieldMetadataMaps([]),
|
||||
standardFlatSearchFieldMetadataMaps:
|
||||
buildFlatSearchFieldMetadataMaps([]),
|
||||
standardFlatSearchFieldMetadataMaps: buildFlatSearchFieldMetadataMaps(
|
||||
[],
|
||||
),
|
||||
customApplicationId: CUSTOM_APPLICATION_ID,
|
||||
});
|
||||
|
||||
|
||||
+23
@@ -0,0 +1,23 @@
|
||||
import { QueryRunner } from 'typeorm';
|
||||
|
||||
import { RegisteredInstanceCommand } from 'src/engine/core-modules/upgrade/decorators/registered-instance-command.decorator';
|
||||
import { FastInstanceCommand } from 'src/engine/core-modules/upgrade/interfaces/fast-instance-command.interface';
|
||||
|
||||
// searchFieldMetadata rows are always system-derived (never user-authored), so the
|
||||
// new column defaults to true, which also backfills every existing row correctly.
|
||||
@RegisteredInstanceCommand('2.20.0', 1783580127637)
|
||||
export class AddIsSystemSideEffectToSearchFieldMetadataFastInstanceCommand
|
||||
implements FastInstanceCommand
|
||||
{
|
||||
public async up(queryRunner: QueryRunner): Promise<void> {
|
||||
await queryRunner.query(
|
||||
'ALTER TABLE "core"."searchFieldMetadata" ADD COLUMN IF NOT EXISTS "isSystemSideEffect" boolean NOT NULL DEFAULT true',
|
||||
);
|
||||
}
|
||||
|
||||
public async down(queryRunner: QueryRunner): Promise<void> {
|
||||
await queryRunner.query(
|
||||
'ALTER TABLE "core"."searchFieldMetadata" DROP COLUMN IF EXISTS "isSystemSideEffect"',
|
||||
);
|
||||
}
|
||||
}
|
||||
+38
@@ -0,0 +1,38 @@
|
||||
import { DataSource, QueryRunner } from 'typeorm';
|
||||
|
||||
import { RegisteredInstanceCommand } from 'src/engine/core-modules/upgrade/decorators/registered-instance-command.decorator';
|
||||
import { SlowInstanceCommand } from 'src/engine/core-modules/upgrade/interfaces/slow-instance-command.interface';
|
||||
|
||||
// The default `name` field used to be provisioned with isSystemSideEffect: true
|
||||
// (2.15 → 2.19). It is now a caller-provided default (isSystemSideEffect: false),
|
||||
// like every other user-owned field, so existing rows must be re-flagged.
|
||||
//
|
||||
// Scoping by name alone is safe: no engine-owned field is named `name` (the
|
||||
// reserved set is id/createdAt/updatedAt/deletedAt/createdBy/updatedBy/position/
|
||||
// searchVector), and API-created custom fields are always false already.
|
||||
//
|
||||
// This is a pure data backfill, so the write lives in runDataMigration() (slow
|
||||
// instance command) rather than up(): keeping the bulk UPDATE out of the fast
|
||||
// schema transaction avoids holding an ACCESS EXCLUSIVE lock that could stall
|
||||
// reads during the deploy. Slow instance commands still run before every
|
||||
// workspace command of the version (order is fast → slow → workspace), so the
|
||||
// fresh value is in place before the 2.20 search reconcile workspace commands
|
||||
// flush and recompute the fieldMetadata flat-entity cache (the UPDATE itself
|
||||
// does not invalidate the per-workspace cache).
|
||||
@RegisteredInstanceCommand('2.20.0', 1783529458168, { type: 'slow' })
|
||||
export class BackfillNameFieldIsSystemSideEffectSlowInstanceCommand implements SlowInstanceCommand {
|
||||
async runDataMigration(dataSource: DataSource): Promise<void> {
|
||||
await dataSource.query(
|
||||
`UPDATE "core"."fieldMetadata" SET "isSystemSideEffect" = false WHERE "name" = 'name' AND "isSystemSideEffect" = true`,
|
||||
);
|
||||
}
|
||||
|
||||
public async up(_queryRunner: QueryRunner): Promise<void> {
|
||||
return;
|
||||
}
|
||||
|
||||
// Intentional no-op: this backfill cannot be safely reversed. Pre-2.15 `name`
|
||||
public async down(_queryRunner: QueryRunner): Promise<void> {
|
||||
return;
|
||||
}
|
||||
}
|
||||
+19
-5
@@ -1,30 +1,44 @@
|
||||
import { Module } from '@nestjs/common';
|
||||
import { TypeOrmModule } from '@nestjs/typeorm';
|
||||
|
||||
import { WorkspaceIteratorModule } from 'src/database/commands/command-runners/workspace-iterator.module';
|
||||
import { BackfillActorSourceEnumValuesCommand } from 'src/database/commands/upgrade-version-command/2-20/2-20-workspace-command-1783499671542-backfill-actor-source-enum-values.command';
|
||||
import { BackfillWorkflowVersionToCoreCommand } from 'src/database/commands/upgrade-version-command/2-20/2-20-workspace-command-1783526282685-backfill-workflow-version-to-core.command';
|
||||
import { AddMessageCampaignStatFieldsCommand } from 'src/database/commands/upgrade-version-command/2-20/2-20-workspace-command-1783525261000-add-message-campaign-stat-fields.command';
|
||||
import { CreateMessageListViewCommand } from 'src/database/commands/upgrade-version-command/2-20/2-20-workspace-command-1783525261001-create-message-list-view.command';
|
||||
import { BackfillWorkflowVersionToCoreCommand } from 'src/database/commands/upgrade-version-command/2-20/2-20-workspace-command-1783526282685-backfill-workflow-version-to-core.command';
|
||||
import { ReconcileSearchVectorGinIndexUniversalIdentifierCommand } from 'src/database/commands/upgrade-version-command/2-20/2-20-workspace-command-1783529458169-reconcile-search-vector-gin-index-universal-identifier.command';
|
||||
import { ReconcileSearchFieldMetadataCommand } from 'src/database/commands/upgrade-version-command/2-20/2-20-workspace-command-1783529458170-reconcile-search-field-metadata.command';
|
||||
import { RebuildInstalledAppSearchVectorsCommand } from 'src/database/commands/upgrade-version-command/2-20/2-20-workspace-command-1783529458171-rebuild-installed-app-search-vectors.command';
|
||||
import { ApplicationModule } from 'src/engine/core-modules/application/application.module';
|
||||
import { WorkflowVersionCoreModule } from 'src/engine/core-modules/workflow/workflow-version-core.module';
|
||||
import { IndexMetadataEntity } from 'src/engine/metadata-modules/index-metadata/index-metadata.entity';
|
||||
import { SearchFieldMetadataEntity } from 'src/engine/metadata-modules/search-field-metadata/search-field-metadata.entity';
|
||||
import { WorkspaceMetadataVersionModule } from 'src/engine/metadata-modules/workspace-metadata-version/workspace-metadata-version.module';
|
||||
import { WorkspaceSchemaManagerModule } from 'src/engine/twenty-orm/workspace-schema-manager/workspace-schema-manager.module';
|
||||
import { WorkspaceCacheModule } from 'src/engine/workspace-cache/workspace-cache.module';
|
||||
import { WorkspaceMigrationRunnerModule } from 'src/engine/workspace-manager/workspace-migration/workspace-migration-runner/workspace-migration-runner.module';
|
||||
import { WorkspaceMigrationModule } from 'src/engine/workspace-manager/workspace-migration/workspace-migration.module';
|
||||
|
||||
@Module({
|
||||
imports: [
|
||||
ApplicationModule,
|
||||
WorkspaceIteratorModule,
|
||||
WorkspaceCacheModule,
|
||||
WorkspaceMigrationModule,
|
||||
WorkspaceSchemaManagerModule,
|
||||
TypeOrmModule.forFeature([IndexMetadataEntity, SearchFieldMetadataEntity]),
|
||||
WorkflowVersionCoreModule,
|
||||
WorkspaceCacheModule,
|
||||
WorkspaceIteratorModule,
|
||||
WorkspaceMetadataVersionModule,
|
||||
WorkspaceMigrationModule,
|
||||
WorkspaceMigrationRunnerModule,
|
||||
WorkspaceSchemaManagerModule,
|
||||
],
|
||||
providers: [
|
||||
AddMessageCampaignStatFieldsCommand,
|
||||
CreateMessageListViewCommand,
|
||||
BackfillActorSourceEnumValuesCommand,
|
||||
BackfillWorkflowVersionToCoreCommand,
|
||||
ReconcileSearchVectorGinIndexUniversalIdentifierCommand,
|
||||
ReconcileSearchFieldMetadataCommand,
|
||||
RebuildInstalledAppSearchVectorsCommand,
|
||||
],
|
||||
})
|
||||
export class V2_20_UpgradeVersionCommandModule {}
|
||||
|
||||
+281
@@ -0,0 +1,281 @@
|
||||
import { InjectRepository } from '@nestjs/typeorm';
|
||||
|
||||
import { Command } from 'nest-commander';
|
||||
import { isDefined } from 'twenty-shared/utils';
|
||||
import { Repository } from 'typeorm';
|
||||
|
||||
import { ActiveOrSuspendedWorkspaceCommandRunner } from 'src/database/commands/command-runners/active-or-suspended-workspace.command-runner';
|
||||
import { WorkspaceIteratorService } from 'src/database/commands/command-runners/workspace-iterator.service';
|
||||
import { type RunOnWorkspaceArgs } from 'src/database/commands/command-runners/workspace.command-runner';
|
||||
import { buildSearchVectorGinIndexBackfillOperations } from 'src/database/commands/upgrade-version-command/2-20/utils/build-search-vector-gin-index-backfill-operations.util';
|
||||
import {
|
||||
buildSearchVectorGinIndexReOwnOperations,
|
||||
type SearchVectorGinIndexUniversalIdentifierUpdate,
|
||||
} from 'src/database/commands/upgrade-version-command/2-20/utils/build-search-vector-gin-index-re-own-operations.util';
|
||||
import { ApplicationService } from 'src/engine/core-modules/application/application.service';
|
||||
import { RegisteredWorkspaceCommand } from 'src/engine/core-modules/upgrade/decorators/registered-workspace-command.decorator';
|
||||
import { getMetadataFlatEntityMapsKey } from 'src/engine/metadata-modules/flat-entity/utils/get-metadata-flat-entity-maps-key.util';
|
||||
import { getMetadataRelatedMetadataNames } from 'src/engine/metadata-modules/flat-entity/utils/get-metadata-related-metadata-names.util';
|
||||
import { getMetadataSerializedRelationNames } from 'src/engine/metadata-modules/flat-entity/utils/get-metadata-serialized-relation-names.util';
|
||||
import { IndexMetadataEntity } from 'src/engine/metadata-modules/index-metadata/index-metadata.entity';
|
||||
import { WorkspaceMetadataVersionService } from 'src/engine/metadata-modules/workspace-metadata-version/services/workspace-metadata-version.service';
|
||||
import { WorkspaceCacheService } from 'src/engine/workspace-cache/services/workspace-cache.service';
|
||||
import { type UniversalFlatIndexMetadata } from 'src/engine/workspace-manager/workspace-migration/universal-flat-entity/types/universal-flat-index-metadata.type';
|
||||
import { WorkspaceMigrationValidateBuildAndRunService } from 'src/engine/workspace-manager/workspace-migration/services/workspace-migration-validate-build-and-run-service';
|
||||
|
||||
type SearchVectorGinIndexReconciliationOperations = {
|
||||
indexUniversalIdentifierUpdates: SearchVectorGinIndexUniversalIdentifierUpdate[];
|
||||
flatIndexesToCreateByApplicationUniversalIdentifier: Record<
|
||||
string,
|
||||
UniversalFlatIndexMetadata[]
|
||||
>;
|
||||
};
|
||||
|
||||
@RegisteredWorkspaceCommand('2.20.0', 1783529458169)
|
||||
@Command({
|
||||
name: 'upgrade:2-20:reconcile-search-vector-gin-index-universal-identifier',
|
||||
description:
|
||||
'Converge every searchVector GIN index universal identifier to its deterministic derivation (re-own, all applications) and create the missing GIN index for installed-app objects (backfill). Idempotent, re-own runs before backfill.',
|
||||
})
|
||||
export class ReconcileSearchVectorGinIndexUniversalIdentifierCommand extends ActiveOrSuspendedWorkspaceCommandRunner {
|
||||
constructor(
|
||||
protected readonly workspaceIteratorService: WorkspaceIteratorService,
|
||||
private readonly applicationService: ApplicationService,
|
||||
private readonly workspaceCacheService: WorkspaceCacheService,
|
||||
private readonly workspaceMetadataVersionService: WorkspaceMetadataVersionService,
|
||||
private readonly workspaceMigrationValidateBuildAndRunService: WorkspaceMigrationValidateBuildAndRunService,
|
||||
// eslint-disable-next-line twenty/prefer-workspace-scoped-repository
|
||||
@InjectRepository(IndexMetadataEntity)
|
||||
private readonly indexMetadataRepository: Repository<IndexMetadataEntity>,
|
||||
) {
|
||||
super(workspaceIteratorService);
|
||||
}
|
||||
|
||||
override async runOnWorkspace({
|
||||
workspaceId,
|
||||
options,
|
||||
}: RunOnWorkspaceArgs): Promise<void> {
|
||||
const isDryRun = options.dryRun ?? false;
|
||||
|
||||
const {
|
||||
indexUniversalIdentifierUpdates,
|
||||
flatIndexesToCreateByApplicationUniversalIdentifier,
|
||||
} = await this.computeOperations({ workspaceId });
|
||||
|
||||
const totalIndexesToBackfill = Object.values(
|
||||
flatIndexesToCreateByApplicationUniversalIdentifier,
|
||||
).reduce((total, flatIndexes) => total + flatIndexes.length, 0);
|
||||
|
||||
if (
|
||||
indexUniversalIdentifierUpdates.length === 0 &&
|
||||
totalIndexesToBackfill === 0
|
||||
) {
|
||||
this.logger.log(
|
||||
`No searchVector GIN index to reconcile for workspace ${workspaceId}`,
|
||||
);
|
||||
|
||||
return;
|
||||
}
|
||||
|
||||
this.logger.log(
|
||||
`${isDryRun ? '[DRY RUN] ' : ''}Reconciling ${indexUniversalIdentifierUpdates.length} searchVector GIN index universal identifier(s) and creating ${totalIndexesToBackfill} missing index(es) across ${Object.keys(flatIndexesToCreateByApplicationUniversalIdentifier).length} application(s) for workspace ${workspaceId}`,
|
||||
);
|
||||
|
||||
if (isDryRun) {
|
||||
return;
|
||||
}
|
||||
|
||||
await this.applyOperations({
|
||||
workspaceId,
|
||||
indexUniversalIdentifierUpdates,
|
||||
flatIndexesToCreateByApplicationUniversalIdentifier,
|
||||
});
|
||||
|
||||
this.logger.log(
|
||||
`Reconciled searchVector GIN indexes for workspace ${workspaceId}`,
|
||||
);
|
||||
}
|
||||
|
||||
private async computeOperations({
|
||||
workspaceId,
|
||||
}: {
|
||||
workspaceId: string;
|
||||
}): Promise<SearchVectorGinIndexReconciliationOperations> {
|
||||
const {
|
||||
flatObjectMetadataMaps,
|
||||
flatFieldMetadataMaps,
|
||||
flatIndexMaps,
|
||||
flatApplicationMaps,
|
||||
} = await this.workspaceCacheService.getOrRecompute(workspaceId, [
|
||||
'flatObjectMetadataMaps',
|
||||
'flatFieldMetadataMaps',
|
||||
'flatIndexMaps',
|
||||
'flatApplicationMaps',
|
||||
]);
|
||||
|
||||
const { twentyStandardFlatApplication, workspaceCustomFlatApplication } =
|
||||
await this.applicationService.findWorkspaceTwentyStandardAndCustomApplicationOrThrow(
|
||||
{ workspaceId },
|
||||
);
|
||||
|
||||
const applicationUniversalIdentifierById = new Map(
|
||||
Object.values(flatApplicationMaps.byId)
|
||||
.filter(isDefined)
|
||||
.map((flatApplication) => [
|
||||
flatApplication.id,
|
||||
flatApplication.universalIdentifier,
|
||||
]),
|
||||
);
|
||||
|
||||
const indexUniversalIdentifierUpdates =
|
||||
buildSearchVectorGinIndexReOwnOperations({
|
||||
flatObjectMetadataMaps,
|
||||
flatFieldMetadataMaps,
|
||||
flatIndexMaps,
|
||||
applicationUniversalIdentifierById,
|
||||
});
|
||||
|
||||
const flatIndexesToCreateByApplicationUniversalIdentifier =
|
||||
buildSearchVectorGinIndexBackfillOperations({
|
||||
flatObjectMetadataMaps,
|
||||
flatFieldMetadataMaps,
|
||||
flatIndexMaps,
|
||||
twentyStandardApplicationId: twentyStandardFlatApplication.id,
|
||||
workspaceCustomApplicationId: workspaceCustomFlatApplication.id,
|
||||
});
|
||||
|
||||
return {
|
||||
indexUniversalIdentifierUpdates,
|
||||
flatIndexesToCreateByApplicationUniversalIdentifier,
|
||||
};
|
||||
}
|
||||
|
||||
private async applyOperations({
|
||||
workspaceId,
|
||||
indexUniversalIdentifierUpdates,
|
||||
flatIndexesToCreateByApplicationUniversalIdentifier,
|
||||
}: {
|
||||
workspaceId: string;
|
||||
} & SearchVectorGinIndexReconciliationOperations): Promise<void> {
|
||||
// Re-own before backfill: a deterministic index created next to a surviving legacy
|
||||
// row would collide on the unique universal identifier constraint.
|
||||
await this.applyReOwn({ workspaceId, indexUniversalIdentifierUpdates });
|
||||
await this.applyBackfill({
|
||||
workspaceId,
|
||||
flatIndexesToCreateByApplicationUniversalIdentifier,
|
||||
});
|
||||
}
|
||||
|
||||
private async applyReOwn({
|
||||
workspaceId,
|
||||
indexUniversalIdentifierUpdates,
|
||||
}: {
|
||||
workspaceId: string;
|
||||
indexUniversalIdentifierUpdates: SearchVectorGinIndexUniversalIdentifierUpdate[];
|
||||
}): Promise<void> {
|
||||
if (indexUniversalIdentifierUpdates.length === 0) {
|
||||
return;
|
||||
}
|
||||
|
||||
try {
|
||||
await this.indexMetadataRepository.manager.transaction(
|
||||
async (entityManager) => {
|
||||
const transactionalIndexMetadataRepository =
|
||||
entityManager.getRepository(IndexMetadataEntity);
|
||||
|
||||
for (const {
|
||||
id,
|
||||
deterministicUniversalIdentifier,
|
||||
} of indexUniversalIdentifierUpdates) {
|
||||
await transactionalIndexMetadataRepository.update(
|
||||
{ id, workspaceId },
|
||||
{ universalIdentifier: deterministicUniversalIdentifier },
|
||||
);
|
||||
}
|
||||
},
|
||||
);
|
||||
} catch (error) {
|
||||
// Abort before backfill: creating a deterministic index next to a surviving legacy
|
||||
// row (re-own rolled back) would collide on the unique universal identifier
|
||||
// constraint.
|
||||
this.logger.error(
|
||||
`Failed to re-own ${indexUniversalIdentifierUpdates.length} searchVector GIN index universal identifier(s) for workspace ${workspaceId}, aborting: ${
|
||||
error instanceof Error ? error.message : String(error)
|
||||
}`,
|
||||
);
|
||||
|
||||
throw error;
|
||||
}
|
||||
|
||||
await this.flushIndexCacheAndBumpMetadataVersion(workspaceId);
|
||||
}
|
||||
|
||||
private async applyBackfill({
|
||||
workspaceId,
|
||||
flatIndexesToCreateByApplicationUniversalIdentifier,
|
||||
}: {
|
||||
workspaceId: string;
|
||||
flatIndexesToCreateByApplicationUniversalIdentifier: Record<
|
||||
string,
|
||||
UniversalFlatIndexMetadata[]
|
||||
>;
|
||||
}): Promise<void> {
|
||||
for (const [
|
||||
applicationUniversalIdentifier,
|
||||
flatIndexesToCreate,
|
||||
] of Object.entries(flatIndexesToCreateByApplicationUniversalIdentifier)) {
|
||||
if (flatIndexesToCreate.length === 0) {
|
||||
continue;
|
||||
}
|
||||
|
||||
const validateAndBuildResult =
|
||||
await this.workspaceMigrationValidateBuildAndRunService.validateBuildAndRunWorkspaceMigration(
|
||||
{
|
||||
isSystemBuild: true,
|
||||
allFlatEntityOperationByMetadataName: {
|
||||
index: {
|
||||
flatEntityToCreate: flatIndexesToCreate,
|
||||
flatEntityToDelete: [],
|
||||
flatEntityToUpdate: [],
|
||||
},
|
||||
},
|
||||
workspaceId,
|
||||
applicationUniversalIdentifier,
|
||||
},
|
||||
);
|
||||
|
||||
if (validateAndBuildResult.status === 'fail') {
|
||||
this.logger.error(
|
||||
`Failed to create searchVector GIN index(es) for application ${applicationUniversalIdentifier}:\n${JSON.stringify(
|
||||
validateAndBuildResult,
|
||||
null,
|
||||
2,
|
||||
)}`,
|
||||
);
|
||||
|
||||
throw new Error(
|
||||
`Failed to create searchVector GIN index(es) for workspace ${workspaceId}`,
|
||||
);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
private async flushIndexCacheAndBumpMetadataVersion(
|
||||
workspaceId: string,
|
||||
): Promise<void> {
|
||||
const indexRelatedMetadataNames = [
|
||||
'index',
|
||||
...getMetadataRelatedMetadataNames('index'),
|
||||
...getMetadataSerializedRelationNames('index'),
|
||||
] as const;
|
||||
const cacheKeysToFlush = [
|
||||
...new Set(indexRelatedMetadataNames.map(getMetadataFlatEntityMapsKey)),
|
||||
];
|
||||
|
||||
await this.workspaceCacheService.flush(workspaceId, cacheKeysToFlush);
|
||||
|
||||
await this.workspaceMetadataVersionService.incrementMetadataVersion(
|
||||
workspaceId,
|
||||
);
|
||||
}
|
||||
}
|
||||
+293
@@ -0,0 +1,293 @@
|
||||
import { InjectRepository } from '@nestjs/typeorm';
|
||||
|
||||
import { Command } from 'nest-commander';
|
||||
import { isDefined } from 'twenty-shared/utils';
|
||||
import { Repository } from 'typeorm';
|
||||
|
||||
import { ActiveOrSuspendedWorkspaceCommandRunner } from 'src/database/commands/command-runners/active-or-suspended-workspace.command-runner';
|
||||
import { WorkspaceIteratorService } from 'src/database/commands/command-runners/workspace-iterator.service';
|
||||
import { type RunOnWorkspaceArgs } from 'src/database/commands/command-runners/workspace.command-runner';
|
||||
import { buildSearchFieldMetadataBackfillOperations } from 'src/database/commands/upgrade-version-command/2-20/utils/build-search-field-metadata-backfill-operations.util';
|
||||
import {
|
||||
buildSearchFieldMetadataReOwnOperations,
|
||||
type SearchFieldMetadataUniversalIdentifierUpdate,
|
||||
} from 'src/database/commands/upgrade-version-command/2-20/utils/build-search-field-metadata-re-own-operations.util';
|
||||
import { ApplicationService } from 'src/engine/core-modules/application/application.service';
|
||||
import { RegisteredWorkspaceCommand } from 'src/engine/core-modules/upgrade/decorators/registered-workspace-command.decorator';
|
||||
import { getMetadataFlatEntityMapsKey } from 'src/engine/metadata-modules/flat-entity/utils/get-metadata-flat-entity-maps-key.util';
|
||||
import { getMetadataRelatedMetadataNames } from 'src/engine/metadata-modules/flat-entity/utils/get-metadata-related-metadata-names.util';
|
||||
import { getMetadataSerializedRelationNames } from 'src/engine/metadata-modules/flat-entity/utils/get-metadata-serialized-relation-names.util';
|
||||
import { SearchFieldMetadataEntity } from 'src/engine/metadata-modules/search-field-metadata/search-field-metadata.entity';
|
||||
import { WorkspaceMetadataVersionService } from 'src/engine/metadata-modules/workspace-metadata-version/services/workspace-metadata-version.service';
|
||||
import { WorkspaceCacheService } from 'src/engine/workspace-cache/services/workspace-cache.service';
|
||||
import { type UniversalFlatSearchFieldMetadata } from 'src/engine/workspace-manager/workspace-migration/universal-flat-entity/types/universal-flat-search-field-metadata.type';
|
||||
import { WorkspaceMigrationValidateBuildAndRunService } from 'src/engine/workspace-manager/workspace-migration/services/workspace-migration-validate-build-and-run-service';
|
||||
|
||||
type SearchFieldMetadataReconciliationOperations = {
|
||||
searchFieldMetadataUniversalIdentifierUpdates: SearchFieldMetadataUniversalIdentifierUpdate[];
|
||||
flatSearchFieldMetadatasToCreateByApplicationUniversalIdentifier: Record<
|
||||
string,
|
||||
UniversalFlatSearchFieldMetadata[]
|
||||
>;
|
||||
};
|
||||
|
||||
@RegisteredWorkspaceCommand('2.20.0', 1783529458170)
|
||||
@Command({
|
||||
name: 'upgrade:2-20:reconcile-search-field-metadata',
|
||||
description:
|
||||
'Converge every searchFieldMetadata universal identifier to its deterministic derivation (re-own, all applications) and create the missing searchFieldMetadata row for installed-app searchable objects (backfill). Idempotent, re-own runs before backfill to avoid a unique-identifier collision.',
|
||||
})
|
||||
export class ReconcileSearchFieldMetadataCommand extends ActiveOrSuspendedWorkspaceCommandRunner {
|
||||
constructor(
|
||||
protected readonly workspaceIteratorService: WorkspaceIteratorService,
|
||||
private readonly applicationService: ApplicationService,
|
||||
private readonly workspaceCacheService: WorkspaceCacheService,
|
||||
private readonly workspaceMetadataVersionService: WorkspaceMetadataVersionService,
|
||||
private readonly workspaceMigrationValidateBuildAndRunService: WorkspaceMigrationValidateBuildAndRunService,
|
||||
// eslint-disable-next-line twenty/prefer-workspace-scoped-repository
|
||||
@InjectRepository(SearchFieldMetadataEntity)
|
||||
private readonly searchFieldMetadataRepository: Repository<SearchFieldMetadataEntity>,
|
||||
) {
|
||||
super(workspaceIteratorService);
|
||||
}
|
||||
|
||||
override async runOnWorkspace({
|
||||
workspaceId,
|
||||
options,
|
||||
}: RunOnWorkspaceArgs): Promise<void> {
|
||||
const isDryRun = options.dryRun ?? false;
|
||||
|
||||
const {
|
||||
searchFieldMetadataUniversalIdentifierUpdates,
|
||||
flatSearchFieldMetadatasToCreateByApplicationUniversalIdentifier,
|
||||
} = await this.computeOperations({ workspaceId });
|
||||
|
||||
const totalRowsToBackfill = Object.values(
|
||||
flatSearchFieldMetadatasToCreateByApplicationUniversalIdentifier,
|
||||
).reduce(
|
||||
(total, flatSearchFieldMetadatas) =>
|
||||
total + flatSearchFieldMetadatas.length,
|
||||
0,
|
||||
);
|
||||
|
||||
if (
|
||||
searchFieldMetadataUniversalIdentifierUpdates.length === 0 &&
|
||||
totalRowsToBackfill === 0
|
||||
) {
|
||||
this.logger.log(
|
||||
`No searchFieldMetadata to reconcile for workspace ${workspaceId}`,
|
||||
);
|
||||
|
||||
return;
|
||||
}
|
||||
|
||||
this.logger.log(
|
||||
`${isDryRun ? '[DRY RUN] ' : ''}Reconciling ${searchFieldMetadataUniversalIdentifierUpdates.length} searchFieldMetadata universal identifier(s) and creating ${totalRowsToBackfill} missing row(s) across ${Object.keys(flatSearchFieldMetadatasToCreateByApplicationUniversalIdentifier).length} application(s) for workspace ${workspaceId}`,
|
||||
);
|
||||
|
||||
if (isDryRun) {
|
||||
return;
|
||||
}
|
||||
|
||||
await this.applyOperations({
|
||||
workspaceId,
|
||||
searchFieldMetadataUniversalIdentifierUpdates,
|
||||
flatSearchFieldMetadatasToCreateByApplicationUniversalIdentifier,
|
||||
});
|
||||
|
||||
this.logger.log(
|
||||
`Reconciled searchFieldMetadata for workspace ${workspaceId}`,
|
||||
);
|
||||
}
|
||||
|
||||
private async computeOperations({
|
||||
workspaceId,
|
||||
}: {
|
||||
workspaceId: string;
|
||||
}): Promise<SearchFieldMetadataReconciliationOperations> {
|
||||
const {
|
||||
flatObjectMetadataMaps,
|
||||
flatFieldMetadataMaps,
|
||||
flatSearchFieldMetadataMaps,
|
||||
flatApplicationMaps,
|
||||
} = await this.workspaceCacheService.getOrRecompute(workspaceId, [
|
||||
'flatObjectMetadataMaps',
|
||||
'flatFieldMetadataMaps',
|
||||
'flatSearchFieldMetadataMaps',
|
||||
'flatApplicationMaps',
|
||||
]);
|
||||
|
||||
const { twentyStandardFlatApplication, workspaceCustomFlatApplication } =
|
||||
await this.applicationService.findWorkspaceTwentyStandardAndCustomApplicationOrThrow(
|
||||
{ workspaceId },
|
||||
);
|
||||
|
||||
const applicationUniversalIdentifierById = new Map(
|
||||
Object.values(flatApplicationMaps.byId)
|
||||
.filter(isDefined)
|
||||
.map((flatApplication) => [
|
||||
flatApplication.id,
|
||||
flatApplication.universalIdentifier,
|
||||
]),
|
||||
);
|
||||
|
||||
const searchFieldMetadataUniversalIdentifierUpdates =
|
||||
buildSearchFieldMetadataReOwnOperations({
|
||||
flatFieldMetadataMaps,
|
||||
flatSearchFieldMetadataMaps,
|
||||
applicationUniversalIdentifierById,
|
||||
});
|
||||
|
||||
const flatSearchFieldMetadatasToCreateByApplicationUniversalIdentifier =
|
||||
buildSearchFieldMetadataBackfillOperations({
|
||||
flatObjectMetadataMaps,
|
||||
flatFieldMetadataMaps,
|
||||
flatSearchFieldMetadataMaps,
|
||||
applicationUniversalIdentifierById,
|
||||
twentyStandardApplicationId: twentyStandardFlatApplication.id,
|
||||
workspaceCustomApplicationId: workspaceCustomFlatApplication.id,
|
||||
});
|
||||
|
||||
return {
|
||||
searchFieldMetadataUniversalIdentifierUpdates,
|
||||
flatSearchFieldMetadatasToCreateByApplicationUniversalIdentifier,
|
||||
};
|
||||
}
|
||||
|
||||
private async applyOperations({
|
||||
workspaceId,
|
||||
searchFieldMetadataUniversalIdentifierUpdates,
|
||||
flatSearchFieldMetadatasToCreateByApplicationUniversalIdentifier,
|
||||
}: {
|
||||
workspaceId: string;
|
||||
} & SearchFieldMetadataReconciliationOperations): Promise<void> {
|
||||
// Re-own before backfill: a deterministic row created next to a surviving legacy v4
|
||||
// row (which the deletion-inference short-circuit keeps alive) would collide on the
|
||||
// unique universal identifier constraint.
|
||||
await this.applyReOwn({
|
||||
workspaceId,
|
||||
searchFieldMetadataUniversalIdentifierUpdates,
|
||||
});
|
||||
await this.applyBackfill({
|
||||
workspaceId,
|
||||
flatSearchFieldMetadatasToCreateByApplicationUniversalIdentifier,
|
||||
});
|
||||
}
|
||||
|
||||
private async applyReOwn({
|
||||
workspaceId,
|
||||
searchFieldMetadataUniversalIdentifierUpdates,
|
||||
}: {
|
||||
workspaceId: string;
|
||||
searchFieldMetadataUniversalIdentifierUpdates: SearchFieldMetadataUniversalIdentifierUpdate[];
|
||||
}): Promise<void> {
|
||||
if (searchFieldMetadataUniversalIdentifierUpdates.length === 0) {
|
||||
return;
|
||||
}
|
||||
|
||||
try {
|
||||
await this.searchFieldMetadataRepository.manager.transaction(
|
||||
async (entityManager) => {
|
||||
const transactionalSearchFieldMetadataRepository =
|
||||
entityManager.getRepository(SearchFieldMetadataEntity);
|
||||
|
||||
for (const {
|
||||
id,
|
||||
deterministicUniversalIdentifier,
|
||||
} of searchFieldMetadataUniversalIdentifierUpdates) {
|
||||
await transactionalSearchFieldMetadataRepository.update(
|
||||
{ id, workspaceId },
|
||||
{ universalIdentifier: deterministicUniversalIdentifier },
|
||||
);
|
||||
}
|
||||
},
|
||||
);
|
||||
} catch (error) {
|
||||
// Abort before backfill: creating a deterministic row next to a surviving legacy
|
||||
// row (re-own rolled back) would collide on the unique universal identifier
|
||||
// constraint.
|
||||
this.logger.error(
|
||||
`Failed to re-own ${searchFieldMetadataUniversalIdentifierUpdates.length} searchFieldMetadata universal identifier(s) for workspace ${workspaceId}, aborting: ${
|
||||
error instanceof Error ? error.message : String(error)
|
||||
}`,
|
||||
);
|
||||
|
||||
throw error;
|
||||
}
|
||||
|
||||
await this.flushSearchFieldMetadataCacheAndBumpMetadataVersion(workspaceId);
|
||||
}
|
||||
|
||||
private async applyBackfill({
|
||||
workspaceId,
|
||||
flatSearchFieldMetadatasToCreateByApplicationUniversalIdentifier,
|
||||
}: {
|
||||
workspaceId: string;
|
||||
flatSearchFieldMetadatasToCreateByApplicationUniversalIdentifier: Record<
|
||||
string,
|
||||
UniversalFlatSearchFieldMetadata[]
|
||||
>;
|
||||
}): Promise<void> {
|
||||
for (const [
|
||||
applicationUniversalIdentifier,
|
||||
flatSearchFieldMetadatasToCreate,
|
||||
] of Object.entries(
|
||||
flatSearchFieldMetadatasToCreateByApplicationUniversalIdentifier,
|
||||
)) {
|
||||
if (flatSearchFieldMetadatasToCreate.length === 0) {
|
||||
continue;
|
||||
}
|
||||
|
||||
const validateAndBuildResult =
|
||||
await this.workspaceMigrationValidateBuildAndRunService.validateBuildAndRunWorkspaceMigration(
|
||||
{
|
||||
isSystemBuild: true,
|
||||
allFlatEntityOperationByMetadataName: {
|
||||
searchFieldMetadata: {
|
||||
flatEntityToCreate: flatSearchFieldMetadatasToCreate,
|
||||
flatEntityToDelete: [],
|
||||
flatEntityToUpdate: [],
|
||||
},
|
||||
},
|
||||
workspaceId,
|
||||
applicationUniversalIdentifier,
|
||||
},
|
||||
);
|
||||
|
||||
if (validateAndBuildResult.status === 'fail') {
|
||||
this.logger.error(
|
||||
`Failed to create searchFieldMetadata row(s) for application ${applicationUniversalIdentifier}:\n${JSON.stringify(
|
||||
validateAndBuildResult,
|
||||
null,
|
||||
2,
|
||||
)}`,
|
||||
);
|
||||
|
||||
throw new Error(
|
||||
`Failed to create searchFieldMetadata row(s) for workspace ${workspaceId}`,
|
||||
);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
private async flushSearchFieldMetadataCacheAndBumpMetadataVersion(
|
||||
workspaceId: string,
|
||||
): Promise<void> {
|
||||
const searchFieldMetadataRelatedMetadataNames = [
|
||||
'searchFieldMetadata',
|
||||
...getMetadataRelatedMetadataNames('searchFieldMetadata'),
|
||||
...getMetadataSerializedRelationNames('searchFieldMetadata'),
|
||||
] as const;
|
||||
const cacheKeysToFlush = [
|
||||
...new Set(
|
||||
searchFieldMetadataRelatedMetadataNames.map(getMetadataFlatEntityMapsKey),
|
||||
),
|
||||
];
|
||||
|
||||
await this.workspaceCacheService.flush(workspaceId, cacheKeysToFlush);
|
||||
|
||||
await this.workspaceMetadataVersionService.incrementMetadataVersion(
|
||||
workspaceId,
|
||||
);
|
||||
}
|
||||
}
|
||||
+99
@@ -0,0 +1,99 @@
|
||||
import { Command } from 'nest-commander';
|
||||
import { FieldMetadataType } from 'twenty-shared/types';
|
||||
import { isDefined } from 'twenty-shared/utils';
|
||||
|
||||
import { ActiveOrSuspendedWorkspaceCommandRunner } from 'src/database/commands/command-runners/active-or-suspended-workspace.command-runner';
|
||||
import { WorkspaceIteratorService } from 'src/database/commands/command-runners/workspace-iterator.service';
|
||||
import { type RunOnWorkspaceArgs } from 'src/database/commands/command-runners/workspace.command-runner';
|
||||
import { ApplicationService } from 'src/engine/core-modules/application/application.service';
|
||||
import { RegisteredWorkspaceCommand } from 'src/engine/core-modules/upgrade/decorators/registered-workspace-command.decorator';
|
||||
import { TWENTY_STANDARD_APPLICATION } from 'src/engine/workspace-manager/twenty-standard-application/constants/twenty-standard-applications';
|
||||
import { WorkspaceCacheService } from 'src/engine/workspace-cache/services/workspace-cache.service';
|
||||
import { type UniversalUpdateFieldAction } from 'src/engine/workspace-manager/workspace-migration/workspace-migration-builder/builders/field/types/workspace-migration-field-action';
|
||||
import { WORKSPACE_MIGRATION_ACTION_TYPE } from 'src/engine/workspace-manager/workspace-migration/workspace-migration-builder/constants/workspace-migration-action-type.constant';
|
||||
import { WorkspaceMigrationRunnerService } from 'src/engine/workspace-manager/workspace-migration/workspace-migration-runner/services/workspace-migration-runner.service';
|
||||
|
||||
@RegisteredWorkspaceCommand('2.20.0', 1783529458171)
|
||||
@Command({
|
||||
name: 'upgrade:2-20:rebuild-installed-app-search-vectors',
|
||||
description:
|
||||
'Rebuild the searchVector column of every installed-app TS_VECTOR field from its searchFieldMetadata rows, now that the GIN index and searchFieldMetadata rows exist (commands 1 and 2). Idempotent.',
|
||||
})
|
||||
export class RebuildInstalledAppSearchVectorsCommand extends ActiveOrSuspendedWorkspaceCommandRunner {
|
||||
constructor(
|
||||
protected readonly workspaceIteratorService: WorkspaceIteratorService,
|
||||
private readonly applicationService: ApplicationService,
|
||||
private readonly workspaceCacheService: WorkspaceCacheService,
|
||||
private readonly workspaceMigrationRunnerService: WorkspaceMigrationRunnerService,
|
||||
) {
|
||||
super(workspaceIteratorService);
|
||||
}
|
||||
|
||||
override async runOnWorkspace({
|
||||
workspaceId,
|
||||
options,
|
||||
}: RunOnWorkspaceArgs): Promise<void> {
|
||||
const isDryRun = options.dryRun ?? false;
|
||||
|
||||
const { flatFieldMetadataMaps } =
|
||||
await this.workspaceCacheService.getOrRecompute(workspaceId, [
|
||||
'flatFieldMetadataMaps',
|
||||
]);
|
||||
|
||||
const { twentyStandardFlatApplication, workspaceCustomFlatApplication } =
|
||||
await this.applicationService.findWorkspaceTwentyStandardAndCustomApplicationOrThrow(
|
||||
{ workspaceId },
|
||||
);
|
||||
|
||||
const installedAppTsVectorFlatFieldMetadatas = Object.values(
|
||||
flatFieldMetadataMaps.byUniversalIdentifier,
|
||||
)
|
||||
.filter(isDefined)
|
||||
.filter(
|
||||
(flatFieldMetadata) =>
|
||||
flatFieldMetadata.type === FieldMetadataType.TS_VECTOR &&
|
||||
flatFieldMetadata.applicationId !==
|
||||
twentyStandardFlatApplication.id &&
|
||||
flatFieldMetadata.applicationId !== workspaceCustomFlatApplication.id,
|
||||
);
|
||||
|
||||
if (installedAppTsVectorFlatFieldMetadatas.length === 0) {
|
||||
this.logger.log(
|
||||
`No installed-app searchVector to rebuild for workspace ${workspaceId}`,
|
||||
);
|
||||
|
||||
return;
|
||||
}
|
||||
|
||||
this.logger.log(
|
||||
`${isDryRun ? '[DRY RUN] ' : ''}Rebuilding ${installedAppTsVectorFlatFieldMetadatas.length} installed-app searchVector(s) for workspace ${workspaceId}`,
|
||||
);
|
||||
|
||||
if (isDryRun) {
|
||||
return;
|
||||
}
|
||||
|
||||
const actions: UniversalUpdateFieldAction[] =
|
||||
installedAppTsVectorFlatFieldMetadatas.map((flatFieldMetadata) => ({
|
||||
type: WORKSPACE_MIGRATION_ACTION_TYPE.update,
|
||||
metadataName: 'fieldMetadata',
|
||||
universalIdentifier: flatFieldMetadata.universalIdentifier,
|
||||
update: {},
|
||||
rebuildSearchVector: true,
|
||||
}));
|
||||
|
||||
await this.workspaceMigrationRunnerService.run({
|
||||
workspaceMigration: {
|
||||
// Cross-app actions; this is only the runner's existence gate, not a scope filter.
|
||||
applicationUniversalIdentifier:
|
||||
TWENTY_STANDARD_APPLICATION.universalIdentifier,
|
||||
actions,
|
||||
},
|
||||
workspaceId,
|
||||
});
|
||||
|
||||
this.logger.log(
|
||||
`Rebuilt ${installedAppTsVectorFlatFieldMetadatas.length} installed-app searchVector(s) for workspace ${workspaceId}`,
|
||||
);
|
||||
}
|
||||
}
|
||||
+2
@@ -0,0 +1,2 @@
|
||||
export const ADD_IS_SYSTEM_SIDE_EFFECT_TO_SEARCH_FIELD_METADATA_UPGRADE_COMMAND_NAME =
|
||||
'2.20.0_AddIsSystemSideEffectToSearchFieldMetadataFastInstanceCommand_1783580127637';
|
||||
+424
@@ -0,0 +1,424 @@
|
||||
import {
|
||||
getFieldUniversalIdentifier,
|
||||
getSearchFieldUniversalIdentifier,
|
||||
} from 'twenty-shared/application';
|
||||
import { FieldMetadataType } from 'twenty-shared/types';
|
||||
|
||||
import { buildSearchFieldMetadataBackfillOperations } from 'src/database/commands/upgrade-version-command/2-20/utils/build-search-field-metadata-backfill-operations.util';
|
||||
import { type FlatEntityMaps } from 'src/engine/metadata-modules/flat-entity/types/flat-entity-maps.type';
|
||||
import { getFlatFieldMetadataMock } from 'src/engine/metadata-modules/flat-field-metadata/__mocks__/get-flat-field-metadata.mock';
|
||||
import { type FlatFieldMetadata } from 'src/engine/metadata-modules/flat-field-metadata/types/flat-field-metadata.type';
|
||||
import { getFlatObjectMetadataMock } from 'src/engine/metadata-modules/flat-object-metadata/__mocks__/get-flat-object-metadata.mock';
|
||||
import { type FlatObjectMetadata } from 'src/engine/metadata-modules/flat-object-metadata/types/flat-object-metadata.type';
|
||||
import { type FlatSearchFieldMetadata } from 'src/engine/metadata-modules/flat-search-field-metadata/types/flat-search-field-metadata.type';
|
||||
import { SEARCH_VECTOR_FIELD } from 'src/engine/metadata-modules/search-field-metadata/constants/search-vector-field.constants';
|
||||
|
||||
// Application universal identifiers must be valid UUIDs: they are used as the v5
|
||||
// namespace by the deterministic identifier helpers.
|
||||
const STANDARD_APPLICATION_ID = 'standard-application-id';
|
||||
const STANDARD_APPLICATION_UID = '11111111-1111-4111-8111-111111111111';
|
||||
const CUSTOM_APPLICATION_ID = 'custom-application-id';
|
||||
const CUSTOM_APPLICATION_UID = '22222222-2222-4222-8222-222222222222';
|
||||
const INSTALLED_APPLICATION_ID = 'installed-application-id';
|
||||
const INSTALLED_APPLICATION_UID = '33333333-3333-4333-8333-333333333333';
|
||||
|
||||
const applicationUniversalIdentifierById = new Map([
|
||||
[STANDARD_APPLICATION_ID, STANDARD_APPLICATION_UID],
|
||||
[CUSTOM_APPLICATION_ID, CUSTOM_APPLICATION_UID],
|
||||
[INSTALLED_APPLICATION_ID, INSTALLED_APPLICATION_UID],
|
||||
]);
|
||||
|
||||
const buildFlatObjectMetadataMaps = (
|
||||
flatObjectMetadatas: FlatObjectMetadata[],
|
||||
): FlatEntityMaps<FlatObjectMetadata> => ({
|
||||
byUniversalIdentifier: Object.fromEntries(
|
||||
flatObjectMetadatas.map((flatObjectMetadata) => [
|
||||
flatObjectMetadata.universalIdentifier,
|
||||
flatObjectMetadata,
|
||||
]),
|
||||
),
|
||||
universalIdentifierById: Object.fromEntries(
|
||||
flatObjectMetadatas.map((flatObjectMetadata) => [
|
||||
flatObjectMetadata.id,
|
||||
flatObjectMetadata.universalIdentifier,
|
||||
]),
|
||||
),
|
||||
universalIdentifiersByApplicationId: {},
|
||||
});
|
||||
|
||||
const buildFlatFieldMetadataMaps = (
|
||||
flatFieldMetadatas: FlatFieldMetadata[],
|
||||
): FlatEntityMaps<FlatFieldMetadata> => ({
|
||||
byUniversalIdentifier: Object.fromEntries(
|
||||
flatFieldMetadatas.map((flatFieldMetadata) => [
|
||||
flatFieldMetadata.universalIdentifier,
|
||||
flatFieldMetadata,
|
||||
]),
|
||||
),
|
||||
universalIdentifierById: Object.fromEntries(
|
||||
flatFieldMetadatas.map((flatFieldMetadata) => [
|
||||
flatFieldMetadata.id,
|
||||
flatFieldMetadata.universalIdentifier,
|
||||
]),
|
||||
),
|
||||
universalIdentifiersByApplicationId: {},
|
||||
});
|
||||
|
||||
const buildFlatSearchFieldMetadataMaps = (
|
||||
flatSearchFieldMetadatas: FlatSearchFieldMetadata[],
|
||||
): FlatEntityMaps<FlatSearchFieldMetadata> => ({
|
||||
byUniversalIdentifier: Object.fromEntries(
|
||||
flatSearchFieldMetadatas.map((flatSearchFieldMetadata) => [
|
||||
flatSearchFieldMetadata.universalIdentifier,
|
||||
flatSearchFieldMetadata,
|
||||
]),
|
||||
),
|
||||
universalIdentifierById: Object.fromEntries(
|
||||
flatSearchFieldMetadatas.map((flatSearchFieldMetadata) => [
|
||||
flatSearchFieldMetadata.id,
|
||||
flatSearchFieldMetadata.universalIdentifier,
|
||||
]),
|
||||
),
|
||||
universalIdentifiersByApplicationId: {},
|
||||
});
|
||||
|
||||
const buildFlatSearchFieldMetadata = ({
|
||||
id,
|
||||
universalIdentifier,
|
||||
objectMetadataId,
|
||||
fieldMetadataId,
|
||||
applicationId,
|
||||
applicationUniversalIdentifier,
|
||||
}: {
|
||||
id: string;
|
||||
universalIdentifier: string;
|
||||
objectMetadataId: string;
|
||||
fieldMetadataId: string;
|
||||
applicationId: string;
|
||||
applicationUniversalIdentifier: string;
|
||||
}): FlatSearchFieldMetadata => {
|
||||
const createdAt = '2024-01-01T00:00:00.000Z';
|
||||
|
||||
return {
|
||||
id,
|
||||
universalIdentifier,
|
||||
objectMetadataId,
|
||||
fieldMetadataId,
|
||||
objectMetadataUniversalIdentifier: `${objectMetadataId}-uid`,
|
||||
fieldMetadataUniversalIdentifier: `${fieldMetadataId}-uid`,
|
||||
tsVectorFieldMetadataId: `${objectMetadataId}-search-vector-field-id`,
|
||||
tsVectorFieldMetadataUniversalIdentifier: `${objectMetadataId}-search-vector-field-uid`,
|
||||
applicationId,
|
||||
applicationUniversalIdentifier,
|
||||
position: 0,
|
||||
isSystemSideEffect: true,
|
||||
workspaceId: 'workspace-id',
|
||||
createdAt,
|
||||
updatedAt: createdAt,
|
||||
};
|
||||
};
|
||||
|
||||
const buildSearchableObjectFixture = ({
|
||||
objectId,
|
||||
objectUniversalIdentifier,
|
||||
applicationId,
|
||||
applicationUniversalIdentifier,
|
||||
labelIdentifierFieldType = FieldMetadataType.TEXT,
|
||||
}: {
|
||||
objectId: string;
|
||||
objectUniversalIdentifier: string;
|
||||
applicationId: string;
|
||||
applicationUniversalIdentifier: string;
|
||||
labelIdentifierFieldType?: FieldMetadataType;
|
||||
}) => {
|
||||
const nameFlatFieldMetadata = getFlatFieldMetadataMock({
|
||||
id: `${objectId}-name-field-id`,
|
||||
universalIdentifier: `${objectUniversalIdentifier}-name-field-uid`,
|
||||
objectMetadataId: objectId,
|
||||
objectMetadataUniversalIdentifier: objectUniversalIdentifier,
|
||||
type: labelIdentifierFieldType,
|
||||
name: 'name',
|
||||
applicationId,
|
||||
applicationUniversalIdentifier,
|
||||
});
|
||||
|
||||
const searchVectorFlatFieldMetadata = getFlatFieldMetadataMock({
|
||||
id: `${objectId}-search-vector-field-id`,
|
||||
universalIdentifier: `${objectUniversalIdentifier}-search-vector-field-uid`,
|
||||
objectMetadataId: objectId,
|
||||
objectMetadataUniversalIdentifier: objectUniversalIdentifier,
|
||||
type: FieldMetadataType.TS_VECTOR,
|
||||
name: SEARCH_VECTOR_FIELD.name,
|
||||
applicationId,
|
||||
applicationUniversalIdentifier,
|
||||
});
|
||||
|
||||
const flatObjectMetadata = getFlatObjectMetadataMock({
|
||||
id: objectId,
|
||||
universalIdentifier: objectUniversalIdentifier,
|
||||
isSearchable: true,
|
||||
applicationId,
|
||||
applicationUniversalIdentifier,
|
||||
labelIdentifierFieldMetadataId: nameFlatFieldMetadata.id,
|
||||
labelIdentifierFieldMetadataUniversalIdentifier:
|
||||
nameFlatFieldMetadata.universalIdentifier,
|
||||
fieldUniversalIdentifiers: [
|
||||
nameFlatFieldMetadata.universalIdentifier,
|
||||
searchVectorFlatFieldMetadata.universalIdentifier,
|
||||
],
|
||||
});
|
||||
|
||||
return {
|
||||
flatObjectMetadata,
|
||||
nameFlatFieldMetadata,
|
||||
searchVectorFlatFieldMetadata,
|
||||
};
|
||||
};
|
||||
|
||||
describe('buildSearchFieldMetadataBackfillOperations', () => {
|
||||
it('creates the searchFieldMetadata row for an installed-app searchable object that has none, grouped under its application', () => {
|
||||
const {
|
||||
flatObjectMetadata,
|
||||
nameFlatFieldMetadata,
|
||||
searchVectorFlatFieldMetadata,
|
||||
} = buildSearchableObjectFixture({
|
||||
objectId: 'installed-object-id',
|
||||
objectUniversalIdentifier: 'installed-object-uid',
|
||||
applicationId: INSTALLED_APPLICATION_ID,
|
||||
applicationUniversalIdentifier: INSTALLED_APPLICATION_UID,
|
||||
});
|
||||
|
||||
const flatSearchFieldMetadatasToCreateByApplicationUniversalIdentifier =
|
||||
buildSearchFieldMetadataBackfillOperations({
|
||||
flatObjectMetadataMaps: buildFlatObjectMetadataMaps([
|
||||
flatObjectMetadata,
|
||||
]),
|
||||
flatFieldMetadataMaps: buildFlatFieldMetadataMaps([
|
||||
nameFlatFieldMetadata,
|
||||
searchVectorFlatFieldMetadata,
|
||||
]),
|
||||
flatSearchFieldMetadataMaps: buildFlatSearchFieldMetadataMaps([]),
|
||||
applicationUniversalIdentifierById,
|
||||
twentyStandardApplicationId: STANDARD_APPLICATION_ID,
|
||||
workspaceCustomApplicationId: CUSTOM_APPLICATION_ID,
|
||||
});
|
||||
|
||||
const createdRows =
|
||||
flatSearchFieldMetadatasToCreateByApplicationUniversalIdentifier[
|
||||
INSTALLED_APPLICATION_UID
|
||||
];
|
||||
|
||||
expect(createdRows).toHaveLength(1);
|
||||
expect(createdRows?.[0].fieldMetadataUniversalIdentifier).toBe(
|
||||
nameFlatFieldMetadata.universalIdentifier,
|
||||
);
|
||||
expect(createdRows?.[0].objectMetadataUniversalIdentifier).toBe(
|
||||
flatObjectMetadata.universalIdentifier,
|
||||
);
|
||||
expect(createdRows?.[0].tsVectorFieldMetadataUniversalIdentifier).toBe(
|
||||
searchVectorFlatFieldMetadata.universalIdentifier,
|
||||
);
|
||||
expect(createdRows?.[0].universalIdentifier).toBe(
|
||||
getSearchFieldUniversalIdentifier({
|
||||
applicationUniversalIdentifier: INSTALLED_APPLICATION_UID,
|
||||
fieldMetadataUniversalIdentifier:
|
||||
nameFlatFieldMetadata.universalIdentifier,
|
||||
}),
|
||||
);
|
||||
});
|
||||
|
||||
it('does not create a row for an installed-app junction object whose label identifier is the UUID id field', () => {
|
||||
const {
|
||||
flatObjectMetadata,
|
||||
nameFlatFieldMetadata,
|
||||
searchVectorFlatFieldMetadata,
|
||||
} = buildSearchableObjectFixture({
|
||||
objectId: 'installed-junction-object-id',
|
||||
objectUniversalIdentifier: 'installed-junction-object-uid',
|
||||
applicationId: INSTALLED_APPLICATION_ID,
|
||||
applicationUniversalIdentifier: INSTALLED_APPLICATION_UID,
|
||||
});
|
||||
|
||||
// The label identifier resolves to the derived `id` field: no search surface.
|
||||
const junctionObject: FlatObjectMetadata = {
|
||||
...flatObjectMetadata,
|
||||
labelIdentifierFieldMetadataUniversalIdentifier:
|
||||
getFieldUniversalIdentifier({
|
||||
applicationUniversalIdentifier: INSTALLED_APPLICATION_UID,
|
||||
objectUniversalIdentifier: flatObjectMetadata.universalIdentifier,
|
||||
name: 'id',
|
||||
}),
|
||||
};
|
||||
|
||||
const flatSearchFieldMetadatasToCreateByApplicationUniversalIdentifier =
|
||||
buildSearchFieldMetadataBackfillOperations({
|
||||
flatObjectMetadataMaps: buildFlatObjectMetadataMaps([junctionObject]),
|
||||
flatFieldMetadataMaps: buildFlatFieldMetadataMaps([
|
||||
nameFlatFieldMetadata,
|
||||
searchVectorFlatFieldMetadata,
|
||||
]),
|
||||
flatSearchFieldMetadataMaps: buildFlatSearchFieldMetadataMaps([]),
|
||||
applicationUniversalIdentifierById,
|
||||
twentyStandardApplicationId: STANDARD_APPLICATION_ID,
|
||||
workspaceCustomApplicationId: CUSTOM_APPLICATION_ID,
|
||||
});
|
||||
|
||||
expect(
|
||||
Object.keys(
|
||||
flatSearchFieldMetadatasToCreateByApplicationUniversalIdentifier,
|
||||
),
|
||||
).toHaveLength(0);
|
||||
});
|
||||
|
||||
it('does not create a row for an installed-app object whose label identifier is a non-searchable type', () => {
|
||||
const {
|
||||
flatObjectMetadata,
|
||||
nameFlatFieldMetadata,
|
||||
searchVectorFlatFieldMetadata,
|
||||
} = buildSearchableObjectFixture({
|
||||
objectId: 'installed-relation-object-id',
|
||||
objectUniversalIdentifier: 'installed-relation-object-uid',
|
||||
applicationId: INSTALLED_APPLICATION_ID,
|
||||
applicationUniversalIdentifier: INSTALLED_APPLICATION_UID,
|
||||
labelIdentifierFieldType: FieldMetadataType.RELATION,
|
||||
});
|
||||
|
||||
const flatSearchFieldMetadatasToCreateByApplicationUniversalIdentifier =
|
||||
buildSearchFieldMetadataBackfillOperations({
|
||||
flatObjectMetadataMaps: buildFlatObjectMetadataMaps([
|
||||
flatObjectMetadata,
|
||||
]),
|
||||
flatFieldMetadataMaps: buildFlatFieldMetadataMaps([
|
||||
nameFlatFieldMetadata,
|
||||
searchVectorFlatFieldMetadata,
|
||||
]),
|
||||
flatSearchFieldMetadataMaps: buildFlatSearchFieldMetadataMaps([]),
|
||||
applicationUniversalIdentifierById,
|
||||
twentyStandardApplicationId: STANDARD_APPLICATION_ID,
|
||||
workspaceCustomApplicationId: CUSTOM_APPLICATION_ID,
|
||||
});
|
||||
|
||||
expect(
|
||||
Object.keys(
|
||||
flatSearchFieldMetadatasToCreateByApplicationUniversalIdentifier,
|
||||
),
|
||||
).toHaveLength(0);
|
||||
});
|
||||
|
||||
it('does not create a row for a twenty-standard searchable object that has none', () => {
|
||||
const {
|
||||
flatObjectMetadata,
|
||||
nameFlatFieldMetadata,
|
||||
searchVectorFlatFieldMetadata,
|
||||
} = buildSearchableObjectFixture({
|
||||
objectId: 'standard-object-id',
|
||||
objectUniversalIdentifier: 'standard-object-uid',
|
||||
applicationId: STANDARD_APPLICATION_ID,
|
||||
applicationUniversalIdentifier: STANDARD_APPLICATION_UID,
|
||||
});
|
||||
|
||||
const flatSearchFieldMetadatasToCreateByApplicationUniversalIdentifier =
|
||||
buildSearchFieldMetadataBackfillOperations({
|
||||
flatObjectMetadataMaps: buildFlatObjectMetadataMaps([
|
||||
flatObjectMetadata,
|
||||
]),
|
||||
flatFieldMetadataMaps: buildFlatFieldMetadataMaps([
|
||||
nameFlatFieldMetadata,
|
||||
searchVectorFlatFieldMetadata,
|
||||
]),
|
||||
flatSearchFieldMetadataMaps: buildFlatSearchFieldMetadataMaps([]),
|
||||
applicationUniversalIdentifierById,
|
||||
twentyStandardApplicationId: STANDARD_APPLICATION_ID,
|
||||
workspaceCustomApplicationId: CUSTOM_APPLICATION_ID,
|
||||
});
|
||||
|
||||
expect(
|
||||
Object.keys(
|
||||
flatSearchFieldMetadatasToCreateByApplicationUniversalIdentifier,
|
||||
),
|
||||
).toHaveLength(0);
|
||||
});
|
||||
|
||||
it('does not create a row for a workspace-custom searchable object that has none', () => {
|
||||
const {
|
||||
flatObjectMetadata,
|
||||
nameFlatFieldMetadata,
|
||||
searchVectorFlatFieldMetadata,
|
||||
} = buildSearchableObjectFixture({
|
||||
objectId: 'custom-object-id',
|
||||
objectUniversalIdentifier: 'custom-object-uid',
|
||||
applicationId: CUSTOM_APPLICATION_ID,
|
||||
applicationUniversalIdentifier: CUSTOM_APPLICATION_UID,
|
||||
});
|
||||
|
||||
const flatSearchFieldMetadatasToCreateByApplicationUniversalIdentifier =
|
||||
buildSearchFieldMetadataBackfillOperations({
|
||||
flatObjectMetadataMaps: buildFlatObjectMetadataMaps([
|
||||
flatObjectMetadata,
|
||||
]),
|
||||
flatFieldMetadataMaps: buildFlatFieldMetadataMaps([
|
||||
nameFlatFieldMetadata,
|
||||
searchVectorFlatFieldMetadata,
|
||||
]),
|
||||
flatSearchFieldMetadataMaps: buildFlatSearchFieldMetadataMaps([]),
|
||||
applicationUniversalIdentifierById,
|
||||
twentyStandardApplicationId: STANDARD_APPLICATION_ID,
|
||||
workspaceCustomApplicationId: CUSTOM_APPLICATION_ID,
|
||||
});
|
||||
|
||||
expect(
|
||||
Object.keys(
|
||||
flatSearchFieldMetadatasToCreateByApplicationUniversalIdentifier,
|
||||
),
|
||||
).toHaveLength(0);
|
||||
});
|
||||
|
||||
it('is idempotent: no create when the installed-app searchFieldMetadata row already exists', () => {
|
||||
const {
|
||||
flatObjectMetadata,
|
||||
nameFlatFieldMetadata,
|
||||
searchVectorFlatFieldMetadata,
|
||||
} = buildSearchableObjectFixture({
|
||||
objectId: 'installed-object-id',
|
||||
objectUniversalIdentifier: 'installed-object-uid',
|
||||
applicationId: INSTALLED_APPLICATION_ID,
|
||||
applicationUniversalIdentifier: INSTALLED_APPLICATION_UID,
|
||||
});
|
||||
|
||||
const existingSearchFieldMetadata = buildFlatSearchFieldMetadata({
|
||||
id: 'existing-search-field-metadata-id',
|
||||
universalIdentifier: getSearchFieldUniversalIdentifier({
|
||||
applicationUniversalIdentifier: INSTALLED_APPLICATION_UID,
|
||||
fieldMetadataUniversalIdentifier:
|
||||
nameFlatFieldMetadata.universalIdentifier,
|
||||
}),
|
||||
objectMetadataId: flatObjectMetadata.id,
|
||||
fieldMetadataId: nameFlatFieldMetadata.id,
|
||||
applicationId: INSTALLED_APPLICATION_ID,
|
||||
applicationUniversalIdentifier: INSTALLED_APPLICATION_UID,
|
||||
});
|
||||
|
||||
const flatSearchFieldMetadatasToCreateByApplicationUniversalIdentifier =
|
||||
buildSearchFieldMetadataBackfillOperations({
|
||||
flatObjectMetadataMaps: buildFlatObjectMetadataMaps([
|
||||
flatObjectMetadata,
|
||||
]),
|
||||
flatFieldMetadataMaps: buildFlatFieldMetadataMaps([
|
||||
nameFlatFieldMetadata,
|
||||
searchVectorFlatFieldMetadata,
|
||||
]),
|
||||
flatSearchFieldMetadataMaps: buildFlatSearchFieldMetadataMaps([
|
||||
existingSearchFieldMetadata,
|
||||
]),
|
||||
applicationUniversalIdentifierById,
|
||||
twentyStandardApplicationId: STANDARD_APPLICATION_ID,
|
||||
workspaceCustomApplicationId: CUSTOM_APPLICATION_ID,
|
||||
});
|
||||
|
||||
expect(
|
||||
Object.keys(
|
||||
flatSearchFieldMetadatasToCreateByApplicationUniversalIdentifier,
|
||||
),
|
||||
).toHaveLength(0);
|
||||
});
|
||||
});
|
||||
+155
@@ -0,0 +1,155 @@
|
||||
import { getSearchFieldUniversalIdentifier } from 'twenty-shared/application';
|
||||
import { FieldMetadataType } from 'twenty-shared/types';
|
||||
|
||||
import { buildSearchFieldMetadataReOwnOperations } from 'src/database/commands/upgrade-version-command/2-20/utils/build-search-field-metadata-re-own-operations.util';
|
||||
import { type FlatEntityMaps } from 'src/engine/metadata-modules/flat-entity/types/flat-entity-maps.type';
|
||||
import { getFlatFieldMetadataMock } from 'src/engine/metadata-modules/flat-field-metadata/__mocks__/get-flat-field-metadata.mock';
|
||||
import { type FlatFieldMetadata } from 'src/engine/metadata-modules/flat-field-metadata/types/flat-field-metadata.type';
|
||||
import { type FlatSearchFieldMetadata } from 'src/engine/metadata-modules/flat-search-field-metadata/types/flat-search-field-metadata.type';
|
||||
|
||||
const STANDARD_APPLICATION_ID = 'standard-application-id';
|
||||
// Application universal identifiers must be valid UUIDs: they are used as the v5
|
||||
// namespace by the deterministic identifier helpers.
|
||||
const STANDARD_APPLICATION_UID = '11111111-1111-4111-8111-111111111111';
|
||||
|
||||
const applicationUniversalIdentifierById = new Map([
|
||||
[STANDARD_APPLICATION_ID, STANDARD_APPLICATION_UID],
|
||||
]);
|
||||
|
||||
const buildFlatFieldMetadataMaps = (
|
||||
flatFieldMetadatas: FlatFieldMetadata[],
|
||||
): FlatEntityMaps<FlatFieldMetadata> => ({
|
||||
byUniversalIdentifier: Object.fromEntries(
|
||||
flatFieldMetadatas.map((flatFieldMetadata) => [
|
||||
flatFieldMetadata.universalIdentifier,
|
||||
flatFieldMetadata,
|
||||
]),
|
||||
),
|
||||
universalIdentifierById: Object.fromEntries(
|
||||
flatFieldMetadatas.map((flatFieldMetadata) => [
|
||||
flatFieldMetadata.id,
|
||||
flatFieldMetadata.universalIdentifier,
|
||||
]),
|
||||
),
|
||||
universalIdentifiersByApplicationId: {},
|
||||
});
|
||||
|
||||
const buildFlatSearchFieldMetadataMaps = (
|
||||
flatSearchFieldMetadatas: FlatSearchFieldMetadata[],
|
||||
): FlatEntityMaps<FlatSearchFieldMetadata> => ({
|
||||
byUniversalIdentifier: Object.fromEntries(
|
||||
flatSearchFieldMetadatas.map((flatSearchFieldMetadata) => [
|
||||
flatSearchFieldMetadata.universalIdentifier,
|
||||
flatSearchFieldMetadata,
|
||||
]),
|
||||
),
|
||||
universalIdentifierById: Object.fromEntries(
|
||||
flatSearchFieldMetadatas.map((flatSearchFieldMetadata) => [
|
||||
flatSearchFieldMetadata.id,
|
||||
flatSearchFieldMetadata.universalIdentifier,
|
||||
]),
|
||||
),
|
||||
universalIdentifiersByApplicationId: {},
|
||||
});
|
||||
|
||||
const buildFlatSearchFieldMetadata = ({
|
||||
id,
|
||||
universalIdentifier,
|
||||
objectMetadataId,
|
||||
fieldMetadataId,
|
||||
}: {
|
||||
id: string;
|
||||
universalIdentifier: string;
|
||||
objectMetadataId: string;
|
||||
fieldMetadataId: string;
|
||||
}): FlatSearchFieldMetadata => {
|
||||
const createdAt = '2024-01-01T00:00:00.000Z';
|
||||
|
||||
return {
|
||||
id,
|
||||
universalIdentifier,
|
||||
objectMetadataId,
|
||||
fieldMetadataId,
|
||||
objectMetadataUniversalIdentifier: `${objectMetadataId}-uid`,
|
||||
fieldMetadataUniversalIdentifier: `${fieldMetadataId}-uid`,
|
||||
tsVectorFieldMetadataId: `${objectMetadataId}-search-vector-field-id`,
|
||||
tsVectorFieldMetadataUniversalIdentifier: `${objectMetadataId}-search-vector-field-uid`,
|
||||
applicationId: STANDARD_APPLICATION_ID,
|
||||
applicationUniversalIdentifier: STANDARD_APPLICATION_UID,
|
||||
position: 0,
|
||||
isSystemSideEffect: true,
|
||||
workspaceId: 'workspace-id',
|
||||
createdAt,
|
||||
updatedAt: createdAt,
|
||||
};
|
||||
};
|
||||
|
||||
const nameFlatFieldMetadata = getFlatFieldMetadataMock({
|
||||
id: 'standard-object-name-field-id',
|
||||
universalIdentifier: 'standard-object-name-field-uid',
|
||||
objectMetadataId: 'standard-object-id',
|
||||
objectMetadataUniversalIdentifier: 'standard-object-uid',
|
||||
type: FieldMetadataType.TEXT,
|
||||
name: 'name',
|
||||
applicationId: STANDARD_APPLICATION_ID,
|
||||
applicationUniversalIdentifier: STANDARD_APPLICATION_UID,
|
||||
});
|
||||
|
||||
describe('buildSearchFieldMetadataReOwnOperations', () => {
|
||||
it('re-owns an existing searchFieldMetadata carrying a non-deterministic universal identifier (all applications)', () => {
|
||||
const existingSearchFieldMetadata = buildFlatSearchFieldMetadata({
|
||||
id: 'existing-search-field-metadata-id',
|
||||
universalIdentifier: 'legacy-v4-search-field-metadata-uid',
|
||||
objectMetadataId: 'standard-object-id',
|
||||
fieldMetadataId: nameFlatFieldMetadata.id,
|
||||
});
|
||||
|
||||
const searchFieldMetadataUniversalIdentifierUpdates =
|
||||
buildSearchFieldMetadataReOwnOperations({
|
||||
flatFieldMetadataMaps: buildFlatFieldMetadataMaps([
|
||||
nameFlatFieldMetadata,
|
||||
]),
|
||||
flatSearchFieldMetadataMaps: buildFlatSearchFieldMetadataMaps([
|
||||
existingSearchFieldMetadata,
|
||||
]),
|
||||
applicationUniversalIdentifierById,
|
||||
});
|
||||
|
||||
expect(searchFieldMetadataUniversalIdentifierUpdates).toEqual([
|
||||
{
|
||||
id: existingSearchFieldMetadata.id,
|
||||
deterministicUniversalIdentifier: getSearchFieldUniversalIdentifier({
|
||||
applicationUniversalIdentifier: STANDARD_APPLICATION_UID,
|
||||
fieldMetadataUniversalIdentifier:
|
||||
nameFlatFieldMetadata.universalIdentifier,
|
||||
}),
|
||||
},
|
||||
]);
|
||||
});
|
||||
|
||||
it('is a no-op for a searchFieldMetadata already carrying its deterministic universal identifier', () => {
|
||||
const existingSearchFieldMetadata = buildFlatSearchFieldMetadata({
|
||||
id: 'existing-search-field-metadata-id',
|
||||
universalIdentifier: getSearchFieldUniversalIdentifier({
|
||||
applicationUniversalIdentifier: STANDARD_APPLICATION_UID,
|
||||
fieldMetadataUniversalIdentifier:
|
||||
nameFlatFieldMetadata.universalIdentifier,
|
||||
}),
|
||||
objectMetadataId: 'standard-object-id',
|
||||
fieldMetadataId: nameFlatFieldMetadata.id,
|
||||
});
|
||||
|
||||
const searchFieldMetadataUniversalIdentifierUpdates =
|
||||
buildSearchFieldMetadataReOwnOperations({
|
||||
flatFieldMetadataMaps: buildFlatFieldMetadataMaps([
|
||||
nameFlatFieldMetadata,
|
||||
]),
|
||||
flatSearchFieldMetadataMaps: buildFlatSearchFieldMetadataMaps([
|
||||
existingSearchFieldMetadata,
|
||||
]),
|
||||
applicationUniversalIdentifierById,
|
||||
});
|
||||
|
||||
expect(searchFieldMetadataUniversalIdentifierUpdates).toHaveLength(0);
|
||||
});
|
||||
});
|
||||
+267
@@ -0,0 +1,267 @@
|
||||
import { FieldMetadataType } from 'twenty-shared/types';
|
||||
|
||||
import { buildSearchVectorGinIndexBackfillOperations } from 'src/database/commands/upgrade-version-command/2-20/utils/build-search-vector-gin-index-backfill-operations.util';
|
||||
import { type FlatEntityMaps } from 'src/engine/metadata-modules/flat-entity/types/flat-entity-maps.type';
|
||||
import { getFlatFieldMetadataMock } from 'src/engine/metadata-modules/flat-field-metadata/__mocks__/get-flat-field-metadata.mock';
|
||||
import { type FlatFieldMetadata } from 'src/engine/metadata-modules/flat-field-metadata/types/flat-field-metadata.type';
|
||||
import { getFlatIndexMetadataMock } from 'src/engine/metadata-modules/flat-index-metadata/__mocks__/get-flat-index-metadata.mock';
|
||||
import {
|
||||
type FlatIndexFieldMetadata,
|
||||
type FlatIndexMetadata,
|
||||
} from 'src/engine/metadata-modules/flat-index-metadata/types/flat-index-metadata.type';
|
||||
import { getFlatObjectMetadataMock } from 'src/engine/metadata-modules/flat-object-metadata/__mocks__/get-flat-object-metadata.mock';
|
||||
import { type FlatObjectMetadata } from 'src/engine/metadata-modules/flat-object-metadata/types/flat-object-metadata.type';
|
||||
import { IndexType } from 'src/engine/metadata-modules/index-metadata/types/indexType.types';
|
||||
import { SEARCH_VECTOR_FIELD } from 'src/engine/metadata-modules/search-field-metadata/constants/search-vector-field.constants';
|
||||
|
||||
// Application universal identifiers must be valid UUIDs: they are used as the v5
|
||||
// namespace by the deterministic identifier helpers.
|
||||
const STANDARD_APPLICATION_ID = 'standard-application-id';
|
||||
const CUSTOM_APPLICATION_ID = 'custom-application-id';
|
||||
const INSTALLED_APPLICATION_ID = 'installed-application-id';
|
||||
const INSTALLED_APPLICATION_UID = '33333333-3333-4333-8333-333333333333';
|
||||
|
||||
const buildFlatObjectMetadataMaps = (
|
||||
flatObjectMetadatas: FlatObjectMetadata[],
|
||||
): FlatEntityMaps<FlatObjectMetadata> => ({
|
||||
byUniversalIdentifier: Object.fromEntries(
|
||||
flatObjectMetadatas.map((flatObjectMetadata) => [
|
||||
flatObjectMetadata.universalIdentifier,
|
||||
flatObjectMetadata,
|
||||
]),
|
||||
),
|
||||
universalIdentifierById: Object.fromEntries(
|
||||
flatObjectMetadatas.map((flatObjectMetadata) => [
|
||||
flatObjectMetadata.id,
|
||||
flatObjectMetadata.universalIdentifier,
|
||||
]),
|
||||
),
|
||||
universalIdentifiersByApplicationId: {},
|
||||
});
|
||||
|
||||
const buildFlatFieldMetadataMaps = (
|
||||
flatFieldMetadatas: FlatFieldMetadata[],
|
||||
): FlatEntityMaps<FlatFieldMetadata> => ({
|
||||
byUniversalIdentifier: Object.fromEntries(
|
||||
flatFieldMetadatas.map((flatFieldMetadata) => [
|
||||
flatFieldMetadata.universalIdentifier,
|
||||
flatFieldMetadata,
|
||||
]),
|
||||
),
|
||||
universalIdentifierById: Object.fromEntries(
|
||||
flatFieldMetadatas.map((flatFieldMetadata) => [
|
||||
flatFieldMetadata.id,
|
||||
flatFieldMetadata.universalIdentifier,
|
||||
]),
|
||||
),
|
||||
universalIdentifiersByApplicationId: {},
|
||||
});
|
||||
|
||||
const buildFlatIndexMaps = (
|
||||
flatIndexMetadatas: FlatIndexMetadata[],
|
||||
): FlatEntityMaps<FlatIndexMetadata> => ({
|
||||
byUniversalIdentifier: Object.fromEntries(
|
||||
flatIndexMetadatas.map((flatIndexMetadata) => [
|
||||
flatIndexMetadata.universalIdentifier,
|
||||
flatIndexMetadata,
|
||||
]),
|
||||
),
|
||||
universalIdentifierById: Object.fromEntries(
|
||||
flatIndexMetadatas.map((flatIndexMetadata) => [
|
||||
flatIndexMetadata.id,
|
||||
flatIndexMetadata.universalIdentifier,
|
||||
]),
|
||||
),
|
||||
universalIdentifiersByApplicationId: {},
|
||||
});
|
||||
|
||||
const buildFlatIndexFieldMetadata = ({
|
||||
fieldMetadataId,
|
||||
}: {
|
||||
fieldMetadataId: string;
|
||||
}): FlatIndexFieldMetadata => {
|
||||
const createdAt = '2024-01-01T00:00:00.000Z';
|
||||
|
||||
return {
|
||||
id: `${fieldMetadataId}-index-field-id`,
|
||||
workspaceId: 'workspace-id',
|
||||
indexMetadataId: 'index-metadata-id',
|
||||
fieldMetadataId,
|
||||
order: 0,
|
||||
subFieldName: null,
|
||||
createdAt,
|
||||
updatedAt: createdAt,
|
||||
};
|
||||
};
|
||||
|
||||
const buildSearchableObjectFixture = ({
|
||||
objectId,
|
||||
objectUniversalIdentifier,
|
||||
applicationId,
|
||||
applicationUniversalIdentifier,
|
||||
}: {
|
||||
objectId: string;
|
||||
objectUniversalIdentifier: string;
|
||||
applicationId: string;
|
||||
applicationUniversalIdentifier: string;
|
||||
}) => {
|
||||
const searchVectorFlatFieldMetadata = getFlatFieldMetadataMock({
|
||||
id: `${objectId}-search-vector-field-id`,
|
||||
universalIdentifier: `${objectUniversalIdentifier}-search-vector-field-uid`,
|
||||
objectMetadataId: objectId,
|
||||
objectMetadataUniversalIdentifier: objectUniversalIdentifier,
|
||||
type: FieldMetadataType.TS_VECTOR,
|
||||
name: SEARCH_VECTOR_FIELD.name,
|
||||
applicationId,
|
||||
applicationUniversalIdentifier,
|
||||
});
|
||||
|
||||
const flatObjectMetadata = getFlatObjectMetadataMock({
|
||||
id: objectId,
|
||||
universalIdentifier: objectUniversalIdentifier,
|
||||
isSearchable: true,
|
||||
applicationId,
|
||||
applicationUniversalIdentifier,
|
||||
fieldUniversalIdentifiers: [
|
||||
searchVectorFlatFieldMetadata.universalIdentifier,
|
||||
],
|
||||
});
|
||||
|
||||
return { flatObjectMetadata, searchVectorFlatFieldMetadata };
|
||||
};
|
||||
|
||||
describe('buildSearchVectorGinIndexBackfillOperations', () => {
|
||||
it('creates a GIN searchVector index for an installed-app object that has none, grouped under its application', () => {
|
||||
const { flatObjectMetadata, searchVectorFlatFieldMetadata } =
|
||||
buildSearchableObjectFixture({
|
||||
objectId: 'installed-object-id',
|
||||
objectUniversalIdentifier: 'installed-object-uid',
|
||||
applicationId: INSTALLED_APPLICATION_ID,
|
||||
applicationUniversalIdentifier: INSTALLED_APPLICATION_UID,
|
||||
});
|
||||
|
||||
const flatIndexesToCreateByApplicationUniversalIdentifier =
|
||||
buildSearchVectorGinIndexBackfillOperations({
|
||||
flatObjectMetadataMaps: buildFlatObjectMetadataMaps([
|
||||
flatObjectMetadata,
|
||||
]),
|
||||
flatFieldMetadataMaps: buildFlatFieldMetadataMaps([
|
||||
searchVectorFlatFieldMetadata,
|
||||
]),
|
||||
flatIndexMaps: buildFlatIndexMaps([]),
|
||||
twentyStandardApplicationId: STANDARD_APPLICATION_ID,
|
||||
workspaceCustomApplicationId: CUSTOM_APPLICATION_ID,
|
||||
});
|
||||
|
||||
const createdIndexes =
|
||||
flatIndexesToCreateByApplicationUniversalIdentifier[
|
||||
INSTALLED_APPLICATION_UID
|
||||
];
|
||||
|
||||
expect(createdIndexes).toHaveLength(1);
|
||||
expect(createdIndexes?.[0].indexType).toBe(IndexType.GIN);
|
||||
expect(createdIndexes?.[0].objectMetadataUniversalIdentifier).toBe(
|
||||
flatObjectMetadata.universalIdentifier,
|
||||
);
|
||||
expect(createdIndexes?.[0].applicationUniversalIdentifier).toBe(
|
||||
INSTALLED_APPLICATION_UID,
|
||||
);
|
||||
});
|
||||
|
||||
it('does not create an index for a twenty-standard object that has none', () => {
|
||||
const { flatObjectMetadata, searchVectorFlatFieldMetadata } =
|
||||
buildSearchableObjectFixture({
|
||||
objectId: 'standard-object-id',
|
||||
objectUniversalIdentifier: 'standard-object-uid',
|
||||
applicationId: STANDARD_APPLICATION_ID,
|
||||
applicationUniversalIdentifier: '11111111-1111-4111-8111-111111111111',
|
||||
});
|
||||
|
||||
const flatIndexesToCreateByApplicationUniversalIdentifier =
|
||||
buildSearchVectorGinIndexBackfillOperations({
|
||||
flatObjectMetadataMaps: buildFlatObjectMetadataMaps([
|
||||
flatObjectMetadata,
|
||||
]),
|
||||
flatFieldMetadataMaps: buildFlatFieldMetadataMaps([
|
||||
searchVectorFlatFieldMetadata,
|
||||
]),
|
||||
flatIndexMaps: buildFlatIndexMaps([]),
|
||||
twentyStandardApplicationId: STANDARD_APPLICATION_ID,
|
||||
workspaceCustomApplicationId: CUSTOM_APPLICATION_ID,
|
||||
});
|
||||
|
||||
expect(
|
||||
Object.keys(flatIndexesToCreateByApplicationUniversalIdentifier),
|
||||
).toHaveLength(0);
|
||||
});
|
||||
|
||||
it('does not create an index for a workspace-custom object that has none', () => {
|
||||
const { flatObjectMetadata, searchVectorFlatFieldMetadata } =
|
||||
buildSearchableObjectFixture({
|
||||
objectId: 'custom-object-id',
|
||||
objectUniversalIdentifier: 'custom-object-uid',
|
||||
applicationId: CUSTOM_APPLICATION_ID,
|
||||
applicationUniversalIdentifier: '22222222-2222-4222-8222-222222222222',
|
||||
});
|
||||
|
||||
const flatIndexesToCreateByApplicationUniversalIdentifier =
|
||||
buildSearchVectorGinIndexBackfillOperations({
|
||||
flatObjectMetadataMaps: buildFlatObjectMetadataMaps([
|
||||
flatObjectMetadata,
|
||||
]),
|
||||
flatFieldMetadataMaps: buildFlatFieldMetadataMaps([
|
||||
searchVectorFlatFieldMetadata,
|
||||
]),
|
||||
flatIndexMaps: buildFlatIndexMaps([]),
|
||||
twentyStandardApplicationId: STANDARD_APPLICATION_ID,
|
||||
workspaceCustomApplicationId: CUSTOM_APPLICATION_ID,
|
||||
});
|
||||
|
||||
expect(
|
||||
Object.keys(flatIndexesToCreateByApplicationUniversalIdentifier),
|
||||
).toHaveLength(0);
|
||||
});
|
||||
|
||||
it('is idempotent: no create when the installed-app object already has a GIN searchVector index', () => {
|
||||
const { flatObjectMetadata, searchVectorFlatFieldMetadata } =
|
||||
buildSearchableObjectFixture({
|
||||
objectId: 'installed-object-id',
|
||||
objectUniversalIdentifier: 'installed-object-uid',
|
||||
applicationId: INSTALLED_APPLICATION_ID,
|
||||
applicationUniversalIdentifier: INSTALLED_APPLICATION_UID,
|
||||
});
|
||||
|
||||
const ginIndex = getFlatIndexMetadataMock({
|
||||
id: 'installed-gin-index-id',
|
||||
universalIdentifier: 'installed-gin-index-uid',
|
||||
objectMetadataId: flatObjectMetadata.id,
|
||||
objectMetadataUniversalIdentifier: flatObjectMetadata.universalIdentifier,
|
||||
applicationId: INSTALLED_APPLICATION_ID,
|
||||
applicationUniversalIdentifier: INSTALLED_APPLICATION_UID,
|
||||
name: 'IDX_SEARCH_VECTOR_INSTALLED',
|
||||
indexType: IndexType.GIN,
|
||||
flatIndexFieldMetadatas: [
|
||||
buildFlatIndexFieldMetadata({
|
||||
fieldMetadataId: searchVectorFlatFieldMetadata.id,
|
||||
}),
|
||||
],
|
||||
});
|
||||
|
||||
const flatIndexesToCreateByApplicationUniversalIdentifier =
|
||||
buildSearchVectorGinIndexBackfillOperations({
|
||||
flatObjectMetadataMaps: buildFlatObjectMetadataMaps([
|
||||
flatObjectMetadata,
|
||||
]),
|
||||
flatFieldMetadataMaps: buildFlatFieldMetadataMaps([
|
||||
searchVectorFlatFieldMetadata,
|
||||
]),
|
||||
flatIndexMaps: buildFlatIndexMaps([ginIndex]),
|
||||
twentyStandardApplicationId: STANDARD_APPLICATION_ID,
|
||||
workspaceCustomApplicationId: CUSTOM_APPLICATION_ID,
|
||||
});
|
||||
|
||||
expect(
|
||||
Object.keys(flatIndexesToCreateByApplicationUniversalIdentifier),
|
||||
).toHaveLength(0);
|
||||
});
|
||||
});
|
||||
+224
@@ -0,0 +1,224 @@
|
||||
import { getIndexUniversalIdentifier } from 'twenty-shared/application';
|
||||
import { FieldMetadataType } from 'twenty-shared/types';
|
||||
|
||||
import { buildSearchVectorGinIndexReOwnOperations } from 'src/database/commands/upgrade-version-command/2-20/utils/build-search-vector-gin-index-re-own-operations.util';
|
||||
import { type FlatEntityMaps } from 'src/engine/metadata-modules/flat-entity/types/flat-entity-maps.type';
|
||||
import { getFlatFieldMetadataMock } from 'src/engine/metadata-modules/flat-field-metadata/__mocks__/get-flat-field-metadata.mock';
|
||||
import { type FlatFieldMetadata } from 'src/engine/metadata-modules/flat-field-metadata/types/flat-field-metadata.type';
|
||||
import { getFlatIndexMetadataMock } from 'src/engine/metadata-modules/flat-index-metadata/__mocks__/get-flat-index-metadata.mock';
|
||||
import {
|
||||
type FlatIndexFieldMetadata,
|
||||
type FlatIndexMetadata,
|
||||
} from 'src/engine/metadata-modules/flat-index-metadata/types/flat-index-metadata.type';
|
||||
import { getFlatObjectMetadataMock } from 'src/engine/metadata-modules/flat-object-metadata/__mocks__/get-flat-object-metadata.mock';
|
||||
import { type FlatObjectMetadata } from 'src/engine/metadata-modules/flat-object-metadata/types/flat-object-metadata.type';
|
||||
import { IndexType } from 'src/engine/metadata-modules/index-metadata/types/indexType.types';
|
||||
import { SEARCH_VECTOR_FIELD } from 'src/engine/metadata-modules/search-field-metadata/constants/search-vector-field.constants';
|
||||
|
||||
const STANDARD_APPLICATION_ID = 'standard-application-id';
|
||||
// Application universal identifiers must be valid UUIDs: they are used as the v5
|
||||
// namespace by the deterministic identifier helpers.
|
||||
const STANDARD_APPLICATION_UID = '11111111-1111-4111-8111-111111111111';
|
||||
|
||||
const applicationUniversalIdentifierById = new Map([
|
||||
[STANDARD_APPLICATION_ID, STANDARD_APPLICATION_UID],
|
||||
]);
|
||||
|
||||
const buildFlatObjectMetadataMaps = (
|
||||
flatObjectMetadatas: FlatObjectMetadata[],
|
||||
): FlatEntityMaps<FlatObjectMetadata> => ({
|
||||
byUniversalIdentifier: Object.fromEntries(
|
||||
flatObjectMetadatas.map((flatObjectMetadata) => [
|
||||
flatObjectMetadata.universalIdentifier,
|
||||
flatObjectMetadata,
|
||||
]),
|
||||
),
|
||||
universalIdentifierById: Object.fromEntries(
|
||||
flatObjectMetadatas.map((flatObjectMetadata) => [
|
||||
flatObjectMetadata.id,
|
||||
flatObjectMetadata.universalIdentifier,
|
||||
]),
|
||||
),
|
||||
universalIdentifiersByApplicationId: {},
|
||||
});
|
||||
|
||||
const buildFlatFieldMetadataMaps = (
|
||||
flatFieldMetadatas: FlatFieldMetadata[],
|
||||
): FlatEntityMaps<FlatFieldMetadata> => ({
|
||||
byUniversalIdentifier: Object.fromEntries(
|
||||
flatFieldMetadatas.map((flatFieldMetadata) => [
|
||||
flatFieldMetadata.universalIdentifier,
|
||||
flatFieldMetadata,
|
||||
]),
|
||||
),
|
||||
universalIdentifierById: Object.fromEntries(
|
||||
flatFieldMetadatas.map((flatFieldMetadata) => [
|
||||
flatFieldMetadata.id,
|
||||
flatFieldMetadata.universalIdentifier,
|
||||
]),
|
||||
),
|
||||
universalIdentifiersByApplicationId: {},
|
||||
});
|
||||
|
||||
const buildFlatIndexMaps = (
|
||||
flatIndexMetadatas: FlatIndexMetadata[],
|
||||
): FlatEntityMaps<FlatIndexMetadata> => ({
|
||||
byUniversalIdentifier: Object.fromEntries(
|
||||
flatIndexMetadatas.map((flatIndexMetadata) => [
|
||||
flatIndexMetadata.universalIdentifier,
|
||||
flatIndexMetadata,
|
||||
]),
|
||||
),
|
||||
universalIdentifierById: Object.fromEntries(
|
||||
flatIndexMetadatas.map((flatIndexMetadata) => [
|
||||
flatIndexMetadata.id,
|
||||
flatIndexMetadata.universalIdentifier,
|
||||
]),
|
||||
),
|
||||
universalIdentifiersByApplicationId: {},
|
||||
});
|
||||
|
||||
const buildFlatIndexFieldMetadata = ({
|
||||
fieldMetadataId,
|
||||
}: {
|
||||
fieldMetadataId: string;
|
||||
}): FlatIndexFieldMetadata => {
|
||||
const createdAt = '2024-01-01T00:00:00.000Z';
|
||||
|
||||
return {
|
||||
id: `${fieldMetadataId}-index-field-id`,
|
||||
workspaceId: 'workspace-id',
|
||||
indexMetadataId: 'index-metadata-id',
|
||||
fieldMetadataId,
|
||||
order: 0,
|
||||
subFieldName: null,
|
||||
createdAt,
|
||||
updatedAt: createdAt,
|
||||
};
|
||||
};
|
||||
|
||||
const buildSearchableObjectFixture = ({
|
||||
objectId,
|
||||
objectUniversalIdentifier,
|
||||
}: {
|
||||
objectId: string;
|
||||
objectUniversalIdentifier: string;
|
||||
}) => {
|
||||
const searchVectorFlatFieldMetadata = getFlatFieldMetadataMock({
|
||||
id: `${objectId}-search-vector-field-id`,
|
||||
universalIdentifier: `${objectUniversalIdentifier}-search-vector-field-uid`,
|
||||
objectMetadataId: objectId,
|
||||
objectMetadataUniversalIdentifier: objectUniversalIdentifier,
|
||||
type: FieldMetadataType.TS_VECTOR,
|
||||
name: SEARCH_VECTOR_FIELD.name,
|
||||
applicationId: STANDARD_APPLICATION_ID,
|
||||
applicationUniversalIdentifier: STANDARD_APPLICATION_UID,
|
||||
});
|
||||
|
||||
const flatObjectMetadata = getFlatObjectMetadataMock({
|
||||
id: objectId,
|
||||
universalIdentifier: objectUniversalIdentifier,
|
||||
isSearchable: true,
|
||||
applicationId: STANDARD_APPLICATION_ID,
|
||||
applicationUniversalIdentifier: STANDARD_APPLICATION_UID,
|
||||
fieldUniversalIdentifiers: [
|
||||
searchVectorFlatFieldMetadata.universalIdentifier,
|
||||
],
|
||||
});
|
||||
|
||||
return { flatObjectMetadata, searchVectorFlatFieldMetadata };
|
||||
};
|
||||
|
||||
describe('buildSearchVectorGinIndexReOwnOperations', () => {
|
||||
it('re-owns an existing GIN searchVector index carrying a non-deterministic universal identifier (all applications)', () => {
|
||||
const { flatObjectMetadata, searchVectorFlatFieldMetadata } =
|
||||
buildSearchableObjectFixture({
|
||||
objectId: 'standard-object-id',
|
||||
objectUniversalIdentifier: 'standard-object-uid',
|
||||
});
|
||||
|
||||
const indexName = 'IDX_SEARCH_VECTOR_STANDARD';
|
||||
const ginIndex = getFlatIndexMetadataMock({
|
||||
id: 'standard-gin-index-id',
|
||||
universalIdentifier: 'legacy-v4-index-uid',
|
||||
objectMetadataId: flatObjectMetadata.id,
|
||||
objectMetadataUniversalIdentifier: flatObjectMetadata.universalIdentifier,
|
||||
applicationId: STANDARD_APPLICATION_ID,
|
||||
applicationUniversalIdentifier: STANDARD_APPLICATION_UID,
|
||||
name: indexName,
|
||||
indexType: IndexType.GIN,
|
||||
flatIndexFieldMetadatas: [
|
||||
buildFlatIndexFieldMetadata({
|
||||
fieldMetadataId: searchVectorFlatFieldMetadata.id,
|
||||
}),
|
||||
],
|
||||
});
|
||||
|
||||
const indexUniversalIdentifierUpdates =
|
||||
buildSearchVectorGinIndexReOwnOperations({
|
||||
flatObjectMetadataMaps: buildFlatObjectMetadataMaps([
|
||||
flatObjectMetadata,
|
||||
]),
|
||||
flatFieldMetadataMaps: buildFlatFieldMetadataMaps([
|
||||
searchVectorFlatFieldMetadata,
|
||||
]),
|
||||
flatIndexMaps: buildFlatIndexMaps([ginIndex]),
|
||||
applicationUniversalIdentifierById,
|
||||
});
|
||||
|
||||
expect(indexUniversalIdentifierUpdates).toEqual([
|
||||
{
|
||||
id: ginIndex.id,
|
||||
name: indexName,
|
||||
deterministicUniversalIdentifier: getIndexUniversalIdentifier({
|
||||
applicationUniversalIdentifier: STANDARD_APPLICATION_UID,
|
||||
objectUniversalIdentifier: flatObjectMetadata.universalIdentifier,
|
||||
name: indexName,
|
||||
}),
|
||||
},
|
||||
]);
|
||||
});
|
||||
|
||||
it('is a no-op for a GIN searchVector index already carrying its deterministic universal identifier', () => {
|
||||
const { flatObjectMetadata, searchVectorFlatFieldMetadata } =
|
||||
buildSearchableObjectFixture({
|
||||
objectId: 'standard-object-id',
|
||||
objectUniversalIdentifier: 'standard-object-uid',
|
||||
});
|
||||
|
||||
const indexName = 'IDX_SEARCH_VECTOR_STANDARD';
|
||||
const ginIndex = getFlatIndexMetadataMock({
|
||||
id: 'standard-gin-index-id',
|
||||
universalIdentifier: getIndexUniversalIdentifier({
|
||||
applicationUniversalIdentifier: STANDARD_APPLICATION_UID,
|
||||
objectUniversalIdentifier: flatObjectMetadata.universalIdentifier,
|
||||
name: indexName,
|
||||
}),
|
||||
objectMetadataId: flatObjectMetadata.id,
|
||||
objectMetadataUniversalIdentifier: flatObjectMetadata.universalIdentifier,
|
||||
applicationId: STANDARD_APPLICATION_ID,
|
||||
applicationUniversalIdentifier: STANDARD_APPLICATION_UID,
|
||||
name: indexName,
|
||||
indexType: IndexType.GIN,
|
||||
flatIndexFieldMetadatas: [
|
||||
buildFlatIndexFieldMetadata({
|
||||
fieldMetadataId: searchVectorFlatFieldMetadata.id,
|
||||
}),
|
||||
],
|
||||
});
|
||||
|
||||
const indexUniversalIdentifierUpdates =
|
||||
buildSearchVectorGinIndexReOwnOperations({
|
||||
flatObjectMetadataMaps: buildFlatObjectMetadataMaps([
|
||||
flatObjectMetadata,
|
||||
]),
|
||||
flatFieldMetadataMaps: buildFlatFieldMetadataMaps([
|
||||
searchVectorFlatFieldMetadata,
|
||||
]),
|
||||
flatIndexMaps: buildFlatIndexMaps([ginIndex]),
|
||||
applicationUniversalIdentifierById,
|
||||
});
|
||||
|
||||
expect(indexUniversalIdentifierUpdates).toHaveLength(0);
|
||||
});
|
||||
});
|
||||
+186
@@ -0,0 +1,186 @@
|
||||
import { FieldMetadataType } from 'twenty-shared/types';
|
||||
|
||||
import { isSearchVectorGinFlatIndexMetadata } from 'src/database/commands/upgrade-version-command/2-20/utils/is-search-vector-gin-flat-index-metadata.util';
|
||||
import { type FlatEntityMaps } from 'src/engine/metadata-modules/flat-entity/types/flat-entity-maps.type';
|
||||
import { getFlatFieldMetadataMock } from 'src/engine/metadata-modules/flat-field-metadata/__mocks__/get-flat-field-metadata.mock';
|
||||
import { type FlatFieldMetadata } from 'src/engine/metadata-modules/flat-field-metadata/types/flat-field-metadata.type';
|
||||
import { getFlatIndexMetadataMock } from 'src/engine/metadata-modules/flat-index-metadata/__mocks__/get-flat-index-metadata.mock';
|
||||
import { type FlatIndexFieldMetadata } from 'src/engine/metadata-modules/flat-index-metadata/types/flat-index-metadata.type';
|
||||
import { IndexType } from 'src/engine/metadata-modules/index-metadata/types/indexType.types';
|
||||
import { SEARCH_VECTOR_FIELD } from 'src/engine/metadata-modules/search-field-metadata/constants/search-vector-field.constants';
|
||||
|
||||
const buildFlatFieldMetadataMaps = (
|
||||
flatFieldMetadatas: FlatFieldMetadata[],
|
||||
): FlatEntityMaps<FlatFieldMetadata> => ({
|
||||
byUniversalIdentifier: Object.fromEntries(
|
||||
flatFieldMetadatas.map((flatFieldMetadata) => [
|
||||
flatFieldMetadata.universalIdentifier,
|
||||
flatFieldMetadata,
|
||||
]),
|
||||
),
|
||||
universalIdentifierById: Object.fromEntries(
|
||||
flatFieldMetadatas.map((flatFieldMetadata) => [
|
||||
flatFieldMetadata.id,
|
||||
flatFieldMetadata.universalIdentifier,
|
||||
]),
|
||||
),
|
||||
universalIdentifiersByApplicationId: {},
|
||||
});
|
||||
|
||||
const buildFlatIndexFieldMetadata = ({
|
||||
fieldMetadataId,
|
||||
}: {
|
||||
fieldMetadataId: string;
|
||||
}): FlatIndexFieldMetadata => {
|
||||
const createdAt = '2024-01-01T00:00:00.000Z';
|
||||
|
||||
return {
|
||||
id: `${fieldMetadataId}-index-field-id`,
|
||||
workspaceId: 'workspace-id',
|
||||
indexMetadataId: 'index-metadata-id',
|
||||
fieldMetadataId,
|
||||
order: 0,
|
||||
subFieldName: null,
|
||||
createdAt,
|
||||
updatedAt: createdAt,
|
||||
};
|
||||
};
|
||||
|
||||
const SEARCH_VECTOR_FIELD_ID = 'search-vector-field-id';
|
||||
const TEXT_FIELD_ID = 'text-field-id';
|
||||
const OTHER_TS_VECTOR_FIELD_ID = 'other-ts-vector-field-id';
|
||||
|
||||
const searchVectorFlatFieldMetadata = getFlatFieldMetadataMock({
|
||||
id: SEARCH_VECTOR_FIELD_ID,
|
||||
universalIdentifier: 'search-vector-field-uid',
|
||||
objectMetadataId: 'object-id',
|
||||
type: FieldMetadataType.TS_VECTOR,
|
||||
name: SEARCH_VECTOR_FIELD.name,
|
||||
});
|
||||
|
||||
const textFlatFieldMetadata = getFlatFieldMetadataMock({
|
||||
id: TEXT_FIELD_ID,
|
||||
universalIdentifier: 'text-field-uid',
|
||||
objectMetadataId: 'object-id',
|
||||
type: FieldMetadataType.TEXT,
|
||||
name: 'name',
|
||||
});
|
||||
|
||||
const otherTsVectorFlatFieldMetadata = getFlatFieldMetadataMock({
|
||||
id: OTHER_TS_VECTOR_FIELD_ID,
|
||||
universalIdentifier: 'other-ts-vector-field-uid',
|
||||
objectMetadataId: 'object-id',
|
||||
type: FieldMetadataType.TS_VECTOR,
|
||||
name: 'customSearchVector',
|
||||
});
|
||||
|
||||
const flatFieldMetadataMaps = buildFlatFieldMetadataMaps([
|
||||
searchVectorFlatFieldMetadata,
|
||||
textFlatFieldMetadata,
|
||||
otherTsVectorFlatFieldMetadata,
|
||||
]);
|
||||
|
||||
describe('isSearchVectorGinFlatIndexMetadata', () => {
|
||||
it('returns true for a single-field GIN index on the TS_VECTOR field', () => {
|
||||
const flatIndexMetadata = getFlatIndexMetadataMock({
|
||||
universalIdentifier: 'gin-index-uid',
|
||||
objectMetadataId: 'object-id',
|
||||
objectMetadataUniversalIdentifier: 'object-uid',
|
||||
applicationUniversalIdentifier: 'application-uid',
|
||||
indexType: IndexType.GIN,
|
||||
flatIndexFieldMetadatas: [
|
||||
buildFlatIndexFieldMetadata({ fieldMetadataId: SEARCH_VECTOR_FIELD_ID }),
|
||||
],
|
||||
});
|
||||
|
||||
expect(
|
||||
isSearchVectorGinFlatIndexMetadata({
|
||||
flatIndexMetadata,
|
||||
flatFieldMetadataMaps,
|
||||
}),
|
||||
).toBe(true);
|
||||
});
|
||||
|
||||
it('returns false for a GIN index on a non-TS_VECTOR field', () => {
|
||||
const flatIndexMetadata = getFlatIndexMetadataMock({
|
||||
universalIdentifier: 'gin-index-uid',
|
||||
objectMetadataId: 'object-id',
|
||||
objectMetadataUniversalIdentifier: 'object-uid',
|
||||
applicationUniversalIdentifier: 'application-uid',
|
||||
indexType: IndexType.GIN,
|
||||
flatIndexFieldMetadatas: [
|
||||
buildFlatIndexFieldMetadata({ fieldMetadataId: TEXT_FIELD_ID }),
|
||||
],
|
||||
});
|
||||
|
||||
expect(
|
||||
isSearchVectorGinFlatIndexMetadata({
|
||||
flatIndexMetadata,
|
||||
flatFieldMetadataMaps,
|
||||
}),
|
||||
).toBe(false);
|
||||
});
|
||||
|
||||
it('returns false for a non-GIN (BTREE) index on the TS_VECTOR field', () => {
|
||||
const flatIndexMetadata = getFlatIndexMetadataMock({
|
||||
universalIdentifier: 'btree-index-uid',
|
||||
objectMetadataId: 'object-id',
|
||||
objectMetadataUniversalIdentifier: 'object-uid',
|
||||
applicationUniversalIdentifier: 'application-uid',
|
||||
indexType: IndexType.BTREE,
|
||||
flatIndexFieldMetadatas: [
|
||||
buildFlatIndexFieldMetadata({ fieldMetadataId: SEARCH_VECTOR_FIELD_ID }),
|
||||
],
|
||||
});
|
||||
|
||||
expect(
|
||||
isSearchVectorGinFlatIndexMetadata({
|
||||
flatIndexMetadata,
|
||||
flatFieldMetadataMaps,
|
||||
}),
|
||||
).toBe(false);
|
||||
});
|
||||
|
||||
it('returns false for a single-field GIN index on a TS_VECTOR field that is not the searchVector field', () => {
|
||||
const flatIndexMetadata = getFlatIndexMetadataMock({
|
||||
universalIdentifier: 'gin-index-uid',
|
||||
objectMetadataId: 'object-id',
|
||||
objectMetadataUniversalIdentifier: 'object-uid',
|
||||
applicationUniversalIdentifier: 'application-uid',
|
||||
indexType: IndexType.GIN,
|
||||
flatIndexFieldMetadatas: [
|
||||
buildFlatIndexFieldMetadata({
|
||||
fieldMetadataId: OTHER_TS_VECTOR_FIELD_ID,
|
||||
}),
|
||||
],
|
||||
});
|
||||
|
||||
expect(
|
||||
isSearchVectorGinFlatIndexMetadata({
|
||||
flatIndexMetadata,
|
||||
flatFieldMetadataMaps,
|
||||
}),
|
||||
).toBe(false);
|
||||
});
|
||||
|
||||
it('returns false for a multi-column GIN index', () => {
|
||||
const flatIndexMetadata = getFlatIndexMetadataMock({
|
||||
universalIdentifier: 'gin-index-uid',
|
||||
objectMetadataId: 'object-id',
|
||||
objectMetadataUniversalIdentifier: 'object-uid',
|
||||
applicationUniversalIdentifier: 'application-uid',
|
||||
indexType: IndexType.GIN,
|
||||
flatIndexFieldMetadatas: [
|
||||
buildFlatIndexFieldMetadata({ fieldMetadataId: SEARCH_VECTOR_FIELD_ID }),
|
||||
buildFlatIndexFieldMetadata({ fieldMetadataId: TEXT_FIELD_ID }),
|
||||
],
|
||||
});
|
||||
|
||||
expect(
|
||||
isSearchVectorGinFlatIndexMetadata({
|
||||
flatIndexMetadata,
|
||||
flatFieldMetadataMaps,
|
||||
}),
|
||||
).toBe(false);
|
||||
});
|
||||
});
|
||||
+139
@@ -0,0 +1,139 @@
|
||||
import { getFieldUniversalIdentifier } from 'twenty-shared/application';
|
||||
import {
|
||||
fromArrayToValuesByKeyRecord,
|
||||
isDefined,
|
||||
isSearchableFieldType,
|
||||
} from 'twenty-shared/utils';
|
||||
|
||||
import { type FlatEntityMaps } from 'src/engine/metadata-modules/flat-entity/types/flat-entity-maps.type';
|
||||
import { type FlatFieldMetadata } from 'src/engine/metadata-modules/flat-field-metadata/types/flat-field-metadata.type';
|
||||
import { type FlatObjectMetadata } from 'src/engine/metadata-modules/flat-object-metadata/types/flat-object-metadata.type';
|
||||
import { type FlatSearchFieldMetadata } from 'src/engine/metadata-modules/flat-search-field-metadata/types/flat-search-field-metadata.type';
|
||||
import { buildFlatSearchFieldMetadataForField } from 'src/engine/metadata-modules/flat-search-field-metadata/utils/build-flat-search-field-metadata-for-field.util';
|
||||
import { findTsVectorFlatFieldMetadataForObject } from 'src/engine/metadata-modules/flat-search-field-metadata/utils/find-ts-vector-flat-field-metadata-for-object.util';
|
||||
import { type UniversalFlatSearchFieldMetadata } from 'src/engine/workspace-manager/workspace-migration/universal-flat-entity/types/universal-flat-search-field-metadata.type';
|
||||
|
||||
type BuildSearchFieldMetadataBackfillOperationsArgs = {
|
||||
flatObjectMetadataMaps: FlatEntityMaps<FlatObjectMetadata>;
|
||||
flatFieldMetadataMaps: FlatEntityMaps<FlatFieldMetadata>;
|
||||
flatSearchFieldMetadataMaps: FlatEntityMaps<FlatSearchFieldMetadata>;
|
||||
applicationUniversalIdentifierById: Map<string, string>;
|
||||
twentyStandardApplicationId: string;
|
||||
workspaceCustomApplicationId: string;
|
||||
};
|
||||
|
||||
// Backfill skips the twenty-standard and workspace-custom applications (their row is
|
||||
// already provisioned by the manifest funnel) and targets every other (installed)
|
||||
// application's searchable objects that have no row yet. The creation rule mirrors the
|
||||
// object-create side effect handler (label-identifier field of a searchable type;
|
||||
// junction/id-label objects have no search surface). Grouped by application universal
|
||||
// identifier for the per-application migration run.
|
||||
export const buildSearchFieldMetadataBackfillOperations = ({
|
||||
flatObjectMetadataMaps,
|
||||
flatFieldMetadataMaps,
|
||||
flatSearchFieldMetadataMaps,
|
||||
applicationUniversalIdentifierById,
|
||||
twentyStandardApplicationId,
|
||||
workspaceCustomApplicationId,
|
||||
}: BuildSearchFieldMetadataBackfillOperationsArgs): Record<
|
||||
string,
|
||||
UniversalFlatSearchFieldMetadata[]
|
||||
> => {
|
||||
const existingSearchFieldMetadataKeys = new Set<string>();
|
||||
|
||||
for (const flatSearchFieldMetadata of Object.values(
|
||||
flatSearchFieldMetadataMaps.byUniversalIdentifier,
|
||||
).filter(isDefined)) {
|
||||
existingSearchFieldMetadataKeys.add(
|
||||
`${flatSearchFieldMetadata.objectMetadataId}:${flatSearchFieldMetadata.fieldMetadataId}`,
|
||||
);
|
||||
}
|
||||
|
||||
const flatSearchFieldMetadatasToCreate: UniversalFlatSearchFieldMetadata[] =
|
||||
[];
|
||||
|
||||
for (const flatObjectMetadata of Object.values(
|
||||
flatObjectMetadataMaps.byUniversalIdentifier,
|
||||
).filter(isDefined)) {
|
||||
if (
|
||||
flatObjectMetadata.applicationId === twentyStandardApplicationId ||
|
||||
flatObjectMetadata.applicationId === workspaceCustomApplicationId
|
||||
) {
|
||||
continue;
|
||||
}
|
||||
|
||||
if (flatObjectMetadata.isSearchable !== true) {
|
||||
continue;
|
||||
}
|
||||
|
||||
const applicationUniversalIdentifier =
|
||||
applicationUniversalIdentifierById.get(flatObjectMetadata.applicationId);
|
||||
|
||||
if (!isDefined(applicationUniversalIdentifier)) {
|
||||
continue;
|
||||
}
|
||||
|
||||
const labelIdentifierFieldMetadataUniversalIdentifier =
|
||||
flatObjectMetadata.labelIdentifierFieldMetadataUniversalIdentifier;
|
||||
|
||||
if (!isDefined(labelIdentifierFieldMetadataUniversalIdentifier)) {
|
||||
continue;
|
||||
}
|
||||
|
||||
const derivedIdFieldUniversalIdentifier = getFieldUniversalIdentifier({
|
||||
applicationUniversalIdentifier,
|
||||
objectUniversalIdentifier: flatObjectMetadata.universalIdentifier,
|
||||
name: 'id',
|
||||
});
|
||||
|
||||
if (
|
||||
labelIdentifierFieldMetadataUniversalIdentifier ===
|
||||
derivedIdFieldUniversalIdentifier
|
||||
) {
|
||||
continue;
|
||||
}
|
||||
|
||||
const labelIdentifierFlatFieldMetadata =
|
||||
flatFieldMetadataMaps.byUniversalIdentifier[
|
||||
labelIdentifierFieldMetadataUniversalIdentifier
|
||||
];
|
||||
|
||||
if (
|
||||
!isDefined(labelIdentifierFlatFieldMetadata) ||
|
||||
!isSearchableFieldType(labelIdentifierFlatFieldMetadata.type)
|
||||
) {
|
||||
continue;
|
||||
}
|
||||
|
||||
if (
|
||||
existingSearchFieldMetadataKeys.has(
|
||||
`${flatObjectMetadata.id}:${labelIdentifierFlatFieldMetadata.id}`,
|
||||
)
|
||||
) {
|
||||
continue;
|
||||
}
|
||||
|
||||
const tsVectorFlatFieldMetadata = findTsVectorFlatFieldMetadataForObject({
|
||||
fieldUniversalIdentifiers: flatObjectMetadata.fieldUniversalIdentifiers,
|
||||
flatFieldMetadataMaps,
|
||||
});
|
||||
|
||||
if (!isDefined(tsVectorFlatFieldMetadata)) {
|
||||
continue;
|
||||
}
|
||||
|
||||
flatSearchFieldMetadatasToCreate.push(
|
||||
buildFlatSearchFieldMetadataForField({
|
||||
flatObjectMetadata,
|
||||
flatFieldMetadata: labelIdentifierFlatFieldMetadata,
|
||||
tsVectorFlatFieldMetadata,
|
||||
position: 0,
|
||||
}),
|
||||
);
|
||||
}
|
||||
|
||||
return fromArrayToValuesByKeyRecord({
|
||||
array: flatSearchFieldMetadatasToCreate,
|
||||
key: 'applicationUniversalIdentifier',
|
||||
});
|
||||
};
|
||||
+70
@@ -0,0 +1,70 @@
|
||||
import { getSearchFieldUniversalIdentifier } from 'twenty-shared/application';
|
||||
import { isDefined } from 'twenty-shared/utils';
|
||||
|
||||
import { type FlatEntityMaps } from 'src/engine/metadata-modules/flat-entity/types/flat-entity-maps.type';
|
||||
import { findFlatEntityByIdInFlatEntityMaps } from 'src/engine/metadata-modules/flat-entity/utils/find-flat-entity-by-id-in-flat-entity-maps.util';
|
||||
import { type FlatFieldMetadata } from 'src/engine/metadata-modules/flat-field-metadata/types/flat-field-metadata.type';
|
||||
import { type FlatSearchFieldMetadata } from 'src/engine/metadata-modules/flat-search-field-metadata/types/flat-search-field-metadata.type';
|
||||
|
||||
export type SearchFieldMetadataUniversalIdentifierUpdate = {
|
||||
id: string;
|
||||
deterministicUniversalIdentifier: string;
|
||||
};
|
||||
|
||||
type BuildSearchFieldMetadataReOwnOperationsArgs = {
|
||||
flatFieldMetadataMaps: FlatEntityMaps<FlatFieldMetadata>;
|
||||
flatSearchFieldMetadataMaps: FlatEntityMaps<FlatSearchFieldMetadata>;
|
||||
applicationUniversalIdentifierById: Map<string, string>;
|
||||
};
|
||||
|
||||
// Re-own is global (every application, including twenty-standard and workspace-custom):
|
||||
// every existing searchFieldMetadata row whose universal identifier drifted from its
|
||||
// getSearchFieldUniversalIdentifier derivation is converged back to it.
|
||||
export const buildSearchFieldMetadataReOwnOperations = ({
|
||||
flatFieldMetadataMaps,
|
||||
flatSearchFieldMetadataMaps,
|
||||
applicationUniversalIdentifierById,
|
||||
}: BuildSearchFieldMetadataReOwnOperationsArgs): SearchFieldMetadataUniversalIdentifierUpdate[] => {
|
||||
const searchFieldMetadataUniversalIdentifierUpdates: SearchFieldMetadataUniversalIdentifierUpdate[] =
|
||||
[];
|
||||
|
||||
for (const flatSearchFieldMetadata of Object.values(
|
||||
flatSearchFieldMetadataMaps.byUniversalIdentifier,
|
||||
).filter(isDefined)) {
|
||||
const applicationUniversalIdentifier =
|
||||
applicationUniversalIdentifierById.get(
|
||||
flatSearchFieldMetadata.applicationId,
|
||||
);
|
||||
const indexedFlatFieldMetadata = findFlatEntityByIdInFlatEntityMaps({
|
||||
flatEntityMaps: flatFieldMetadataMaps,
|
||||
flatEntityId: flatSearchFieldMetadata.fieldMetadataId,
|
||||
});
|
||||
|
||||
if (
|
||||
!isDefined(applicationUniversalIdentifier) ||
|
||||
!isDefined(indexedFlatFieldMetadata)
|
||||
) {
|
||||
continue;
|
||||
}
|
||||
|
||||
const deterministicUniversalIdentifier = getSearchFieldUniversalIdentifier({
|
||||
applicationUniversalIdentifier,
|
||||
fieldMetadataUniversalIdentifier:
|
||||
indexedFlatFieldMetadata.universalIdentifier,
|
||||
});
|
||||
|
||||
if (
|
||||
deterministicUniversalIdentifier ===
|
||||
flatSearchFieldMetadata.universalIdentifier
|
||||
) {
|
||||
continue;
|
||||
}
|
||||
|
||||
searchFieldMetadataUniversalIdentifierUpdates.push({
|
||||
id: flatSearchFieldMetadata.id,
|
||||
deterministicUniversalIdentifier,
|
||||
});
|
||||
}
|
||||
|
||||
return searchFieldMetadataUniversalIdentifierUpdates;
|
||||
};
|
||||
+88
@@ -0,0 +1,88 @@
|
||||
import { fromArrayToValuesByKeyRecord, isDefined } from 'twenty-shared/utils';
|
||||
|
||||
import { isSearchVectorGinFlatIndexMetadata } from 'src/database/commands/upgrade-version-command/2-20/utils/is-search-vector-gin-flat-index-metadata.util';
|
||||
import { type FlatEntityMaps } from 'src/engine/metadata-modules/flat-entity/types/flat-entity-maps.type';
|
||||
import { type FlatFieldMetadata } from 'src/engine/metadata-modules/flat-field-metadata/types/flat-field-metadata.type';
|
||||
import { type FlatIndexMetadata } from 'src/engine/metadata-modules/flat-index-metadata/types/flat-index-metadata.type';
|
||||
import { type FlatObjectMetadata } from 'src/engine/metadata-modules/flat-object-metadata/types/flat-object-metadata.type';
|
||||
import { findTsVectorFlatFieldMetadataForObject } from 'src/engine/metadata-modules/flat-search-field-metadata/utils/find-ts-vector-flat-field-metadata-for-object.util';
|
||||
import { buildSearchVectorGinIndexForCustomObject } from 'src/engine/metadata-modules/object-metadata/utils/build-search-vector-gin-index-for-custom-object.util';
|
||||
import { type UniversalFlatIndexMetadata } from 'src/engine/workspace-manager/workspace-migration/universal-flat-entity/types/universal-flat-index-metadata.type';
|
||||
|
||||
type BuildSearchVectorGinIndexBackfillOperationsArgs = {
|
||||
flatObjectMetadataMaps: FlatEntityMaps<FlatObjectMetadata>;
|
||||
flatFieldMetadataMaps: FlatEntityMaps<FlatFieldMetadata>;
|
||||
flatIndexMaps: FlatEntityMaps<FlatIndexMetadata>;
|
||||
twentyStandardApplicationId: string;
|
||||
workspaceCustomApplicationId: string;
|
||||
};
|
||||
|
||||
// Backfill skips the twenty-standard and workspace-custom applications (their GIN index is
|
||||
// already provisioned by the manifest funnel) and targets every other (installed)
|
||||
// application object that has a searchVector field but no GIN index yet. Grouped by
|
||||
// application universal identifier for the per-application migration run.
|
||||
export const buildSearchVectorGinIndexBackfillOperations = ({
|
||||
flatObjectMetadataMaps,
|
||||
flatFieldMetadataMaps,
|
||||
flatIndexMaps,
|
||||
twentyStandardApplicationId,
|
||||
workspaceCustomApplicationId,
|
||||
}: BuildSearchVectorGinIndexBackfillOperationsArgs): Record<
|
||||
string,
|
||||
UniversalFlatIndexMetadata[]
|
||||
> => {
|
||||
const objectMetadataIdsWithSearchVectorGinIndex = new Set<string>();
|
||||
|
||||
for (const flatIndexMetadata of Object.values(
|
||||
flatIndexMaps.byUniversalIdentifier,
|
||||
).filter(isDefined)) {
|
||||
if (
|
||||
isSearchVectorGinFlatIndexMetadata({
|
||||
flatIndexMetadata,
|
||||
flatFieldMetadataMaps,
|
||||
})
|
||||
) {
|
||||
objectMetadataIdsWithSearchVectorGinIndex.add(
|
||||
flatIndexMetadata.objectMetadataId,
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
const flatIndexesToCreate: UniversalFlatIndexMetadata[] = [];
|
||||
|
||||
for (const flatObjectMetadata of Object.values(
|
||||
flatObjectMetadataMaps.byUniversalIdentifier,
|
||||
).filter(isDefined)) {
|
||||
if (
|
||||
flatObjectMetadata.applicationId === twentyStandardApplicationId ||
|
||||
flatObjectMetadata.applicationId === workspaceCustomApplicationId
|
||||
) {
|
||||
continue;
|
||||
}
|
||||
|
||||
if (objectMetadataIdsWithSearchVectorGinIndex.has(flatObjectMetadata.id)) {
|
||||
continue;
|
||||
}
|
||||
|
||||
const tsVectorFlatFieldMetadata = findTsVectorFlatFieldMetadataForObject({
|
||||
fieldUniversalIdentifiers: flatObjectMetadata.fieldUniversalIdentifiers,
|
||||
flatFieldMetadataMaps,
|
||||
});
|
||||
|
||||
if (!isDefined(tsVectorFlatFieldMetadata)) {
|
||||
continue;
|
||||
}
|
||||
|
||||
flatIndexesToCreate.push(
|
||||
buildSearchVectorGinIndexForCustomObject({
|
||||
flatObjectMetadata,
|
||||
searchVectorFlatFieldMetadata: tsVectorFlatFieldMetadata,
|
||||
}),
|
||||
);
|
||||
}
|
||||
|
||||
return fromArrayToValuesByKeyRecord({
|
||||
array: flatIndexesToCreate,
|
||||
key: 'applicationUniversalIdentifier',
|
||||
});
|
||||
};
|
||||
+82
@@ -0,0 +1,82 @@
|
||||
import { getIndexUniversalIdentifier } from 'twenty-shared/application';
|
||||
import { isDefined } from 'twenty-shared/utils';
|
||||
|
||||
import { isSearchVectorGinFlatIndexMetadata } from 'src/database/commands/upgrade-version-command/2-20/utils/is-search-vector-gin-flat-index-metadata.util';
|
||||
import { type FlatEntityMaps } from 'src/engine/metadata-modules/flat-entity/types/flat-entity-maps.type';
|
||||
import { findFlatEntityByIdInFlatEntityMaps } from 'src/engine/metadata-modules/flat-entity/utils/find-flat-entity-by-id-in-flat-entity-maps.util';
|
||||
import { type FlatFieldMetadata } from 'src/engine/metadata-modules/flat-field-metadata/types/flat-field-metadata.type';
|
||||
import { type FlatIndexMetadata } from 'src/engine/metadata-modules/flat-index-metadata/types/flat-index-metadata.type';
|
||||
import { type FlatObjectMetadata } from 'src/engine/metadata-modules/flat-object-metadata/types/flat-object-metadata.type';
|
||||
|
||||
export type SearchVectorGinIndexUniversalIdentifierUpdate = {
|
||||
id: string;
|
||||
name: string;
|
||||
deterministicUniversalIdentifier: string;
|
||||
};
|
||||
|
||||
type BuildSearchVectorGinIndexReOwnOperationsArgs = {
|
||||
flatObjectMetadataMaps: FlatEntityMaps<FlatObjectMetadata>;
|
||||
flatFieldMetadataMaps: FlatEntityMaps<FlatFieldMetadata>;
|
||||
flatIndexMaps: FlatEntityMaps<FlatIndexMetadata>;
|
||||
applicationUniversalIdentifierById: Map<string, string>;
|
||||
};
|
||||
|
||||
// Re-own is global (every application, including twenty-standard and workspace-custom):
|
||||
// every existing searchVector GIN index whose universal identifier drifted from its
|
||||
// getIndexUniversalIdentifier derivation is converged back to it.
|
||||
export const buildSearchVectorGinIndexReOwnOperations = ({
|
||||
flatObjectMetadataMaps,
|
||||
flatFieldMetadataMaps,
|
||||
flatIndexMaps,
|
||||
applicationUniversalIdentifierById,
|
||||
}: BuildSearchVectorGinIndexReOwnOperationsArgs): SearchVectorGinIndexUniversalIdentifierUpdate[] => {
|
||||
const indexUniversalIdentifierUpdates: SearchVectorGinIndexUniversalIdentifierUpdate[] =
|
||||
[];
|
||||
|
||||
for (const flatIndexMetadata of Object.values(
|
||||
flatIndexMaps.byUniversalIdentifier,
|
||||
).filter(isDefined)) {
|
||||
if (
|
||||
!isSearchVectorGinFlatIndexMetadata({
|
||||
flatIndexMetadata,
|
||||
flatFieldMetadataMaps,
|
||||
})
|
||||
) {
|
||||
continue;
|
||||
}
|
||||
|
||||
const applicationUniversalIdentifier =
|
||||
applicationUniversalIdentifierById.get(flatIndexMetadata.applicationId);
|
||||
const flatObjectMetadata = findFlatEntityByIdInFlatEntityMaps({
|
||||
flatEntityMaps: flatObjectMetadataMaps,
|
||||
flatEntityId: flatIndexMetadata.objectMetadataId,
|
||||
});
|
||||
|
||||
if (
|
||||
!isDefined(applicationUniversalIdentifier) ||
|
||||
!isDefined(flatObjectMetadata)
|
||||
) {
|
||||
continue;
|
||||
}
|
||||
|
||||
const deterministicUniversalIdentifier = getIndexUniversalIdentifier({
|
||||
applicationUniversalIdentifier,
|
||||
objectUniversalIdentifier: flatObjectMetadata.universalIdentifier,
|
||||
name: flatIndexMetadata.name,
|
||||
});
|
||||
|
||||
if (
|
||||
deterministicUniversalIdentifier === flatIndexMetadata.universalIdentifier
|
||||
) {
|
||||
continue;
|
||||
}
|
||||
|
||||
indexUniversalIdentifierUpdates.push({
|
||||
id: flatIndexMetadata.id,
|
||||
name: flatIndexMetadata.name,
|
||||
deterministicUniversalIdentifier,
|
||||
});
|
||||
}
|
||||
|
||||
return indexUniversalIdentifierUpdates;
|
||||
};
|
||||
+38
@@ -0,0 +1,38 @@
|
||||
import { FieldMetadataType } from 'twenty-shared/types';
|
||||
import { isDefined } from 'twenty-shared/utils';
|
||||
|
||||
import { type FlatEntityMaps } from 'src/engine/metadata-modules/flat-entity/types/flat-entity-maps.type';
|
||||
import { findFlatEntityByIdInFlatEntityMaps } from 'src/engine/metadata-modules/flat-entity/utils/find-flat-entity-by-id-in-flat-entity-maps.util';
|
||||
import { type FlatFieldMetadata } from 'src/engine/metadata-modules/flat-field-metadata/types/flat-field-metadata.type';
|
||||
import { type FlatIndexMetadata } from 'src/engine/metadata-modules/flat-index-metadata/types/flat-index-metadata.type';
|
||||
import { IndexType } from 'src/engine/metadata-modules/index-metadata/types/indexType.types';
|
||||
import { SEARCH_VECTOR_FIELD } from 'src/engine/metadata-modules/search-field-metadata/constants/search-vector-field.constants';
|
||||
|
||||
// The searchVector GIN index is the single-column GIN index whose only field is
|
||||
// the object's TS_VECTOR searchVector field (see build-search-vector-gin-index-for-custom-object.util).
|
||||
export const isSearchVectorGinFlatIndexMetadata = ({
|
||||
flatIndexMetadata,
|
||||
flatFieldMetadataMaps,
|
||||
}: {
|
||||
flatIndexMetadata: FlatIndexMetadata;
|
||||
flatFieldMetadataMaps: FlatEntityMaps<FlatFieldMetadata>;
|
||||
}): boolean => {
|
||||
if (flatIndexMetadata.indexType !== IndexType.GIN) {
|
||||
return false;
|
||||
}
|
||||
|
||||
if (flatIndexMetadata.flatIndexFieldMetadatas.length !== 1) {
|
||||
return false;
|
||||
}
|
||||
|
||||
const indexedFlatFieldMetadata = findFlatEntityByIdInFlatEntityMaps({
|
||||
flatEntityMaps: flatFieldMetadataMaps,
|
||||
flatEntityId: flatIndexMetadata.flatIndexFieldMetadatas[0].fieldMetadataId,
|
||||
});
|
||||
|
||||
return (
|
||||
isDefined(indexedFlatFieldMetadata) &&
|
||||
indexedFlatFieldMetadata.type === FieldMetadataType.TS_VECTOR &&
|
||||
indexedFlatFieldMetadata.name === SEARCH_VECTOR_FIELD.name
|
||||
);
|
||||
};
|
||||
+4
@@ -101,7 +101,9 @@ import { BackfillDisplayFieldsOnApplicationRegistrationSlowInstanceCommand } fro
|
||||
import { BackfillIsFeaturedOnApplicationRegistrationSlowInstanceCommand } from './2-19/2-19-instance-command-slow-1783120000000-backfill-is-featured-on-application-registration';
|
||||
import { AllowServerScopedFileFastInstanceCommand } from 'src/database/commands/upgrade-version-command/2-20/2-20-instance-command-fast-1783499671541-allow-server-scoped-file';
|
||||
import { CreateWorkflowVersionCoreTableFastInstanceCommand } from './2-20/2-20-instance-command-fast-1783512000000-create-workflow-version-core-table';
|
||||
import { BackfillNameFieldIsSystemSideEffectSlowInstanceCommand } from './2-20/2-20-instance-command-slow-1783529458168-backfill-name-field-is-system-side-effect';
|
||||
import { RenameIsFeaturedToIsVettedOnApplicationRegistrationFastInstanceCommand } from './2-20/2-20-instance-command-fast-1783527064000-rename-is-featured-to-is-vetted-on-application-registration';
|
||||
import { AddIsSystemSideEffectToSearchFieldMetadataFastInstanceCommand } from './2-20/2-20-instance-command-fast-1783580127637-add-is-system-side-effect-to-search-field-metadata';
|
||||
|
||||
export const INSTANCE_COMMANDS = [
|
||||
AddViewFieldGroupIdIndexOnViewFieldFastInstanceCommand,
|
||||
@@ -205,5 +207,7 @@ export const INSTANCE_COMMANDS = [
|
||||
BackfillIsFeaturedOnApplicationRegistrationSlowInstanceCommand,
|
||||
AllowServerScopedFileFastInstanceCommand,
|
||||
CreateWorkflowVersionCoreTableFastInstanceCommand,
|
||||
BackfillNameFieldIsSystemSideEffectSlowInstanceCommand,
|
||||
RenameIsFeaturedToIsVettedOnApplicationRegistrationFastInstanceCommand,
|
||||
AddIsSystemSideEffectToSearchFieldMetadataFastInstanceCommand,
|
||||
];
|
||||
|
||||
Reference in New Issue
Block a user