Centralize system field side effects + search field metadata (#22594)

## Introduction

Closes twentyhq/core-team-issues#2635 and twentyhq/core-team-issues#2642
and twentyhq/core-team-issues#2589

Object system fields (`searchVector` + its GIN index +
`searchFieldMetadata`, the reserved system fields, default relations)
were provisioned through several scattered, path-specific code paths. As
a result the **app-manifest sync path** authored objects with an
empty/`NULL` `searchVector` and **zero `searchFieldMetadata`**, so
app-owned objects shipped a broken generated search column (see #22657).
The generation logic also lived partly in imperative services rather
than in the metadata side-effect engine, and relied on non-deterministic
(`v4`) universal identifiers that `twenty apply` could not converge,
destroying manually backfilled rows.

This PR centralizes every object-creation system side effect into the
**metadata side-effect engine**, extends the engine to keep search
metadata consistent on field delete and object relabel, makes the
standard app's search identifiers deterministic, and ships upgrade
commands to reconcile existing workspaces.

## What changed

### Side effects moved into the metadata side-effect engine

New dedicated, self-contained handlers — so every write path (API and
app manifest) gets identical results, and side effects never trigger
other side effects.

**Object create / delete** (`handlers/object-metadata`)

* **`objectSystemFieldsOnCreate`** — generates the 7 reserved system
fields (`id`, `createdAt`, `updatedAt`, `deletedAt`, `createdBy`,
`updatedBy`, `position`).
* **`objectSearchVectorOnCreate`** — provisions the full-text search
surface as one unit: the `searchVector` `TS_VECTOR` field, its backing
GIN index, and the `searchFieldMetadata` row (for searchable objects
whose label identifier is a searchable field) that keeps `searchVector`
populated instead of `NULL`.
* **`objectSystemSideEffectsOnDelete`** — tears the above down on object
deletion.

**Search-metadata consistency on relabel / field delete** (new — these
are what close the manifest-path gaps)

* **`objectSearchVectorOnUpdate`** (`handlers/object-metadata`) — when a
searchable object is relabeled onto a new searchable field, provisions
the `searchFieldMetadata` row that indexes it. Relabeling is
**additive**: existing rows (e.g. the provisioned `name` row) are
preserved, so the previous label identifier stays searchable. Mirrors
the API update path so a manifest re-sync that changes the label
identifier reaches search parity. No-ops for junction objects (`id`
label identifier) and non-searchable field types.
* **`fieldSearchFieldMetadataOnDelete`** (`handlers/field-metadata`) —
when a field is deleted, cascade-deletes every `searchFieldMetadata` row
that indexes it. `searchFieldMetadata` is excluded from manifest
deletion inference, so this explicit cascade is what covers **both the
API and manifest paths** (the object-scoped DB cascade only fires on
object deletion). Uses the `searchFieldMetadataUniversalIdentifiers`
aggregator on the flat field for an O(k) lookup instead of scanning all
rows.

The **default `name` field and default relations are now caller-provided
default fields** (SDK autocomplete on the manifest path, input
transpiler on the API path) rather than system side effects — removing
duplicate name generation, the imperative
`build-default-*-for-custom-object` utilities, and the ad-hoc
system-field integrity validator.

### Deterministic identifiers for the standard app

The twenty-standard search GIN index and `searchFieldMetadata` now
derive deterministic universal identifiers
(`getIndexUniversalIdentifier` / `getSearchFieldUniversalIdentifier`)
instead of `v4`, so `twenty apply` converges instead of recreating.

### Upgrade commands (`2-20`) to reconcile existing workspaces

**Instance commands** (run once per instance; ordered fast → slow →
workspace):

1. **`AddIsSystemSideEffectToSearchFieldMetadata`** (fast) — adds the
`isSystemSideEffect` column to `core.searchFieldMetadata`. Defaults to
`true`, which also correctly backfills every existing row since
`searchFieldMetadata` is always system-derived (never user-authored).
2. **`BackfillNameFieldIsSystemSideEffect`** (slow) — re-flags existing
`name` fields from `isSystemSideEffect: true` → `false`, since the
default `name` field is now a caller-provided default like any other
user-owned field (it was provisioned as `true` in 2.15 → 2.19). This is
a pure data backfill, so the bulk `UPDATE` lives in `runDataMigration()`
rather than `up()` — keeping it out of the fast schema transaction
avoids holding an `ACCESS EXCLUSIVE` lock that could stall reads during
the deploy. Slow instance commands still run before every workspace
command of the version, so the fresh value is in place before the
search-reconcile workspace commands recompute the `fieldMetadata`
flat-entity cache. Scoping by name alone is safe (no engine-owned field
is named `name`); `down()` is best-effort (pre-2.15 `false` rows are
indistinguishable from flipped ones).

**Workspace commands** (idempotent, dry-run supported):

1. **`reconcile-search-vector-gin-index-universal-identifier`** —
re-owns every searchVector GIN index UID to its deterministic value (all
applications), then backfills the missing GIN index for installed-app
objects.
2. **`reconcile-search-field-metadata`** — re-owns every
`searchFieldMetadata` UID (all applications), then backfills the missing
rows for installed-app searchable objects.
3. **`rebuild-installed-app-search-vectors`** — rebuilds the
`searchVector` column of every installed-app `TS_VECTOR` field, once the
index and rows exist.

Design notes:

* **Re-own is global** (twenty-standard, workspace-custom, installed) —
a UID convergence keyed on each row's own application.
* **Backfill is installed-app only** — standard/custom objects already
have these rows via the manifest funnel.
* Re-own runs **before** backfill and is transaction-guarded; a failure
aborts that workspace to avoid a unique-identifier collision.

## Tests

* Integration: app manifest sync now asserts system fields + searchable
objects (searchVector, GIN index, searchFieldMetadata) are created; a
new relabel suite drives three manifest syncs and asserts records stay
searchable through the old + new label identifiers and lose
searchability when a field is removed; removed the obsolete
system-fields-integrity suite/snapshots.
* Unit: per-handler side-effect specs (including the new
`objectSearchVectorOnUpdate` and `fieldSearchFieldMetadataOnDelete`
handlers), and per-util specs for the re-own / backfill operation
builders and the GIN-index classifier.

## Upgrade / migration notes

* Existing workspaces converge on the next upgrade run via the `2-20`
instance + workspace commands (idempotent, dry-run supported).
* Backfill and rebuild go through the workspace-migration runner
(automatic cache invalidation); the re-own step invalidates only the
affected flat-entity maps directly.
* The cross-version upgrade CI now flushes the cache before running the
upgrade, so the new version recomputes every flat-entity map from the
database instead of reading blobs the old version serialized in an older
shape.

## Follow-up

* `object-metadata.service.ts` still carries a `TODO: remove once
default view fields move to the metadata side effect engine` — default
view fields are the next candidate to move into the engine.
* A single manifest sync cannot yet both create a field and relabel the
object onto it, because `objectMetadata.update` is ordered before
`fieldMetadata.create` in the migration runner. Tracked in
twentyhq/core-team-issues#2655; to be fixed in a follow-up.
This commit is contained in:
Paul Rastoin
2026-07-09 16:59:54 +02:00
committed by GitHub
parent b47e64c9e3
commit 60fd322b49
88 changed files with 5246 additions and 3095 deletions
@@ -5,15 +5,16 @@ import { buildSearchFieldMetadataBackfillOperations } from 'src/database/command
import { type FlatEntityMaps } from 'src/engine/metadata-modules/flat-entity/types/flat-entity-maps.type';
import { getFlatFieldMetadataMock } from 'src/engine/metadata-modules/flat-field-metadata/__mocks__/get-flat-field-metadata.mock';
import { type FlatFieldMetadata } from 'src/engine/metadata-modules/flat-field-metadata/types/flat-field-metadata.type';
import { SEARCH_VECTOR_FIELD } from 'src/engine/metadata-modules/search-field-metadata/constants/search-vector-field.constants';
import {
getFlatObjectMetadataMock,
getStandardFlatObjectMetadataMock,
} from 'src/engine/metadata-modules/flat-object-metadata/__mocks__/get-flat-object-metadata.mock';
import { type FlatObjectMetadata } from 'src/engine/metadata-modules/flat-object-metadata/types/flat-object-metadata.type';
import { type FlatSearchFieldMetadata } from 'src/engine/metadata-modules/flat-search-field-metadata/types/flat-search-field-metadata.type';
import { SEARCH_VECTOR_FIELD } from 'src/engine/metadata-modules/search-field-metadata/constants/search-vector-field.constants';
const CUSTOM_APPLICATION_UNIVERSAL_IDENTIFIER = 'custom-application-uid';
const CUSTOM_APPLICATION_UNIVERSAL_IDENTIFIER =
'c0c1c2c3-c4c5-4000-8000-000000000001';
const CUSTOM_APPLICATION_ID = 'custom-application-id';
const buildUniversalIdentifiersByApplicationId = (
@@ -129,6 +130,7 @@ const buildSearchFieldMetadata = ({
applicationId: 'unused-application-id',
applicationUniversalIdentifier,
position,
isSystemSideEffect: true,
workspaceId: 'workspace-id',
createdAt,
updatedAt: createdAt,
@@ -221,8 +223,9 @@ describe('buildSearchFieldMetadataBackfillOperations', () => {
searchVectorField,
]),
flatSearchFieldMetadataMaps: buildFlatSearchFieldMetadataMaps([]),
standardFlatSearchFieldMetadataMaps:
buildFlatSearchFieldMetadataMaps([]),
standardFlatSearchFieldMetadataMaps: buildFlatSearchFieldMetadataMaps(
[],
),
customApplicationId: CUSTOM_APPLICATION_ID,
});
@@ -288,8 +291,9 @@ describe('buildSearchFieldMetadataBackfillOperations', () => {
flatObjectMetadataMaps: buildFlatObjectMetadataMaps([junctionObject]),
flatFieldMetadataMaps: buildFlatFieldMetadataMaps([idField]),
flatSearchFieldMetadataMaps: buildFlatSearchFieldMetadataMaps([]),
standardFlatSearchFieldMetadataMaps:
buildFlatSearchFieldMetadataMaps([]),
standardFlatSearchFieldMetadataMaps: buildFlatSearchFieldMetadataMaps(
[],
),
customApplicationId: CUSTOM_APPLICATION_ID,
});
@@ -592,8 +596,7 @@ describe('buildSearchFieldMetadataBackfillOperations', () => {
fieldMetadataId: nameField.id,
objectMetadataUniversalIdentifier: customObject.universalIdentifier,
fieldMetadataUniversalIdentifier: nameField.universalIdentifier,
applicationUniversalIdentifier:
CUSTOM_APPLICATION_UNIVERSAL_IDENTIFIER,
applicationUniversalIdentifier: CUSTOM_APPLICATION_UNIVERSAL_IDENTIFIER,
});
const { flatSearchFieldMetadatasToCreateByApplicationUniversalIdentifier } =
@@ -606,8 +609,9 @@ describe('buildSearchFieldMetadataBackfillOperations', () => {
flatSearchFieldMetadataMaps: buildFlatSearchFieldMetadataMaps([
existingSearchFieldMetadata,
]),
standardFlatSearchFieldMetadataMaps:
buildFlatSearchFieldMetadataMaps([]),
standardFlatSearchFieldMetadataMaps: buildFlatSearchFieldMetadataMaps(
[],
),
customApplicationId: CUSTOM_APPLICATION_ID,
});
@@ -650,8 +654,9 @@ describe('buildSearchFieldMetadataBackfillOperations', () => {
flatObjectMetadataMaps: buildFlatObjectMetadataMaps([customObject]),
flatFieldMetadataMaps: buildFlatFieldMetadataMaps([relationNameField]),
flatSearchFieldMetadataMaps: buildFlatSearchFieldMetadataMaps([]),
standardFlatSearchFieldMetadataMaps:
buildFlatSearchFieldMetadataMaps([]),
standardFlatSearchFieldMetadataMaps: buildFlatSearchFieldMetadataMaps(
[],
),
customApplicationId: CUSTOM_APPLICATION_ID,
});
@@ -685,8 +690,9 @@ describe('buildSearchFieldMetadataBackfillOperations', () => {
nameDescriptionField,
]),
flatSearchFieldMetadataMaps: buildFlatSearchFieldMetadataMaps([]),
standardFlatSearchFieldMetadataMaps:
buildFlatSearchFieldMetadataMaps([]),
standardFlatSearchFieldMetadataMaps: buildFlatSearchFieldMetadataMaps(
[],
),
customApplicationId: CUSTOM_APPLICATION_ID,
});
@@ -716,8 +722,9 @@ describe('buildSearchFieldMetadataBackfillOperations', () => {
nameDescriptionField,
]),
flatSearchFieldMetadataMaps: buildFlatSearchFieldMetadataMaps([]),
standardFlatSearchFieldMetadataMaps:
buildFlatSearchFieldMetadataMaps([]),
standardFlatSearchFieldMetadataMaps: buildFlatSearchFieldMetadataMaps(
[],
),
customApplicationId: CUSTOM_APPLICATION_ID,
});
@@ -0,0 +1,23 @@
import { QueryRunner } from 'typeorm';
import { RegisteredInstanceCommand } from 'src/engine/core-modules/upgrade/decorators/registered-instance-command.decorator';
import { FastInstanceCommand } from 'src/engine/core-modules/upgrade/interfaces/fast-instance-command.interface';
// searchFieldMetadata rows are always system-derived (never user-authored), so the
// new column defaults to true, which also backfills every existing row correctly.
@RegisteredInstanceCommand('2.20.0', 1783580127637)
export class AddIsSystemSideEffectToSearchFieldMetadataFastInstanceCommand
implements FastInstanceCommand
{
public async up(queryRunner: QueryRunner): Promise<void> {
await queryRunner.query(
'ALTER TABLE "core"."searchFieldMetadata" ADD COLUMN IF NOT EXISTS "isSystemSideEffect" boolean NOT NULL DEFAULT true',
);
}
public async down(queryRunner: QueryRunner): Promise<void> {
await queryRunner.query(
'ALTER TABLE "core"."searchFieldMetadata" DROP COLUMN IF EXISTS "isSystemSideEffect"',
);
}
}
@@ -0,0 +1,38 @@
import { DataSource, QueryRunner } from 'typeorm';
import { RegisteredInstanceCommand } from 'src/engine/core-modules/upgrade/decorators/registered-instance-command.decorator';
import { SlowInstanceCommand } from 'src/engine/core-modules/upgrade/interfaces/slow-instance-command.interface';
// The default `name` field used to be provisioned with isSystemSideEffect: true
// (2.15 → 2.19). It is now a caller-provided default (isSystemSideEffect: false),
// like every other user-owned field, so existing rows must be re-flagged.
//
// Scoping by name alone is safe: no engine-owned field is named `name` (the
// reserved set is id/createdAt/updatedAt/deletedAt/createdBy/updatedBy/position/
// searchVector), and API-created custom fields are always false already.
//
// This is a pure data backfill, so the write lives in runDataMigration() (slow
// instance command) rather than up(): keeping the bulk UPDATE out of the fast
// schema transaction avoids holding an ACCESS EXCLUSIVE lock that could stall
// reads during the deploy. Slow instance commands still run before every
// workspace command of the version (order is fast → slow → workspace), so the
// fresh value is in place before the 2.20 search reconcile workspace commands
// flush and recompute the fieldMetadata flat-entity cache (the UPDATE itself
// does not invalidate the per-workspace cache).
@RegisteredInstanceCommand('2.20.0', 1783529458168, { type: 'slow' })
export class BackfillNameFieldIsSystemSideEffectSlowInstanceCommand implements SlowInstanceCommand {
async runDataMigration(dataSource: DataSource): Promise<void> {
await dataSource.query(
`UPDATE "core"."fieldMetadata" SET "isSystemSideEffect" = false WHERE "name" = 'name' AND "isSystemSideEffect" = true`,
);
}
public async up(_queryRunner: QueryRunner): Promise<void> {
return;
}
// Intentional no-op: this backfill cannot be safely reversed. Pre-2.15 `name`
public async down(_queryRunner: QueryRunner): Promise<void> {
return;
}
}
@@ -1,30 +1,44 @@
import { Module } from '@nestjs/common';
import { TypeOrmModule } from '@nestjs/typeorm';
import { WorkspaceIteratorModule } from 'src/database/commands/command-runners/workspace-iterator.module';
import { BackfillActorSourceEnumValuesCommand } from 'src/database/commands/upgrade-version-command/2-20/2-20-workspace-command-1783499671542-backfill-actor-source-enum-values.command';
import { BackfillWorkflowVersionToCoreCommand } from 'src/database/commands/upgrade-version-command/2-20/2-20-workspace-command-1783526282685-backfill-workflow-version-to-core.command';
import { AddMessageCampaignStatFieldsCommand } from 'src/database/commands/upgrade-version-command/2-20/2-20-workspace-command-1783525261000-add-message-campaign-stat-fields.command';
import { CreateMessageListViewCommand } from 'src/database/commands/upgrade-version-command/2-20/2-20-workspace-command-1783525261001-create-message-list-view.command';
import { BackfillWorkflowVersionToCoreCommand } from 'src/database/commands/upgrade-version-command/2-20/2-20-workspace-command-1783526282685-backfill-workflow-version-to-core.command';
import { ReconcileSearchVectorGinIndexUniversalIdentifierCommand } from 'src/database/commands/upgrade-version-command/2-20/2-20-workspace-command-1783529458169-reconcile-search-vector-gin-index-universal-identifier.command';
import { ReconcileSearchFieldMetadataCommand } from 'src/database/commands/upgrade-version-command/2-20/2-20-workspace-command-1783529458170-reconcile-search-field-metadata.command';
import { RebuildInstalledAppSearchVectorsCommand } from 'src/database/commands/upgrade-version-command/2-20/2-20-workspace-command-1783529458171-rebuild-installed-app-search-vectors.command';
import { ApplicationModule } from 'src/engine/core-modules/application/application.module';
import { WorkflowVersionCoreModule } from 'src/engine/core-modules/workflow/workflow-version-core.module';
import { IndexMetadataEntity } from 'src/engine/metadata-modules/index-metadata/index-metadata.entity';
import { SearchFieldMetadataEntity } from 'src/engine/metadata-modules/search-field-metadata/search-field-metadata.entity';
import { WorkspaceMetadataVersionModule } from 'src/engine/metadata-modules/workspace-metadata-version/workspace-metadata-version.module';
import { WorkspaceSchemaManagerModule } from 'src/engine/twenty-orm/workspace-schema-manager/workspace-schema-manager.module';
import { WorkspaceCacheModule } from 'src/engine/workspace-cache/workspace-cache.module';
import { WorkspaceMigrationRunnerModule } from 'src/engine/workspace-manager/workspace-migration/workspace-migration-runner/workspace-migration-runner.module';
import { WorkspaceMigrationModule } from 'src/engine/workspace-manager/workspace-migration/workspace-migration.module';
@Module({
imports: [
ApplicationModule,
WorkspaceIteratorModule,
WorkspaceCacheModule,
WorkspaceMigrationModule,
WorkspaceSchemaManagerModule,
TypeOrmModule.forFeature([IndexMetadataEntity, SearchFieldMetadataEntity]),
WorkflowVersionCoreModule,
WorkspaceCacheModule,
WorkspaceIteratorModule,
WorkspaceMetadataVersionModule,
WorkspaceMigrationModule,
WorkspaceMigrationRunnerModule,
WorkspaceSchemaManagerModule,
],
providers: [
AddMessageCampaignStatFieldsCommand,
CreateMessageListViewCommand,
BackfillActorSourceEnumValuesCommand,
BackfillWorkflowVersionToCoreCommand,
ReconcileSearchVectorGinIndexUniversalIdentifierCommand,
ReconcileSearchFieldMetadataCommand,
RebuildInstalledAppSearchVectorsCommand,
],
})
export class V2_20_UpgradeVersionCommandModule {}
@@ -0,0 +1,281 @@
import { InjectRepository } from '@nestjs/typeorm';
import { Command } from 'nest-commander';
import { isDefined } from 'twenty-shared/utils';
import { Repository } from 'typeorm';
import { ActiveOrSuspendedWorkspaceCommandRunner } from 'src/database/commands/command-runners/active-or-suspended-workspace.command-runner';
import { WorkspaceIteratorService } from 'src/database/commands/command-runners/workspace-iterator.service';
import { type RunOnWorkspaceArgs } from 'src/database/commands/command-runners/workspace.command-runner';
import { buildSearchVectorGinIndexBackfillOperations } from 'src/database/commands/upgrade-version-command/2-20/utils/build-search-vector-gin-index-backfill-operations.util';
import {
buildSearchVectorGinIndexReOwnOperations,
type SearchVectorGinIndexUniversalIdentifierUpdate,
} from 'src/database/commands/upgrade-version-command/2-20/utils/build-search-vector-gin-index-re-own-operations.util';
import { ApplicationService } from 'src/engine/core-modules/application/application.service';
import { RegisteredWorkspaceCommand } from 'src/engine/core-modules/upgrade/decorators/registered-workspace-command.decorator';
import { getMetadataFlatEntityMapsKey } from 'src/engine/metadata-modules/flat-entity/utils/get-metadata-flat-entity-maps-key.util';
import { getMetadataRelatedMetadataNames } from 'src/engine/metadata-modules/flat-entity/utils/get-metadata-related-metadata-names.util';
import { getMetadataSerializedRelationNames } from 'src/engine/metadata-modules/flat-entity/utils/get-metadata-serialized-relation-names.util';
import { IndexMetadataEntity } from 'src/engine/metadata-modules/index-metadata/index-metadata.entity';
import { WorkspaceMetadataVersionService } from 'src/engine/metadata-modules/workspace-metadata-version/services/workspace-metadata-version.service';
import { WorkspaceCacheService } from 'src/engine/workspace-cache/services/workspace-cache.service';
import { type UniversalFlatIndexMetadata } from 'src/engine/workspace-manager/workspace-migration/universal-flat-entity/types/universal-flat-index-metadata.type';
import { WorkspaceMigrationValidateBuildAndRunService } from 'src/engine/workspace-manager/workspace-migration/services/workspace-migration-validate-build-and-run-service';
type SearchVectorGinIndexReconciliationOperations = {
indexUniversalIdentifierUpdates: SearchVectorGinIndexUniversalIdentifierUpdate[];
flatIndexesToCreateByApplicationUniversalIdentifier: Record<
string,
UniversalFlatIndexMetadata[]
>;
};
@RegisteredWorkspaceCommand('2.20.0', 1783529458169)
@Command({
name: 'upgrade:2-20:reconcile-search-vector-gin-index-universal-identifier',
description:
'Converge every searchVector GIN index universal identifier to its deterministic derivation (re-own, all applications) and create the missing GIN index for installed-app objects (backfill). Idempotent, re-own runs before backfill.',
})
export class ReconcileSearchVectorGinIndexUniversalIdentifierCommand extends ActiveOrSuspendedWorkspaceCommandRunner {
constructor(
protected readonly workspaceIteratorService: WorkspaceIteratorService,
private readonly applicationService: ApplicationService,
private readonly workspaceCacheService: WorkspaceCacheService,
private readonly workspaceMetadataVersionService: WorkspaceMetadataVersionService,
private readonly workspaceMigrationValidateBuildAndRunService: WorkspaceMigrationValidateBuildAndRunService,
// eslint-disable-next-line twenty/prefer-workspace-scoped-repository
@InjectRepository(IndexMetadataEntity)
private readonly indexMetadataRepository: Repository<IndexMetadataEntity>,
) {
super(workspaceIteratorService);
}
override async runOnWorkspace({
workspaceId,
options,
}: RunOnWorkspaceArgs): Promise<void> {
const isDryRun = options.dryRun ?? false;
const {
indexUniversalIdentifierUpdates,
flatIndexesToCreateByApplicationUniversalIdentifier,
} = await this.computeOperations({ workspaceId });
const totalIndexesToBackfill = Object.values(
flatIndexesToCreateByApplicationUniversalIdentifier,
).reduce((total, flatIndexes) => total + flatIndexes.length, 0);
if (
indexUniversalIdentifierUpdates.length === 0 &&
totalIndexesToBackfill === 0
) {
this.logger.log(
`No searchVector GIN index to reconcile for workspace ${workspaceId}`,
);
return;
}
this.logger.log(
`${isDryRun ? '[DRY RUN] ' : ''}Reconciling ${indexUniversalIdentifierUpdates.length} searchVector GIN index universal identifier(s) and creating ${totalIndexesToBackfill} missing index(es) across ${Object.keys(flatIndexesToCreateByApplicationUniversalIdentifier).length} application(s) for workspace ${workspaceId}`,
);
if (isDryRun) {
return;
}
await this.applyOperations({
workspaceId,
indexUniversalIdentifierUpdates,
flatIndexesToCreateByApplicationUniversalIdentifier,
});
this.logger.log(
`Reconciled searchVector GIN indexes for workspace ${workspaceId}`,
);
}
private async computeOperations({
workspaceId,
}: {
workspaceId: string;
}): Promise<SearchVectorGinIndexReconciliationOperations> {
const {
flatObjectMetadataMaps,
flatFieldMetadataMaps,
flatIndexMaps,
flatApplicationMaps,
} = await this.workspaceCacheService.getOrRecompute(workspaceId, [
'flatObjectMetadataMaps',
'flatFieldMetadataMaps',
'flatIndexMaps',
'flatApplicationMaps',
]);
const { twentyStandardFlatApplication, workspaceCustomFlatApplication } =
await this.applicationService.findWorkspaceTwentyStandardAndCustomApplicationOrThrow(
{ workspaceId },
);
const applicationUniversalIdentifierById = new Map(
Object.values(flatApplicationMaps.byId)
.filter(isDefined)
.map((flatApplication) => [
flatApplication.id,
flatApplication.universalIdentifier,
]),
);
const indexUniversalIdentifierUpdates =
buildSearchVectorGinIndexReOwnOperations({
flatObjectMetadataMaps,
flatFieldMetadataMaps,
flatIndexMaps,
applicationUniversalIdentifierById,
});
const flatIndexesToCreateByApplicationUniversalIdentifier =
buildSearchVectorGinIndexBackfillOperations({
flatObjectMetadataMaps,
flatFieldMetadataMaps,
flatIndexMaps,
twentyStandardApplicationId: twentyStandardFlatApplication.id,
workspaceCustomApplicationId: workspaceCustomFlatApplication.id,
});
return {
indexUniversalIdentifierUpdates,
flatIndexesToCreateByApplicationUniversalIdentifier,
};
}
private async applyOperations({
workspaceId,
indexUniversalIdentifierUpdates,
flatIndexesToCreateByApplicationUniversalIdentifier,
}: {
workspaceId: string;
} & SearchVectorGinIndexReconciliationOperations): Promise<void> {
// Re-own before backfill: a deterministic index created next to a surviving legacy
// row would collide on the unique universal identifier constraint.
await this.applyReOwn({ workspaceId, indexUniversalIdentifierUpdates });
await this.applyBackfill({
workspaceId,
flatIndexesToCreateByApplicationUniversalIdentifier,
});
}
private async applyReOwn({
workspaceId,
indexUniversalIdentifierUpdates,
}: {
workspaceId: string;
indexUniversalIdentifierUpdates: SearchVectorGinIndexUniversalIdentifierUpdate[];
}): Promise<void> {
if (indexUniversalIdentifierUpdates.length === 0) {
return;
}
try {
await this.indexMetadataRepository.manager.transaction(
async (entityManager) => {
const transactionalIndexMetadataRepository =
entityManager.getRepository(IndexMetadataEntity);
for (const {
id,
deterministicUniversalIdentifier,
} of indexUniversalIdentifierUpdates) {
await transactionalIndexMetadataRepository.update(
{ id, workspaceId },
{ universalIdentifier: deterministicUniversalIdentifier },
);
}
},
);
} catch (error) {
// Abort before backfill: creating a deterministic index next to a surviving legacy
// row (re-own rolled back) would collide on the unique universal identifier
// constraint.
this.logger.error(
`Failed to re-own ${indexUniversalIdentifierUpdates.length} searchVector GIN index universal identifier(s) for workspace ${workspaceId}, aborting: ${
error instanceof Error ? error.message : String(error)
}`,
);
throw error;
}
await this.flushIndexCacheAndBumpMetadataVersion(workspaceId);
}
private async applyBackfill({
workspaceId,
flatIndexesToCreateByApplicationUniversalIdentifier,
}: {
workspaceId: string;
flatIndexesToCreateByApplicationUniversalIdentifier: Record<
string,
UniversalFlatIndexMetadata[]
>;
}): Promise<void> {
for (const [
applicationUniversalIdentifier,
flatIndexesToCreate,
] of Object.entries(flatIndexesToCreateByApplicationUniversalIdentifier)) {
if (flatIndexesToCreate.length === 0) {
continue;
}
const validateAndBuildResult =
await this.workspaceMigrationValidateBuildAndRunService.validateBuildAndRunWorkspaceMigration(
{
isSystemBuild: true,
allFlatEntityOperationByMetadataName: {
index: {
flatEntityToCreate: flatIndexesToCreate,
flatEntityToDelete: [],
flatEntityToUpdate: [],
},
},
workspaceId,
applicationUniversalIdentifier,
},
);
if (validateAndBuildResult.status === 'fail') {
this.logger.error(
`Failed to create searchVector GIN index(es) for application ${applicationUniversalIdentifier}:\n${JSON.stringify(
validateAndBuildResult,
null,
2,
)}`,
);
throw new Error(
`Failed to create searchVector GIN index(es) for workspace ${workspaceId}`,
);
}
}
}
private async flushIndexCacheAndBumpMetadataVersion(
workspaceId: string,
): Promise<void> {
const indexRelatedMetadataNames = [
'index',
...getMetadataRelatedMetadataNames('index'),
...getMetadataSerializedRelationNames('index'),
] as const;
const cacheKeysToFlush = [
...new Set(indexRelatedMetadataNames.map(getMetadataFlatEntityMapsKey)),
];
await this.workspaceCacheService.flush(workspaceId, cacheKeysToFlush);
await this.workspaceMetadataVersionService.incrementMetadataVersion(
workspaceId,
);
}
}
@@ -0,0 +1,293 @@
import { InjectRepository } from '@nestjs/typeorm';
import { Command } from 'nest-commander';
import { isDefined } from 'twenty-shared/utils';
import { Repository } from 'typeorm';
import { ActiveOrSuspendedWorkspaceCommandRunner } from 'src/database/commands/command-runners/active-or-suspended-workspace.command-runner';
import { WorkspaceIteratorService } from 'src/database/commands/command-runners/workspace-iterator.service';
import { type RunOnWorkspaceArgs } from 'src/database/commands/command-runners/workspace.command-runner';
import { buildSearchFieldMetadataBackfillOperations } from 'src/database/commands/upgrade-version-command/2-20/utils/build-search-field-metadata-backfill-operations.util';
import {
buildSearchFieldMetadataReOwnOperations,
type SearchFieldMetadataUniversalIdentifierUpdate,
} from 'src/database/commands/upgrade-version-command/2-20/utils/build-search-field-metadata-re-own-operations.util';
import { ApplicationService } from 'src/engine/core-modules/application/application.service';
import { RegisteredWorkspaceCommand } from 'src/engine/core-modules/upgrade/decorators/registered-workspace-command.decorator';
import { getMetadataFlatEntityMapsKey } from 'src/engine/metadata-modules/flat-entity/utils/get-metadata-flat-entity-maps-key.util';
import { getMetadataRelatedMetadataNames } from 'src/engine/metadata-modules/flat-entity/utils/get-metadata-related-metadata-names.util';
import { getMetadataSerializedRelationNames } from 'src/engine/metadata-modules/flat-entity/utils/get-metadata-serialized-relation-names.util';
import { SearchFieldMetadataEntity } from 'src/engine/metadata-modules/search-field-metadata/search-field-metadata.entity';
import { WorkspaceMetadataVersionService } from 'src/engine/metadata-modules/workspace-metadata-version/services/workspace-metadata-version.service';
import { WorkspaceCacheService } from 'src/engine/workspace-cache/services/workspace-cache.service';
import { type UniversalFlatSearchFieldMetadata } from 'src/engine/workspace-manager/workspace-migration/universal-flat-entity/types/universal-flat-search-field-metadata.type';
import { WorkspaceMigrationValidateBuildAndRunService } from 'src/engine/workspace-manager/workspace-migration/services/workspace-migration-validate-build-and-run-service';
type SearchFieldMetadataReconciliationOperations = {
searchFieldMetadataUniversalIdentifierUpdates: SearchFieldMetadataUniversalIdentifierUpdate[];
flatSearchFieldMetadatasToCreateByApplicationUniversalIdentifier: Record<
string,
UniversalFlatSearchFieldMetadata[]
>;
};
@RegisteredWorkspaceCommand('2.20.0', 1783529458170)
@Command({
name: 'upgrade:2-20:reconcile-search-field-metadata',
description:
'Converge every searchFieldMetadata universal identifier to its deterministic derivation (re-own, all applications) and create the missing searchFieldMetadata row for installed-app searchable objects (backfill). Idempotent, re-own runs before backfill to avoid a unique-identifier collision.',
})
export class ReconcileSearchFieldMetadataCommand extends ActiveOrSuspendedWorkspaceCommandRunner {
constructor(
protected readonly workspaceIteratorService: WorkspaceIteratorService,
private readonly applicationService: ApplicationService,
private readonly workspaceCacheService: WorkspaceCacheService,
private readonly workspaceMetadataVersionService: WorkspaceMetadataVersionService,
private readonly workspaceMigrationValidateBuildAndRunService: WorkspaceMigrationValidateBuildAndRunService,
// eslint-disable-next-line twenty/prefer-workspace-scoped-repository
@InjectRepository(SearchFieldMetadataEntity)
private readonly searchFieldMetadataRepository: Repository<SearchFieldMetadataEntity>,
) {
super(workspaceIteratorService);
}
override async runOnWorkspace({
workspaceId,
options,
}: RunOnWorkspaceArgs): Promise<void> {
const isDryRun = options.dryRun ?? false;
const {
searchFieldMetadataUniversalIdentifierUpdates,
flatSearchFieldMetadatasToCreateByApplicationUniversalIdentifier,
} = await this.computeOperations({ workspaceId });
const totalRowsToBackfill = Object.values(
flatSearchFieldMetadatasToCreateByApplicationUniversalIdentifier,
).reduce(
(total, flatSearchFieldMetadatas) =>
total + flatSearchFieldMetadatas.length,
0,
);
if (
searchFieldMetadataUniversalIdentifierUpdates.length === 0 &&
totalRowsToBackfill === 0
) {
this.logger.log(
`No searchFieldMetadata to reconcile for workspace ${workspaceId}`,
);
return;
}
this.logger.log(
`${isDryRun ? '[DRY RUN] ' : ''}Reconciling ${searchFieldMetadataUniversalIdentifierUpdates.length} searchFieldMetadata universal identifier(s) and creating ${totalRowsToBackfill} missing row(s) across ${Object.keys(flatSearchFieldMetadatasToCreateByApplicationUniversalIdentifier).length} application(s) for workspace ${workspaceId}`,
);
if (isDryRun) {
return;
}
await this.applyOperations({
workspaceId,
searchFieldMetadataUniversalIdentifierUpdates,
flatSearchFieldMetadatasToCreateByApplicationUniversalIdentifier,
});
this.logger.log(
`Reconciled searchFieldMetadata for workspace ${workspaceId}`,
);
}
private async computeOperations({
workspaceId,
}: {
workspaceId: string;
}): Promise<SearchFieldMetadataReconciliationOperations> {
const {
flatObjectMetadataMaps,
flatFieldMetadataMaps,
flatSearchFieldMetadataMaps,
flatApplicationMaps,
} = await this.workspaceCacheService.getOrRecompute(workspaceId, [
'flatObjectMetadataMaps',
'flatFieldMetadataMaps',
'flatSearchFieldMetadataMaps',
'flatApplicationMaps',
]);
const { twentyStandardFlatApplication, workspaceCustomFlatApplication } =
await this.applicationService.findWorkspaceTwentyStandardAndCustomApplicationOrThrow(
{ workspaceId },
);
const applicationUniversalIdentifierById = new Map(
Object.values(flatApplicationMaps.byId)
.filter(isDefined)
.map((flatApplication) => [
flatApplication.id,
flatApplication.universalIdentifier,
]),
);
const searchFieldMetadataUniversalIdentifierUpdates =
buildSearchFieldMetadataReOwnOperations({
flatFieldMetadataMaps,
flatSearchFieldMetadataMaps,
applicationUniversalIdentifierById,
});
const flatSearchFieldMetadatasToCreateByApplicationUniversalIdentifier =
buildSearchFieldMetadataBackfillOperations({
flatObjectMetadataMaps,
flatFieldMetadataMaps,
flatSearchFieldMetadataMaps,
applicationUniversalIdentifierById,
twentyStandardApplicationId: twentyStandardFlatApplication.id,
workspaceCustomApplicationId: workspaceCustomFlatApplication.id,
});
return {
searchFieldMetadataUniversalIdentifierUpdates,
flatSearchFieldMetadatasToCreateByApplicationUniversalIdentifier,
};
}
private async applyOperations({
workspaceId,
searchFieldMetadataUniversalIdentifierUpdates,
flatSearchFieldMetadatasToCreateByApplicationUniversalIdentifier,
}: {
workspaceId: string;
} & SearchFieldMetadataReconciliationOperations): Promise<void> {
// Re-own before backfill: a deterministic row created next to a surviving legacy v4
// row (which the deletion-inference short-circuit keeps alive) would collide on the
// unique universal identifier constraint.
await this.applyReOwn({
workspaceId,
searchFieldMetadataUniversalIdentifierUpdates,
});
await this.applyBackfill({
workspaceId,
flatSearchFieldMetadatasToCreateByApplicationUniversalIdentifier,
});
}
private async applyReOwn({
workspaceId,
searchFieldMetadataUniversalIdentifierUpdates,
}: {
workspaceId: string;
searchFieldMetadataUniversalIdentifierUpdates: SearchFieldMetadataUniversalIdentifierUpdate[];
}): Promise<void> {
if (searchFieldMetadataUniversalIdentifierUpdates.length === 0) {
return;
}
try {
await this.searchFieldMetadataRepository.manager.transaction(
async (entityManager) => {
const transactionalSearchFieldMetadataRepository =
entityManager.getRepository(SearchFieldMetadataEntity);
for (const {
id,
deterministicUniversalIdentifier,
} of searchFieldMetadataUniversalIdentifierUpdates) {
await transactionalSearchFieldMetadataRepository.update(
{ id, workspaceId },
{ universalIdentifier: deterministicUniversalIdentifier },
);
}
},
);
} catch (error) {
// Abort before backfill: creating a deterministic row next to a surviving legacy
// row (re-own rolled back) would collide on the unique universal identifier
// constraint.
this.logger.error(
`Failed to re-own ${searchFieldMetadataUniversalIdentifierUpdates.length} searchFieldMetadata universal identifier(s) for workspace ${workspaceId}, aborting: ${
error instanceof Error ? error.message : String(error)
}`,
);
throw error;
}
await this.flushSearchFieldMetadataCacheAndBumpMetadataVersion(workspaceId);
}
private async applyBackfill({
workspaceId,
flatSearchFieldMetadatasToCreateByApplicationUniversalIdentifier,
}: {
workspaceId: string;
flatSearchFieldMetadatasToCreateByApplicationUniversalIdentifier: Record<
string,
UniversalFlatSearchFieldMetadata[]
>;
}): Promise<void> {
for (const [
applicationUniversalIdentifier,
flatSearchFieldMetadatasToCreate,
] of Object.entries(
flatSearchFieldMetadatasToCreateByApplicationUniversalIdentifier,
)) {
if (flatSearchFieldMetadatasToCreate.length === 0) {
continue;
}
const validateAndBuildResult =
await this.workspaceMigrationValidateBuildAndRunService.validateBuildAndRunWorkspaceMigration(
{
isSystemBuild: true,
allFlatEntityOperationByMetadataName: {
searchFieldMetadata: {
flatEntityToCreate: flatSearchFieldMetadatasToCreate,
flatEntityToDelete: [],
flatEntityToUpdate: [],
},
},
workspaceId,
applicationUniversalIdentifier,
},
);
if (validateAndBuildResult.status === 'fail') {
this.logger.error(
`Failed to create searchFieldMetadata row(s) for application ${applicationUniversalIdentifier}:\n${JSON.stringify(
validateAndBuildResult,
null,
2,
)}`,
);
throw new Error(
`Failed to create searchFieldMetadata row(s) for workspace ${workspaceId}`,
);
}
}
}
private async flushSearchFieldMetadataCacheAndBumpMetadataVersion(
workspaceId: string,
): Promise<void> {
const searchFieldMetadataRelatedMetadataNames = [
'searchFieldMetadata',
...getMetadataRelatedMetadataNames('searchFieldMetadata'),
...getMetadataSerializedRelationNames('searchFieldMetadata'),
] as const;
const cacheKeysToFlush = [
...new Set(
searchFieldMetadataRelatedMetadataNames.map(getMetadataFlatEntityMapsKey),
),
];
await this.workspaceCacheService.flush(workspaceId, cacheKeysToFlush);
await this.workspaceMetadataVersionService.incrementMetadataVersion(
workspaceId,
);
}
}
@@ -0,0 +1,99 @@
import { Command } from 'nest-commander';
import { FieldMetadataType } from 'twenty-shared/types';
import { isDefined } from 'twenty-shared/utils';
import { ActiveOrSuspendedWorkspaceCommandRunner } from 'src/database/commands/command-runners/active-or-suspended-workspace.command-runner';
import { WorkspaceIteratorService } from 'src/database/commands/command-runners/workspace-iterator.service';
import { type RunOnWorkspaceArgs } from 'src/database/commands/command-runners/workspace.command-runner';
import { ApplicationService } from 'src/engine/core-modules/application/application.service';
import { RegisteredWorkspaceCommand } from 'src/engine/core-modules/upgrade/decorators/registered-workspace-command.decorator';
import { TWENTY_STANDARD_APPLICATION } from 'src/engine/workspace-manager/twenty-standard-application/constants/twenty-standard-applications';
import { WorkspaceCacheService } from 'src/engine/workspace-cache/services/workspace-cache.service';
import { type UniversalUpdateFieldAction } from 'src/engine/workspace-manager/workspace-migration/workspace-migration-builder/builders/field/types/workspace-migration-field-action';
import { WORKSPACE_MIGRATION_ACTION_TYPE } from 'src/engine/workspace-manager/workspace-migration/workspace-migration-builder/constants/workspace-migration-action-type.constant';
import { WorkspaceMigrationRunnerService } from 'src/engine/workspace-manager/workspace-migration/workspace-migration-runner/services/workspace-migration-runner.service';
@RegisteredWorkspaceCommand('2.20.0', 1783529458171)
@Command({
name: 'upgrade:2-20:rebuild-installed-app-search-vectors',
description:
'Rebuild the searchVector column of every installed-app TS_VECTOR field from its searchFieldMetadata rows, now that the GIN index and searchFieldMetadata rows exist (commands 1 and 2). Idempotent.',
})
export class RebuildInstalledAppSearchVectorsCommand extends ActiveOrSuspendedWorkspaceCommandRunner {
constructor(
protected readonly workspaceIteratorService: WorkspaceIteratorService,
private readonly applicationService: ApplicationService,
private readonly workspaceCacheService: WorkspaceCacheService,
private readonly workspaceMigrationRunnerService: WorkspaceMigrationRunnerService,
) {
super(workspaceIteratorService);
}
override async runOnWorkspace({
workspaceId,
options,
}: RunOnWorkspaceArgs): Promise<void> {
const isDryRun = options.dryRun ?? false;
const { flatFieldMetadataMaps } =
await this.workspaceCacheService.getOrRecompute(workspaceId, [
'flatFieldMetadataMaps',
]);
const { twentyStandardFlatApplication, workspaceCustomFlatApplication } =
await this.applicationService.findWorkspaceTwentyStandardAndCustomApplicationOrThrow(
{ workspaceId },
);
const installedAppTsVectorFlatFieldMetadatas = Object.values(
flatFieldMetadataMaps.byUniversalIdentifier,
)
.filter(isDefined)
.filter(
(flatFieldMetadata) =>
flatFieldMetadata.type === FieldMetadataType.TS_VECTOR &&
flatFieldMetadata.applicationId !==
twentyStandardFlatApplication.id &&
flatFieldMetadata.applicationId !== workspaceCustomFlatApplication.id,
);
if (installedAppTsVectorFlatFieldMetadatas.length === 0) {
this.logger.log(
`No installed-app searchVector to rebuild for workspace ${workspaceId}`,
);
return;
}
this.logger.log(
`${isDryRun ? '[DRY RUN] ' : ''}Rebuilding ${installedAppTsVectorFlatFieldMetadatas.length} installed-app searchVector(s) for workspace ${workspaceId}`,
);
if (isDryRun) {
return;
}
const actions: UniversalUpdateFieldAction[] =
installedAppTsVectorFlatFieldMetadatas.map((flatFieldMetadata) => ({
type: WORKSPACE_MIGRATION_ACTION_TYPE.update,
metadataName: 'fieldMetadata',
universalIdentifier: flatFieldMetadata.universalIdentifier,
update: {},
rebuildSearchVector: true,
}));
await this.workspaceMigrationRunnerService.run({
workspaceMigration: {
// Cross-app actions; this is only the runner's existence gate, not a scope filter.
applicationUniversalIdentifier:
TWENTY_STANDARD_APPLICATION.universalIdentifier,
actions,
},
workspaceId,
});
this.logger.log(
`Rebuilt ${installedAppTsVectorFlatFieldMetadatas.length} installed-app searchVector(s) for workspace ${workspaceId}`,
);
}
}
@@ -0,0 +1,2 @@
export const ADD_IS_SYSTEM_SIDE_EFFECT_TO_SEARCH_FIELD_METADATA_UPGRADE_COMMAND_NAME =
'2.20.0_AddIsSystemSideEffectToSearchFieldMetadataFastInstanceCommand_1783580127637';
@@ -0,0 +1,424 @@
import {
getFieldUniversalIdentifier,
getSearchFieldUniversalIdentifier,
} from 'twenty-shared/application';
import { FieldMetadataType } from 'twenty-shared/types';
import { buildSearchFieldMetadataBackfillOperations } from 'src/database/commands/upgrade-version-command/2-20/utils/build-search-field-metadata-backfill-operations.util';
import { type FlatEntityMaps } from 'src/engine/metadata-modules/flat-entity/types/flat-entity-maps.type';
import { getFlatFieldMetadataMock } from 'src/engine/metadata-modules/flat-field-metadata/__mocks__/get-flat-field-metadata.mock';
import { type FlatFieldMetadata } from 'src/engine/metadata-modules/flat-field-metadata/types/flat-field-metadata.type';
import { getFlatObjectMetadataMock } from 'src/engine/metadata-modules/flat-object-metadata/__mocks__/get-flat-object-metadata.mock';
import { type FlatObjectMetadata } from 'src/engine/metadata-modules/flat-object-metadata/types/flat-object-metadata.type';
import { type FlatSearchFieldMetadata } from 'src/engine/metadata-modules/flat-search-field-metadata/types/flat-search-field-metadata.type';
import { SEARCH_VECTOR_FIELD } from 'src/engine/metadata-modules/search-field-metadata/constants/search-vector-field.constants';
// Application universal identifiers must be valid UUIDs: they are used as the v5
// namespace by the deterministic identifier helpers.
const STANDARD_APPLICATION_ID = 'standard-application-id';
const STANDARD_APPLICATION_UID = '11111111-1111-4111-8111-111111111111';
const CUSTOM_APPLICATION_ID = 'custom-application-id';
const CUSTOM_APPLICATION_UID = '22222222-2222-4222-8222-222222222222';
const INSTALLED_APPLICATION_ID = 'installed-application-id';
const INSTALLED_APPLICATION_UID = '33333333-3333-4333-8333-333333333333';
const applicationUniversalIdentifierById = new Map([
[STANDARD_APPLICATION_ID, STANDARD_APPLICATION_UID],
[CUSTOM_APPLICATION_ID, CUSTOM_APPLICATION_UID],
[INSTALLED_APPLICATION_ID, INSTALLED_APPLICATION_UID],
]);
const buildFlatObjectMetadataMaps = (
flatObjectMetadatas: FlatObjectMetadata[],
): FlatEntityMaps<FlatObjectMetadata> => ({
byUniversalIdentifier: Object.fromEntries(
flatObjectMetadatas.map((flatObjectMetadata) => [
flatObjectMetadata.universalIdentifier,
flatObjectMetadata,
]),
),
universalIdentifierById: Object.fromEntries(
flatObjectMetadatas.map((flatObjectMetadata) => [
flatObjectMetadata.id,
flatObjectMetadata.universalIdentifier,
]),
),
universalIdentifiersByApplicationId: {},
});
const buildFlatFieldMetadataMaps = (
flatFieldMetadatas: FlatFieldMetadata[],
): FlatEntityMaps<FlatFieldMetadata> => ({
byUniversalIdentifier: Object.fromEntries(
flatFieldMetadatas.map((flatFieldMetadata) => [
flatFieldMetadata.universalIdentifier,
flatFieldMetadata,
]),
),
universalIdentifierById: Object.fromEntries(
flatFieldMetadatas.map((flatFieldMetadata) => [
flatFieldMetadata.id,
flatFieldMetadata.universalIdentifier,
]),
),
universalIdentifiersByApplicationId: {},
});
const buildFlatSearchFieldMetadataMaps = (
flatSearchFieldMetadatas: FlatSearchFieldMetadata[],
): FlatEntityMaps<FlatSearchFieldMetadata> => ({
byUniversalIdentifier: Object.fromEntries(
flatSearchFieldMetadatas.map((flatSearchFieldMetadata) => [
flatSearchFieldMetadata.universalIdentifier,
flatSearchFieldMetadata,
]),
),
universalIdentifierById: Object.fromEntries(
flatSearchFieldMetadatas.map((flatSearchFieldMetadata) => [
flatSearchFieldMetadata.id,
flatSearchFieldMetadata.universalIdentifier,
]),
),
universalIdentifiersByApplicationId: {},
});
const buildFlatSearchFieldMetadata = ({
id,
universalIdentifier,
objectMetadataId,
fieldMetadataId,
applicationId,
applicationUniversalIdentifier,
}: {
id: string;
universalIdentifier: string;
objectMetadataId: string;
fieldMetadataId: string;
applicationId: string;
applicationUniversalIdentifier: string;
}): FlatSearchFieldMetadata => {
const createdAt = '2024-01-01T00:00:00.000Z';
return {
id,
universalIdentifier,
objectMetadataId,
fieldMetadataId,
objectMetadataUniversalIdentifier: `${objectMetadataId}-uid`,
fieldMetadataUniversalIdentifier: `${fieldMetadataId}-uid`,
tsVectorFieldMetadataId: `${objectMetadataId}-search-vector-field-id`,
tsVectorFieldMetadataUniversalIdentifier: `${objectMetadataId}-search-vector-field-uid`,
applicationId,
applicationUniversalIdentifier,
position: 0,
isSystemSideEffect: true,
workspaceId: 'workspace-id',
createdAt,
updatedAt: createdAt,
};
};
const buildSearchableObjectFixture = ({
objectId,
objectUniversalIdentifier,
applicationId,
applicationUniversalIdentifier,
labelIdentifierFieldType = FieldMetadataType.TEXT,
}: {
objectId: string;
objectUniversalIdentifier: string;
applicationId: string;
applicationUniversalIdentifier: string;
labelIdentifierFieldType?: FieldMetadataType;
}) => {
const nameFlatFieldMetadata = getFlatFieldMetadataMock({
id: `${objectId}-name-field-id`,
universalIdentifier: `${objectUniversalIdentifier}-name-field-uid`,
objectMetadataId: objectId,
objectMetadataUniversalIdentifier: objectUniversalIdentifier,
type: labelIdentifierFieldType,
name: 'name',
applicationId,
applicationUniversalIdentifier,
});
const searchVectorFlatFieldMetadata = getFlatFieldMetadataMock({
id: `${objectId}-search-vector-field-id`,
universalIdentifier: `${objectUniversalIdentifier}-search-vector-field-uid`,
objectMetadataId: objectId,
objectMetadataUniversalIdentifier: objectUniversalIdentifier,
type: FieldMetadataType.TS_VECTOR,
name: SEARCH_VECTOR_FIELD.name,
applicationId,
applicationUniversalIdentifier,
});
const flatObjectMetadata = getFlatObjectMetadataMock({
id: objectId,
universalIdentifier: objectUniversalIdentifier,
isSearchable: true,
applicationId,
applicationUniversalIdentifier,
labelIdentifierFieldMetadataId: nameFlatFieldMetadata.id,
labelIdentifierFieldMetadataUniversalIdentifier:
nameFlatFieldMetadata.universalIdentifier,
fieldUniversalIdentifiers: [
nameFlatFieldMetadata.universalIdentifier,
searchVectorFlatFieldMetadata.universalIdentifier,
],
});
return {
flatObjectMetadata,
nameFlatFieldMetadata,
searchVectorFlatFieldMetadata,
};
};
describe('buildSearchFieldMetadataBackfillOperations', () => {
it('creates the searchFieldMetadata row for an installed-app searchable object that has none, grouped under its application', () => {
const {
flatObjectMetadata,
nameFlatFieldMetadata,
searchVectorFlatFieldMetadata,
} = buildSearchableObjectFixture({
objectId: 'installed-object-id',
objectUniversalIdentifier: 'installed-object-uid',
applicationId: INSTALLED_APPLICATION_ID,
applicationUniversalIdentifier: INSTALLED_APPLICATION_UID,
});
const flatSearchFieldMetadatasToCreateByApplicationUniversalIdentifier =
buildSearchFieldMetadataBackfillOperations({
flatObjectMetadataMaps: buildFlatObjectMetadataMaps([
flatObjectMetadata,
]),
flatFieldMetadataMaps: buildFlatFieldMetadataMaps([
nameFlatFieldMetadata,
searchVectorFlatFieldMetadata,
]),
flatSearchFieldMetadataMaps: buildFlatSearchFieldMetadataMaps([]),
applicationUniversalIdentifierById,
twentyStandardApplicationId: STANDARD_APPLICATION_ID,
workspaceCustomApplicationId: CUSTOM_APPLICATION_ID,
});
const createdRows =
flatSearchFieldMetadatasToCreateByApplicationUniversalIdentifier[
INSTALLED_APPLICATION_UID
];
expect(createdRows).toHaveLength(1);
expect(createdRows?.[0].fieldMetadataUniversalIdentifier).toBe(
nameFlatFieldMetadata.universalIdentifier,
);
expect(createdRows?.[0].objectMetadataUniversalIdentifier).toBe(
flatObjectMetadata.universalIdentifier,
);
expect(createdRows?.[0].tsVectorFieldMetadataUniversalIdentifier).toBe(
searchVectorFlatFieldMetadata.universalIdentifier,
);
expect(createdRows?.[0].universalIdentifier).toBe(
getSearchFieldUniversalIdentifier({
applicationUniversalIdentifier: INSTALLED_APPLICATION_UID,
fieldMetadataUniversalIdentifier:
nameFlatFieldMetadata.universalIdentifier,
}),
);
});
it('does not create a row for an installed-app junction object whose label identifier is the UUID id field', () => {
const {
flatObjectMetadata,
nameFlatFieldMetadata,
searchVectorFlatFieldMetadata,
} = buildSearchableObjectFixture({
objectId: 'installed-junction-object-id',
objectUniversalIdentifier: 'installed-junction-object-uid',
applicationId: INSTALLED_APPLICATION_ID,
applicationUniversalIdentifier: INSTALLED_APPLICATION_UID,
});
// The label identifier resolves to the derived `id` field: no search surface.
const junctionObject: FlatObjectMetadata = {
...flatObjectMetadata,
labelIdentifierFieldMetadataUniversalIdentifier:
getFieldUniversalIdentifier({
applicationUniversalIdentifier: INSTALLED_APPLICATION_UID,
objectUniversalIdentifier: flatObjectMetadata.universalIdentifier,
name: 'id',
}),
};
const flatSearchFieldMetadatasToCreateByApplicationUniversalIdentifier =
buildSearchFieldMetadataBackfillOperations({
flatObjectMetadataMaps: buildFlatObjectMetadataMaps([junctionObject]),
flatFieldMetadataMaps: buildFlatFieldMetadataMaps([
nameFlatFieldMetadata,
searchVectorFlatFieldMetadata,
]),
flatSearchFieldMetadataMaps: buildFlatSearchFieldMetadataMaps([]),
applicationUniversalIdentifierById,
twentyStandardApplicationId: STANDARD_APPLICATION_ID,
workspaceCustomApplicationId: CUSTOM_APPLICATION_ID,
});
expect(
Object.keys(
flatSearchFieldMetadatasToCreateByApplicationUniversalIdentifier,
),
).toHaveLength(0);
});
it('does not create a row for an installed-app object whose label identifier is a non-searchable type', () => {
const {
flatObjectMetadata,
nameFlatFieldMetadata,
searchVectorFlatFieldMetadata,
} = buildSearchableObjectFixture({
objectId: 'installed-relation-object-id',
objectUniversalIdentifier: 'installed-relation-object-uid',
applicationId: INSTALLED_APPLICATION_ID,
applicationUniversalIdentifier: INSTALLED_APPLICATION_UID,
labelIdentifierFieldType: FieldMetadataType.RELATION,
});
const flatSearchFieldMetadatasToCreateByApplicationUniversalIdentifier =
buildSearchFieldMetadataBackfillOperations({
flatObjectMetadataMaps: buildFlatObjectMetadataMaps([
flatObjectMetadata,
]),
flatFieldMetadataMaps: buildFlatFieldMetadataMaps([
nameFlatFieldMetadata,
searchVectorFlatFieldMetadata,
]),
flatSearchFieldMetadataMaps: buildFlatSearchFieldMetadataMaps([]),
applicationUniversalIdentifierById,
twentyStandardApplicationId: STANDARD_APPLICATION_ID,
workspaceCustomApplicationId: CUSTOM_APPLICATION_ID,
});
expect(
Object.keys(
flatSearchFieldMetadatasToCreateByApplicationUniversalIdentifier,
),
).toHaveLength(0);
});
it('does not create a row for a twenty-standard searchable object that has none', () => {
const {
flatObjectMetadata,
nameFlatFieldMetadata,
searchVectorFlatFieldMetadata,
} = buildSearchableObjectFixture({
objectId: 'standard-object-id',
objectUniversalIdentifier: 'standard-object-uid',
applicationId: STANDARD_APPLICATION_ID,
applicationUniversalIdentifier: STANDARD_APPLICATION_UID,
});
const flatSearchFieldMetadatasToCreateByApplicationUniversalIdentifier =
buildSearchFieldMetadataBackfillOperations({
flatObjectMetadataMaps: buildFlatObjectMetadataMaps([
flatObjectMetadata,
]),
flatFieldMetadataMaps: buildFlatFieldMetadataMaps([
nameFlatFieldMetadata,
searchVectorFlatFieldMetadata,
]),
flatSearchFieldMetadataMaps: buildFlatSearchFieldMetadataMaps([]),
applicationUniversalIdentifierById,
twentyStandardApplicationId: STANDARD_APPLICATION_ID,
workspaceCustomApplicationId: CUSTOM_APPLICATION_ID,
});
expect(
Object.keys(
flatSearchFieldMetadatasToCreateByApplicationUniversalIdentifier,
),
).toHaveLength(0);
});
it('does not create a row for a workspace-custom searchable object that has none', () => {
const {
flatObjectMetadata,
nameFlatFieldMetadata,
searchVectorFlatFieldMetadata,
} = buildSearchableObjectFixture({
objectId: 'custom-object-id',
objectUniversalIdentifier: 'custom-object-uid',
applicationId: CUSTOM_APPLICATION_ID,
applicationUniversalIdentifier: CUSTOM_APPLICATION_UID,
});
const flatSearchFieldMetadatasToCreateByApplicationUniversalIdentifier =
buildSearchFieldMetadataBackfillOperations({
flatObjectMetadataMaps: buildFlatObjectMetadataMaps([
flatObjectMetadata,
]),
flatFieldMetadataMaps: buildFlatFieldMetadataMaps([
nameFlatFieldMetadata,
searchVectorFlatFieldMetadata,
]),
flatSearchFieldMetadataMaps: buildFlatSearchFieldMetadataMaps([]),
applicationUniversalIdentifierById,
twentyStandardApplicationId: STANDARD_APPLICATION_ID,
workspaceCustomApplicationId: CUSTOM_APPLICATION_ID,
});
expect(
Object.keys(
flatSearchFieldMetadatasToCreateByApplicationUniversalIdentifier,
),
).toHaveLength(0);
});
it('is idempotent: no create when the installed-app searchFieldMetadata row already exists', () => {
const {
flatObjectMetadata,
nameFlatFieldMetadata,
searchVectorFlatFieldMetadata,
} = buildSearchableObjectFixture({
objectId: 'installed-object-id',
objectUniversalIdentifier: 'installed-object-uid',
applicationId: INSTALLED_APPLICATION_ID,
applicationUniversalIdentifier: INSTALLED_APPLICATION_UID,
});
const existingSearchFieldMetadata = buildFlatSearchFieldMetadata({
id: 'existing-search-field-metadata-id',
universalIdentifier: getSearchFieldUniversalIdentifier({
applicationUniversalIdentifier: INSTALLED_APPLICATION_UID,
fieldMetadataUniversalIdentifier:
nameFlatFieldMetadata.universalIdentifier,
}),
objectMetadataId: flatObjectMetadata.id,
fieldMetadataId: nameFlatFieldMetadata.id,
applicationId: INSTALLED_APPLICATION_ID,
applicationUniversalIdentifier: INSTALLED_APPLICATION_UID,
});
const flatSearchFieldMetadatasToCreateByApplicationUniversalIdentifier =
buildSearchFieldMetadataBackfillOperations({
flatObjectMetadataMaps: buildFlatObjectMetadataMaps([
flatObjectMetadata,
]),
flatFieldMetadataMaps: buildFlatFieldMetadataMaps([
nameFlatFieldMetadata,
searchVectorFlatFieldMetadata,
]),
flatSearchFieldMetadataMaps: buildFlatSearchFieldMetadataMaps([
existingSearchFieldMetadata,
]),
applicationUniversalIdentifierById,
twentyStandardApplicationId: STANDARD_APPLICATION_ID,
workspaceCustomApplicationId: CUSTOM_APPLICATION_ID,
});
expect(
Object.keys(
flatSearchFieldMetadatasToCreateByApplicationUniversalIdentifier,
),
).toHaveLength(0);
});
});
@@ -0,0 +1,155 @@
import { getSearchFieldUniversalIdentifier } from 'twenty-shared/application';
import { FieldMetadataType } from 'twenty-shared/types';
import { buildSearchFieldMetadataReOwnOperations } from 'src/database/commands/upgrade-version-command/2-20/utils/build-search-field-metadata-re-own-operations.util';
import { type FlatEntityMaps } from 'src/engine/metadata-modules/flat-entity/types/flat-entity-maps.type';
import { getFlatFieldMetadataMock } from 'src/engine/metadata-modules/flat-field-metadata/__mocks__/get-flat-field-metadata.mock';
import { type FlatFieldMetadata } from 'src/engine/metadata-modules/flat-field-metadata/types/flat-field-metadata.type';
import { type FlatSearchFieldMetadata } from 'src/engine/metadata-modules/flat-search-field-metadata/types/flat-search-field-metadata.type';
const STANDARD_APPLICATION_ID = 'standard-application-id';
// Application universal identifiers must be valid UUIDs: they are used as the v5
// namespace by the deterministic identifier helpers.
const STANDARD_APPLICATION_UID = '11111111-1111-4111-8111-111111111111';
const applicationUniversalIdentifierById = new Map([
[STANDARD_APPLICATION_ID, STANDARD_APPLICATION_UID],
]);
const buildFlatFieldMetadataMaps = (
flatFieldMetadatas: FlatFieldMetadata[],
): FlatEntityMaps<FlatFieldMetadata> => ({
byUniversalIdentifier: Object.fromEntries(
flatFieldMetadatas.map((flatFieldMetadata) => [
flatFieldMetadata.universalIdentifier,
flatFieldMetadata,
]),
),
universalIdentifierById: Object.fromEntries(
flatFieldMetadatas.map((flatFieldMetadata) => [
flatFieldMetadata.id,
flatFieldMetadata.universalIdentifier,
]),
),
universalIdentifiersByApplicationId: {},
});
const buildFlatSearchFieldMetadataMaps = (
flatSearchFieldMetadatas: FlatSearchFieldMetadata[],
): FlatEntityMaps<FlatSearchFieldMetadata> => ({
byUniversalIdentifier: Object.fromEntries(
flatSearchFieldMetadatas.map((flatSearchFieldMetadata) => [
flatSearchFieldMetadata.universalIdentifier,
flatSearchFieldMetadata,
]),
),
universalIdentifierById: Object.fromEntries(
flatSearchFieldMetadatas.map((flatSearchFieldMetadata) => [
flatSearchFieldMetadata.id,
flatSearchFieldMetadata.universalIdentifier,
]),
),
universalIdentifiersByApplicationId: {},
});
const buildFlatSearchFieldMetadata = ({
id,
universalIdentifier,
objectMetadataId,
fieldMetadataId,
}: {
id: string;
universalIdentifier: string;
objectMetadataId: string;
fieldMetadataId: string;
}): FlatSearchFieldMetadata => {
const createdAt = '2024-01-01T00:00:00.000Z';
return {
id,
universalIdentifier,
objectMetadataId,
fieldMetadataId,
objectMetadataUniversalIdentifier: `${objectMetadataId}-uid`,
fieldMetadataUniversalIdentifier: `${fieldMetadataId}-uid`,
tsVectorFieldMetadataId: `${objectMetadataId}-search-vector-field-id`,
tsVectorFieldMetadataUniversalIdentifier: `${objectMetadataId}-search-vector-field-uid`,
applicationId: STANDARD_APPLICATION_ID,
applicationUniversalIdentifier: STANDARD_APPLICATION_UID,
position: 0,
isSystemSideEffect: true,
workspaceId: 'workspace-id',
createdAt,
updatedAt: createdAt,
};
};
const nameFlatFieldMetadata = getFlatFieldMetadataMock({
id: 'standard-object-name-field-id',
universalIdentifier: 'standard-object-name-field-uid',
objectMetadataId: 'standard-object-id',
objectMetadataUniversalIdentifier: 'standard-object-uid',
type: FieldMetadataType.TEXT,
name: 'name',
applicationId: STANDARD_APPLICATION_ID,
applicationUniversalIdentifier: STANDARD_APPLICATION_UID,
});
describe('buildSearchFieldMetadataReOwnOperations', () => {
it('re-owns an existing searchFieldMetadata carrying a non-deterministic universal identifier (all applications)', () => {
const existingSearchFieldMetadata = buildFlatSearchFieldMetadata({
id: 'existing-search-field-metadata-id',
universalIdentifier: 'legacy-v4-search-field-metadata-uid',
objectMetadataId: 'standard-object-id',
fieldMetadataId: nameFlatFieldMetadata.id,
});
const searchFieldMetadataUniversalIdentifierUpdates =
buildSearchFieldMetadataReOwnOperations({
flatFieldMetadataMaps: buildFlatFieldMetadataMaps([
nameFlatFieldMetadata,
]),
flatSearchFieldMetadataMaps: buildFlatSearchFieldMetadataMaps([
existingSearchFieldMetadata,
]),
applicationUniversalIdentifierById,
});
expect(searchFieldMetadataUniversalIdentifierUpdates).toEqual([
{
id: existingSearchFieldMetadata.id,
deterministicUniversalIdentifier: getSearchFieldUniversalIdentifier({
applicationUniversalIdentifier: STANDARD_APPLICATION_UID,
fieldMetadataUniversalIdentifier:
nameFlatFieldMetadata.universalIdentifier,
}),
},
]);
});
it('is a no-op for a searchFieldMetadata already carrying its deterministic universal identifier', () => {
const existingSearchFieldMetadata = buildFlatSearchFieldMetadata({
id: 'existing-search-field-metadata-id',
universalIdentifier: getSearchFieldUniversalIdentifier({
applicationUniversalIdentifier: STANDARD_APPLICATION_UID,
fieldMetadataUniversalIdentifier:
nameFlatFieldMetadata.universalIdentifier,
}),
objectMetadataId: 'standard-object-id',
fieldMetadataId: nameFlatFieldMetadata.id,
});
const searchFieldMetadataUniversalIdentifierUpdates =
buildSearchFieldMetadataReOwnOperations({
flatFieldMetadataMaps: buildFlatFieldMetadataMaps([
nameFlatFieldMetadata,
]),
flatSearchFieldMetadataMaps: buildFlatSearchFieldMetadataMaps([
existingSearchFieldMetadata,
]),
applicationUniversalIdentifierById,
});
expect(searchFieldMetadataUniversalIdentifierUpdates).toHaveLength(0);
});
});
@@ -0,0 +1,267 @@
import { FieldMetadataType } from 'twenty-shared/types';
import { buildSearchVectorGinIndexBackfillOperations } from 'src/database/commands/upgrade-version-command/2-20/utils/build-search-vector-gin-index-backfill-operations.util';
import { type FlatEntityMaps } from 'src/engine/metadata-modules/flat-entity/types/flat-entity-maps.type';
import { getFlatFieldMetadataMock } from 'src/engine/metadata-modules/flat-field-metadata/__mocks__/get-flat-field-metadata.mock';
import { type FlatFieldMetadata } from 'src/engine/metadata-modules/flat-field-metadata/types/flat-field-metadata.type';
import { getFlatIndexMetadataMock } from 'src/engine/metadata-modules/flat-index-metadata/__mocks__/get-flat-index-metadata.mock';
import {
type FlatIndexFieldMetadata,
type FlatIndexMetadata,
} from 'src/engine/metadata-modules/flat-index-metadata/types/flat-index-metadata.type';
import { getFlatObjectMetadataMock } from 'src/engine/metadata-modules/flat-object-metadata/__mocks__/get-flat-object-metadata.mock';
import { type FlatObjectMetadata } from 'src/engine/metadata-modules/flat-object-metadata/types/flat-object-metadata.type';
import { IndexType } from 'src/engine/metadata-modules/index-metadata/types/indexType.types';
import { SEARCH_VECTOR_FIELD } from 'src/engine/metadata-modules/search-field-metadata/constants/search-vector-field.constants';
// Application universal identifiers must be valid UUIDs: they are used as the v5
// namespace by the deterministic identifier helpers.
const STANDARD_APPLICATION_ID = 'standard-application-id';
const CUSTOM_APPLICATION_ID = 'custom-application-id';
const INSTALLED_APPLICATION_ID = 'installed-application-id';
const INSTALLED_APPLICATION_UID = '33333333-3333-4333-8333-333333333333';
const buildFlatObjectMetadataMaps = (
flatObjectMetadatas: FlatObjectMetadata[],
): FlatEntityMaps<FlatObjectMetadata> => ({
byUniversalIdentifier: Object.fromEntries(
flatObjectMetadatas.map((flatObjectMetadata) => [
flatObjectMetadata.universalIdentifier,
flatObjectMetadata,
]),
),
universalIdentifierById: Object.fromEntries(
flatObjectMetadatas.map((flatObjectMetadata) => [
flatObjectMetadata.id,
flatObjectMetadata.universalIdentifier,
]),
),
universalIdentifiersByApplicationId: {},
});
const buildFlatFieldMetadataMaps = (
flatFieldMetadatas: FlatFieldMetadata[],
): FlatEntityMaps<FlatFieldMetadata> => ({
byUniversalIdentifier: Object.fromEntries(
flatFieldMetadatas.map((flatFieldMetadata) => [
flatFieldMetadata.universalIdentifier,
flatFieldMetadata,
]),
),
universalIdentifierById: Object.fromEntries(
flatFieldMetadatas.map((flatFieldMetadata) => [
flatFieldMetadata.id,
flatFieldMetadata.universalIdentifier,
]),
),
universalIdentifiersByApplicationId: {},
});
const buildFlatIndexMaps = (
flatIndexMetadatas: FlatIndexMetadata[],
): FlatEntityMaps<FlatIndexMetadata> => ({
byUniversalIdentifier: Object.fromEntries(
flatIndexMetadatas.map((flatIndexMetadata) => [
flatIndexMetadata.universalIdentifier,
flatIndexMetadata,
]),
),
universalIdentifierById: Object.fromEntries(
flatIndexMetadatas.map((flatIndexMetadata) => [
flatIndexMetadata.id,
flatIndexMetadata.universalIdentifier,
]),
),
universalIdentifiersByApplicationId: {},
});
const buildFlatIndexFieldMetadata = ({
fieldMetadataId,
}: {
fieldMetadataId: string;
}): FlatIndexFieldMetadata => {
const createdAt = '2024-01-01T00:00:00.000Z';
return {
id: `${fieldMetadataId}-index-field-id`,
workspaceId: 'workspace-id',
indexMetadataId: 'index-metadata-id',
fieldMetadataId,
order: 0,
subFieldName: null,
createdAt,
updatedAt: createdAt,
};
};
const buildSearchableObjectFixture = ({
objectId,
objectUniversalIdentifier,
applicationId,
applicationUniversalIdentifier,
}: {
objectId: string;
objectUniversalIdentifier: string;
applicationId: string;
applicationUniversalIdentifier: string;
}) => {
const searchVectorFlatFieldMetadata = getFlatFieldMetadataMock({
id: `${objectId}-search-vector-field-id`,
universalIdentifier: `${objectUniversalIdentifier}-search-vector-field-uid`,
objectMetadataId: objectId,
objectMetadataUniversalIdentifier: objectUniversalIdentifier,
type: FieldMetadataType.TS_VECTOR,
name: SEARCH_VECTOR_FIELD.name,
applicationId,
applicationUniversalIdentifier,
});
const flatObjectMetadata = getFlatObjectMetadataMock({
id: objectId,
universalIdentifier: objectUniversalIdentifier,
isSearchable: true,
applicationId,
applicationUniversalIdentifier,
fieldUniversalIdentifiers: [
searchVectorFlatFieldMetadata.universalIdentifier,
],
});
return { flatObjectMetadata, searchVectorFlatFieldMetadata };
};
describe('buildSearchVectorGinIndexBackfillOperations', () => {
it('creates a GIN searchVector index for an installed-app object that has none, grouped under its application', () => {
const { flatObjectMetadata, searchVectorFlatFieldMetadata } =
buildSearchableObjectFixture({
objectId: 'installed-object-id',
objectUniversalIdentifier: 'installed-object-uid',
applicationId: INSTALLED_APPLICATION_ID,
applicationUniversalIdentifier: INSTALLED_APPLICATION_UID,
});
const flatIndexesToCreateByApplicationUniversalIdentifier =
buildSearchVectorGinIndexBackfillOperations({
flatObjectMetadataMaps: buildFlatObjectMetadataMaps([
flatObjectMetadata,
]),
flatFieldMetadataMaps: buildFlatFieldMetadataMaps([
searchVectorFlatFieldMetadata,
]),
flatIndexMaps: buildFlatIndexMaps([]),
twentyStandardApplicationId: STANDARD_APPLICATION_ID,
workspaceCustomApplicationId: CUSTOM_APPLICATION_ID,
});
const createdIndexes =
flatIndexesToCreateByApplicationUniversalIdentifier[
INSTALLED_APPLICATION_UID
];
expect(createdIndexes).toHaveLength(1);
expect(createdIndexes?.[0].indexType).toBe(IndexType.GIN);
expect(createdIndexes?.[0].objectMetadataUniversalIdentifier).toBe(
flatObjectMetadata.universalIdentifier,
);
expect(createdIndexes?.[0].applicationUniversalIdentifier).toBe(
INSTALLED_APPLICATION_UID,
);
});
it('does not create an index for a twenty-standard object that has none', () => {
const { flatObjectMetadata, searchVectorFlatFieldMetadata } =
buildSearchableObjectFixture({
objectId: 'standard-object-id',
objectUniversalIdentifier: 'standard-object-uid',
applicationId: STANDARD_APPLICATION_ID,
applicationUniversalIdentifier: '11111111-1111-4111-8111-111111111111',
});
const flatIndexesToCreateByApplicationUniversalIdentifier =
buildSearchVectorGinIndexBackfillOperations({
flatObjectMetadataMaps: buildFlatObjectMetadataMaps([
flatObjectMetadata,
]),
flatFieldMetadataMaps: buildFlatFieldMetadataMaps([
searchVectorFlatFieldMetadata,
]),
flatIndexMaps: buildFlatIndexMaps([]),
twentyStandardApplicationId: STANDARD_APPLICATION_ID,
workspaceCustomApplicationId: CUSTOM_APPLICATION_ID,
});
expect(
Object.keys(flatIndexesToCreateByApplicationUniversalIdentifier),
).toHaveLength(0);
});
it('does not create an index for a workspace-custom object that has none', () => {
const { flatObjectMetadata, searchVectorFlatFieldMetadata } =
buildSearchableObjectFixture({
objectId: 'custom-object-id',
objectUniversalIdentifier: 'custom-object-uid',
applicationId: CUSTOM_APPLICATION_ID,
applicationUniversalIdentifier: '22222222-2222-4222-8222-222222222222',
});
const flatIndexesToCreateByApplicationUniversalIdentifier =
buildSearchVectorGinIndexBackfillOperations({
flatObjectMetadataMaps: buildFlatObjectMetadataMaps([
flatObjectMetadata,
]),
flatFieldMetadataMaps: buildFlatFieldMetadataMaps([
searchVectorFlatFieldMetadata,
]),
flatIndexMaps: buildFlatIndexMaps([]),
twentyStandardApplicationId: STANDARD_APPLICATION_ID,
workspaceCustomApplicationId: CUSTOM_APPLICATION_ID,
});
expect(
Object.keys(flatIndexesToCreateByApplicationUniversalIdentifier),
).toHaveLength(0);
});
it('is idempotent: no create when the installed-app object already has a GIN searchVector index', () => {
const { flatObjectMetadata, searchVectorFlatFieldMetadata } =
buildSearchableObjectFixture({
objectId: 'installed-object-id',
objectUniversalIdentifier: 'installed-object-uid',
applicationId: INSTALLED_APPLICATION_ID,
applicationUniversalIdentifier: INSTALLED_APPLICATION_UID,
});
const ginIndex = getFlatIndexMetadataMock({
id: 'installed-gin-index-id',
universalIdentifier: 'installed-gin-index-uid',
objectMetadataId: flatObjectMetadata.id,
objectMetadataUniversalIdentifier: flatObjectMetadata.universalIdentifier,
applicationId: INSTALLED_APPLICATION_ID,
applicationUniversalIdentifier: INSTALLED_APPLICATION_UID,
name: 'IDX_SEARCH_VECTOR_INSTALLED',
indexType: IndexType.GIN,
flatIndexFieldMetadatas: [
buildFlatIndexFieldMetadata({
fieldMetadataId: searchVectorFlatFieldMetadata.id,
}),
],
});
const flatIndexesToCreateByApplicationUniversalIdentifier =
buildSearchVectorGinIndexBackfillOperations({
flatObjectMetadataMaps: buildFlatObjectMetadataMaps([
flatObjectMetadata,
]),
flatFieldMetadataMaps: buildFlatFieldMetadataMaps([
searchVectorFlatFieldMetadata,
]),
flatIndexMaps: buildFlatIndexMaps([ginIndex]),
twentyStandardApplicationId: STANDARD_APPLICATION_ID,
workspaceCustomApplicationId: CUSTOM_APPLICATION_ID,
});
expect(
Object.keys(flatIndexesToCreateByApplicationUniversalIdentifier),
).toHaveLength(0);
});
});
@@ -0,0 +1,224 @@
import { getIndexUniversalIdentifier } from 'twenty-shared/application';
import { FieldMetadataType } from 'twenty-shared/types';
import { buildSearchVectorGinIndexReOwnOperations } from 'src/database/commands/upgrade-version-command/2-20/utils/build-search-vector-gin-index-re-own-operations.util';
import { type FlatEntityMaps } from 'src/engine/metadata-modules/flat-entity/types/flat-entity-maps.type';
import { getFlatFieldMetadataMock } from 'src/engine/metadata-modules/flat-field-metadata/__mocks__/get-flat-field-metadata.mock';
import { type FlatFieldMetadata } from 'src/engine/metadata-modules/flat-field-metadata/types/flat-field-metadata.type';
import { getFlatIndexMetadataMock } from 'src/engine/metadata-modules/flat-index-metadata/__mocks__/get-flat-index-metadata.mock';
import {
type FlatIndexFieldMetadata,
type FlatIndexMetadata,
} from 'src/engine/metadata-modules/flat-index-metadata/types/flat-index-metadata.type';
import { getFlatObjectMetadataMock } from 'src/engine/metadata-modules/flat-object-metadata/__mocks__/get-flat-object-metadata.mock';
import { type FlatObjectMetadata } from 'src/engine/metadata-modules/flat-object-metadata/types/flat-object-metadata.type';
import { IndexType } from 'src/engine/metadata-modules/index-metadata/types/indexType.types';
import { SEARCH_VECTOR_FIELD } from 'src/engine/metadata-modules/search-field-metadata/constants/search-vector-field.constants';
const STANDARD_APPLICATION_ID = 'standard-application-id';
// Application universal identifiers must be valid UUIDs: they are used as the v5
// namespace by the deterministic identifier helpers.
const STANDARD_APPLICATION_UID = '11111111-1111-4111-8111-111111111111';
const applicationUniversalIdentifierById = new Map([
[STANDARD_APPLICATION_ID, STANDARD_APPLICATION_UID],
]);
const buildFlatObjectMetadataMaps = (
flatObjectMetadatas: FlatObjectMetadata[],
): FlatEntityMaps<FlatObjectMetadata> => ({
byUniversalIdentifier: Object.fromEntries(
flatObjectMetadatas.map((flatObjectMetadata) => [
flatObjectMetadata.universalIdentifier,
flatObjectMetadata,
]),
),
universalIdentifierById: Object.fromEntries(
flatObjectMetadatas.map((flatObjectMetadata) => [
flatObjectMetadata.id,
flatObjectMetadata.universalIdentifier,
]),
),
universalIdentifiersByApplicationId: {},
});
const buildFlatFieldMetadataMaps = (
flatFieldMetadatas: FlatFieldMetadata[],
): FlatEntityMaps<FlatFieldMetadata> => ({
byUniversalIdentifier: Object.fromEntries(
flatFieldMetadatas.map((flatFieldMetadata) => [
flatFieldMetadata.universalIdentifier,
flatFieldMetadata,
]),
),
universalIdentifierById: Object.fromEntries(
flatFieldMetadatas.map((flatFieldMetadata) => [
flatFieldMetadata.id,
flatFieldMetadata.universalIdentifier,
]),
),
universalIdentifiersByApplicationId: {},
});
const buildFlatIndexMaps = (
flatIndexMetadatas: FlatIndexMetadata[],
): FlatEntityMaps<FlatIndexMetadata> => ({
byUniversalIdentifier: Object.fromEntries(
flatIndexMetadatas.map((flatIndexMetadata) => [
flatIndexMetadata.universalIdentifier,
flatIndexMetadata,
]),
),
universalIdentifierById: Object.fromEntries(
flatIndexMetadatas.map((flatIndexMetadata) => [
flatIndexMetadata.id,
flatIndexMetadata.universalIdentifier,
]),
),
universalIdentifiersByApplicationId: {},
});
const buildFlatIndexFieldMetadata = ({
fieldMetadataId,
}: {
fieldMetadataId: string;
}): FlatIndexFieldMetadata => {
const createdAt = '2024-01-01T00:00:00.000Z';
return {
id: `${fieldMetadataId}-index-field-id`,
workspaceId: 'workspace-id',
indexMetadataId: 'index-metadata-id',
fieldMetadataId,
order: 0,
subFieldName: null,
createdAt,
updatedAt: createdAt,
};
};
const buildSearchableObjectFixture = ({
objectId,
objectUniversalIdentifier,
}: {
objectId: string;
objectUniversalIdentifier: string;
}) => {
const searchVectorFlatFieldMetadata = getFlatFieldMetadataMock({
id: `${objectId}-search-vector-field-id`,
universalIdentifier: `${objectUniversalIdentifier}-search-vector-field-uid`,
objectMetadataId: objectId,
objectMetadataUniversalIdentifier: objectUniversalIdentifier,
type: FieldMetadataType.TS_VECTOR,
name: SEARCH_VECTOR_FIELD.name,
applicationId: STANDARD_APPLICATION_ID,
applicationUniversalIdentifier: STANDARD_APPLICATION_UID,
});
const flatObjectMetadata = getFlatObjectMetadataMock({
id: objectId,
universalIdentifier: objectUniversalIdentifier,
isSearchable: true,
applicationId: STANDARD_APPLICATION_ID,
applicationUniversalIdentifier: STANDARD_APPLICATION_UID,
fieldUniversalIdentifiers: [
searchVectorFlatFieldMetadata.universalIdentifier,
],
});
return { flatObjectMetadata, searchVectorFlatFieldMetadata };
};
describe('buildSearchVectorGinIndexReOwnOperations', () => {
it('re-owns an existing GIN searchVector index carrying a non-deterministic universal identifier (all applications)', () => {
const { flatObjectMetadata, searchVectorFlatFieldMetadata } =
buildSearchableObjectFixture({
objectId: 'standard-object-id',
objectUniversalIdentifier: 'standard-object-uid',
});
const indexName = 'IDX_SEARCH_VECTOR_STANDARD';
const ginIndex = getFlatIndexMetadataMock({
id: 'standard-gin-index-id',
universalIdentifier: 'legacy-v4-index-uid',
objectMetadataId: flatObjectMetadata.id,
objectMetadataUniversalIdentifier: flatObjectMetadata.universalIdentifier,
applicationId: STANDARD_APPLICATION_ID,
applicationUniversalIdentifier: STANDARD_APPLICATION_UID,
name: indexName,
indexType: IndexType.GIN,
flatIndexFieldMetadatas: [
buildFlatIndexFieldMetadata({
fieldMetadataId: searchVectorFlatFieldMetadata.id,
}),
],
});
const indexUniversalIdentifierUpdates =
buildSearchVectorGinIndexReOwnOperations({
flatObjectMetadataMaps: buildFlatObjectMetadataMaps([
flatObjectMetadata,
]),
flatFieldMetadataMaps: buildFlatFieldMetadataMaps([
searchVectorFlatFieldMetadata,
]),
flatIndexMaps: buildFlatIndexMaps([ginIndex]),
applicationUniversalIdentifierById,
});
expect(indexUniversalIdentifierUpdates).toEqual([
{
id: ginIndex.id,
name: indexName,
deterministicUniversalIdentifier: getIndexUniversalIdentifier({
applicationUniversalIdentifier: STANDARD_APPLICATION_UID,
objectUniversalIdentifier: flatObjectMetadata.universalIdentifier,
name: indexName,
}),
},
]);
});
it('is a no-op for a GIN searchVector index already carrying its deterministic universal identifier', () => {
const { flatObjectMetadata, searchVectorFlatFieldMetadata } =
buildSearchableObjectFixture({
objectId: 'standard-object-id',
objectUniversalIdentifier: 'standard-object-uid',
});
const indexName = 'IDX_SEARCH_VECTOR_STANDARD';
const ginIndex = getFlatIndexMetadataMock({
id: 'standard-gin-index-id',
universalIdentifier: getIndexUniversalIdentifier({
applicationUniversalIdentifier: STANDARD_APPLICATION_UID,
objectUniversalIdentifier: flatObjectMetadata.universalIdentifier,
name: indexName,
}),
objectMetadataId: flatObjectMetadata.id,
objectMetadataUniversalIdentifier: flatObjectMetadata.universalIdentifier,
applicationId: STANDARD_APPLICATION_ID,
applicationUniversalIdentifier: STANDARD_APPLICATION_UID,
name: indexName,
indexType: IndexType.GIN,
flatIndexFieldMetadatas: [
buildFlatIndexFieldMetadata({
fieldMetadataId: searchVectorFlatFieldMetadata.id,
}),
],
});
const indexUniversalIdentifierUpdates =
buildSearchVectorGinIndexReOwnOperations({
flatObjectMetadataMaps: buildFlatObjectMetadataMaps([
flatObjectMetadata,
]),
flatFieldMetadataMaps: buildFlatFieldMetadataMaps([
searchVectorFlatFieldMetadata,
]),
flatIndexMaps: buildFlatIndexMaps([ginIndex]),
applicationUniversalIdentifierById,
});
expect(indexUniversalIdentifierUpdates).toHaveLength(0);
});
});
@@ -0,0 +1,186 @@
import { FieldMetadataType } from 'twenty-shared/types';
import { isSearchVectorGinFlatIndexMetadata } from 'src/database/commands/upgrade-version-command/2-20/utils/is-search-vector-gin-flat-index-metadata.util';
import { type FlatEntityMaps } from 'src/engine/metadata-modules/flat-entity/types/flat-entity-maps.type';
import { getFlatFieldMetadataMock } from 'src/engine/metadata-modules/flat-field-metadata/__mocks__/get-flat-field-metadata.mock';
import { type FlatFieldMetadata } from 'src/engine/metadata-modules/flat-field-metadata/types/flat-field-metadata.type';
import { getFlatIndexMetadataMock } from 'src/engine/metadata-modules/flat-index-metadata/__mocks__/get-flat-index-metadata.mock';
import { type FlatIndexFieldMetadata } from 'src/engine/metadata-modules/flat-index-metadata/types/flat-index-metadata.type';
import { IndexType } from 'src/engine/metadata-modules/index-metadata/types/indexType.types';
import { SEARCH_VECTOR_FIELD } from 'src/engine/metadata-modules/search-field-metadata/constants/search-vector-field.constants';
const buildFlatFieldMetadataMaps = (
flatFieldMetadatas: FlatFieldMetadata[],
): FlatEntityMaps<FlatFieldMetadata> => ({
byUniversalIdentifier: Object.fromEntries(
flatFieldMetadatas.map((flatFieldMetadata) => [
flatFieldMetadata.universalIdentifier,
flatFieldMetadata,
]),
),
universalIdentifierById: Object.fromEntries(
flatFieldMetadatas.map((flatFieldMetadata) => [
flatFieldMetadata.id,
flatFieldMetadata.universalIdentifier,
]),
),
universalIdentifiersByApplicationId: {},
});
const buildFlatIndexFieldMetadata = ({
fieldMetadataId,
}: {
fieldMetadataId: string;
}): FlatIndexFieldMetadata => {
const createdAt = '2024-01-01T00:00:00.000Z';
return {
id: `${fieldMetadataId}-index-field-id`,
workspaceId: 'workspace-id',
indexMetadataId: 'index-metadata-id',
fieldMetadataId,
order: 0,
subFieldName: null,
createdAt,
updatedAt: createdAt,
};
};
const SEARCH_VECTOR_FIELD_ID = 'search-vector-field-id';
const TEXT_FIELD_ID = 'text-field-id';
const OTHER_TS_VECTOR_FIELD_ID = 'other-ts-vector-field-id';
const searchVectorFlatFieldMetadata = getFlatFieldMetadataMock({
id: SEARCH_VECTOR_FIELD_ID,
universalIdentifier: 'search-vector-field-uid',
objectMetadataId: 'object-id',
type: FieldMetadataType.TS_VECTOR,
name: SEARCH_VECTOR_FIELD.name,
});
const textFlatFieldMetadata = getFlatFieldMetadataMock({
id: TEXT_FIELD_ID,
universalIdentifier: 'text-field-uid',
objectMetadataId: 'object-id',
type: FieldMetadataType.TEXT,
name: 'name',
});
const otherTsVectorFlatFieldMetadata = getFlatFieldMetadataMock({
id: OTHER_TS_VECTOR_FIELD_ID,
universalIdentifier: 'other-ts-vector-field-uid',
objectMetadataId: 'object-id',
type: FieldMetadataType.TS_VECTOR,
name: 'customSearchVector',
});
const flatFieldMetadataMaps = buildFlatFieldMetadataMaps([
searchVectorFlatFieldMetadata,
textFlatFieldMetadata,
otherTsVectorFlatFieldMetadata,
]);
describe('isSearchVectorGinFlatIndexMetadata', () => {
it('returns true for a single-field GIN index on the TS_VECTOR field', () => {
const flatIndexMetadata = getFlatIndexMetadataMock({
universalIdentifier: 'gin-index-uid',
objectMetadataId: 'object-id',
objectMetadataUniversalIdentifier: 'object-uid',
applicationUniversalIdentifier: 'application-uid',
indexType: IndexType.GIN,
flatIndexFieldMetadatas: [
buildFlatIndexFieldMetadata({ fieldMetadataId: SEARCH_VECTOR_FIELD_ID }),
],
});
expect(
isSearchVectorGinFlatIndexMetadata({
flatIndexMetadata,
flatFieldMetadataMaps,
}),
).toBe(true);
});
it('returns false for a GIN index on a non-TS_VECTOR field', () => {
const flatIndexMetadata = getFlatIndexMetadataMock({
universalIdentifier: 'gin-index-uid',
objectMetadataId: 'object-id',
objectMetadataUniversalIdentifier: 'object-uid',
applicationUniversalIdentifier: 'application-uid',
indexType: IndexType.GIN,
flatIndexFieldMetadatas: [
buildFlatIndexFieldMetadata({ fieldMetadataId: TEXT_FIELD_ID }),
],
});
expect(
isSearchVectorGinFlatIndexMetadata({
flatIndexMetadata,
flatFieldMetadataMaps,
}),
).toBe(false);
});
it('returns false for a non-GIN (BTREE) index on the TS_VECTOR field', () => {
const flatIndexMetadata = getFlatIndexMetadataMock({
universalIdentifier: 'btree-index-uid',
objectMetadataId: 'object-id',
objectMetadataUniversalIdentifier: 'object-uid',
applicationUniversalIdentifier: 'application-uid',
indexType: IndexType.BTREE,
flatIndexFieldMetadatas: [
buildFlatIndexFieldMetadata({ fieldMetadataId: SEARCH_VECTOR_FIELD_ID }),
],
});
expect(
isSearchVectorGinFlatIndexMetadata({
flatIndexMetadata,
flatFieldMetadataMaps,
}),
).toBe(false);
});
it('returns false for a single-field GIN index on a TS_VECTOR field that is not the searchVector field', () => {
const flatIndexMetadata = getFlatIndexMetadataMock({
universalIdentifier: 'gin-index-uid',
objectMetadataId: 'object-id',
objectMetadataUniversalIdentifier: 'object-uid',
applicationUniversalIdentifier: 'application-uid',
indexType: IndexType.GIN,
flatIndexFieldMetadatas: [
buildFlatIndexFieldMetadata({
fieldMetadataId: OTHER_TS_VECTOR_FIELD_ID,
}),
],
});
expect(
isSearchVectorGinFlatIndexMetadata({
flatIndexMetadata,
flatFieldMetadataMaps,
}),
).toBe(false);
});
it('returns false for a multi-column GIN index', () => {
const flatIndexMetadata = getFlatIndexMetadataMock({
universalIdentifier: 'gin-index-uid',
objectMetadataId: 'object-id',
objectMetadataUniversalIdentifier: 'object-uid',
applicationUniversalIdentifier: 'application-uid',
indexType: IndexType.GIN,
flatIndexFieldMetadatas: [
buildFlatIndexFieldMetadata({ fieldMetadataId: SEARCH_VECTOR_FIELD_ID }),
buildFlatIndexFieldMetadata({ fieldMetadataId: TEXT_FIELD_ID }),
],
});
expect(
isSearchVectorGinFlatIndexMetadata({
flatIndexMetadata,
flatFieldMetadataMaps,
}),
).toBe(false);
});
});
@@ -0,0 +1,139 @@
import { getFieldUniversalIdentifier } from 'twenty-shared/application';
import {
fromArrayToValuesByKeyRecord,
isDefined,
isSearchableFieldType,
} from 'twenty-shared/utils';
import { type FlatEntityMaps } from 'src/engine/metadata-modules/flat-entity/types/flat-entity-maps.type';
import { type FlatFieldMetadata } from 'src/engine/metadata-modules/flat-field-metadata/types/flat-field-metadata.type';
import { type FlatObjectMetadata } from 'src/engine/metadata-modules/flat-object-metadata/types/flat-object-metadata.type';
import { type FlatSearchFieldMetadata } from 'src/engine/metadata-modules/flat-search-field-metadata/types/flat-search-field-metadata.type';
import { buildFlatSearchFieldMetadataForField } from 'src/engine/metadata-modules/flat-search-field-metadata/utils/build-flat-search-field-metadata-for-field.util';
import { findTsVectorFlatFieldMetadataForObject } from 'src/engine/metadata-modules/flat-search-field-metadata/utils/find-ts-vector-flat-field-metadata-for-object.util';
import { type UniversalFlatSearchFieldMetadata } from 'src/engine/workspace-manager/workspace-migration/universal-flat-entity/types/universal-flat-search-field-metadata.type';
type BuildSearchFieldMetadataBackfillOperationsArgs = {
flatObjectMetadataMaps: FlatEntityMaps<FlatObjectMetadata>;
flatFieldMetadataMaps: FlatEntityMaps<FlatFieldMetadata>;
flatSearchFieldMetadataMaps: FlatEntityMaps<FlatSearchFieldMetadata>;
applicationUniversalIdentifierById: Map<string, string>;
twentyStandardApplicationId: string;
workspaceCustomApplicationId: string;
};
// Backfill skips the twenty-standard and workspace-custom applications (their row is
// already provisioned by the manifest funnel) and targets every other (installed)
// application's searchable objects that have no row yet. The creation rule mirrors the
// object-create side effect handler (label-identifier field of a searchable type;
// junction/id-label objects have no search surface). Grouped by application universal
// identifier for the per-application migration run.
export const buildSearchFieldMetadataBackfillOperations = ({
flatObjectMetadataMaps,
flatFieldMetadataMaps,
flatSearchFieldMetadataMaps,
applicationUniversalIdentifierById,
twentyStandardApplicationId,
workspaceCustomApplicationId,
}: BuildSearchFieldMetadataBackfillOperationsArgs): Record<
string,
UniversalFlatSearchFieldMetadata[]
> => {
const existingSearchFieldMetadataKeys = new Set<string>();
for (const flatSearchFieldMetadata of Object.values(
flatSearchFieldMetadataMaps.byUniversalIdentifier,
).filter(isDefined)) {
existingSearchFieldMetadataKeys.add(
`${flatSearchFieldMetadata.objectMetadataId}:${flatSearchFieldMetadata.fieldMetadataId}`,
);
}
const flatSearchFieldMetadatasToCreate: UniversalFlatSearchFieldMetadata[] =
[];
for (const flatObjectMetadata of Object.values(
flatObjectMetadataMaps.byUniversalIdentifier,
).filter(isDefined)) {
if (
flatObjectMetadata.applicationId === twentyStandardApplicationId ||
flatObjectMetadata.applicationId === workspaceCustomApplicationId
) {
continue;
}
if (flatObjectMetadata.isSearchable !== true) {
continue;
}
const applicationUniversalIdentifier =
applicationUniversalIdentifierById.get(flatObjectMetadata.applicationId);
if (!isDefined(applicationUniversalIdentifier)) {
continue;
}
const labelIdentifierFieldMetadataUniversalIdentifier =
flatObjectMetadata.labelIdentifierFieldMetadataUniversalIdentifier;
if (!isDefined(labelIdentifierFieldMetadataUniversalIdentifier)) {
continue;
}
const derivedIdFieldUniversalIdentifier = getFieldUniversalIdentifier({
applicationUniversalIdentifier,
objectUniversalIdentifier: flatObjectMetadata.universalIdentifier,
name: 'id',
});
if (
labelIdentifierFieldMetadataUniversalIdentifier ===
derivedIdFieldUniversalIdentifier
) {
continue;
}
const labelIdentifierFlatFieldMetadata =
flatFieldMetadataMaps.byUniversalIdentifier[
labelIdentifierFieldMetadataUniversalIdentifier
];
if (
!isDefined(labelIdentifierFlatFieldMetadata) ||
!isSearchableFieldType(labelIdentifierFlatFieldMetadata.type)
) {
continue;
}
if (
existingSearchFieldMetadataKeys.has(
`${flatObjectMetadata.id}:${labelIdentifierFlatFieldMetadata.id}`,
)
) {
continue;
}
const tsVectorFlatFieldMetadata = findTsVectorFlatFieldMetadataForObject({
fieldUniversalIdentifiers: flatObjectMetadata.fieldUniversalIdentifiers,
flatFieldMetadataMaps,
});
if (!isDefined(tsVectorFlatFieldMetadata)) {
continue;
}
flatSearchFieldMetadatasToCreate.push(
buildFlatSearchFieldMetadataForField({
flatObjectMetadata,
flatFieldMetadata: labelIdentifierFlatFieldMetadata,
tsVectorFlatFieldMetadata,
position: 0,
}),
);
}
return fromArrayToValuesByKeyRecord({
array: flatSearchFieldMetadatasToCreate,
key: 'applicationUniversalIdentifier',
});
};
@@ -0,0 +1,70 @@
import { getSearchFieldUniversalIdentifier } from 'twenty-shared/application';
import { isDefined } from 'twenty-shared/utils';
import { type FlatEntityMaps } from 'src/engine/metadata-modules/flat-entity/types/flat-entity-maps.type';
import { findFlatEntityByIdInFlatEntityMaps } from 'src/engine/metadata-modules/flat-entity/utils/find-flat-entity-by-id-in-flat-entity-maps.util';
import { type FlatFieldMetadata } from 'src/engine/metadata-modules/flat-field-metadata/types/flat-field-metadata.type';
import { type FlatSearchFieldMetadata } from 'src/engine/metadata-modules/flat-search-field-metadata/types/flat-search-field-metadata.type';
export type SearchFieldMetadataUniversalIdentifierUpdate = {
id: string;
deterministicUniversalIdentifier: string;
};
type BuildSearchFieldMetadataReOwnOperationsArgs = {
flatFieldMetadataMaps: FlatEntityMaps<FlatFieldMetadata>;
flatSearchFieldMetadataMaps: FlatEntityMaps<FlatSearchFieldMetadata>;
applicationUniversalIdentifierById: Map<string, string>;
};
// Re-own is global (every application, including twenty-standard and workspace-custom):
// every existing searchFieldMetadata row whose universal identifier drifted from its
// getSearchFieldUniversalIdentifier derivation is converged back to it.
export const buildSearchFieldMetadataReOwnOperations = ({
flatFieldMetadataMaps,
flatSearchFieldMetadataMaps,
applicationUniversalIdentifierById,
}: BuildSearchFieldMetadataReOwnOperationsArgs): SearchFieldMetadataUniversalIdentifierUpdate[] => {
const searchFieldMetadataUniversalIdentifierUpdates: SearchFieldMetadataUniversalIdentifierUpdate[] =
[];
for (const flatSearchFieldMetadata of Object.values(
flatSearchFieldMetadataMaps.byUniversalIdentifier,
).filter(isDefined)) {
const applicationUniversalIdentifier =
applicationUniversalIdentifierById.get(
flatSearchFieldMetadata.applicationId,
);
const indexedFlatFieldMetadata = findFlatEntityByIdInFlatEntityMaps({
flatEntityMaps: flatFieldMetadataMaps,
flatEntityId: flatSearchFieldMetadata.fieldMetadataId,
});
if (
!isDefined(applicationUniversalIdentifier) ||
!isDefined(indexedFlatFieldMetadata)
) {
continue;
}
const deterministicUniversalIdentifier = getSearchFieldUniversalIdentifier({
applicationUniversalIdentifier,
fieldMetadataUniversalIdentifier:
indexedFlatFieldMetadata.universalIdentifier,
});
if (
deterministicUniversalIdentifier ===
flatSearchFieldMetadata.universalIdentifier
) {
continue;
}
searchFieldMetadataUniversalIdentifierUpdates.push({
id: flatSearchFieldMetadata.id,
deterministicUniversalIdentifier,
});
}
return searchFieldMetadataUniversalIdentifierUpdates;
};
@@ -0,0 +1,88 @@
import { fromArrayToValuesByKeyRecord, isDefined } from 'twenty-shared/utils';
import { isSearchVectorGinFlatIndexMetadata } from 'src/database/commands/upgrade-version-command/2-20/utils/is-search-vector-gin-flat-index-metadata.util';
import { type FlatEntityMaps } from 'src/engine/metadata-modules/flat-entity/types/flat-entity-maps.type';
import { type FlatFieldMetadata } from 'src/engine/metadata-modules/flat-field-metadata/types/flat-field-metadata.type';
import { type FlatIndexMetadata } from 'src/engine/metadata-modules/flat-index-metadata/types/flat-index-metadata.type';
import { type FlatObjectMetadata } from 'src/engine/metadata-modules/flat-object-metadata/types/flat-object-metadata.type';
import { findTsVectorFlatFieldMetadataForObject } from 'src/engine/metadata-modules/flat-search-field-metadata/utils/find-ts-vector-flat-field-metadata-for-object.util';
import { buildSearchVectorGinIndexForCustomObject } from 'src/engine/metadata-modules/object-metadata/utils/build-search-vector-gin-index-for-custom-object.util';
import { type UniversalFlatIndexMetadata } from 'src/engine/workspace-manager/workspace-migration/universal-flat-entity/types/universal-flat-index-metadata.type';
type BuildSearchVectorGinIndexBackfillOperationsArgs = {
flatObjectMetadataMaps: FlatEntityMaps<FlatObjectMetadata>;
flatFieldMetadataMaps: FlatEntityMaps<FlatFieldMetadata>;
flatIndexMaps: FlatEntityMaps<FlatIndexMetadata>;
twentyStandardApplicationId: string;
workspaceCustomApplicationId: string;
};
// Backfill skips the twenty-standard and workspace-custom applications (their GIN index is
// already provisioned by the manifest funnel) and targets every other (installed)
// application object that has a searchVector field but no GIN index yet. Grouped by
// application universal identifier for the per-application migration run.
export const buildSearchVectorGinIndexBackfillOperations = ({
flatObjectMetadataMaps,
flatFieldMetadataMaps,
flatIndexMaps,
twentyStandardApplicationId,
workspaceCustomApplicationId,
}: BuildSearchVectorGinIndexBackfillOperationsArgs): Record<
string,
UniversalFlatIndexMetadata[]
> => {
const objectMetadataIdsWithSearchVectorGinIndex = new Set<string>();
for (const flatIndexMetadata of Object.values(
flatIndexMaps.byUniversalIdentifier,
).filter(isDefined)) {
if (
isSearchVectorGinFlatIndexMetadata({
flatIndexMetadata,
flatFieldMetadataMaps,
})
) {
objectMetadataIdsWithSearchVectorGinIndex.add(
flatIndexMetadata.objectMetadataId,
);
}
}
const flatIndexesToCreate: UniversalFlatIndexMetadata[] = [];
for (const flatObjectMetadata of Object.values(
flatObjectMetadataMaps.byUniversalIdentifier,
).filter(isDefined)) {
if (
flatObjectMetadata.applicationId === twentyStandardApplicationId ||
flatObjectMetadata.applicationId === workspaceCustomApplicationId
) {
continue;
}
if (objectMetadataIdsWithSearchVectorGinIndex.has(flatObjectMetadata.id)) {
continue;
}
const tsVectorFlatFieldMetadata = findTsVectorFlatFieldMetadataForObject({
fieldUniversalIdentifiers: flatObjectMetadata.fieldUniversalIdentifiers,
flatFieldMetadataMaps,
});
if (!isDefined(tsVectorFlatFieldMetadata)) {
continue;
}
flatIndexesToCreate.push(
buildSearchVectorGinIndexForCustomObject({
flatObjectMetadata,
searchVectorFlatFieldMetadata: tsVectorFlatFieldMetadata,
}),
);
}
return fromArrayToValuesByKeyRecord({
array: flatIndexesToCreate,
key: 'applicationUniversalIdentifier',
});
};
@@ -0,0 +1,82 @@
import { getIndexUniversalIdentifier } from 'twenty-shared/application';
import { isDefined } from 'twenty-shared/utils';
import { isSearchVectorGinFlatIndexMetadata } from 'src/database/commands/upgrade-version-command/2-20/utils/is-search-vector-gin-flat-index-metadata.util';
import { type FlatEntityMaps } from 'src/engine/metadata-modules/flat-entity/types/flat-entity-maps.type';
import { findFlatEntityByIdInFlatEntityMaps } from 'src/engine/metadata-modules/flat-entity/utils/find-flat-entity-by-id-in-flat-entity-maps.util';
import { type FlatFieldMetadata } from 'src/engine/metadata-modules/flat-field-metadata/types/flat-field-metadata.type';
import { type FlatIndexMetadata } from 'src/engine/metadata-modules/flat-index-metadata/types/flat-index-metadata.type';
import { type FlatObjectMetadata } from 'src/engine/metadata-modules/flat-object-metadata/types/flat-object-metadata.type';
export type SearchVectorGinIndexUniversalIdentifierUpdate = {
id: string;
name: string;
deterministicUniversalIdentifier: string;
};
type BuildSearchVectorGinIndexReOwnOperationsArgs = {
flatObjectMetadataMaps: FlatEntityMaps<FlatObjectMetadata>;
flatFieldMetadataMaps: FlatEntityMaps<FlatFieldMetadata>;
flatIndexMaps: FlatEntityMaps<FlatIndexMetadata>;
applicationUniversalIdentifierById: Map<string, string>;
};
// Re-own is global (every application, including twenty-standard and workspace-custom):
// every existing searchVector GIN index whose universal identifier drifted from its
// getIndexUniversalIdentifier derivation is converged back to it.
export const buildSearchVectorGinIndexReOwnOperations = ({
flatObjectMetadataMaps,
flatFieldMetadataMaps,
flatIndexMaps,
applicationUniversalIdentifierById,
}: BuildSearchVectorGinIndexReOwnOperationsArgs): SearchVectorGinIndexUniversalIdentifierUpdate[] => {
const indexUniversalIdentifierUpdates: SearchVectorGinIndexUniversalIdentifierUpdate[] =
[];
for (const flatIndexMetadata of Object.values(
flatIndexMaps.byUniversalIdentifier,
).filter(isDefined)) {
if (
!isSearchVectorGinFlatIndexMetadata({
flatIndexMetadata,
flatFieldMetadataMaps,
})
) {
continue;
}
const applicationUniversalIdentifier =
applicationUniversalIdentifierById.get(flatIndexMetadata.applicationId);
const flatObjectMetadata = findFlatEntityByIdInFlatEntityMaps({
flatEntityMaps: flatObjectMetadataMaps,
flatEntityId: flatIndexMetadata.objectMetadataId,
});
if (
!isDefined(applicationUniversalIdentifier) ||
!isDefined(flatObjectMetadata)
) {
continue;
}
const deterministicUniversalIdentifier = getIndexUniversalIdentifier({
applicationUniversalIdentifier,
objectUniversalIdentifier: flatObjectMetadata.universalIdentifier,
name: flatIndexMetadata.name,
});
if (
deterministicUniversalIdentifier === flatIndexMetadata.universalIdentifier
) {
continue;
}
indexUniversalIdentifierUpdates.push({
id: flatIndexMetadata.id,
name: flatIndexMetadata.name,
deterministicUniversalIdentifier,
});
}
return indexUniversalIdentifierUpdates;
};
@@ -0,0 +1,38 @@
import { FieldMetadataType } from 'twenty-shared/types';
import { isDefined } from 'twenty-shared/utils';
import { type FlatEntityMaps } from 'src/engine/metadata-modules/flat-entity/types/flat-entity-maps.type';
import { findFlatEntityByIdInFlatEntityMaps } from 'src/engine/metadata-modules/flat-entity/utils/find-flat-entity-by-id-in-flat-entity-maps.util';
import { type FlatFieldMetadata } from 'src/engine/metadata-modules/flat-field-metadata/types/flat-field-metadata.type';
import { type FlatIndexMetadata } from 'src/engine/metadata-modules/flat-index-metadata/types/flat-index-metadata.type';
import { IndexType } from 'src/engine/metadata-modules/index-metadata/types/indexType.types';
import { SEARCH_VECTOR_FIELD } from 'src/engine/metadata-modules/search-field-metadata/constants/search-vector-field.constants';
// The searchVector GIN index is the single-column GIN index whose only field is
// the object's TS_VECTOR searchVector field (see build-search-vector-gin-index-for-custom-object.util).
export const isSearchVectorGinFlatIndexMetadata = ({
flatIndexMetadata,
flatFieldMetadataMaps,
}: {
flatIndexMetadata: FlatIndexMetadata;
flatFieldMetadataMaps: FlatEntityMaps<FlatFieldMetadata>;
}): boolean => {
if (flatIndexMetadata.indexType !== IndexType.GIN) {
return false;
}
if (flatIndexMetadata.flatIndexFieldMetadatas.length !== 1) {
return false;
}
const indexedFlatFieldMetadata = findFlatEntityByIdInFlatEntityMaps({
flatEntityMaps: flatFieldMetadataMaps,
flatEntityId: flatIndexMetadata.flatIndexFieldMetadatas[0].fieldMetadataId,
});
return (
isDefined(indexedFlatFieldMetadata) &&
indexedFlatFieldMetadata.type === FieldMetadataType.TS_VECTOR &&
indexedFlatFieldMetadata.name === SEARCH_VECTOR_FIELD.name
);
};
@@ -101,7 +101,9 @@ import { BackfillDisplayFieldsOnApplicationRegistrationSlowInstanceCommand } fro
import { BackfillIsFeaturedOnApplicationRegistrationSlowInstanceCommand } from './2-19/2-19-instance-command-slow-1783120000000-backfill-is-featured-on-application-registration';
import { AllowServerScopedFileFastInstanceCommand } from 'src/database/commands/upgrade-version-command/2-20/2-20-instance-command-fast-1783499671541-allow-server-scoped-file';
import { CreateWorkflowVersionCoreTableFastInstanceCommand } from './2-20/2-20-instance-command-fast-1783512000000-create-workflow-version-core-table';
import { BackfillNameFieldIsSystemSideEffectSlowInstanceCommand } from './2-20/2-20-instance-command-slow-1783529458168-backfill-name-field-is-system-side-effect';
import { RenameIsFeaturedToIsVettedOnApplicationRegistrationFastInstanceCommand } from './2-20/2-20-instance-command-fast-1783527064000-rename-is-featured-to-is-vetted-on-application-registration';
import { AddIsSystemSideEffectToSearchFieldMetadataFastInstanceCommand } from './2-20/2-20-instance-command-fast-1783580127637-add-is-system-side-effect-to-search-field-metadata';
export const INSTANCE_COMMANDS = [
AddViewFieldGroupIdIndexOnViewFieldFastInstanceCommand,
@@ -205,5 +207,7 @@ export const INSTANCE_COMMANDS = [
BackfillIsFeaturedOnApplicationRegistrationSlowInstanceCommand,
AllowServerScopedFileFastInstanceCommand,
CreateWorkflowVersionCoreTableFastInstanceCommand,
BackfillNameFieldIsSystemSideEffectSlowInstanceCommand,
RenameIsFeaturedToIsVettedOnApplicationRegistrationFastInstanceCommand,
AddIsSystemSideEffectToSearchFieldMetadataFastInstanceCommand,
];