feat(data-model): custom-indexes management UI and mutations (#20846)

## Summary

Brings indexes management into the per-object Settings tab as a section
under Search (no feature flag, advanced mode only). Admins can create /
delete non-unique indexes with the UI; apps can declare indexes in code
with `defineIndex`. Composite-typed fields are now indexable by picking
a specific sub-column (e.g. `Address > City`).

A few related polish items also land here (invite-user dropdown lands on
the Invite tab; standard warning callout above the new-index form).

## What ships

### UI — custom indexes on per-object Settings
- New section directly under Search, wrapped in
`AdvancedSettingsWrapper`.
- Filter dropdown on the search bar toggles system-index visibility
(shown by default since advanced mode).
- **+ Add Index** button (disabled with tooltip once the per-object cap
is reached) navigates to a dedicated `SettingsObjectNewIndex` page
(matches the field-creation pattern, not a modal):
- Field picker mirrors the webhook event-form layout (rows of dropdowns,
implicit trailing empty row).
- Composite fields surface their sub-properties (`Address > City`,
`Currency > Amount`, …).
  - BTREE / GIN type selector.
- Standard warning Callout: "Use indexes sparingly — each one speeds
reads but slows writes."
- Trash icon on `isCustom: true` rows → confirmation modal →
`deleteOneIndex`.

### Server — `createOneIndex` / `deleteOneIndex` mutations
- Gated by `SettingsPermissionGuard(DATA_MODEL)`.
- `IndexMetadataService` wraps the existing migration runner via
`WorkspaceMigrationValidateBuildAndRunService` so the metadata row and
the SQL index land atomically.
- Validation: rejects empty fields, duplicate `(fieldMetadataId,
subFieldName)` pairs, fields not on the object, requires `subFieldName`
for composite parents, forbids `subFieldName` on scalar/relation,
enforces `MAX_CUSTOM_INDEXES_PER_OBJECT = 10`.
- Delete refuses on `isCustom: false` rows so system indexes can't be
removed via this API.
- Dedicated GraphQL exception handler maps each typed error to the right
transport error class.

### Composite sub-field indexing
- Adds `subFieldName: string | null` column to
`IndexFieldMetadataEntity` (fast instance command).
- The flat-entity flow (`UniversalFlatIndexFieldMetadata`,
`FlatIndexFieldMetadata`, `from-universal-flat-index-to-flat-index`,
runner column resolution) all carry `subFieldName` through.
- For composite parents, the runner uses
`computeCompositeColumnName({...}, property)` for the picked sub-column;
for non-composite parents, behavior is unchanged.
- The `'::'` separator encodes `(fieldMetadataId, subFieldName)` for
dedup on the wire; the frontend uses the same separator inside the
Select component's string value.

### Apps can declare indexes in code (`defineIndex`)
- New `IndexManifest` + `IndexFieldManifest` types in
`twenty-shared/application` wired into the `Manifest` type.
- `defineIndex` SDK helper + `IndexConfig`. CLI manifest builder +
extractor recognize `defineIndex` / `ManifestEntityKey.Indexes`.
- Server: `from-index-manifest-to-universal-flat-index` converter
resolves field IDs, validates composite/scalar `subFieldName` rules, and
delegates to `generateFlatIndexMetadataWithNameOrThrow` for the
deterministic name.
- Orchestrator wires the loop after the field-resolution pass;
per-object cap enforced inline against the manifest.
- Cascade on uninstall is automatic — when an app disappears its indexes
drop with it (universal-flat-entity diff handles it).
- Rich-app fixture ships a real `defineIndex` on `PostCard.status`,
exercising the full manifest → install path in CI.

### Closed for now (open later if needed)
- Apps cannot declare `isUnique` indexes — unique constraints stay with
the field-creation flow.
- Apps cannot use a partial-`indexWhereClause` — the UI surface keeps
the framework's hardcoded allowlist.
- UI cannot create unique or partial indexes either; same reasons.

### Cleanups along the way
- Reused the existing `getCompositeSubFieldLabel` +
`COMPOSITE_FIELD_SUB_FIELD_LABELS` (deleted the duplicates I'd created
early in the PR).
- Moved `MAX_CUSTOM_INDEXES_PER_OBJECT` to `twenty-shared/constants`
(single source for FE + BE).
- Replaced inline `isDefined(x) && x !== ''` with `isNonEmptyString`
(from `@sniptt/guards`).
- Hoisted the per-object fields Map + inlined the cap counter into the
indexes orchestrator loop (drops the install scan from O(indexes ×
totalFields) to O(totalFields + indexes)).
- Per design-feedback: page-based create flow (not a modal), filter
dropdown on the SearchInput (not a separate toggle), webhook-style
picker, field icons.

### Unrelated polish that lands here
- "Invite user" link in the multi-workspace dropdown now lands on the
Invite tab directly (`#invite`) instead of the first tab of the members
page.

## Test plan
- [ ] `npx nx typecheck twenty-server / twenty-front / twenty-sdk /
twenty-shared` — passes
- [ ] `npx nx lint:diff-with-main twenty-server / twenty-front` — clean
- [ ] `npx jest index-metadata.service.spec` — green
- [ ] `npx jest from-index-manifest-to-universal-flat-index` — green
(new converter spec, 8 cases)
- [ ] `npx vitest run
src/sdk/define/indexes/__tests__/define-index.spec.ts` (twenty-sdk) —
green (6 cases)
- [ ] `npx vitest run --config vitest.integration.config.ts -t
"rich-app"` — green (rich-app app-dev integration exercises the new
manifest path with the PostCard.status index)
- [ ] Advanced mode → Settings → any object → Settings tab → Indexes
section is visible under Search
- [ ] Create a single-field BTREE index, confirm SQL index exists
(verify via `pg_indexes`)
- [ ] Create a composite-field index (`Address > City`) and confirm the
column is `addressAddressCity`
- [ ] Create an index spanning two columns; column order matches the
picker order
- [ ] Attempt to create an 11th custom index → button is disabled with
tooltip
- [ ] Delete a custom index → confirmation modal → row disappears, PG
index dropped
- [ ] System indexes have no trash icon and are hidden by default
This commit is contained in:
Félix Malfait
2026-05-25 17:47:09 +02:00
committed by GitHub
parent 69d89f8cfc
commit d602f35cbd
98 changed files with 3742 additions and 387 deletions
@@ -0,0 +1,280 @@
import { FieldMetadataType } from 'twenty-shared/types';
import { fromIndexManifestToUniversalFlatIndex } from 'src/engine/core-modules/application/application-manifest/converters/from-index-manifest-to-universal-flat-index.util';
import { IndexType } from 'src/engine/metadata-modules/index-metadata/types/indexType.types';
import { type UniversalFlatFieldMetadata } from 'src/engine/workspace-manager/workspace-migration/universal-flat-entity/types/universal-flat-field-metadata.type';
import { type UniversalFlatObjectMetadata } from 'src/engine/workspace-manager/workspace-migration/universal-flat-entity/types/universal-flat-object-metadata.type';
describe('fromIndexManifestToUniversalFlatIndex', () => {
const now = '2026-01-01T00:00:00.000Z';
const applicationUniversalIdentifier = 'app-uuid-1';
const flatObjectMetadata = {
universalIdentifier: 'obj-uuid-1',
nameSingular: 'company',
isCustom: false,
} as UniversalFlatObjectMetadata;
const scalarField = {
universalIdentifier: 'field-uuid-scalar',
name: 'industry',
type: FieldMetadataType.TEXT,
} as UniversalFlatFieldMetadata;
const compositeField = {
universalIdentifier: 'field-uuid-address',
name: 'address',
type: FieldMetadataType.ADDRESS,
} as UniversalFlatFieldMetadata;
const baseManifest = {
universalIdentifier: 'idx-uuid-1',
objectUniversalIdentifier: 'obj-uuid-1',
indexType: 'BTREE' as const,
};
it('builds a flat index from a scalar field', () => {
const result = fromIndexManifestToUniversalFlatIndex({
indexManifest: {
...baseManifest,
fields: [
{
universalIdentifier: 'field-entry-1',
fieldUniversalIdentifier: scalarField.universalIdentifier,
},
],
},
flatObjectMetadata,
objectFlatFieldMetadatas: [scalarField],
applicationUniversalIdentifier,
now,
});
expect(result.universalIdentifier).toBe('idx-uuid-1');
expect(result.applicationUniversalIdentifier).toBe(
applicationUniversalIdentifier,
);
expect(result.indexType).toBe(IndexType.BTREE);
expect(result.isUnique).toBe(false);
expect(result.indexWhereClause).toBeNull();
expect(result.isCustom).toBe(false);
expect(result.universalFlatIndexFieldMetadatas).toHaveLength(1);
expect(result.universalFlatIndexFieldMetadatas[0]).toMatchObject({
order: 0,
subFieldName: null,
fieldMetadataUniversalIdentifier: scalarField.universalIdentifier,
indexMetadataUniversalIdentifier: 'idx-uuid-1',
});
expect(result.name).toMatch(/^IDX_/);
});
it('builds a flat index from a composite field with subFieldName', () => {
const result = fromIndexManifestToUniversalFlatIndex({
indexManifest: {
...baseManifest,
fields: [
{
universalIdentifier: 'field-entry-1',
fieldUniversalIdentifier: compositeField.universalIdentifier,
subFieldName: 'addressCity',
},
],
},
flatObjectMetadata,
objectFlatFieldMetadatas: [compositeField],
applicationUniversalIdentifier,
now,
});
expect(result.universalFlatIndexFieldMetadatas[0].subFieldName).toBe(
'addressCity',
);
});
it('forwards isUnique from the manifest', () => {
const result = fromIndexManifestToUniversalFlatIndex({
indexManifest: {
...baseManifest,
isUnique: true,
fields: [
{
universalIdentifier: 'field-entry-1',
fieldUniversalIdentifier: scalarField.universalIdentifier,
},
],
},
flatObjectMetadata,
objectFlatFieldMetadatas: [scalarField],
applicationUniversalIdentifier,
now,
});
expect(result.isUnique).toBe(true);
});
it('throws when a composite field is referenced without a subFieldName', () => {
expect(() =>
fromIndexManifestToUniversalFlatIndex({
indexManifest: {
...baseManifest,
fields: [
{
universalIdentifier: 'field-entry-1',
fieldUniversalIdentifier: compositeField.universalIdentifier,
},
],
},
flatObjectMetadata,
objectFlatFieldMetadatas: [compositeField],
applicationUniversalIdentifier,
now,
}),
).toThrow(/requires a subFieldName/);
});
it('throws when a scalar field is given a subFieldName', () => {
expect(() =>
fromIndexManifestToUniversalFlatIndex({
indexManifest: {
...baseManifest,
fields: [
{
universalIdentifier: 'field-entry-1',
fieldUniversalIdentifier: scalarField.universalIdentifier,
subFieldName: 'something',
},
],
},
flatObjectMetadata,
objectFlatFieldMetadatas: [scalarField],
applicationUniversalIdentifier,
now,
}),
).toThrow(/is not composite/);
});
it('throws when a composite sub-field is unknown', () => {
expect(() =>
fromIndexManifestToUniversalFlatIndex({
indexManifest: {
...baseManifest,
fields: [
{
universalIdentifier: 'field-entry-1',
fieldUniversalIdentifier: compositeField.universalIdentifier,
subFieldName: 'unknownSubField',
},
],
},
flatObjectMetadata,
objectFlatFieldMetadatas: [compositeField],
applicationUniversalIdentifier,
now,
}),
).toThrow(/not found on composite field/);
});
it('throws when a referenced field does not exist on the object', () => {
expect(() =>
fromIndexManifestToUniversalFlatIndex({
indexManifest: {
...baseManifest,
fields: [
{
universalIdentifier: 'field-entry-1',
fieldUniversalIdentifier: 'missing-field-uuid',
},
],
},
flatObjectMetadata,
objectFlatFieldMetadatas: [scalarField],
applicationUniversalIdentifier,
now,
}),
).toThrow(/references unknown field/);
});
it('throws when the same (field, subFieldName) pair appears twice', () => {
expect(() =>
fromIndexManifestToUniversalFlatIndex({
indexManifest: {
...baseManifest,
fields: [
{
universalIdentifier: 'field-entry-1',
fieldUniversalIdentifier: scalarField.universalIdentifier,
},
{
universalIdentifier: 'field-entry-2',
fieldUniversalIdentifier: scalarField.universalIdentifier,
},
],
},
flatObjectMetadata,
objectFlatFieldMetadatas: [scalarField],
applicationUniversalIdentifier,
now,
}),
).toThrow(/same column twice/);
});
it('throws when GIN is requested on a scalar (non-GIN-compatible) field', () => {
expect(() =>
fromIndexManifestToUniversalFlatIndex({
indexManifest: {
...baseManifest,
indexType: 'GIN',
fields: [
{
universalIdentifier: 'field-entry-1',
fieldUniversalIdentifier: scalarField.universalIdentifier,
},
],
},
flatObjectMetadata,
objectFlatFieldMetadatas: [scalarField],
applicationUniversalIdentifier,
now,
}),
).toThrow(/GIN index does not support/);
});
it('accepts GIN on a TS_VECTOR field', () => {
const tsVectorField = {
universalIdentifier: 'field-uuid-tsvector',
name: 'searchVector',
type: FieldMetadataType.TS_VECTOR,
} as UniversalFlatFieldMetadata;
const result = fromIndexManifestToUniversalFlatIndex({
indexManifest: {
...baseManifest,
indexType: 'GIN',
fields: [
{
universalIdentifier: 'field-entry-1',
fieldUniversalIdentifier: tsVectorField.universalIdentifier,
},
],
},
flatObjectMetadata,
objectFlatFieldMetadatas: [tsVectorField],
applicationUniversalIdentifier,
now,
});
expect(result.indexType).toBe(IndexType.GIN);
});
it('throws when fields is empty', () => {
expect(() =>
fromIndexManifestToUniversalFlatIndex({
indexManifest: { ...baseManifest, fields: [] },
flatObjectMetadata,
objectFlatFieldMetadatas: [],
applicationUniversalIdentifier,
now,
}),
).toThrow(/at least one field/);
});
});
@@ -0,0 +1,135 @@
import { isNonEmptyString } from '@sniptt/guards';
import { type IndexManifest } from 'twenty-shared/application';
import {
compositeTypeDefinitions,
type FieldMetadataType,
} from 'twenty-shared/types';
import { isDefined } from 'twenty-shared/utils';
import { isCompositeFieldMetadataType } from 'src/engine/metadata-modules/field-metadata/utils/is-composite-field-metadata-type.util';
import { IndexType } from 'src/engine/metadata-modules/index-metadata/types/indexType.types';
import { generateFlatIndexMetadataWithNameOrThrow } from 'src/engine/metadata-modules/index-metadata/utils/generate-flat-index.util';
import { validateIndexTypeAgainstFieldsOrThrow } from 'src/engine/metadata-modules/index-metadata/utils/validate-index-type-against-fields.util';
import { type UniversalFlatFieldMetadata } from 'src/engine/workspace-manager/workspace-migration/universal-flat-entity/types/universal-flat-field-metadata.type';
import { type UniversalFlatIndexMetadata } from 'src/engine/workspace-manager/workspace-migration/universal-flat-entity/types/universal-flat-index-metadata.type';
import { type UniversalFlatObjectMetadata } from 'src/engine/workspace-manager/workspace-migration/universal-flat-entity/types/universal-flat-object-metadata.type';
export const fromIndexManifestToUniversalFlatIndex = ({
indexManifest,
flatObjectMetadata,
objectFlatFieldMetadatas,
applicationUniversalIdentifier,
now,
}: {
indexManifest: IndexManifest;
flatObjectMetadata: UniversalFlatObjectMetadata;
objectFlatFieldMetadatas: UniversalFlatFieldMetadata[];
applicationUniversalIdentifier: string;
now: string;
}): UniversalFlatIndexMetadata => {
if (indexManifest.fields.length === 0) {
throw new Error(
`Index "${indexManifest.universalIdentifier}" must reference at least one field`,
);
}
const dedupKeys = indexManifest.fields.map(
(entry) => `${entry.fieldUniversalIdentifier}::${entry.subFieldName ?? ''}`,
);
if (new Set(dedupKeys).size !== dedupKeys.length) {
throw new Error(
`Index "${indexManifest.universalIdentifier}" lists the same column twice`,
);
}
const resolvedIndexType = (indexManifest.indexType ?? 'BTREE') as IndexType;
const resolvedFieldsForValidation: Array<{
type: FieldMetadataType;
name: string;
label: string;
subFieldName: string | null;
}> = [];
const universalFlatIndexFieldMetadatas = indexManifest.fields.map(
(entry, order) => {
const flatField = objectFlatFieldMetadatas.find(
(candidate) =>
candidate.universalIdentifier === entry.fieldUniversalIdentifier,
);
if (!isDefined(flatField)) {
throw new Error(
`Index "${indexManifest.universalIdentifier}" references unknown field ${entry.fieldUniversalIdentifier} on object ${flatObjectMetadata.universalIdentifier}`,
);
}
const isComposite = isCompositeFieldMetadataType(flatField.type);
if (isComposite) {
if (!isNonEmptyString(entry.subFieldName)) {
throw new Error(
`Composite field "${flatField.name}" requires a subFieldName in index "${indexManifest.universalIdentifier}"`,
);
}
const property = compositeTypeDefinitions
.get(flatField.type)
?.properties.find(
(compositeProperty) =>
compositeProperty.name === entry.subFieldName,
);
if (!isDefined(property)) {
throw new Error(
`Sub-field "${entry.subFieldName}" not found on composite field "${flatField.name}" in index "${indexManifest.universalIdentifier}"`,
);
}
} else if (isNonEmptyString(entry.subFieldName)) {
throw new Error(
`Field "${flatField.name}" is not composite — subFieldName must be omitted in index "${indexManifest.universalIdentifier}"`,
);
}
const subFieldName = isComposite ? (entry.subFieldName ?? null) : null;
resolvedFieldsForValidation.push({
type: flatField.type,
name: flatField.name,
label: flatField.label,
subFieldName,
});
return {
createdAt: now,
updatedAt: now,
order,
subFieldName,
fieldMetadataUniversalIdentifier: flatField.universalIdentifier,
indexMetadataUniversalIdentifier: indexManifest.universalIdentifier,
};
},
);
validateIndexTypeAgainstFieldsOrThrow({
indexType: resolvedIndexType,
fields: resolvedFieldsForValidation,
});
return generateFlatIndexMetadataWithNameOrThrow({
flatObjectMetadata,
objectFlatFieldMetadatas,
flatIndex: {
createdAt: now,
updatedAt: now,
universalIdentifier: indexManifest.universalIdentifier,
applicationUniversalIdentifier,
objectMetadataUniversalIdentifier: flatObjectMetadata.universalIdentifier,
indexType: resolvedIndexType,
indexWhereClause: null,
isCustom: false,
isUnique: indexManifest.isUnique ?? false,
universalFlatIndexFieldMetadatas,
},
});
};
@@ -1,4 +1,5 @@
import { type Manifest } from 'twenty-shared/application';
import { MAX_CUSTOM_INDEXES_PER_OBJECT } from 'twenty-shared/constants';
import { FieldMetadataType } from 'twenty-shared/types';
import { isDefined } from 'twenty-shared/utils';
@@ -10,6 +11,7 @@ import { fromConnectionProviderManifestToUniversalFlatConnectionProvider } from
import { fromFieldManifestToUniversalFlatFieldMetadata } from 'src/engine/core-modules/application/application-manifest/converters/from-field-manifest-to-universal-flat-field-metadata.util';
import { fromFieldPermissionManifestToUniversalFlatFieldPermission } from 'src/engine/core-modules/application/application-manifest/converters/from-field-permission-manifest-to-universal-flat-field-permission.util';
import { fromFrontComponentManifestToUniversalFlatFrontComponent } from 'src/engine/core-modules/application/application-manifest/converters/from-front-component-manifest-to-universal-flat-front-component.util';
import { fromIndexManifestToUniversalFlatIndex } from 'src/engine/core-modules/application/application-manifest/converters/from-index-manifest-to-universal-flat-index.util';
import { fromLogicFunctionManifestToUniversalFlatLogicFunction } from 'src/engine/core-modules/application/application-manifest/converters/from-logic-function-manifest-to-universal-flat-logic-function.util';
import { fromNavigationMenuItemManifestToUniversalFlatNavigationMenuItem } from 'src/engine/core-modules/application/application-manifest/converters/from-navigation-menu-item-manifest-to-universal-flat-navigation-menu-item.util';
import { fromObjectManifestToUniversalFlatObjectMetadata } from 'src/engine/core-modules/application/application-manifest/converters/from-object-manifest-to-universal-flat-object-metadata.util';
@@ -32,6 +34,7 @@ import { type FlatApplication } from 'src/engine/core-modules/application/types/
import { fromAgentManifestToUniversalFlatAgent } from 'src/engine/core-modules/application/utils/from-agent-manifest-to-universal-flat-agent.util';
import { createEmptyAllFlatEntityMaps } from 'src/engine/metadata-modules/flat-entity/constant/create-empty-all-flat-entity-maps.constant';
import { type AllFlatEntityMaps } from 'src/engine/metadata-modules/flat-entity/types/all-flat-entity-maps.type';
import { type UniversalFlatFieldMetadata } from 'src/engine/workspace-manager/workspace-migration/universal-flat-entity/types/universal-flat-field-metadata.type';
import { addUniversalFlatEntityToUniversalFlatEntityMapsThroughMutationOrThrow } from 'src/engine/workspace-manager/workspace-migration/universal-flat-entity/utils/add-universal-flat-entity-to-universal-flat-entity-maps-through-mutation-or-throw.util';
export const computeApplicationManifestAllUniversalFlatEntityMaps = ({
@@ -135,6 +138,75 @@ export const computeApplicationManifestAllUniversalFlatEntityMaps = ({
}
}
const indexCountByObjectUniversalIdentifier = new Map<string, number>();
const fieldsByObjectUniversalIdentifier = new Map<
string,
UniversalFlatFieldMetadata[]
>();
for (const flatField of Object.values(
allUniversalFlatEntityMaps.flatFieldMetadataMaps.byUniversalIdentifier,
)) {
if (!isDefined(flatField)) continue;
const bucket =
fieldsByObjectUniversalIdentifier.get(
flatField.objectMetadataUniversalIdentifier,
) ?? [];
if (bucket.length === 0) {
fieldsByObjectUniversalIdentifier.set(
flatField.objectMetadataUniversalIdentifier,
bucket,
);
}
bucket.push(flatField);
}
for (const indexManifest of manifest.indexes ?? []) {
const flatObjectMetadata =
allUniversalFlatEntityMaps.flatObjectMetadataMaps.byUniversalIdentifier[
indexManifest.objectUniversalIdentifier
];
if (!isDefined(flatObjectMetadata)) {
throw new Error(
`Index "${indexManifest.universalIdentifier}" references unknown object ${indexManifest.objectUniversalIdentifier}`,
);
}
const nextCount =
(indexCountByObjectUniversalIdentifier.get(
indexManifest.objectUniversalIdentifier,
) ?? 0) + 1;
if (nextCount > MAX_CUSTOM_INDEXES_PER_OBJECT) {
throw new Error(
`Application declares more than ${MAX_CUSTOM_INDEXES_PER_OBJECT} indexes on object ${indexManifest.objectUniversalIdentifier}`,
);
}
indexCountByObjectUniversalIdentifier.set(
indexManifest.objectUniversalIdentifier,
nextCount,
);
addUniversalFlatEntityToUniversalFlatEntityMapsThroughMutationOrThrow({
universalFlatEntity: fromIndexManifestToUniversalFlatIndex({
indexManifest,
flatObjectMetadata,
objectFlatFieldMetadatas:
fieldsByObjectUniversalIdentifier.get(
flatObjectMetadata.universalIdentifier,
) ?? [],
applicationUniversalIdentifier,
now,
}),
universalFlatEntityMapsToMutate: allUniversalFlatEntityMaps.flatIndexMaps,
});
}
for (const logicFunctionManifest of manifest.logicFunctions) {
addUniversalFlatEntityToUniversalFlatEntityMapsThroughMutationOrThrow({
universalFlatEntity:
@@ -411,19 +411,20 @@ export class DataloaderService {
return [];
}
return indexMetadataEntity.flatIndexFieldMetadatas.map(
(indexFieldMetadata) => {
return [...indexMetadataEntity.flatIndexFieldMetadatas]
.sort((a, b) => a.order - b.order)
.map((indexFieldMetadata) => {
return {
id: indexFieldMetadata.id,
fieldMetadataId: indexFieldMetadata.fieldMetadataId,
subFieldName: indexFieldMetadata.subFieldName ?? undefined,
order: indexFieldMetadata.order,
createdAt: new Date(indexFieldMetadata.createdAt),
updatedAt: new Date(indexFieldMetadata.updatedAt),
indexMetadataId,
workspaceId,
};
},
);
});
},
);
});
@@ -45,6 +45,8 @@ import {
import { FieldMetadataRestApiExceptionFilter } from 'src/engine/metadata-modules/field-metadata/filters/field-metadata-rest-api-exception.filter';
import { FieldMetadataService } from 'src/engine/metadata-modules/field-metadata/services/field-metadata.service';
import { fromFieldMetadataEntityToFieldMetadataDto } from 'src/engine/metadata-modules/field-metadata/utils/from-field-metadata-entity-to-field-metadata-dto.util';
import { WorkspaceManyOrAllFlatEntityMapsCacheService } from 'src/engine/metadata-modules/flat-entity/services/workspace-many-or-all-flat-entity-maps-cache.service';
import { computeUniqueFieldMetadataIdsFromFlatIndexMaps } from 'src/engine/metadata-modules/index-metadata/utils/compute-unique-field-metadata-ids-from-flat-index-maps.util';
import {
toLegacyFieldMetadataCreateResponse,
toLegacyFieldMetadataDeleteResponse,
@@ -72,8 +74,20 @@ export class FieldMetadataController {
private readonly fieldMetadataRepository: Repository<FieldMetadataEntity>,
private readonly fieldMetadataService: FieldMetadataService,
private readonly featureFlagService: FeatureFlagService,
private readonly flatEntityMapsCacheService: WorkspaceManyOrAllFlatEntityMapsCacheService,
) {}
private async loadUniqueFieldMetadataIds(
workspaceId: string,
): Promise<ReadonlySet<string>> {
const { flatIndexMaps } =
await this.flatEntityMapsCacheService.getOrRecomputeManyOrAllFlatEntityMaps(
{ workspaceId, flatMapsKeys: ['flatIndexMaps'] },
);
return computeUniqueFieldMetadataIdsFromFlatIndexMaps(flatIndexMaps);
}
@Get()
async findMany(
@Req() request: AuthenticatedRequest,
@@ -87,12 +101,17 @@ export class FieldMetadataController {
endingBefore: parseEndingBeforeRestRequest(request),
});
const uniqueFieldMetadataIds =
await this.loadUniqueFieldMetadataIds(workspaceId);
const result: {
data: FieldMetadataDTO[];
pageInfo: RestCursorPageInfo;
totalCount: number;
} = {
data: items.map(fromFieldMetadataEntityToFieldMetadataDto),
data: items.map((item) =>
fromFieldMetadataEntityToFieldMetadataDto(item, uniqueFieldMetadataIds),
),
pageInfo,
totalCount,
};
@@ -118,7 +137,12 @@ export class FieldMetadataController {
);
}
const result = fromFieldMetadataEntityToFieldMetadataDto(field);
const uniqueFieldMetadataIds =
await this.loadUniqueFieldMetadataIds(workspaceId);
const result = fromFieldMetadataEntityToFieldMetadataDto(
field,
uniqueFieldMetadataIds,
);
return (await this.isNewMetadataFormat(workspaceId))
? result
@@ -122,8 +122,11 @@ export class FieldMetadataEntity<
@Column({ nullable: true, default: true, type: 'boolean' })
isNullable: boolean | null;
// Is this really nullable ?
@Column({ nullable: true, default: false, type: 'boolean' })
// Derived at flat-entity cache build time from the existence of a
// single-field UNIQUE IndexMetadata covering this field — never persisted
// on this entity. Kept on the type so flat-entity consumers continue to
// read field.isUnique without per-call derivation; the PG column was
// dropped by 1798300000000-drop-field-metadata-is-unique-column.ts.
isUnique: boolean | null;
@Column({ default: false })
@@ -16,7 +16,7 @@ import { workspaceMigrationBuilderGraphqlApiExceptionHandler } from 'src/engine/
export const fieldMetadataGraphqlApiExceptionHandler = (error: Error) => {
if (error instanceof WorkspaceMigrationBuilderException) {
workspaceMigrationBuilderGraphqlApiExceptionHandler(error);
return workspaceMigrationBuilderGraphqlApiExceptionHandler(error);
}
if (error instanceof InvalidMetadataException) {
@@ -1,8 +1,13 @@
import { type FieldMetadataDTO } from 'src/engine/metadata-modules/field-metadata/dtos/field-metadata.dto';
import { type FieldMetadataEntity } from 'src/engine/metadata-modules/field-metadata/field-metadata.entity';
// isUnique is derived from IndexMetadata rather than stored on the field
// entity; callers that need an accurate value (e.g. the REST controller)
// pass the precomputed Set<fieldMetadataId>. Callers in pure-entity
// contexts that don't care about uniqueness can omit it.
export const fromFieldMetadataEntityToFieldMetadataDto = (
entity: FieldMetadataEntity,
uniqueFieldMetadataIds?: ReadonlySet<string>,
): FieldMetadataDTO => ({
id: entity.id,
universalIdentifier: entity.universalIdentifier,
@@ -18,7 +23,7 @@ export const fromFieldMetadataEntityToFieldMetadataDto = (
isSystem: entity.isSystem,
isUIReadOnly: entity.isUIReadOnly,
isNullable: entity.isNullable ?? false,
isUnique: entity.isUnique ?? false,
isUnique: uniqueFieldMetadataIds?.has(entity.id) ?? false,
defaultValue: entity.defaultValue ?? undefined,
options: entity.options ?? undefined,
settings: entity.settings ?? undefined,
@@ -61,6 +61,12 @@ export const ALL_ENTITY_PROPERTIES_CONFIGURATION_BY_METADATA_NAME = {
toStringify: false,
universalProperty: undefined,
},
// isUnique is derived from IndexMetadata at cache build time and is
// not a column on the fieldMetadata table. It stays in
// propertiesToCompare so per-type validators see the proposed
// change (e.g. rejecting unique on FILES), but the field-metadata
// runner drops it before issuing the SQL UPDATE — the actual state
// change rides on the side-effect index create/delete.
isUnique: {
toCompare: true,
toStringify: false,
@@ -276,6 +276,7 @@ exports[`fromCreateFieldInputToFlatFieldMetadatasToCreate MORPH_RELATION test su
"fieldMetadataUniversalIdentifier": Any<String>,
"indexMetadataUniversalIdentifier": Any<String>,
"order": 0,
"subFieldName": null,
"updatedAt": Any<String>,
},
],
@@ -297,6 +298,7 @@ exports[`fromCreateFieldInputToFlatFieldMetadatasToCreate MORPH_RELATION test su
"fieldMetadataUniversalIdentifier": Any<String>,
"indexMetadataUniversalIdentifier": Any<String>,
"order": 0,
"subFieldName": null,
"updatedAt": Any<String>,
},
],
@@ -11,6 +11,8 @@ import { createEmptyFlatEntityMaps } from 'src/engine/metadata-modules/flat-enti
import { FlatEntityMaps } from 'src/engine/metadata-modules/flat-entity/types/flat-entity-maps.type';
import { FlatFieldMetadata } from 'src/engine/metadata-modules/flat-field-metadata/types/flat-field-metadata.type';
import { fromFieldMetadataEntityToFlatFieldMetadata } from 'src/engine/metadata-modules/flat-field-metadata/utils/from-field-metadata-entity-to-flat-field-metadata.util';
import { IndexMetadataEntity } from 'src/engine/metadata-modules/index-metadata/index-metadata.entity';
import { computeUniqueFieldMetadataIdsFromIndexEntities } from 'src/engine/metadata-modules/index-metadata/utils/compute-unique-field-metadata-ids-from-index-entities.util';
import { ObjectMetadataEntity } from 'src/engine/metadata-modules/object-metadata/object-metadata.entity';
import { ViewFieldEntity } from 'src/engine/metadata-modules/view-field/entities/view-field.entity';
import { ViewFilterEntity } from 'src/engine/metadata-modules/view-filter/entities/view-filter.entity';
@@ -30,6 +32,8 @@ export class WorkspaceFlatFieldMetadataMapCacheService extends WorkspaceCachePro
constructor(
@InjectRepository(FieldMetadataEntity)
private readonly fieldMetadataRepository: Repository<FieldMetadataEntity>,
@InjectRepository(IndexMetadataEntity)
private readonly indexMetadataRepository: Repository<IndexMetadataEntity>,
@InjectRepository(ObjectMetadataEntity)
private readonly objectMetadataRepository: Repository<ObjectMetadataEntity>,
@InjectRepository(ApplicationEntity)
@@ -53,6 +57,7 @@ export class WorkspaceFlatFieldMetadataMapCacheService extends WorkspaceCachePro
): Promise<FlatEntityMaps<FlatFieldMetadata>> {
const [
fieldMetadatas,
indexMetadatas,
objectMetadatas,
applications,
viewFields,
@@ -64,6 +69,11 @@ export class WorkspaceFlatFieldMetadataMapCacheService extends WorkspaceCachePro
where: { workspaceId },
withDeleted: true,
}),
this.indexMetadataRepository.find({
where: { workspaceId, isUnique: true },
relations: ['indexFieldMetadatas'],
withDeleted: true,
}),
this.objectMetadataRepository.find({
where: { workspaceId },
select: ['id', 'universalIdentifier'],
@@ -145,6 +155,9 @@ export class WorkspaceFlatFieldMetadataMapCacheService extends WorkspaceCachePro
const applicationIdToUniversalIdentifierMap =
createIdToUniversalIdentifierMap(applications);
const uniqueFieldMetadataIds =
computeUniqueFieldMetadataIdsFromIndexEntities(indexMetadatas);
const flatFieldMetadataMaps = createEmptyFlatEntityMaps();
for (const fieldMetadataEntity of fieldMetadatas) {
@@ -169,7 +182,10 @@ export class WorkspaceFlatFieldMetadataMapCacheService extends WorkspaceCachePro
});
addFlatEntityToFlatEntityMapsThroughMutationOrThrow({
flatEntity: flatFieldMetadata,
flatEntity: {
...flatFieldMetadata,
isUnique: uniqueFieldMetadataIds.has(fieldMetadataEntity.id),
},
flatEntityMapsToMutate: flatFieldMetadataMaps,
});
}
@@ -30,6 +30,7 @@ export const generateIndexForFlatFieldMetadata = ({
flatFieldMetadata.universalIdentifier,
indexMetadataUniversalIdentifier,
order: 0,
subFieldName: null,
updatedAt: createdAt,
},
],
@@ -0,0 +1,19 @@
import { type FlatIndexMetadata } from 'src/engine/metadata-modules/flat-index-metadata/types/flat-index-metadata.type';
import { type IndexMetadataDTO } from 'src/engine/metadata-modules/index-metadata/dtos/index-metadata.dto';
export const fromFlatIndexMetadataToIndexMetadataDto = (
flatIndexMetadata: FlatIndexMetadata,
): IndexMetadataDTO => {
return {
id: flatIndexMetadata.id,
name: flatIndexMetadata.name,
isCustom: flatIndexMetadata.isCustom,
isUnique: flatIndexMetadata.isUnique,
indexWhereClause: flatIndexMetadata.indexWhereClause ?? undefined,
indexType: flatIndexMetadata.indexType,
objectMetadataId: flatIndexMetadata.objectMetadataId,
workspaceId: flatIndexMetadata.workspaceId,
createdAt: new Date(flatIndexMetadata.createdAt),
updatedAt: new Date(flatIndexMetadata.updatedAt),
};
};
@@ -78,6 +78,7 @@ export const fromIndexMetadataEntityToFlatIndexMetadata = ({
return {
order: indexFieldMetadata.order,
subFieldName: indexFieldMetadata.subFieldName,
createdAt: indexFieldMetadata.createdAt.toISOString(),
updatedAt: indexFieldMetadata.updatedAt.toISOString(),
indexMetadataUniversalIdentifier:
@@ -0,0 +1,19 @@
import { Field, InputType } from '@nestjs/graphql';
import { IsOptional, IsString, IsUUID } from 'class-validator';
import { UUIDScalarType } from 'src/engine/api/graphql/workspace-schema-builder/graphql-types/scalars';
@InputType()
export class CreateIndexFieldInput {
@IsUUID()
@Field(() => UUIDScalarType)
fieldMetadataId!: string;
// Composite sub-property name (e.g. 'addressCity'). Required for composite
// parents, must be absent for scalar/relation parents.
@IsOptional()
@IsString()
@Field({ nullable: true })
subFieldName?: string;
}
@@ -0,0 +1,38 @@
import { Field, InputType } from '@nestjs/graphql';
import { Type } from 'class-transformer';
import {
ArrayMinSize,
ArrayNotEmpty,
IsArray,
IsEnum,
IsUUID,
ValidateNested,
} from 'class-validator';
import { UUIDScalarType } from 'src/engine/api/graphql/workspace-schema-builder/graphql-types/scalars';
import { CreateIndexFieldInput } from 'src/engine/metadata-modules/index-metadata/dtos/create-index-field.input';
import { IndexType } from 'src/engine/metadata-modules/index-metadata/types/indexType.types';
@InputType()
export class CreateIndexInput {
@IsUUID()
@Field(() => UUIDScalarType)
objectMetadataId!: string;
// Order matters: Postgres uses the leading column first.
@IsArray()
@ArrayNotEmpty()
@ArrayMinSize(1)
@Type(() => CreateIndexFieldInput)
@ValidateNested({ each: true })
@Field(() => [CreateIndexFieldInput])
fields!: CreateIndexFieldInput[];
@IsEnum(IndexType)
@Field(() => IndexType, { defaultValue: IndexType.BTREE })
indexType!: IndexType;
// indexWhereClause is not exposed: the validator only allows a hardcoded
// allowlist, so a free-text field on the user-facing API would mislead.
}
@@ -0,0 +1,16 @@
import { Field, InputType } from '@nestjs/graphql';
import { Type } from 'class-transformer';
import { ValidateNested } from 'class-validator';
import { CreateIndexInput } from 'src/engine/metadata-modules/index-metadata/dtos/create-index.input';
@InputType()
export class CreateOneIndexInput {
@Type(() => CreateIndexInput)
@ValidateNested()
@Field(() => CreateIndexInput, {
description: 'The custom index to create',
})
index!: CreateIndexInput;
}
@@ -0,0 +1,15 @@
import { InputType } from '@nestjs/graphql';
import { IDField } from '@ptc-org/nestjs-query-graphql';
import { IsUUID } from 'class-validator';
import { UUIDScalarType } from 'src/engine/api/graphql/workspace-schema-builder/graphql-types/scalars';
@InputType()
export class DeleteOneIndexInput {
@IDField(() => UUIDScalarType, {
description: 'The id of the custom index to delete.',
})
@IsUUID()
id!: string;
}
@@ -7,7 +7,14 @@ import {
QueryOptions,
Relation,
} from '@ptc-org/nestjs-query-graphql';
import { IsDateString, IsNotEmpty, IsNumber, IsUUID } from 'class-validator';
import {
IsDateString,
IsNotEmpty,
IsNumber,
IsOptional,
IsString,
IsUUID,
} from 'class-validator';
import { UUIDScalarType } from 'src/engine/api/graphql/workspace-schema-builder/graphql-types/scalars';
import { FieldMetadataDTO } from 'src/engine/metadata-modules/field-metadata/dtos/field-metadata.dto';
@@ -50,6 +57,11 @@ export class IndexFieldMetadataDTO {
@Field()
order: number;
@IsOptional()
@IsString()
@Field({ nullable: true })
subFieldName?: string;
@IsDateString()
@Field()
createdAt: Date;
@@ -57,6 +57,12 @@ export class IndexFieldMetadataEntity implements Required<IndexFieldMetadataEnti
@Column({ nullable: false })
order: number;
// Null for scalar/relation fields. Set to the composite sub-property name
// (e.g. 'addressCity') when the index targets a single column of a
// composite-type parent.
@Column({ type: 'text', nullable: true })
subFieldName: string | null;
@CreateDateColumn({ type: 'timestamptz' })
createdAt: Date;
@@ -20,4 +20,13 @@ export enum IndexMetadataExceptionCode {
INDEX_CREATION_FAILED = 'INDEX_CREATION_FAILED',
INDEX_NOT_SUPPORTED_FOR_COMPOSITE_FIELD = 'INDEX_NOT_SUPPORTED_FOR_COMPOSITE_FIELD',
INDEX_NOT_SUPPORTED_FOR_MORH_RELATION_FIELD_AND_RELATION_FIELD = 'INDEX_NOT_SUPPORTED_FOR_MORH_RELATION_FIELD_AND_RELATION_FIELD',
CUSTOM_INDEX_LIMIT_REACHED = 'CUSTOM_INDEX_LIMIT_REACHED',
CANNOT_DELETE_SYSTEM_INDEX = 'CANNOT_DELETE_SYSTEM_INDEX',
INDEX_FIELDS_REQUIRED = 'INDEX_FIELDS_REQUIRED',
DUPLICATE_INDEX_FIELDS = 'DUPLICATE_INDEX_FIELDS',
INDEX_OBJECT_NOT_FOUND = 'INDEX_OBJECT_NOT_FOUND',
INDEX_FIELD_NOT_FOUND_ON_OBJECT = 'INDEX_FIELD_NOT_FOUND_ON_OBJECT',
INDEX_NOT_FOUND = 'INDEX_NOT_FOUND',
INDEX_TYPE_NOT_SUPPORTED_FOR_FIELD_TYPE = 'INDEX_TYPE_NOT_SUPPORTED_FOR_FIELD_TYPE',
DUPLICATE_UNIQUE_INDEX = 'DUPLICATE_UNIQUE_INDEX',
}
@@ -5,16 +5,25 @@ import { SortDirection } from '@ptc-org/nestjs-query-core';
import { NestjsQueryGraphQLModule } from '@ptc-org/nestjs-query-graphql';
import { NestjsQueryTypeOrmModule } from '@ptc-org/nestjs-query-typeorm';
import { ApplicationModule } from 'src/engine/core-modules/application/application.module';
import { WorkspaceAuthGuard } from 'src/engine/guards/workspace-auth.guard';
import { WorkspaceManyOrAllFlatEntityMapsCacheModule } from 'src/engine/metadata-modules/flat-entity/services/workspace-many-or-all-flat-entity-maps-cache.module';
import { IndexMetadataDTO } from 'src/engine/metadata-modules/index-metadata/dtos/index-metadata.dto';
import { IndexFieldMetadataEntity } from 'src/engine/metadata-modules/index-metadata/index-field-metadata.entity';
import { IndexMetadataEntity } from 'src/engine/metadata-modules/index-metadata/index-metadata.entity';
import { IndexMetadataResolver } from 'src/engine/metadata-modules/index-metadata/index-metadata.resolver';
import { IndexMetadataService } from 'src/engine/metadata-modules/index-metadata/services/index-metadata.service';
import { ObjectMetadataGraphqlApiExceptionInterceptor } from 'src/engine/metadata-modules/object-metadata/interceptors/object-metadata-graphql-api-exception.interceptor';
import { PermissionsModule } from 'src/engine/metadata-modules/permissions/permissions.module';
import { WorkspaceMigrationModule } from 'src/engine/workspace-manager/workspace-migration/workspace-migration.module';
@Module({
imports: [
TypeOrmModule.forFeature([IndexMetadataEntity]),
ApplicationModule,
PermissionsModule,
WorkspaceMigrationModule,
WorkspaceManyOrAllFlatEntityMapsCacheModule,
NestjsQueryGraphQLModule.forFeature({
imports: [
NestjsQueryTypeOrmModule.forFeature([
@@ -43,7 +52,7 @@ import { ObjectMetadataGraphqlApiExceptionInterceptor } from 'src/engine/metadat
],
}),
],
providers: [IndexMetadataResolver],
exports: [],
providers: [IndexMetadataResolver, IndexMetadataService],
exports: [IndexMetadataService],
})
export class IndexMetadataModule {}
@@ -1,5 +1,7 @@
import { UseFilters, UseGuards, UsePipes } from '@nestjs/common';
import { Context, Parent, ResolveField } from '@nestjs/graphql';
import { Args, Context, Mutation, Parent, ResolveField } from '@nestjs/graphql';
import { PermissionFlagType } from 'twenty-shared/constants';
import { MetadataResolver } from 'src/engine/api/graphql/graphql-config/decorators/metadata-resolver.decorator';
import { PreventNestToAutoLogGraphqlErrorsFilter } from 'src/engine/core-modules/graphql/filters/prevent-nest-to-auto-log-graphql-errors.filter';
@@ -7,9 +9,15 @@ import { ResolverValidationPipe } from 'src/engine/core-modules/graphql/pipes/re
import { WorkspaceEntity } from 'src/engine/core-modules/workspace/workspace.entity';
import { type IDataloaders } from 'src/engine/dataloaders/dataloader.interface';
import { AuthWorkspace } from 'src/engine/decorators/auth/auth-workspace.decorator';
import { SettingsPermissionGuard } from 'src/engine/guards/settings-permission.guard';
import { WorkspaceAuthGuard } from 'src/engine/guards/workspace-auth.guard';
import { fromFlatIndexMetadataToIndexMetadataDto } from 'src/engine/metadata-modules/flat-index-metadata/utils/from-flat-index-metadata-to-index-metadata-dto.util';
import { CreateOneIndexInput } from 'src/engine/metadata-modules/index-metadata/dtos/create-one-index.input';
import { DeleteOneIndexInput } from 'src/engine/metadata-modules/index-metadata/dtos/delete-index.input';
import { IndexFieldMetadataDTO } from 'src/engine/metadata-modules/index-metadata/dtos/index-field-metadata.dto';
import { IndexMetadataDTO } from 'src/engine/metadata-modules/index-metadata/dtos/index-metadata.dto';
import { IndexMetadataService } from 'src/engine/metadata-modules/index-metadata/services/index-metadata.service';
import { indexMetadataGraphqlApiExceptionHandler } from 'src/engine/metadata-modules/index-metadata/utils/index-metadata-graphql-api-exception-handler.util';
import { objectMetadataGraphqlApiExceptionHandler } from 'src/engine/metadata-modules/object-metadata/utils/object-metadata-graphql-api-exception-handler.util';
import { PermissionsGraphqlApiExceptionFilter } from 'src/engine/metadata-modules/permissions/utils/permissions-graphql-api-exception.filter';
@@ -21,6 +29,8 @@ import { PermissionsGraphqlApiExceptionFilter } from 'src/engine/metadata-module
PermissionsGraphqlApiExceptionFilter,
)
export class IndexMetadataResolver {
constructor(private readonly indexMetadataService: IndexMetadataService) {}
@ResolveField(() => [IndexFieldMetadataDTO], { nullable: false })
async indexFieldMetadataList(
@AuthWorkspace() workspace: WorkspaceEntity,
@@ -42,4 +52,40 @@ export class IndexMetadataResolver {
return [];
}
}
@UseGuards(SettingsPermissionGuard(PermissionFlagType.DATA_MODEL))
@Mutation(() => IndexMetadataDTO)
async createOneIndex(
@Args('input') input: CreateOneIndexInput,
@AuthWorkspace() { id: workspaceId }: WorkspaceEntity,
): Promise<IndexMetadataDTO> {
try {
const flatIndexMetadata = await this.indexMetadataService.createOne({
createIndexInput: input.index,
workspaceId,
});
return fromFlatIndexMetadataToIndexMetadataDto(flatIndexMetadata);
} catch (error) {
return indexMetadataGraphqlApiExceptionHandler(error);
}
}
@UseGuards(SettingsPermissionGuard(PermissionFlagType.DATA_MODEL))
@Mutation(() => IndexMetadataDTO)
async deleteOneIndex(
@Args('input') input: DeleteOneIndexInput,
@AuthWorkspace() { id: workspaceId }: WorkspaceEntity,
): Promise<IndexMetadataDTO> {
try {
const flatIndexMetadata = await this.indexMetadataService.deleteOne({
id: input.id,
workspaceId,
});
return fromFlatIndexMetadataToIndexMetadataDto(flatIndexMetadata);
} catch (error) {
return indexMetadataGraphqlApiExceptionHandler(error);
}
}
}
@@ -0,0 +1,450 @@
import { Test, type TestingModule } from '@nestjs/testing';
import { FieldMetadataType } from 'twenty-shared/types';
import { ApplicationService } from 'src/engine/core-modules/application/application.service';
import { createEmptyFlatEntityMaps } from 'src/engine/metadata-modules/flat-entity/constant/create-empty-flat-entity-maps.constant';
import { WorkspaceManyOrAllFlatEntityMapsCacheService } from 'src/engine/metadata-modules/flat-entity/services/workspace-many-or-all-flat-entity-maps-cache.service';
import { getFlatIndexMetadataMock } from 'src/engine/metadata-modules/flat-index-metadata/__mocks__/get-flat-index-metadata.mock';
import { MAX_CUSTOM_INDEXES_PER_OBJECT } from 'twenty-shared/constants';
import {
IndexMetadataException,
IndexMetadataExceptionCode,
} from 'src/engine/metadata-modules/index-metadata/index-field-metadata.exception';
import { IndexMetadataService } from 'src/engine/metadata-modules/index-metadata/services/index-metadata.service';
import { IndexType } from 'src/engine/metadata-modules/index-metadata/types/indexType.types';
import { WorkspaceMigrationValidateBuildAndRunService } from 'src/engine/workspace-manager/workspace-migration/services/workspace-migration-validate-build-and-run-service';
const WORKSPACE_ID = 'workspace-id';
const OBJECT_ID = 'object-id';
const OBJECT_UNIVERSAL_ID = 'object-universal-id';
const APPLICATION_UNIVERSAL_ID = 'app-universal-id';
const buildFlatObjectMetadataMaps = () => {
const maps = createEmptyFlatEntityMaps() as ReturnType<
typeof createEmptyFlatEntityMaps
> & {
byUniversalIdentifier: Record<string, unknown>;
universalIdentifierById: Record<string, string>;
};
maps.byUniversalIdentifier = {
[OBJECT_UNIVERSAL_ID]: {
id: OBJECT_ID,
universalIdentifier: OBJECT_UNIVERSAL_ID,
nameSingular: 'company',
isCustom: false,
},
};
maps.universalIdentifierById = { [OBJECT_ID]: OBJECT_UNIVERSAL_ID };
return maps;
};
const buildFlatFieldMetadataMaps = (
fields: {
id: string;
universalIdentifier: string;
name: string;
type?: FieldMetadataType;
}[],
) => {
const maps = createEmptyFlatEntityMaps() as ReturnType<
typeof createEmptyFlatEntityMaps
> & {
byUniversalIdentifier: Record<string, unknown>;
universalIdentifierById: Record<string, string>;
};
for (const field of fields) {
maps.byUniversalIdentifier[field.universalIdentifier] = {
id: field.id,
universalIdentifier: field.universalIdentifier,
name: field.name,
label: field.name,
type: field.type ?? FieldMetadataType.TEXT,
objectMetadataId: OBJECT_ID,
isUnique: false,
isCustom: true,
isActive: true,
isSystem: false,
isNullable: true,
};
maps.universalIdentifierById[field.id] = field.universalIdentifier;
}
return maps;
};
const buildFlatIndexMaps = (
indexes: Array<{
id: string;
universalIdentifier: string;
isCustom: boolean;
}>,
) => {
const maps = createEmptyFlatEntityMaps() as ReturnType<
typeof createEmptyFlatEntityMaps
> & {
byUniversalIdentifier: Record<string, unknown>;
universalIdentifierById: Record<string, string>;
};
for (const index of indexes) {
maps.byUniversalIdentifier[index.universalIdentifier] =
getFlatIndexMetadataMock({
id: index.id,
universalIdentifier: index.universalIdentifier,
objectMetadataId: OBJECT_ID,
objectMetadataUniversalIdentifier: OBJECT_UNIVERSAL_ID,
applicationUniversalIdentifier: APPLICATION_UNIVERSAL_ID,
isCustom: index.isCustom,
});
maps.universalIdentifierById[index.id] = index.universalIdentifier;
}
return maps;
};
describe('IndexMetadataService', () => {
let service: IndexMetadataService;
let cacheService: jest.Mocked<WorkspaceManyOrAllFlatEntityMapsCacheService>;
let migrationService: jest.Mocked<WorkspaceMigrationValidateBuildAndRunService>;
let applicationService: jest.Mocked<ApplicationService>;
const setupCacheReturn = ({
fieldIds = [],
customIndexCount = 0,
}: {
fieldIds?: {
id: string;
universalIdentifier: string;
name: string;
type?: FieldMetadataType;
}[];
customIndexCount?: number;
} = {}) => {
const indexes = Array.from({ length: customIndexCount }).map((_, i) => ({
id: `idx-${i}`,
universalIdentifier: `idx-universal-${i}`,
isCustom: true,
}));
cacheService.getOrRecomputeManyOrAllFlatEntityMaps.mockResolvedValue({
flatObjectMetadataMaps: buildFlatObjectMetadataMaps(),
flatFieldMetadataMaps: buildFlatFieldMetadataMaps(fieldIds),
flatIndexMaps: buildFlatIndexMaps(indexes),
// oxlint-disable-next-line @typescript-eslint/no-explicit-any
} as any);
};
beforeEach(async () => {
const module: TestingModule = await Test.createTestingModule({
providers: [
IndexMetadataService,
{
provide: WorkspaceManyOrAllFlatEntityMapsCacheService,
useValue: {
getOrRecomputeManyOrAllFlatEntityMaps: jest.fn(),
invalidateFlatEntityMaps: jest.fn(),
},
},
{
provide: WorkspaceMigrationValidateBuildAndRunService,
useValue: {
validateBuildAndRunWorkspaceMigration: jest
.fn()
.mockResolvedValue({ status: 'success' }),
},
},
{
provide: ApplicationService,
useValue: {
findWorkspaceTwentyStandardAndCustomApplicationOrThrow: jest
.fn()
.mockResolvedValue({
workspaceCustomFlatApplication: {
universalIdentifier: APPLICATION_UNIVERSAL_ID,
},
}),
},
},
],
}).compile();
service = module.get(IndexMetadataService);
cacheService = module.get(WorkspaceManyOrAllFlatEntityMapsCacheService);
migrationService = module.get(WorkspaceMigrationValidateBuildAndRunService);
applicationService = module.get(ApplicationService);
});
describe('createOne validation', () => {
it('rejects empty fields', async () => {
setupCacheReturn();
await expect(
service.createOne({
workspaceId: WORKSPACE_ID,
createIndexInput: {
objectMetadataId: OBJECT_ID,
fields: [],
indexType: IndexType.BTREE,
},
}),
).rejects.toMatchObject({
code: IndexMetadataExceptionCode.INDEX_FIELDS_REQUIRED,
});
});
it('rejects duplicate (fieldMetadataId + subFieldName) pairs', async () => {
setupCacheReturn();
await expect(
service.createOne({
workspaceId: WORKSPACE_ID,
createIndexInput: {
objectMetadataId: OBJECT_ID,
fields: [
{ fieldMetadataId: 'field-1' },
{ fieldMetadataId: 'field-1' },
],
indexType: IndexType.BTREE,
},
}),
).rejects.toMatchObject({
code: IndexMetadataExceptionCode.DUPLICATE_INDEX_FIELDS,
});
});
it('rejects when object does not exist', async () => {
setupCacheReturn();
await expect(
service.createOne({
workspaceId: WORKSPACE_ID,
createIndexInput: {
objectMetadataId: 'unknown-object',
fields: [{ fieldMetadataId: 'field-1' }],
indexType: IndexType.BTREE,
},
}),
).rejects.toMatchObject({
code: IndexMetadataExceptionCode.INDEX_OBJECT_NOT_FOUND,
});
});
it('rejects composite-type fields without subFieldName', async () => {
setupCacheReturn({
fieldIds: [
{
id: 'field-currency',
universalIdentifier: 'field-currency-universal',
name: 'annualRecurringRevenue',
type: FieldMetadataType.CURRENCY,
},
],
});
await expect(
service.createOne({
workspaceId: WORKSPACE_ID,
createIndexInput: {
objectMetadataId: OBJECT_ID,
fields: [{ fieldMetadataId: 'field-currency' }],
indexType: IndexType.BTREE,
},
}),
).rejects.toMatchObject({
code: IndexMetadataExceptionCode.INDEX_NOT_SUPPORTED_FOR_COMPOSITE_FIELD,
});
expect(
migrationService.validateBuildAndRunWorkspaceMigration,
).not.toHaveBeenCalled();
});
it('rejects composite-type fields with an unknown subFieldName', async () => {
setupCacheReturn({
fieldIds: [
{
id: 'field-currency',
universalIdentifier: 'field-currency-universal',
name: 'annualRecurringRevenue',
type: FieldMetadataType.CURRENCY,
},
],
});
await expect(
service.createOne({
workspaceId: WORKSPACE_ID,
createIndexInput: {
objectMetadataId: OBJECT_ID,
fields: [
{
fieldMetadataId: 'field-currency',
subFieldName: 'notARealProp',
},
],
indexType: IndexType.BTREE,
},
}),
).rejects.toMatchObject({
code: IndexMetadataExceptionCode.INDEX_NOT_SUPPORTED_FOR_COMPOSITE_FIELD,
});
});
it('rejects subFieldName on a scalar field', async () => {
setupCacheReturn({
fieldIds: [
{
id: 'field-text',
universalIdentifier: 'field-text-universal',
name: 'someText',
type: FieldMetadataType.TEXT,
},
],
});
await expect(
service.createOne({
workspaceId: WORKSPACE_ID,
createIndexInput: {
objectMetadataId: OBJECT_ID,
fields: [
{ fieldMetadataId: 'field-text', subFieldName: 'whatever' },
],
indexType: IndexType.BTREE,
},
}),
).rejects.toMatchObject({
code: IndexMetadataExceptionCode.INDEX_NOT_SUPPORTED_FOR_COMPOSITE_FIELD,
});
});
it('rejects when a field does not belong to the object', async () => {
setupCacheReturn();
await expect(
service.createOne({
workspaceId: WORKSPACE_ID,
createIndexInput: {
objectMetadataId: OBJECT_ID,
fields: [{ fieldMetadataId: 'unknown-field' }],
indexType: IndexType.BTREE,
},
}),
).rejects.toMatchObject({
code: IndexMetadataExceptionCode.INDEX_FIELD_NOT_FOUND_ON_OBJECT,
});
});
it('rejects when custom index count is at the cap', async () => {
setupCacheReturn({
fieldIds: [
{
id: 'field-1',
universalIdentifier: 'field-1-universal',
name: 'someColumn',
},
],
customIndexCount: MAX_CUSTOM_INDEXES_PER_OBJECT,
});
await expect(
service.createOne({
workspaceId: WORKSPACE_ID,
createIndexInput: {
objectMetadataId: OBJECT_ID,
fields: [{ fieldMetadataId: 'field-1' }],
indexType: IndexType.BTREE,
},
}),
).rejects.toMatchObject({
code: IndexMetadataExceptionCode.CUSTOM_INDEX_LIMIT_REACHED,
});
expect(
applicationService.findWorkspaceTwentyStandardAndCustomApplicationOrThrow,
).not.toHaveBeenCalled();
});
});
describe('deleteOne validation', () => {
it('throws INDEX_NOT_FOUND (not the generic flat-entity error) for an unknown id', async () => {
cacheService.getOrRecomputeManyOrAllFlatEntityMaps.mockResolvedValue({
flatIndexMaps: buildFlatIndexMaps([]),
// oxlint-disable-next-line @typescript-eslint/no-explicit-any
} as any);
await expect(
service.deleteOne({ id: 'unknown-idx', workspaceId: WORKSPACE_ID }),
).rejects.toMatchObject({
code: IndexMetadataExceptionCode.INDEX_NOT_FOUND,
});
expect(
migrationService.validateBuildAndRunWorkspaceMigration,
).not.toHaveBeenCalled();
});
it('refuses to delete a system index', async () => {
cacheService.getOrRecomputeManyOrAllFlatEntityMaps.mockResolvedValue({
flatIndexMaps: buildFlatIndexMaps([
{
id: 'system-idx',
universalIdentifier: 'system-idx-universal',
isCustom: false,
},
]),
// oxlint-disable-next-line @typescript-eslint/no-explicit-any
} as any);
await expect(
service.deleteOne({ id: 'system-idx', workspaceId: WORKSPACE_ID }),
).rejects.toMatchObject({
code: IndexMetadataExceptionCode.CANNOT_DELETE_SYSTEM_INDEX,
});
expect(
migrationService.validateBuildAndRunWorkspaceMigration,
).not.toHaveBeenCalled();
});
it('runs the migration when deleting a custom index', async () => {
cacheService.getOrRecomputeManyOrAllFlatEntityMaps.mockResolvedValue({
flatIndexMaps: buildFlatIndexMaps([
{
id: 'custom-idx',
universalIdentifier: 'custom-idx-universal',
isCustom: true,
},
]),
// oxlint-disable-next-line @typescript-eslint/no-explicit-any
} as any);
await service.deleteOne({
id: 'custom-idx',
workspaceId: WORKSPACE_ID,
});
expect(
migrationService.validateBuildAndRunWorkspaceMigration,
).toHaveBeenCalledTimes(1);
const call =
migrationService.validateBuildAndRunWorkspaceMigration.mock.calls[0][0];
expect(
call.allFlatEntityOperationByMetadataName.index?.flatEntityToDelete,
).toHaveLength(1);
});
});
it('IndexMetadataException can be thrown', () => {
expect(
new IndexMetadataException(
'msg',
IndexMetadataExceptionCode.INDEX_NOT_FOUND,
),
).toBeInstanceOf(IndexMetadataException);
});
});
@@ -0,0 +1,385 @@
import { Injectable } from '@nestjs/common';
import { msg } from '@lingui/core/macro';
import { compositeTypeDefinitions, RelationType } from 'twenty-shared/types';
import { isNonEmptyString } from '@sniptt/guards';
import { isDefined } from 'twenty-shared/utils';
import { v4 } from 'uuid';
import { ApplicationService } from 'src/engine/core-modules/application/application.service';
import { isCompositeFieldMetadataType } from 'src/engine/metadata-modules/field-metadata/utils/is-composite-field-metadata-type.util';
import { isMorphOrRelationFlatFieldMetadata } from 'src/engine/metadata-modules/flat-field-metadata/utils/is-morph-or-relation-flat-field-metadata.util';
import { WorkspaceManyOrAllFlatEntityMapsCacheService } from 'src/engine/metadata-modules/flat-entity/services/workspace-many-or-all-flat-entity-maps-cache.service';
import { findFlatEntityByIdInFlatEntityMaps } from 'src/engine/metadata-modules/flat-entity/utils/find-flat-entity-by-id-in-flat-entity-maps.util';
import { findFlatEntityByUniversalIdentifier } from 'src/engine/metadata-modules/flat-entity/utils/find-flat-entity-by-universal-identifier.util';
import { type FlatIndexMetadata } from 'src/engine/metadata-modules/flat-index-metadata/types/flat-index-metadata.type';
import { MAX_CUSTOM_INDEXES_PER_OBJECT } from 'twenty-shared/constants';
import { type CreateIndexInput } from 'src/engine/metadata-modules/index-metadata/dtos/create-index.input';
import {
IndexMetadataException,
IndexMetadataExceptionCode,
} from 'src/engine/metadata-modules/index-metadata/index-field-metadata.exception';
import { generateFlatIndexMetadataWithNameOrThrow } from 'src/engine/metadata-modules/index-metadata/utils/generate-flat-index.util';
import { validateIndexTypeAgainstFieldsOrThrow } from 'src/engine/metadata-modules/index-metadata/utils/validate-index-type-against-fields.util';
import { validateNoDuplicateUniqueIndexOrThrow } from 'src/engine/metadata-modules/index-metadata/utils/validate-no-duplicate-unique-index.util';
import { WorkspaceMigrationBuilderException } from 'src/engine/workspace-manager/workspace-migration/exceptions/workspace-migration-builder-exception';
import { WorkspaceMigrationValidateBuildAndRunService } from 'src/engine/workspace-manager/workspace-migration/services/workspace-migration-validate-build-and-run-service';
@Injectable()
export class IndexMetadataService {
constructor(
private readonly flatEntityMapsCacheService: WorkspaceManyOrAllFlatEntityMapsCacheService,
private readonly workspaceMigrationValidateBuildAndRunService: WorkspaceMigrationValidateBuildAndRunService,
private readonly applicationService: ApplicationService,
) {}
async createOne({
createIndexInput,
workspaceId,
}: {
createIndexInput: CreateIndexInput;
workspaceId: string;
}): Promise<FlatIndexMetadata> {
const { fields: fieldInputs } = createIndexInput;
if (fieldInputs.length === 0) {
throw new IndexMetadataException(
'At least one field is required to create an index',
IndexMetadataExceptionCode.INDEX_FIELDS_REQUIRED,
{
userFriendlyMessage: msg`Pick at least one field for the index.`,
},
);
}
// Duplicate check considers (fieldMetadataId, subFieldName) pair so the
// user CAN pick "Address > City" and "Address > Postcode" in the same
// composite index, but not the exact same column twice.
const dedupKeys = fieldInputs.map(
(input) => `${input.fieldMetadataId}::${input.subFieldName ?? ''}`,
);
if (new Set(dedupKeys).size !== dedupKeys.length) {
throw new IndexMetadataException(
'Duplicate field+sub-field in index field list',
IndexMetadataExceptionCode.DUPLICATE_INDEX_FIELDS,
{
userFriendlyMessage: msg`The same column cannot appear twice in an index.`,
},
);
}
const {
flatObjectMetadataMaps: existingFlatObjectMetadataMaps,
flatFieldMetadataMaps: existingFlatFieldMetadataMaps,
flatIndexMaps: existingFlatIndexMaps,
} = await this.flatEntityMapsCacheService.getOrRecomputeManyOrAllFlatEntityMaps(
{
workspaceId,
flatMapsKeys: [
'flatObjectMetadataMaps',
'flatFieldMetadataMaps',
'flatIndexMaps',
],
},
);
const flatObjectMetadata = findFlatEntityByIdInFlatEntityMaps({
flatEntityMaps: existingFlatObjectMetadataMaps,
flatEntityId: createIndexInput.objectMetadataId,
});
if (!isDefined(flatObjectMetadata)) {
throw new IndexMetadataException(
`Object metadata ${createIndexInput.objectMetadataId} not found`,
IndexMetadataExceptionCode.INDEX_OBJECT_NOT_FOUND,
{
userFriendlyMessage: msg`Could not find the object for this index.`,
},
);
}
// Resolve each input to a flat field + validated subFieldName (or null
// for scalar/relation parents).
const resolvedInputs = fieldInputs.map((input) => {
const flatField = findFlatEntityByIdInFlatEntityMaps({
flatEntityMaps: existingFlatFieldMetadataMaps,
flatEntityId: input.fieldMetadataId,
});
if (
!isDefined(flatField) ||
flatField.objectMetadataId !== createIndexInput.objectMetadataId
) {
throw new IndexMetadataException(
`Field ${input.fieldMetadataId} not found on object ${createIndexInput.objectMetadataId}`,
IndexMetadataExceptionCode.INDEX_FIELD_NOT_FOUND_ON_OBJECT,
{
userFriendlyMessage: msg`One of the selected fields does not belong to this object.`,
},
);
}
if (
isMorphOrRelationFlatFieldMetadata(flatField) &&
flatField.settings?.relationType !== RelationType.MANY_TO_ONE
) {
throw new IndexMetadataException(
`Field ${flatField.name} is a non-MANY_TO_ONE relation and has no join column to index`,
IndexMetadataExceptionCode.INDEX_NOT_SUPPORTED_FOR_MORH_RELATION_FIELD_AND_RELATION_FIELD,
{
userFriendlyMessage: msg`"${flatField.label}" is a one-to-many relation and can't be indexed directly. Index the foreign-key side instead.`,
},
);
}
const isComposite = isCompositeFieldMetadataType(flatField.type);
if (isComposite) {
if (!isNonEmptyString(input.subFieldName)) {
throw new IndexMetadataException(
`Composite field ${flatField.name} requires a sub-field selection`,
IndexMetadataExceptionCode.INDEX_NOT_SUPPORTED_FOR_COMPOSITE_FIELD,
{
userFriendlyMessage: msg`Pick a specific sub-field of "${flatField.label}" — composite fields can't be indexed as a whole.`,
},
);
}
const compositeType = compositeTypeDefinitions.get(flatField.type);
const knownProperty = compositeType?.properties.find(
(property) => property.name === input.subFieldName,
);
if (!isDefined(knownProperty)) {
throw new IndexMetadataException(
`Unknown sub-field ${input.subFieldName} on composite field ${flatField.name}`,
IndexMetadataExceptionCode.INDEX_NOT_SUPPORTED_FOR_COMPOSITE_FIELD,
{
userFriendlyMessage: msg`"${input.subFieldName}" is not a valid sub-field of "${flatField.label}".`,
},
);
}
} else if (isNonEmptyString(input.subFieldName)) {
// Scalar / relation parent — sub-field doesn't apply.
throw new IndexMetadataException(
`Field ${flatField.name} is not composite — subFieldName must not be set`,
IndexMetadataExceptionCode.INDEX_NOT_SUPPORTED_FOR_COMPOSITE_FIELD,
{
userFriendlyMessage: msg`"${flatField.label}" is not a composite field — remove the sub-field selection.`,
},
);
}
return {
flatField,
subFieldName: isComposite ? (input.subFieldName ?? null) : null,
};
});
const objectFlatFieldMetadatas = resolvedInputs.map(
({ flatField }) => flatField,
);
validateIndexTypeAgainstFieldsOrThrow({
indexType: createIndexInput.indexType,
fields: resolvedInputs.map(({ flatField, subFieldName }) => ({
type: flatField.type,
name: flatField.name,
label: flatField.label,
subFieldName,
})),
});
validateNoDuplicateUniqueIndexOrThrow({
proposed: {
isUnique: false,
fields: resolvedInputs.map(({ flatField, subFieldName }) => ({
fieldMetadataId: flatField.id,
subFieldName,
})),
},
existingFlatIndexMaps,
objectMetadataId: createIndexInput.objectMetadataId,
});
const existingCustomIndexCount = Object.values(
existingFlatIndexMaps.byUniversalIdentifier,
)
.filter(isDefined)
.filter(
(flatIndex) =>
flatIndex.objectMetadataId === createIndexInput.objectMetadataId &&
flatIndex.isCustom,
).length;
if (existingCustomIndexCount >= MAX_CUSTOM_INDEXES_PER_OBJECT) {
throw new IndexMetadataException(
`Custom index limit of ${MAX_CUSTOM_INDEXES_PER_OBJECT} reached for object ${createIndexInput.objectMetadataId}`,
IndexMetadataExceptionCode.CUSTOM_INDEX_LIMIT_REACHED,
{
userFriendlyMessage: msg`You can have at most ${MAX_CUSTOM_INDEXES_PER_OBJECT} custom indexes per object. Delete one before creating a new one.`,
},
);
}
const { workspaceCustomFlatApplication } =
await this.applicationService.findWorkspaceTwentyStandardAndCustomApplicationOrThrow(
{ workspaceId },
);
const indexMetadataUniversalIdentifier = v4();
const createdAt = new Date().toISOString();
const universalFlatIndexMetadata = generateFlatIndexMetadataWithNameOrThrow(
{
flatObjectMetadata,
objectFlatFieldMetadatas,
flatIndex: {
createdAt,
updatedAt: createdAt,
indexType: createIndexInput.indexType,
// WHERE clause is system-only — see CreateIndexInput for rationale.
indexWhereClause: null,
isCustom: true,
isUnique: false,
objectMetadataUniversalIdentifier:
flatObjectMetadata.universalIdentifier,
universalIdentifier: indexMetadataUniversalIdentifier,
applicationUniversalIdentifier:
workspaceCustomFlatApplication.universalIdentifier,
universalFlatIndexFieldMetadatas: resolvedInputs.map(
({ flatField, subFieldName }, order) => ({
createdAt,
updatedAt: createdAt,
order,
subFieldName,
fieldMetadataUniversalIdentifier: flatField.universalIdentifier,
indexMetadataUniversalIdentifier,
}),
),
},
},
);
const validateAndBuildResult =
await this.workspaceMigrationValidateBuildAndRunService.validateBuildAndRunWorkspaceMigration(
{
allFlatEntityOperationByMetadataName: {
index: {
flatEntityToCreate: [universalFlatIndexMetadata],
flatEntityToDelete: [],
flatEntityToUpdate: [],
},
},
workspaceId,
applicationUniversalIdentifier:
workspaceCustomFlatApplication.universalIdentifier,
},
);
if (validateAndBuildResult.status === 'fail') {
throw new WorkspaceMigrationBuilderException(
validateAndBuildResult,
'Validation errors occurred while creating index',
);
}
const { flatIndexMaps: recomputedFlatIndexMaps } =
await this.flatEntityMapsCacheService.getOrRecomputeManyOrAllFlatEntityMaps(
{
workspaceId,
flatMapsKeys: ['flatIndexMaps'],
},
);
const createdFlatIndexMetadata = findFlatEntityByUniversalIdentifier({
universalIdentifier: indexMetadataUniversalIdentifier,
flatEntityMaps: recomputedFlatIndexMaps,
});
if (!isDefined(createdFlatIndexMetadata)) {
throw new IndexMetadataException(
`Index ${indexMetadataUniversalIdentifier} was created but is missing from the recomputed cache`,
IndexMetadataExceptionCode.INDEX_CREATION_FAILED,
);
}
return createdFlatIndexMetadata;
}
async deleteOne({
id,
workspaceId,
}: {
id: string;
workspaceId: string;
}): Promise<FlatIndexMetadata> {
const { flatIndexMaps: existingFlatIndexMaps } =
await this.flatEntityMapsCacheService.getOrRecomputeManyOrAllFlatEntityMaps(
{
workspaceId,
flatMapsKeys: ['flatIndexMaps'],
},
);
const flatIndexToDelete = findFlatEntityByIdInFlatEntityMaps({
flatEntityMaps: existingFlatIndexMaps,
flatEntityId: id,
});
// Map "no such index" to a domain-specific error so the GraphQL handler
// can return a NotFoundError instead of leaking a FlatEntityMapsException.
if (!isDefined(flatIndexToDelete)) {
throw new IndexMetadataException(
`Index ${id} not found`,
IndexMetadataExceptionCode.INDEX_NOT_FOUND,
{
userFriendlyMessage: msg`This index does not exist or has already been deleted.`,
},
);
}
// Protect system indexes — they back uniqueness constraints, FK lookups,
// and search performance. Dropping one corrupts the data model.
if (!flatIndexToDelete.isCustom) {
throw new IndexMetadataException(
`Index ${id} is a system index and cannot be deleted`,
IndexMetadataExceptionCode.CANNOT_DELETE_SYSTEM_INDEX,
{
userFriendlyMessage: msg`System indexes are required for Twenty to work and cannot be deleted.`,
},
);
}
const { workspaceCustomFlatApplication } =
await this.applicationService.findWorkspaceTwentyStandardAndCustomApplicationOrThrow(
{ workspaceId },
);
const validateAndBuildResult =
await this.workspaceMigrationValidateBuildAndRunService.validateBuildAndRunWorkspaceMigration(
{
allFlatEntityOperationByMetadataName: {
index: {
flatEntityToCreate: [],
flatEntityToDelete: [flatIndexToDelete],
flatEntityToUpdate: [],
},
},
workspaceId,
applicationUniversalIdentifier:
workspaceCustomFlatApplication.universalIdentifier,
},
);
if (validateAndBuildResult.status === 'fail') {
throw new WorkspaceMigrationBuilderException(
validateAndBuildResult,
'Validation errors occurred while deleting index',
);
}
return flatIndexToDelete;
}
}
@@ -0,0 +1,11 @@
// Jest Snapshot v1, https://jestjs.io/docs/snapshot-testing
exports[`generateFlatIndexMetadataWithNameOrThrow pins names for representative standard-index shapes 1`] = `
{
"manyToOneJoin": "IDX_f719a95179070eac397ba18dc70",
"partialUnique": "IDX_UNIQUE_10f41e6e837c3c40fa07ca9d76f",
"scalarNonUnique": "IDX_f6529884d22eaf60dd0bfc9f831",
"scalarUnique": "IDX_UNIQUE_2a32339058d0b6910b0834ddf81",
"searchVectorGin": "IDX_fb1f4905546cfc6d70a971c76f7",
}
`;
@@ -0,0 +1,148 @@
import { FieldMetadataType, RelationType } from 'twenty-shared/types';
import { IndexType } from 'src/engine/metadata-modules/index-metadata/types/indexType.types';
import { generateFlatIndexMetadataWithNameOrThrow } from 'src/engine/metadata-modules/index-metadata/utils/generate-flat-index.util';
import { type UniversalFlatFieldMetadata } from 'src/engine/workspace-manager/workspace-migration/universal-flat-entity/types/universal-flat-field-metadata.type';
import { type UniversalFlatObjectMetadata } from 'src/engine/workspace-manager/workspace-migration/universal-flat-entity/types/universal-flat-object-metadata.type';
// Pin the deterministic index names produced by the shared engine for
// canonical input shapes. Any future engine refactor that perturbs a hash
// trips here locally instead of after a workspace migration starts failing
// on a customer install.
describe('generateFlatIndexMetadataWithNameOrThrow', () => {
const now = '2026-05-25T00:00:00.000Z';
const companyObject = {
universalIdentifier: 'obj-company',
nameSingular: 'company',
isCustom: false,
} as UniversalFlatObjectMetadata;
const scalarUniqueField = {
universalIdentifier: 'field-domain',
name: 'domainName',
type: FieldMetadataType.TEXT,
isUnique: true,
} as UniversalFlatFieldMetadata;
const scalarNonUniqueField = {
universalIdentifier: 'field-employees',
name: 'employees',
type: FieldMetadataType.NUMBER,
isUnique: false,
} as UniversalFlatFieldMetadata;
const tsVectorField = {
universalIdentifier: 'field-search',
name: 'searchVector',
type: FieldMetadataType.TS_VECTOR,
isUnique: false,
} as UniversalFlatFieldMetadata;
const manyToOneRelationField = {
universalIdentifier: 'field-account-owner',
name: 'accountOwner',
type: FieldMetadataType.RELATION,
isUnique: false,
universalSettings: { relationType: RelationType.MANY_TO_ONE },
} as unknown as UniversalFlatFieldMetadata;
const buildIndex = (overrides: {
universalIdentifier: string;
isUnique: boolean;
indexType: IndexType;
fieldIds: string[];
indexWhereClause?: string | null;
}) => ({
createdAt: now,
updatedAt: now,
universalIdentifier: overrides.universalIdentifier,
applicationUniversalIdentifier: 'app-standard',
objectMetadataUniversalIdentifier: companyObject.universalIdentifier,
indexType: overrides.indexType,
indexWhereClause: overrides.indexWhereClause ?? null,
isCustom: false,
isUnique: overrides.isUnique,
universalFlatIndexFieldMetadatas: overrides.fieldIds.map((id, order) => ({
createdAt: now,
updatedAt: now,
order,
subFieldName: null,
fieldMetadataUniversalIdentifier: id,
indexMetadataUniversalIdentifier: overrides.universalIdentifier,
})),
});
it('pins names for representative standard-index shapes', () => {
const fields = [
scalarUniqueField,
scalarNonUniqueField,
tsVectorField,
manyToOneRelationField,
];
const scalarUnique = generateFlatIndexMetadataWithNameOrThrow({
flatObjectMetadata: companyObject,
objectFlatFieldMetadatas: fields,
flatIndex: buildIndex({
universalIdentifier: 'idx-scalar-unique',
isUnique: true,
indexType: IndexType.BTREE,
fieldIds: [scalarUniqueField.universalIdentifier],
}),
});
const scalarNonUnique = generateFlatIndexMetadataWithNameOrThrow({
flatObjectMetadata: companyObject,
objectFlatFieldMetadatas: fields,
flatIndex: buildIndex({
universalIdentifier: 'idx-scalar-non-unique',
isUnique: false,
indexType: IndexType.BTREE,
fieldIds: [scalarNonUniqueField.universalIdentifier],
}),
});
const searchVectorGin = generateFlatIndexMetadataWithNameOrThrow({
flatObjectMetadata: companyObject,
objectFlatFieldMetadatas: fields,
flatIndex: buildIndex({
universalIdentifier: 'idx-search-vector',
isUnique: false,
indexType: IndexType.GIN,
fieldIds: [tsVectorField.universalIdentifier],
}),
});
const manyToOneJoin = generateFlatIndexMetadataWithNameOrThrow({
flatObjectMetadata: companyObject,
objectFlatFieldMetadatas: fields,
flatIndex: buildIndex({
universalIdentifier: 'idx-account-owner',
isUnique: false,
indexType: IndexType.BTREE,
fieldIds: [manyToOneRelationField.universalIdentifier],
}),
});
const partialUnique = generateFlatIndexMetadataWithNameOrThrow({
flatObjectMetadata: companyObject,
objectFlatFieldMetadatas: fields,
flatIndex: buildIndex({
universalIdentifier: 'idx-scalar-unique-partial',
isUnique: true,
indexType: IndexType.BTREE,
fieldIds: [scalarUniqueField.universalIdentifier],
indexWhereClause: '"deletedAt" IS NULL',
}),
});
expect({
scalarUnique: scalarUnique.name,
scalarNonUnique: scalarNonUnique.name,
searchVectorGin: searchVectorGin.name,
manyToOneJoin: manyToOneJoin.name,
partialUnique: partialUnique.name,
}).toMatchSnapshot();
});
});
@@ -0,0 +1,12 @@
import { isDefined } from 'twenty-shared/utils';
import { type FlatEntityMaps } from 'src/engine/metadata-modules/flat-entity/types/flat-entity-maps.type';
import { type FlatIndexMetadata } from 'src/engine/metadata-modules/flat-index-metadata/types/flat-index-metadata.type';
import { computeUniqueFieldMetadataIdsFromIndexes } from 'src/engine/metadata-modules/index-metadata/utils/compute-unique-field-metadata-ids-from-indexes.util';
export const computeUniqueFieldMetadataIdsFromFlatIndexMaps = (
flatIndexMaps: FlatEntityMaps<FlatIndexMetadata>,
): Set<string> =>
computeUniqueFieldMetadataIdsFromIndexes(
Object.values(flatIndexMaps.byUniversalIdentifier).filter(isDefined),
);
@@ -0,0 +1,6 @@
import { type IndexMetadataEntity } from 'src/engine/metadata-modules/index-metadata/index-metadata.entity';
import { computeUniqueFieldMetadataIdsFromIndexes } from 'src/engine/metadata-modules/index-metadata/utils/compute-unique-field-metadata-ids-from-indexes.util';
export const computeUniqueFieldMetadataIdsFromIndexEntities = (
indexEntities: ReadonlyArray<IndexMetadataEntity>,
): Set<string> => computeUniqueFieldMetadataIdsFromIndexes(indexEntities);
@@ -0,0 +1,33 @@
type IndexLike = {
isUnique: boolean;
flatIndexFieldMetadatas?: Array<{
fieldMetadataId: string;
subFieldName: string | null;
}>;
indexFieldMetadatas?: Array<{
fieldMetadataId: string;
subFieldName: string | null;
}>;
};
// A field is "unique" iff there exists a UNIQUE IndexMetadata whose single
// member is exactly that field (no composite sub-field). Centralized so the
// cache builder, REST controller, and any future consumer agree on the rule.
export const computeUniqueFieldMetadataIdsFromIndexes = (
indexes: ReadonlyArray<IndexLike>,
): Set<string> => {
const set = new Set<string>();
for (const index of indexes) {
if (!index.isUnique) continue;
const fields = index.flatIndexFieldMetadatas ?? index.indexFieldMetadatas;
if (fields?.length !== 1) continue;
if (fields[0].subFieldName !== null) continue;
set.add(fields[0].fieldMetadataId);
}
return set;
};
@@ -10,11 +10,16 @@ type GenerateDeterministicIndexNameArgs = {
>;
isUnique?: boolean;
orderedIndexColumnNames: string[];
// Include the WHERE clause in the hash so a partial index on the same
// columns doesn't collide with the non-partial one (Postgres lets them
// coexist; the unique-name constraint must not block that).
indexWhereClause?: string | null;
};
export const generateDeterministicIndexNameV2 = ({
orderedIndexColumnNames,
flatObjectMetadata,
isUnique = false,
indexWhereClause,
}: GenerateDeterministicIndexNameArgs): string => {
const hash = createHash('sha256');
@@ -27,5 +32,9 @@ export const generateDeterministicIndexNameV2 = ({
hash.update(column);
});
if (indexWhereClause) {
hash.update(indexWhereClause);
}
return `IDX_${isUnique ? 'UNIQUE_' : ''}${hash.digest('hex').slice(0, 27)}`;
};
@@ -1,11 +1,13 @@
import { RelationType } from 'twenty-shared/types';
import { compositeTypeDefinitions, RelationType } from 'twenty-shared/types';
import { isDefined } from 'twenty-shared/utils';
import { computeCompositeColumnName } from 'src/engine/metadata-modules/field-metadata/utils/compute-column-name.util';
import { computeMorphOrRelationFieldJoinColumnName } from 'src/engine/metadata-modules/field-metadata/utils/compute-morph-or-relation-field-join-column-name.util';
import { isCompositeFieldMetadataType } from 'src/engine/metadata-modules/field-metadata/utils/is-composite-field-metadata-type.util';
import {
FlatEntityMapsException,
FlatEntityMapsExceptionCode,
} from 'src/engine/metadata-modules/flat-entity/exceptions/flat-entity-maps.exception';
import { computeMorphOrRelationFieldJoinColumnName } from 'src/engine/metadata-modules/field-metadata/utils/compute-morph-or-relation-field-join-column-name.util';
import { isMorphOrRelationUniversalFlatFieldMetadata } from 'src/engine/metadata-modules/flat-field-metadata/utils/is-morph-or-relation-flat-field-metadata.util';
import { generateDeterministicIndexNameV2 } from 'src/engine/metadata-modules/index-metadata/utils/generate-deterministic-index-name-v2';
import { type UniversalFlatFieldMetadata } from 'src/engine/workspace-manager/workspace-migration/universal-flat-entity/types/universal-flat-field-metadata.type';
@@ -17,12 +19,13 @@ export type GenerateFlatIndexArgs = {
objectFlatFieldMetadatas: UniversalFlatFieldMetadata[];
flatIndex: Omit<UniversalFlatIndexMetadata, 'name'>;
};
export const generateFlatIndexMetadataWithNameOrThrow = ({
flatObjectMetadata,
objectFlatFieldMetadatas,
flatIndex,
}: GenerateFlatIndexArgs): UniversalFlatIndexMetadata => {
const orderedFlatFields = flatIndex.universalFlatIndexFieldMetadatas
const orderedIndexColumnNames = flatIndex.universalFlatIndexFieldMetadatas
.sort((a, b) => a.order - b.order)
.map((flatIndexField) => {
const relatedFlatFieldMetadata = objectFlatFieldMetadatas.find(
@@ -38,36 +41,59 @@ export const generateFlatIndexMetadataWithNameOrThrow = ({
);
}
// Composite parent with an explicit sub-field → single sub-column.
// Composite parent without sub-field falls through to the legacy
// scalar branch below, which produces a deterministic name based on
// the parent name (the runner handles the multi-column SQL expansion
// via isIncludedInUniqueConstraint).
if (
isCompositeFieldMetadataType(relatedFlatFieldMetadata.type) &&
isDefined(flatIndexField.subFieldName)
) {
const property = compositeTypeDefinitions
.get(relatedFlatFieldMetadata.type)
?.properties.find(
(compositeProperty) =>
compositeProperty.name === flatIndexField.subFieldName,
);
if (!isDefined(property)) {
throw new FlatEntityMapsException(
`Composite sub-field "${flatIndexField.subFieldName}" not found on ${relatedFlatFieldMetadata.name}`,
FlatEntityMapsExceptionCode.ENTITY_NOT_FOUND,
);
}
return computeCompositeColumnName(
{
name: relatedFlatFieldMetadata.name,
type: relatedFlatFieldMetadata.type,
},
property,
);
}
const isManyToOneRelation =
isMorphOrRelationUniversalFlatFieldMetadata(relatedFlatFieldMetadata) &&
relatedFlatFieldMetadata.universalSettings?.relationType ===
RelationType.MANY_TO_ONE;
const name = isManyToOneRelation
return isManyToOneRelation
? computeMorphOrRelationFieldJoinColumnName({
name: relatedFlatFieldMetadata.name,
})
: relatedFlatFieldMetadata.name;
return {
name,
isUnique: relatedFlatFieldMetadata.isUnique,
};
});
const isUnique = orderedFlatFields.some((flatField) => flatField.isUnique);
const orderedIndexColumnNames = orderedFlatFields.map(
(flatField) => flatField.name,
);
const name = generateDeterministicIndexNameV2({
flatObjectMetadata,
orderedIndexColumnNames,
isUnique,
isUnique: flatIndex.isUnique,
indexWhereClause: flatIndex.indexWhereClause,
});
return {
...flatIndex,
name,
isUnique,
};
};
@@ -0,0 +1,48 @@
import { assertUnreachable } from 'twenty-shared/utils';
import {
ConflictError,
ForbiddenError,
InternalServerError,
NotFoundError,
UserInputError,
} from 'src/engine/core-modules/graphql/utils/graphql-errors.util';
import {
IndexMetadataException,
IndexMetadataExceptionCode,
} from 'src/engine/metadata-modules/index-metadata/index-field-metadata.exception';
import { WorkspaceMigrationBuilderException } from 'src/engine/workspace-manager/workspace-migration/exceptions/workspace-migration-builder-exception';
import { workspaceMigrationBuilderGraphqlApiExceptionHandler } from 'src/engine/workspace-manager/workspace-migration/interceptors/utils/workspace-migration-builder-graphql-api-exception-handler.util';
export const indexMetadataGraphqlApiExceptionHandler = (error: Error) => {
if (error instanceof WorkspaceMigrationBuilderException) {
return workspaceMigrationBuilderGraphqlApiExceptionHandler(error);
}
if (error instanceof IndexMetadataException) {
switch (error.code) {
case IndexMetadataExceptionCode.INDEX_OBJECT_NOT_FOUND:
case IndexMetadataExceptionCode.INDEX_NOT_FOUND:
throw new NotFoundError(error);
case IndexMetadataExceptionCode.INDEX_FIELDS_REQUIRED:
case IndexMetadataExceptionCode.DUPLICATE_INDEX_FIELDS:
case IndexMetadataExceptionCode.INDEX_FIELD_NOT_FOUND_ON_OBJECT:
case IndexMetadataExceptionCode.INDEX_NOT_SUPPORTED_FOR_COMPOSITE_FIELD:
case IndexMetadataExceptionCode.INDEX_NOT_SUPPORTED_FOR_MORH_RELATION_FIELD_AND_RELATION_FIELD:
case IndexMetadataExceptionCode.INDEX_TYPE_NOT_SUPPORTED_FOR_FIELD_TYPE:
case IndexMetadataExceptionCode.DUPLICATE_UNIQUE_INDEX:
throw new UserInputError(error);
case IndexMetadataExceptionCode.CANNOT_DELETE_SYSTEM_INDEX:
throw new ForbiddenError(error);
case IndexMetadataExceptionCode.CUSTOM_INDEX_LIMIT_REACHED:
throw new ConflictError(error);
case IndexMetadataExceptionCode.INDEX_CREATION_FAILED:
throw new InternalServerError(error);
default: {
return assertUnreachable(error.code);
}
}
}
throw error;
};
@@ -0,0 +1,54 @@
import { msg } from '@lingui/core/macro';
import { GIN_COMPATIBLE_FIELD_TYPES } from 'twenty-shared/constants';
import { type FieldMetadataType } from 'twenty-shared/types';
import {
IndexMetadataException,
IndexMetadataExceptionCode,
} from 'src/engine/metadata-modules/index-metadata/index-field-metadata.exception';
import { IndexType } from 'src/engine/metadata-modules/index-metadata/types/indexType.types';
type IndexFieldForValidation = {
type: FieldMetadataType;
name: string;
label: string;
subFieldName: string | null;
};
// GIN requires an operator class for each column. Postgres ships a default
// opclass only for jsonb (RAW_JSON), text[]/varchar[] (ARRAY, MULTI_SELECT),
// and tsvector (TS_VECTOR). Composite parents resolve to scalar text/numeric
// sub-columns, none of which have a default GIN opclass.
export const validateIndexTypeAgainstFieldsOrThrow = ({
indexType,
fields,
}: {
indexType: IndexType;
fields: IndexFieldForValidation[];
}): void => {
if (indexType !== IndexType.GIN) {
return;
}
for (const field of fields) {
if (field.subFieldName !== null) {
throw new IndexMetadataException(
`GIN index does not support composite sub-property ${field.name}.${field.subFieldName}`,
IndexMetadataExceptionCode.INDEX_TYPE_NOT_SUPPORTED_FOR_FIELD_TYPE,
{
userFriendlyMessage: msg`GIN indexes work on multi-select, array, JSON, and search-vector columns. "${field.label}" sub-properties don't qualify.`,
},
);
}
if (!GIN_COMPATIBLE_FIELD_TYPES.has(field.type)) {
throw new IndexMetadataException(
`GIN index does not support field ${field.name} of type ${field.type}`,
IndexMetadataExceptionCode.INDEX_TYPE_NOT_SUPPORTED_FOR_FIELD_TYPE,
{
userFriendlyMessage: msg`GIN indexes work on multi-select, array, JSON, and search-vector columns. "${field.label}" doesn't qualify.`,
},
);
}
}
};
@@ -0,0 +1,63 @@
import { msg } from '@lingui/core/macro';
import { isDefined } from 'twenty-shared/utils';
import { type FlatEntityMaps } from 'src/engine/metadata-modules/flat-entity/types/flat-entity-maps.type';
import { type FlatIndexMetadata } from 'src/engine/metadata-modules/flat-index-metadata/types/flat-index-metadata.type';
import {
IndexMetadataException,
IndexMetadataExceptionCode,
} from 'src/engine/metadata-modules/index-metadata/index-field-metadata.exception';
type ProposedUniqueIndex = {
isUnique: boolean;
fields: Array<{ fieldMetadataId: string; subFieldName: string | null }>;
};
// Field-level uniqueness is now expressed exclusively as a single-field
// UNIQUE IndexMetadata. Two unique indexes on the same single (field,
// subFieldName) pair are redundant and waste write throughput, so reject.
export const validateNoDuplicateUniqueIndexOrThrow = ({
proposed,
existingFlatIndexMaps,
objectMetadataId,
ignoreIndexId,
}: {
proposed: ProposedUniqueIndex;
existingFlatIndexMaps: FlatEntityMaps<FlatIndexMetadata>;
objectMetadataId: string;
ignoreIndexId?: string;
}): void => {
if (!proposed.isUnique || proposed.fields.length !== 1) {
return;
}
const [proposedField] = proposed.fields;
const duplicate = Object.values(
existingFlatIndexMaps.byUniversalIdentifier,
).find((flatIndex) => {
if (!isDefined(flatIndex)) return false;
if (flatIndex.id === ignoreIndexId) return false;
if (!flatIndex.isUnique) return false;
if (flatIndex.objectMetadataId !== objectMetadataId) return false;
if (flatIndex.flatIndexFieldMetadatas.length !== 1) return false;
const existingField = flatIndex.flatIndexFieldMetadatas[0];
return (
existingField.fieldMetadataId === proposedField.fieldMetadataId &&
(existingField.subFieldName ?? null) ===
(proposedField.subFieldName ?? null)
);
});
if (isDefined(duplicate)) {
throw new IndexMetadataException(
`A UNIQUE index already covers this column (${duplicate.name})`,
IndexMetadataExceptionCode.DUPLICATE_UNIQUE_INDEX,
{
userFriendlyMessage: msg`This column is already marked as unique. Toggle the field's "Unique" off first if you want to manage the constraint here.`,
},
);
}
};
@@ -36,6 +36,8 @@ import { SettingsPermissionGuard } from 'src/engine/guards/settings-permission.g
import { WorkspaceAuthGuard } from 'src/engine/guards/workspace-auth.guard';
import { FieldMetadataEntity } from 'src/engine/metadata-modules/field-metadata/field-metadata.entity';
import { fromFieldMetadataEntityToFieldMetadataDto } from 'src/engine/metadata-modules/field-metadata/utils/from-field-metadata-entity-to-field-metadata-dto.util';
import { WorkspaceManyOrAllFlatEntityMapsCacheService } from 'src/engine/metadata-modules/flat-entity/services/workspace-many-or-all-flat-entity-maps-cache.service';
import { computeUniqueFieldMetadataIdsFromFlatIndexMaps } from 'src/engine/metadata-modules/index-metadata/utils/compute-unique-field-metadata-ids-from-flat-index-maps.util';
import { fromFlatObjectMetadataToObjectMetadataDto } from 'src/engine/metadata-modules/flat-object-metadata/utils/from-flat-object-metadata-to-object-metadata-dto.util';
import { CreateObjectInput } from 'src/engine/metadata-modules/object-metadata/dtos/create-object.input';
import { type ObjectMetadataWithFieldsDTO } from 'src/engine/metadata-modules/object-metadata/dtos/object-metadata-with-fields.dto';
@@ -76,8 +78,20 @@ export class ObjectMetadataController {
private readonly fieldMetadataRepository: Repository<FieldMetadataEntity>,
private readonly objectMetadataService: ObjectMetadataService,
private readonly featureFlagService: FeatureFlagService,
private readonly flatEntityMapsCacheService: WorkspaceManyOrAllFlatEntityMapsCacheService,
) {}
private async loadUniqueFieldMetadataIds(
workspaceId: string,
): Promise<ReadonlySet<string>> {
const { flatIndexMaps } =
await this.flatEntityMapsCacheService.getOrRecomputeManyOrAllFlatEntityMaps(
{ workspaceId, flatMapsKeys: ['flatIndexMaps'] },
);
return computeUniqueFieldMetadataIdsFromFlatIndexMaps(flatIndexMaps);
}
@Get()
async findMany(
@Req() request: AuthenticatedRequest,
@@ -91,13 +105,20 @@ export class ObjectMetadataController {
endingBefore: parseEndingBeforeRestRequest(request),
});
const fields = await this.findFieldsForObjectIds(
workspaceId,
items.map((object) => object.id),
);
const [fields, uniqueFieldMetadataIds] = await Promise.all([
this.findFieldsForObjectIds(
workspaceId,
items.map((object) => object.id),
),
this.loadUniqueFieldMetadataIds(workspaceId),
]);
const data = items.map((object) =>
this.toObjectWithFieldsDto(object, fields.get(object.id) ?? []),
this.toObjectWithFieldsDto(
object,
fields.get(object.id) ?? [],
uniqueFieldMetadataIds,
),
);
const result: {
@@ -127,11 +148,18 @@ export class ObjectMetadataController {
);
}
const fields = await this.fieldMetadataRepository.find({
where: { objectMetadataId: object.id, workspaceId },
});
const [fields, uniqueFieldMetadataIds] = await Promise.all([
this.fieldMetadataRepository.find({
where: { objectMetadataId: object.id, workspaceId },
}),
this.loadUniqueFieldMetadataIds(workspaceId),
]);
const result = this.toObjectWithFieldsDto(object, fields);
const result = this.toObjectWithFieldsDto(
object,
fields,
uniqueFieldMetadataIds,
);
return (await this.isNewMetadataFormat(workspaceId))
? result
@@ -148,13 +176,21 @@ export class ObjectMetadataController {
workspaceId,
});
const fields = await this.fieldMetadataRepository.find({
where: { objectMetadataId: flatObject.id, workspaceId },
});
const [fields, uniqueFieldMetadataIds] = await Promise.all([
this.fieldMetadataRepository.find({
where: { objectMetadataId: flatObject.id, workspaceId },
}),
this.loadUniqueFieldMetadataIds(workspaceId),
]);
const result: ObjectMetadataWithFieldsDTO = {
...fromFlatObjectMetadataToObjectMetadataDto(flatObject),
fields: fields.map(fromFieldMetadataEntityToFieldMetadataDto),
fields: fields.map((field) =>
fromFieldMetadataEntityToFieldMetadataDto(
field,
uniqueFieldMetadataIds,
),
),
};
return (await this.isNewMetadataFormat(workspaceId))
@@ -211,13 +247,21 @@ export class ObjectMetadataController {
workspaceId,
});
const fields = await this.fieldMetadataRepository.find({
where: { objectMetadataId: flatObject.id, workspaceId },
});
const [fields, uniqueFieldMetadataIds] = await Promise.all([
this.fieldMetadataRepository.find({
where: { objectMetadataId: flatObject.id, workspaceId },
}),
this.loadUniqueFieldMetadataIds(workspaceId),
]);
const result: ObjectMetadataWithFieldsDTO = {
...fromFlatObjectMetadataToObjectMetadataDto(flatObject),
fields: fields.map(fromFieldMetadataEntityToFieldMetadataDto),
fields: fields.map((field) =>
fromFieldMetadataEntityToFieldMetadataDto(
field,
uniqueFieldMetadataIds,
),
),
};
return (await this.isNewMetadataFormat(workspaceId))
@@ -262,10 +306,16 @@ export class ObjectMetadataController {
private toObjectWithFieldsDto(
object: ObjectMetadataEntity,
fields: FieldMetadataEntity[],
uniqueFieldMetadataIds: ReadonlySet<string>,
): ObjectMetadataWithFieldsDTO {
return {
...fromObjectMetadataEntityToObjectMetadataDto(object),
fields: fields.map(fromFieldMetadataEntityToFieldMetadataDto),
fields: fields.map((field) =>
fromFieldMetadataEntityToFieldMetadataDto(
field,
uniqueFieldMetadataIds,
),
),
};
}
}
@@ -31,6 +31,7 @@ export const buildDefaultIndexesForCustomObject = ({
indexMetadataUniversalIdentifier:
tsFlatVectorIndexUniversalIdentifier,
order: 0,
subFieldName: null,
updatedAt: createdAt.toISOString(),
},
],
@@ -17,7 +17,7 @@ import { workspaceMigrationBuilderGraphqlApiExceptionHandler } from 'src/engine/
export const objectMetadataGraphqlApiExceptionHandler = (error: Error) => {
if (error instanceof WorkspaceMigrationBuilderException) {
workspaceMigrationBuilderGraphqlApiExceptionHandler(error);
return workspaceMigrationBuilderGraphqlApiExceptionHandler(error);
}
if (error instanceof InvalidMetadataException) {
@@ -4,7 +4,6 @@ export type WorkspaceSchemaColumnDefinition = {
isNullable?: boolean;
default?: string | number | boolean | null;
isPrimary?: boolean;
isUnique?: boolean;
isArray?: boolean;
asExpression?: string;
generatedType?: 'STORED' | 'VIRTUAL';
@@ -108,6 +108,7 @@ export const createStandardIndexFlatMetadata = <
) => ({
createdAt: now,
order: index,
subFieldName: null,
updatedAt: now,
fieldMetadataUniversalIdentifier,
indexMetadataUniversalIdentifier:
@@ -130,6 +131,7 @@ export const createStandardIndexFlatMetadata = <
id: v4(),
indexMetadataId: indexId,
order: index,
subFieldName: null,
updatedAt: now,
workspaceId,
}),
@@ -4,6 +4,7 @@ import { msg, t } from '@lingui/core/macro';
import { ALL_METADATA_NAME } from 'twenty-shared/metadata';
import {
FieldMetadataType,
RelationType,
compositeTypeDefinitions,
} from 'twenty-shared/types';
import { isDefined } from 'twenty-shared/utils';
@@ -11,6 +12,7 @@ import { isDefined } from 'twenty-shared/utils';
import { FlatEntityMapsExceptionCode } from 'src/engine/metadata-modules/flat-entity/exceptions/flat-entity-maps.exception';
import { findFlatEntityByUniversalIdentifier } from 'src/engine/metadata-modules/flat-entity/utils/find-flat-entity-by-universal-identifier.util';
import { isCompositeUniversalFlatFieldMetadata } from 'src/engine/metadata-modules/flat-field-metadata/utils/is-composite-flat-field-metadata.util';
import { isMorphOrRelationUniversalFlatFieldMetadata } from 'src/engine/metadata-modules/flat-field-metadata/utils/is-morph-or-relation-flat-field-metadata.util';
import { IndexExceptionCode } from 'src/engine/metadata-modules/flat-index-metadata/exceptions/index-exception-code';
import { FailedFlatEntityValidation } from 'src/engine/workspace-manager/workspace-migration/workspace-migration-builder/builders/types/failed-flat-entity-validation.type';
import { getEmptyFlatEntityValidationError } from 'src/engine/workspace-manager/workspace-migration/workspace-migration-builder/builders/utils/get-flat-entity-validation-error.util';
@@ -184,11 +186,18 @@ export class FlatIndexValidatorService {
(property) => property.isIncludedInUniqueConstraint,
);
// MORPH_RELATION resolves to multiple join columns and can't
// satisfy a single-column UNIQUE constraint. RELATION is only
// accepted as MANY_TO_ONE: that side owns a single join column
// (UUID) which Postgres can uniquely index.
const isUnindexableRelation =
isMorphOrRelationUniversalFlatFieldMetadata(relatedFlatField) &&
(relatedFlatField.type === FieldMetadataType.MORPH_RELATION ||
relatedFlatField.universalSettings?.relationType !==
RelationType.MANY_TO_ONE);
if (
[
FieldMetadataType.MORPH_RELATION,
FieldMetadataType.RELATION,
].includes(relatedFlatField.type) ||
isUnindexableRelation ||
isCompositeFieldWithNonIncludedUniqueConstraint
) {
const fieldType = relatedFlatField.type;
@@ -138,7 +138,21 @@ export class UpdateFieldActionHandlerService extends WorkspaceMigrationRunnerAct
const { entityId, update } = flatAction;
await fieldMetadataRepository.update({ id: entityId, workspaceId }, update);
// isUnique is derived from IndexMetadata at cache build time and has
// no underlying column on fieldMetadata. It travels in the update
// payload only so per-type validators (e.g. FILES rejection) can run
// — the actual state change is handled by the side-effect index
// create/delete in handleIndexChangesDuringFieldUpdate.
const { isUnique: _droppedIsUnique, ...persistedUpdate } = update;
if (Object.keys(persistedUpdate).length === 0) {
return;
}
await fieldMetadataRepository.update(
{ id: entityId, workspaceId },
persistedUpdate,
);
}
async executeForWorkspaceSchema(
@@ -59,6 +59,7 @@ export const fromUniversalFlatIndexToFlatIndex = ({
indexMetadataId,
fieldMetadataId: fieldMetadata.id,
order: universalFlatIndexFieldMetadata.order,
subFieldName: universalFlatIndexFieldMetadata.subFieldName,
createdAt: now,
updatedAt: now,
workspaceId,
@@ -28,57 +28,81 @@ export const computeFlatIndexFieldColumnNames = ({
flatIndexFieldMetadatas: FlatIndexFieldMetadata[];
flatFieldMetadataMaps: MetadataFlatEntityMaps<'fieldMetadata'>;
}): string[] => {
return flatIndexFieldMetadatas.flatMap(({ fieldMetadataId }) => {
const flatFieldMetadata = findFlatEntityByIdInFlatEntityMapsOrThrow({
flatEntityId: fieldMetadataId,
flatEntityMaps: flatFieldMetadataMaps,
});
return flatIndexFieldMetadatas.flatMap(
({ fieldMetadataId, subFieldName }) => {
const flatFieldMetadata = findFlatEntityByIdInFlatEntityMapsOrThrow({
flatEntityId: fieldMetadataId,
flatEntityMaps: flatFieldMetadataMaps,
});
if (!isDefined(flatFieldMetadata)) {
throw new FlatEntityMapsException(
'Index field related field metadata not found',
FlatEntityMapsExceptionCode.ENTITY_NOT_FOUND,
);
}
if (isMorphOrRelationFlatFieldMetadata(flatFieldMetadata)) {
if (
flatFieldMetadata.settings?.relationType !== RelationType.MANY_TO_ONE
) {
if (!isDefined(flatFieldMetadata)) {
throw new FlatEntityMapsException(
'Cannot index a relation field that has no join column',
'Index field related field metadata not found',
FlatEntityMapsExceptionCode.ENTITY_NOT_FOUND,
);
}
return computeMorphOrRelationFieldJoinColumnName({
name: flatFieldMetadata.name,
});
}
if (isMorphOrRelationFlatFieldMetadata(flatFieldMetadata)) {
if (
flatFieldMetadata.settings?.relationType !== RelationType.MANY_TO_ONE
) {
throw new FlatEntityMapsException(
'Cannot index a relation field that has no join column',
FlatEntityMapsExceptionCode.ENTITY_NOT_FOUND,
);
}
if (isCompositeFieldMetadataType(flatFieldMetadata.type)) {
const compositeType = compositeTypeDefinitions.get(
flatFieldMetadata.type,
);
return computeMorphOrRelationFieldJoinColumnName({
name: flatFieldMetadata.name,
});
}
if (!compositeType) {
throw new FlatEntityMapsException(
'Composite type not found',
FlatEntityMapsExceptionCode.INTERNAL_SERVER_ERROR,
if (isCompositeFieldMetadataType(flatFieldMetadata.type)) {
const compositeType = compositeTypeDefinitions.get(
flatFieldMetadata.type,
);
if (!compositeType) {
throw new FlatEntityMapsException(
'Composite type not found',
FlatEntityMapsExceptionCode.INTERNAL_SERVER_ERROR,
);
}
if (isDefined(subFieldName)) {
const property = compositeType.properties.find(
(compositeProperty) => compositeProperty.name === subFieldName,
);
if (!isDefined(property)) {
throw new FlatEntityMapsException(
`Composite sub-field "${subFieldName}" not found on ${flatFieldMetadata.name}`,
FlatEntityMapsExceptionCode.ENTITY_NOT_FOUND,
);
}
return [
computeCompositeColumnName(
{ name: flatFieldMetadata.name, type: flatFieldMetadata.type },
property,
),
];
}
// System indexes (no subFieldName) project the composite parent onto
// every property flagged isIncludedInUniqueConstraint.
const uniqueCompositeProperties = compositeType.properties.filter(
(property) => property.isIncludedInUniqueConstraint,
);
return uniqueCompositeProperties.map((subField) =>
computeCompositeColumnName(flatFieldMetadata.name, subField),
);
}
const uniqueCompositeProperties = compositeType.properties.filter(
(property) => property.isIncludedInUniqueConstraint,
);
return uniqueCompositeProperties.map((subField) =>
computeCompositeColumnName(flatFieldMetadata.name, subField),
);
}
return flatFieldMetadata.name;
});
return flatFieldMetadata.name;
},
);
};
export const deleteIndexMetadata = async ({
@@ -59,7 +59,6 @@ describe('Generate Column Definitions', () => {
isArray: false,
isNullable: true,
isPrimary: false,
isUnique: false,
default: 'NULL',
});
});
@@ -104,7 +103,6 @@ describe('Generate Column Definitions', () => {
isArray: true,
isNullable: true,
isPrimary: false,
isUnique: false,
default: 'NULL',
});
});
@@ -157,7 +155,6 @@ describe('Generate Column Definitions', () => {
type: 'uuid',
isNullable: true,
isPrimary: false,
isUnique: false,
default: null,
isArray: false,
});
@@ -199,7 +196,6 @@ describe('Generate Column Definitions', () => {
columns.forEach((column) => {
expect(column.isNullable).toBe(true);
expect(column.isPrimary).toBe(false);
expect(column.isUnique).toBe(false);
expect(column.default).toBe('NULL');
});
});
@@ -233,7 +229,6 @@ describe('Generate Column Definitions', () => {
type: 'numeric',
isNullable: true,
isPrimary: false,
isUnique: false,
default: "'100000000'::numeric",
});
@@ -242,7 +237,6 @@ describe('Generate Column Definitions', () => {
type: 'text',
isNullable: true,
isPrimary: false,
isUnique: false,
default: "'USD'::text",
});
});
@@ -354,7 +348,6 @@ describe('Generate Column Definitions', () => {
type: 'text',
isNullable: true,
isPrimary: false,
isUnique: false,
default: 'NULL',
isArray: false,
},
@@ -382,7 +375,6 @@ describe('Generate Column Definitions', () => {
type: 'boolean',
isNullable: true,
isPrimary: false,
isUnique: false,
default: "'true'::boolean",
isArray: false,
},
@@ -411,7 +403,6 @@ describe('Generate Column Definitions', () => {
type: 'text',
isNullable: true,
isPrimary: false,
isUnique: false,
default: 'NULL',
isArray: false,
},
@@ -438,7 +429,6 @@ describe('Generate Column Definitions', () => {
type: 'uuid',
isNullable: true,
isPrimary: false,
isUnique: false,
default: 'NULL',
isArray: false,
},
@@ -89,7 +89,6 @@ export const generateCompositeColumnDefinition = ({
: columnType,
isNullable:
parentFlatFieldMetadata.isNullable || !compositeProperty.isRequired,
isUnique: parentFlatFieldMetadata.isUnique ?? false,
default: serializedDefaultValue,
isArray: isArrayFlag,
isPrimary: false,
@@ -108,7 +107,6 @@ const generateTsVectorColumnDefinition = (
type: fieldMetadataTypeToColumnType(flatFieldMetadata.type),
isNullable: true,
isArray: false,
isUnique: false,
default: null,
asExpression: flatFieldMetadata.settings?.asExpression ?? undefined,
generatedType: flatFieldMetadata.settings?.generatedType ?? undefined,
@@ -134,7 +132,6 @@ const generateRelationColumnDefinition = (
type: fieldMetadataTypeToColumnType(FieldMetadataType.UUID),
isNullable: true,
isArray: false,
isUnique: false,
default: null,
isPrimary: false,
};
@@ -171,7 +168,6 @@ const generateColumnDefinition = ({
isArray:
flatFieldMetadata.type === FieldMetadataType.ARRAY ||
flatFieldMetadata.type === FieldMetadataType.MULTI_SELECT,
isUnique: flatFieldMetadata.isUnique ?? false,
default: serializedDefaultValue,
isPrimary: flatFieldMetadata.name === 'id',
};