feat(data-model): custom-indexes management UI and mutations (#20846)

## Summary

Brings indexes management into the per-object Settings tab as a section
under Search (no feature flag, advanced mode only). Admins can create /
delete non-unique indexes with the UI; apps can declare indexes in code
with `defineIndex`. Composite-typed fields are now indexable by picking
a specific sub-column (e.g. `Address > City`).

A few related polish items also land here (invite-user dropdown lands on
the Invite tab; standard warning callout above the new-index form).

## What ships

### UI — custom indexes on per-object Settings
- New section directly under Search, wrapped in
`AdvancedSettingsWrapper`.
- Filter dropdown on the search bar toggles system-index visibility
(shown by default since advanced mode).
- **+ Add Index** button (disabled with tooltip once the per-object cap
is reached) navigates to a dedicated `SettingsObjectNewIndex` page
(matches the field-creation pattern, not a modal):
- Field picker mirrors the webhook event-form layout (rows of dropdowns,
implicit trailing empty row).
- Composite fields surface their sub-properties (`Address > City`,
`Currency > Amount`, …).
  - BTREE / GIN type selector.
- Standard warning Callout: "Use indexes sparingly — each one speeds
reads but slows writes."
- Trash icon on `isCustom: true` rows → confirmation modal →
`deleteOneIndex`.

### Server — `createOneIndex` / `deleteOneIndex` mutations
- Gated by `SettingsPermissionGuard(DATA_MODEL)`.
- `IndexMetadataService` wraps the existing migration runner via
`WorkspaceMigrationValidateBuildAndRunService` so the metadata row and
the SQL index land atomically.
- Validation: rejects empty fields, duplicate `(fieldMetadataId,
subFieldName)` pairs, fields not on the object, requires `subFieldName`
for composite parents, forbids `subFieldName` on scalar/relation,
enforces `MAX_CUSTOM_INDEXES_PER_OBJECT = 10`.
- Delete refuses on `isCustom: false` rows so system indexes can't be
removed via this API.
- Dedicated GraphQL exception handler maps each typed error to the right
transport error class.

### Composite sub-field indexing
- Adds `subFieldName: string | null` column to
`IndexFieldMetadataEntity` (fast instance command).
- The flat-entity flow (`UniversalFlatIndexFieldMetadata`,
`FlatIndexFieldMetadata`, `from-universal-flat-index-to-flat-index`,
runner column resolution) all carry `subFieldName` through.
- For composite parents, the runner uses
`computeCompositeColumnName({...}, property)` for the picked sub-column;
for non-composite parents, behavior is unchanged.
- The `'::'` separator encodes `(fieldMetadataId, subFieldName)` for
dedup on the wire; the frontend uses the same separator inside the
Select component's string value.

### Apps can declare indexes in code (`defineIndex`)
- New `IndexManifest` + `IndexFieldManifest` types in
`twenty-shared/application` wired into the `Manifest` type.
- `defineIndex` SDK helper + `IndexConfig`. CLI manifest builder +
extractor recognize `defineIndex` / `ManifestEntityKey.Indexes`.
- Server: `from-index-manifest-to-universal-flat-index` converter
resolves field IDs, validates composite/scalar `subFieldName` rules, and
delegates to `generateFlatIndexMetadataWithNameOrThrow` for the
deterministic name.
- Orchestrator wires the loop after the field-resolution pass;
per-object cap enforced inline against the manifest.
- Cascade on uninstall is automatic — when an app disappears its indexes
drop with it (universal-flat-entity diff handles it).
- Rich-app fixture ships a real `defineIndex` on `PostCard.status`,
exercising the full manifest → install path in CI.

### Closed for now (open later if needed)
- Apps cannot declare `isUnique` indexes — unique constraints stay with
the field-creation flow.
- Apps cannot use a partial-`indexWhereClause` — the UI surface keeps
the framework's hardcoded allowlist.
- UI cannot create unique or partial indexes either; same reasons.

### Cleanups along the way
- Reused the existing `getCompositeSubFieldLabel` +
`COMPOSITE_FIELD_SUB_FIELD_LABELS` (deleted the duplicates I'd created
early in the PR).
- Moved `MAX_CUSTOM_INDEXES_PER_OBJECT` to `twenty-shared/constants`
(single source for FE + BE).
- Replaced inline `isDefined(x) && x !== ''` with `isNonEmptyString`
(from `@sniptt/guards`).
- Hoisted the per-object fields Map + inlined the cap counter into the
indexes orchestrator loop (drops the install scan from O(indexes ×
totalFields) to O(totalFields + indexes)).
- Per design-feedback: page-based create flow (not a modal), filter
dropdown on the SearchInput (not a separate toggle), webhook-style
picker, field icons.

### Unrelated polish that lands here
- "Invite user" link in the multi-workspace dropdown now lands on the
Invite tab directly (`#invite`) instead of the first tab of the members
page.

## Test plan
- [ ] `npx nx typecheck twenty-server / twenty-front / twenty-sdk /
twenty-shared` — passes
- [ ] `npx nx lint:diff-with-main twenty-server / twenty-front` — clean
- [ ] `npx jest index-metadata.service.spec` — green
- [ ] `npx jest from-index-manifest-to-universal-flat-index` — green
(new converter spec, 8 cases)
- [ ] `npx vitest run
src/sdk/define/indexes/__tests__/define-index.spec.ts` (twenty-sdk) —
green (6 cases)
- [ ] `npx vitest run --config vitest.integration.config.ts -t
"rich-app"` — green (rich-app app-dev integration exercises the new
manifest path with the PostCard.status index)
- [ ] Advanced mode → Settings → any object → Settings tab → Indexes
section is visible under Search
- [ ] Create a single-field BTREE index, confirm SQL index exists
(verify via `pg_indexes`)
- [ ] Create a composite-field index (`Address > City`) and confirm the
column is `addressAddressCity`
- [ ] Create an index spanning two columns; column order matches the
picker order
- [ ] Attempt to create an 11th custom index → button is disabled with
tooltip
- [ ] Delete a custom index → confirmation modal → row disappears, PG
index dropped
- [ ] System indexes have no trash icon and are hidden by default
This commit is contained in:
Félix Malfait
2026-05-25 17:47:09 +02:00
committed by GitHub
parent 69d89f8cfc
commit d602f35cbd
98 changed files with 3742 additions and 387 deletions
@@ -0,0 +1,280 @@
import { FieldMetadataType } from 'twenty-shared/types';
import { fromIndexManifestToUniversalFlatIndex } from 'src/engine/core-modules/application/application-manifest/converters/from-index-manifest-to-universal-flat-index.util';
import { IndexType } from 'src/engine/metadata-modules/index-metadata/types/indexType.types';
import { type UniversalFlatFieldMetadata } from 'src/engine/workspace-manager/workspace-migration/universal-flat-entity/types/universal-flat-field-metadata.type';
import { type UniversalFlatObjectMetadata } from 'src/engine/workspace-manager/workspace-migration/universal-flat-entity/types/universal-flat-object-metadata.type';
describe('fromIndexManifestToUniversalFlatIndex', () => {
const now = '2026-01-01T00:00:00.000Z';
const applicationUniversalIdentifier = 'app-uuid-1';
const flatObjectMetadata = {
universalIdentifier: 'obj-uuid-1',
nameSingular: 'company',
isCustom: false,
} as UniversalFlatObjectMetadata;
const scalarField = {
universalIdentifier: 'field-uuid-scalar',
name: 'industry',
type: FieldMetadataType.TEXT,
} as UniversalFlatFieldMetadata;
const compositeField = {
universalIdentifier: 'field-uuid-address',
name: 'address',
type: FieldMetadataType.ADDRESS,
} as UniversalFlatFieldMetadata;
const baseManifest = {
universalIdentifier: 'idx-uuid-1',
objectUniversalIdentifier: 'obj-uuid-1',
indexType: 'BTREE' as const,
};
it('builds a flat index from a scalar field', () => {
const result = fromIndexManifestToUniversalFlatIndex({
indexManifest: {
...baseManifest,
fields: [
{
universalIdentifier: 'field-entry-1',
fieldUniversalIdentifier: scalarField.universalIdentifier,
},
],
},
flatObjectMetadata,
objectFlatFieldMetadatas: [scalarField],
applicationUniversalIdentifier,
now,
});
expect(result.universalIdentifier).toBe('idx-uuid-1');
expect(result.applicationUniversalIdentifier).toBe(
applicationUniversalIdentifier,
);
expect(result.indexType).toBe(IndexType.BTREE);
expect(result.isUnique).toBe(false);
expect(result.indexWhereClause).toBeNull();
expect(result.isCustom).toBe(false);
expect(result.universalFlatIndexFieldMetadatas).toHaveLength(1);
expect(result.universalFlatIndexFieldMetadatas[0]).toMatchObject({
order: 0,
subFieldName: null,
fieldMetadataUniversalIdentifier: scalarField.universalIdentifier,
indexMetadataUniversalIdentifier: 'idx-uuid-1',
});
expect(result.name).toMatch(/^IDX_/);
});
it('builds a flat index from a composite field with subFieldName', () => {
const result = fromIndexManifestToUniversalFlatIndex({
indexManifest: {
...baseManifest,
fields: [
{
universalIdentifier: 'field-entry-1',
fieldUniversalIdentifier: compositeField.universalIdentifier,
subFieldName: 'addressCity',
},
],
},
flatObjectMetadata,
objectFlatFieldMetadatas: [compositeField],
applicationUniversalIdentifier,
now,
});
expect(result.universalFlatIndexFieldMetadatas[0].subFieldName).toBe(
'addressCity',
);
});
it('forwards isUnique from the manifest', () => {
const result = fromIndexManifestToUniversalFlatIndex({
indexManifest: {
...baseManifest,
isUnique: true,
fields: [
{
universalIdentifier: 'field-entry-1',
fieldUniversalIdentifier: scalarField.universalIdentifier,
},
],
},
flatObjectMetadata,
objectFlatFieldMetadatas: [scalarField],
applicationUniversalIdentifier,
now,
});
expect(result.isUnique).toBe(true);
});
it('throws when a composite field is referenced without a subFieldName', () => {
expect(() =>
fromIndexManifestToUniversalFlatIndex({
indexManifest: {
...baseManifest,
fields: [
{
universalIdentifier: 'field-entry-1',
fieldUniversalIdentifier: compositeField.universalIdentifier,
},
],
},
flatObjectMetadata,
objectFlatFieldMetadatas: [compositeField],
applicationUniversalIdentifier,
now,
}),
).toThrow(/requires a subFieldName/);
});
it('throws when a scalar field is given a subFieldName', () => {
expect(() =>
fromIndexManifestToUniversalFlatIndex({
indexManifest: {
...baseManifest,
fields: [
{
universalIdentifier: 'field-entry-1',
fieldUniversalIdentifier: scalarField.universalIdentifier,
subFieldName: 'something',
},
],
},
flatObjectMetadata,
objectFlatFieldMetadatas: [scalarField],
applicationUniversalIdentifier,
now,
}),
).toThrow(/is not composite/);
});
it('throws when a composite sub-field is unknown', () => {
expect(() =>
fromIndexManifestToUniversalFlatIndex({
indexManifest: {
...baseManifest,
fields: [
{
universalIdentifier: 'field-entry-1',
fieldUniversalIdentifier: compositeField.universalIdentifier,
subFieldName: 'unknownSubField',
},
],
},
flatObjectMetadata,
objectFlatFieldMetadatas: [compositeField],
applicationUniversalIdentifier,
now,
}),
).toThrow(/not found on composite field/);
});
it('throws when a referenced field does not exist on the object', () => {
expect(() =>
fromIndexManifestToUniversalFlatIndex({
indexManifest: {
...baseManifest,
fields: [
{
universalIdentifier: 'field-entry-1',
fieldUniversalIdentifier: 'missing-field-uuid',
},
],
},
flatObjectMetadata,
objectFlatFieldMetadatas: [scalarField],
applicationUniversalIdentifier,
now,
}),
).toThrow(/references unknown field/);
});
it('throws when the same (field, subFieldName) pair appears twice', () => {
expect(() =>
fromIndexManifestToUniversalFlatIndex({
indexManifest: {
...baseManifest,
fields: [
{
universalIdentifier: 'field-entry-1',
fieldUniversalIdentifier: scalarField.universalIdentifier,
},
{
universalIdentifier: 'field-entry-2',
fieldUniversalIdentifier: scalarField.universalIdentifier,
},
],
},
flatObjectMetadata,
objectFlatFieldMetadatas: [scalarField],
applicationUniversalIdentifier,
now,
}),
).toThrow(/same column twice/);
});
it('throws when GIN is requested on a scalar (non-GIN-compatible) field', () => {
expect(() =>
fromIndexManifestToUniversalFlatIndex({
indexManifest: {
...baseManifest,
indexType: 'GIN',
fields: [
{
universalIdentifier: 'field-entry-1',
fieldUniversalIdentifier: scalarField.universalIdentifier,
},
],
},
flatObjectMetadata,
objectFlatFieldMetadatas: [scalarField],
applicationUniversalIdentifier,
now,
}),
).toThrow(/GIN index does not support/);
});
it('accepts GIN on a TS_VECTOR field', () => {
const tsVectorField = {
universalIdentifier: 'field-uuid-tsvector',
name: 'searchVector',
type: FieldMetadataType.TS_VECTOR,
} as UniversalFlatFieldMetadata;
const result = fromIndexManifestToUniversalFlatIndex({
indexManifest: {
...baseManifest,
indexType: 'GIN',
fields: [
{
universalIdentifier: 'field-entry-1',
fieldUniversalIdentifier: tsVectorField.universalIdentifier,
},
],
},
flatObjectMetadata,
objectFlatFieldMetadatas: [tsVectorField],
applicationUniversalIdentifier,
now,
});
expect(result.indexType).toBe(IndexType.GIN);
});
it('throws when fields is empty', () => {
expect(() =>
fromIndexManifestToUniversalFlatIndex({
indexManifest: { ...baseManifest, fields: [] },
flatObjectMetadata,
objectFlatFieldMetadatas: [],
applicationUniversalIdentifier,
now,
}),
).toThrow(/at least one field/);
});
});
@@ -0,0 +1,135 @@
import { isNonEmptyString } from '@sniptt/guards';
import { type IndexManifest } from 'twenty-shared/application';
import {
compositeTypeDefinitions,
type FieldMetadataType,
} from 'twenty-shared/types';
import { isDefined } from 'twenty-shared/utils';
import { isCompositeFieldMetadataType } from 'src/engine/metadata-modules/field-metadata/utils/is-composite-field-metadata-type.util';
import { IndexType } from 'src/engine/metadata-modules/index-metadata/types/indexType.types';
import { generateFlatIndexMetadataWithNameOrThrow } from 'src/engine/metadata-modules/index-metadata/utils/generate-flat-index.util';
import { validateIndexTypeAgainstFieldsOrThrow } from 'src/engine/metadata-modules/index-metadata/utils/validate-index-type-against-fields.util';
import { type UniversalFlatFieldMetadata } from 'src/engine/workspace-manager/workspace-migration/universal-flat-entity/types/universal-flat-field-metadata.type';
import { type UniversalFlatIndexMetadata } from 'src/engine/workspace-manager/workspace-migration/universal-flat-entity/types/universal-flat-index-metadata.type';
import { type UniversalFlatObjectMetadata } from 'src/engine/workspace-manager/workspace-migration/universal-flat-entity/types/universal-flat-object-metadata.type';
export const fromIndexManifestToUniversalFlatIndex = ({
indexManifest,
flatObjectMetadata,
objectFlatFieldMetadatas,
applicationUniversalIdentifier,
now,
}: {
indexManifest: IndexManifest;
flatObjectMetadata: UniversalFlatObjectMetadata;
objectFlatFieldMetadatas: UniversalFlatFieldMetadata[];
applicationUniversalIdentifier: string;
now: string;
}): UniversalFlatIndexMetadata => {
if (indexManifest.fields.length === 0) {
throw new Error(
`Index "${indexManifest.universalIdentifier}" must reference at least one field`,
);
}
const dedupKeys = indexManifest.fields.map(
(entry) => `${entry.fieldUniversalIdentifier}::${entry.subFieldName ?? ''}`,
);
if (new Set(dedupKeys).size !== dedupKeys.length) {
throw new Error(
`Index "${indexManifest.universalIdentifier}" lists the same column twice`,
);
}
const resolvedIndexType = (indexManifest.indexType ?? 'BTREE') as IndexType;
const resolvedFieldsForValidation: Array<{
type: FieldMetadataType;
name: string;
label: string;
subFieldName: string | null;
}> = [];
const universalFlatIndexFieldMetadatas = indexManifest.fields.map(
(entry, order) => {
const flatField = objectFlatFieldMetadatas.find(
(candidate) =>
candidate.universalIdentifier === entry.fieldUniversalIdentifier,
);
if (!isDefined(flatField)) {
throw new Error(
`Index "${indexManifest.universalIdentifier}" references unknown field ${entry.fieldUniversalIdentifier} on object ${flatObjectMetadata.universalIdentifier}`,
);
}
const isComposite = isCompositeFieldMetadataType(flatField.type);
if (isComposite) {
if (!isNonEmptyString(entry.subFieldName)) {
throw new Error(
`Composite field "${flatField.name}" requires a subFieldName in index "${indexManifest.universalIdentifier}"`,
);
}
const property = compositeTypeDefinitions
.get(flatField.type)
?.properties.find(
(compositeProperty) =>
compositeProperty.name === entry.subFieldName,
);
if (!isDefined(property)) {
throw new Error(
`Sub-field "${entry.subFieldName}" not found on composite field "${flatField.name}" in index "${indexManifest.universalIdentifier}"`,
);
}
} else if (isNonEmptyString(entry.subFieldName)) {
throw new Error(
`Field "${flatField.name}" is not composite — subFieldName must be omitted in index "${indexManifest.universalIdentifier}"`,
);
}
const subFieldName = isComposite ? (entry.subFieldName ?? null) : null;
resolvedFieldsForValidation.push({
type: flatField.type,
name: flatField.name,
label: flatField.label,
subFieldName,
});
return {
createdAt: now,
updatedAt: now,
order,
subFieldName,
fieldMetadataUniversalIdentifier: flatField.universalIdentifier,
indexMetadataUniversalIdentifier: indexManifest.universalIdentifier,
};
},
);
validateIndexTypeAgainstFieldsOrThrow({
indexType: resolvedIndexType,
fields: resolvedFieldsForValidation,
});
return generateFlatIndexMetadataWithNameOrThrow({
flatObjectMetadata,
objectFlatFieldMetadatas,
flatIndex: {
createdAt: now,
updatedAt: now,
universalIdentifier: indexManifest.universalIdentifier,
applicationUniversalIdentifier,
objectMetadataUniversalIdentifier: flatObjectMetadata.universalIdentifier,
indexType: resolvedIndexType,
indexWhereClause: null,
isCustom: false,
isUnique: indexManifest.isUnique ?? false,
universalFlatIndexFieldMetadatas,
},
});
};
@@ -1,4 +1,5 @@
import { type Manifest } from 'twenty-shared/application';
import { MAX_CUSTOM_INDEXES_PER_OBJECT } from 'twenty-shared/constants';
import { FieldMetadataType } from 'twenty-shared/types';
import { isDefined } from 'twenty-shared/utils';
@@ -10,6 +11,7 @@ import { fromConnectionProviderManifestToUniversalFlatConnectionProvider } from
import { fromFieldManifestToUniversalFlatFieldMetadata } from 'src/engine/core-modules/application/application-manifest/converters/from-field-manifest-to-universal-flat-field-metadata.util';
import { fromFieldPermissionManifestToUniversalFlatFieldPermission } from 'src/engine/core-modules/application/application-manifest/converters/from-field-permission-manifest-to-universal-flat-field-permission.util';
import { fromFrontComponentManifestToUniversalFlatFrontComponent } from 'src/engine/core-modules/application/application-manifest/converters/from-front-component-manifest-to-universal-flat-front-component.util';
import { fromIndexManifestToUniversalFlatIndex } from 'src/engine/core-modules/application/application-manifest/converters/from-index-manifest-to-universal-flat-index.util';
import { fromLogicFunctionManifestToUniversalFlatLogicFunction } from 'src/engine/core-modules/application/application-manifest/converters/from-logic-function-manifest-to-universal-flat-logic-function.util';
import { fromNavigationMenuItemManifestToUniversalFlatNavigationMenuItem } from 'src/engine/core-modules/application/application-manifest/converters/from-navigation-menu-item-manifest-to-universal-flat-navigation-menu-item.util';
import { fromObjectManifestToUniversalFlatObjectMetadata } from 'src/engine/core-modules/application/application-manifest/converters/from-object-manifest-to-universal-flat-object-metadata.util';
@@ -32,6 +34,7 @@ import { type FlatApplication } from 'src/engine/core-modules/application/types/
import { fromAgentManifestToUniversalFlatAgent } from 'src/engine/core-modules/application/utils/from-agent-manifest-to-universal-flat-agent.util';
import { createEmptyAllFlatEntityMaps } from 'src/engine/metadata-modules/flat-entity/constant/create-empty-all-flat-entity-maps.constant';
import { type AllFlatEntityMaps } from 'src/engine/metadata-modules/flat-entity/types/all-flat-entity-maps.type';
import { type UniversalFlatFieldMetadata } from 'src/engine/workspace-manager/workspace-migration/universal-flat-entity/types/universal-flat-field-metadata.type';
import { addUniversalFlatEntityToUniversalFlatEntityMapsThroughMutationOrThrow } from 'src/engine/workspace-manager/workspace-migration/universal-flat-entity/utils/add-universal-flat-entity-to-universal-flat-entity-maps-through-mutation-or-throw.util';
export const computeApplicationManifestAllUniversalFlatEntityMaps = ({
@@ -135,6 +138,75 @@ export const computeApplicationManifestAllUniversalFlatEntityMaps = ({
}
}
const indexCountByObjectUniversalIdentifier = new Map<string, number>();
const fieldsByObjectUniversalIdentifier = new Map<
string,
UniversalFlatFieldMetadata[]
>();
for (const flatField of Object.values(
allUniversalFlatEntityMaps.flatFieldMetadataMaps.byUniversalIdentifier,
)) {
if (!isDefined(flatField)) continue;
const bucket =
fieldsByObjectUniversalIdentifier.get(
flatField.objectMetadataUniversalIdentifier,
) ?? [];
if (bucket.length === 0) {
fieldsByObjectUniversalIdentifier.set(
flatField.objectMetadataUniversalIdentifier,
bucket,
);
}
bucket.push(flatField);
}
for (const indexManifest of manifest.indexes ?? []) {
const flatObjectMetadata =
allUniversalFlatEntityMaps.flatObjectMetadataMaps.byUniversalIdentifier[
indexManifest.objectUniversalIdentifier
];
if (!isDefined(flatObjectMetadata)) {
throw new Error(
`Index "${indexManifest.universalIdentifier}" references unknown object ${indexManifest.objectUniversalIdentifier}`,
);
}
const nextCount =
(indexCountByObjectUniversalIdentifier.get(
indexManifest.objectUniversalIdentifier,
) ?? 0) + 1;
if (nextCount > MAX_CUSTOM_INDEXES_PER_OBJECT) {
throw new Error(
`Application declares more than ${MAX_CUSTOM_INDEXES_PER_OBJECT} indexes on object ${indexManifest.objectUniversalIdentifier}`,
);
}
indexCountByObjectUniversalIdentifier.set(
indexManifest.objectUniversalIdentifier,
nextCount,
);
addUniversalFlatEntityToUniversalFlatEntityMapsThroughMutationOrThrow({
universalFlatEntity: fromIndexManifestToUniversalFlatIndex({
indexManifest,
flatObjectMetadata,
objectFlatFieldMetadatas:
fieldsByObjectUniversalIdentifier.get(
flatObjectMetadata.universalIdentifier,
) ?? [],
applicationUniversalIdentifier,
now,
}),
universalFlatEntityMapsToMutate: allUniversalFlatEntityMaps.flatIndexMaps,
});
}
for (const logicFunctionManifest of manifest.logicFunctions) {
addUniversalFlatEntityToUniversalFlatEntityMapsThroughMutationOrThrow({
universalFlatEntity: