feat(data-model): custom-indexes management UI and mutations (#20846)

## Summary

Brings indexes management into the per-object Settings tab as a section
under Search (no feature flag, advanced mode only). Admins can create /
delete non-unique indexes with the UI; apps can declare indexes in code
with `defineIndex`. Composite-typed fields are now indexable by picking
a specific sub-column (e.g. `Address > City`).

A few related polish items also land here (invite-user dropdown lands on
the Invite tab; standard warning callout above the new-index form).

## What ships

### UI — custom indexes on per-object Settings
- New section directly under Search, wrapped in
`AdvancedSettingsWrapper`.
- Filter dropdown on the search bar toggles system-index visibility
(shown by default since advanced mode).
- **+ Add Index** button (disabled with tooltip once the per-object cap
is reached) navigates to a dedicated `SettingsObjectNewIndex` page
(matches the field-creation pattern, not a modal):
- Field picker mirrors the webhook event-form layout (rows of dropdowns,
implicit trailing empty row).
- Composite fields surface their sub-properties (`Address > City`,
`Currency > Amount`, …).
  - BTREE / GIN type selector.
- Standard warning Callout: "Use indexes sparingly — each one speeds
reads but slows writes."
- Trash icon on `isCustom: true` rows → confirmation modal →
`deleteOneIndex`.

### Server — `createOneIndex` / `deleteOneIndex` mutations
- Gated by `SettingsPermissionGuard(DATA_MODEL)`.
- `IndexMetadataService` wraps the existing migration runner via
`WorkspaceMigrationValidateBuildAndRunService` so the metadata row and
the SQL index land atomically.
- Validation: rejects empty fields, duplicate `(fieldMetadataId,
subFieldName)` pairs, fields not on the object, requires `subFieldName`
for composite parents, forbids `subFieldName` on scalar/relation,
enforces `MAX_CUSTOM_INDEXES_PER_OBJECT = 10`.
- Delete refuses on `isCustom: false` rows so system indexes can't be
removed via this API.
- Dedicated GraphQL exception handler maps each typed error to the right
transport error class.

### Composite sub-field indexing
- Adds `subFieldName: string | null` column to
`IndexFieldMetadataEntity` (fast instance command).
- The flat-entity flow (`UniversalFlatIndexFieldMetadata`,
`FlatIndexFieldMetadata`, `from-universal-flat-index-to-flat-index`,
runner column resolution) all carry `subFieldName` through.
- For composite parents, the runner uses
`computeCompositeColumnName({...}, property)` for the picked sub-column;
for non-composite parents, behavior is unchanged.
- The `'::'` separator encodes `(fieldMetadataId, subFieldName)` for
dedup on the wire; the frontend uses the same separator inside the
Select component's string value.

### Apps can declare indexes in code (`defineIndex`)
- New `IndexManifest` + `IndexFieldManifest` types in
`twenty-shared/application` wired into the `Manifest` type.
- `defineIndex` SDK helper + `IndexConfig`. CLI manifest builder +
extractor recognize `defineIndex` / `ManifestEntityKey.Indexes`.
- Server: `from-index-manifest-to-universal-flat-index` converter
resolves field IDs, validates composite/scalar `subFieldName` rules, and
delegates to `generateFlatIndexMetadataWithNameOrThrow` for the
deterministic name.
- Orchestrator wires the loop after the field-resolution pass;
per-object cap enforced inline against the manifest.
- Cascade on uninstall is automatic — when an app disappears its indexes
drop with it (universal-flat-entity diff handles it).
- Rich-app fixture ships a real `defineIndex` on `PostCard.status`,
exercising the full manifest → install path in CI.

### Closed for now (open later if needed)
- Apps cannot declare `isUnique` indexes — unique constraints stay with
the field-creation flow.
- Apps cannot use a partial-`indexWhereClause` — the UI surface keeps
the framework's hardcoded allowlist.
- UI cannot create unique or partial indexes either; same reasons.

### Cleanups along the way
- Reused the existing `getCompositeSubFieldLabel` +
`COMPOSITE_FIELD_SUB_FIELD_LABELS` (deleted the duplicates I'd created
early in the PR).
- Moved `MAX_CUSTOM_INDEXES_PER_OBJECT` to `twenty-shared/constants`
(single source for FE + BE).
- Replaced inline `isDefined(x) && x !== ''` with `isNonEmptyString`
(from `@sniptt/guards`).
- Hoisted the per-object fields Map + inlined the cap counter into the
indexes orchestrator loop (drops the install scan from O(indexes ×
totalFields) to O(totalFields + indexes)).
- Per design-feedback: page-based create flow (not a modal), filter
dropdown on the SearchInput (not a separate toggle), webhook-style
picker, field icons.

### Unrelated polish that lands here
- "Invite user" link in the multi-workspace dropdown now lands on the
Invite tab directly (`#invite`) instead of the first tab of the members
page.

## Test plan
- [ ] `npx nx typecheck twenty-server / twenty-front / twenty-sdk /
twenty-shared` — passes
- [ ] `npx nx lint:diff-with-main twenty-server / twenty-front` — clean
- [ ] `npx jest index-metadata.service.spec` — green
- [ ] `npx jest from-index-manifest-to-universal-flat-index` — green
(new converter spec, 8 cases)
- [ ] `npx vitest run
src/sdk/define/indexes/__tests__/define-index.spec.ts` (twenty-sdk) —
green (6 cases)
- [ ] `npx vitest run --config vitest.integration.config.ts -t
"rich-app"` — green (rich-app app-dev integration exercises the new
manifest path with the PostCard.status index)
- [ ] Advanced mode → Settings → any object → Settings tab → Indexes
section is visible under Search
- [ ] Create a single-field BTREE index, confirm SQL index exists
(verify via `pg_indexes`)
- [ ] Create a composite-field index (`Address > City`) and confirm the
column is `addressAddressCity`
- [ ] Create an index spanning two columns; column order matches the
picker order
- [ ] Attempt to create an 11th custom index → button is disabled with
tooltip
- [ ] Delete a custom index → confirmation modal → row disappears, PG
index dropped
- [ ] System indexes have no trash icon and are hidden by default
This commit is contained in:
Félix Malfait
2026-05-25 17:47:09 +02:00
committed by GitHub
parent 69d89f8cfc
commit d602f35cbd
98 changed files with 3742 additions and 387 deletions
@@ -108,6 +108,7 @@ export const createStandardIndexFlatMetadata = <
) => ({
createdAt: now,
order: index,
subFieldName: null,
updatedAt: now,
fieldMetadataUniversalIdentifier,
indexMetadataUniversalIdentifier:
@@ -130,6 +131,7 @@ export const createStandardIndexFlatMetadata = <
id: v4(),
indexMetadataId: indexId,
order: index,
subFieldName: null,
updatedAt: now,
workspaceId,
}),
@@ -4,6 +4,7 @@ import { msg, t } from '@lingui/core/macro';
import { ALL_METADATA_NAME } from 'twenty-shared/metadata';
import {
FieldMetadataType,
RelationType,
compositeTypeDefinitions,
} from 'twenty-shared/types';
import { isDefined } from 'twenty-shared/utils';
@@ -11,6 +12,7 @@ import { isDefined } from 'twenty-shared/utils';
import { FlatEntityMapsExceptionCode } from 'src/engine/metadata-modules/flat-entity/exceptions/flat-entity-maps.exception';
import { findFlatEntityByUniversalIdentifier } from 'src/engine/metadata-modules/flat-entity/utils/find-flat-entity-by-universal-identifier.util';
import { isCompositeUniversalFlatFieldMetadata } from 'src/engine/metadata-modules/flat-field-metadata/utils/is-composite-flat-field-metadata.util';
import { isMorphOrRelationUniversalFlatFieldMetadata } from 'src/engine/metadata-modules/flat-field-metadata/utils/is-morph-or-relation-flat-field-metadata.util';
import { IndexExceptionCode } from 'src/engine/metadata-modules/flat-index-metadata/exceptions/index-exception-code';
import { FailedFlatEntityValidation } from 'src/engine/workspace-manager/workspace-migration/workspace-migration-builder/builders/types/failed-flat-entity-validation.type';
import { getEmptyFlatEntityValidationError } from 'src/engine/workspace-manager/workspace-migration/workspace-migration-builder/builders/utils/get-flat-entity-validation-error.util';
@@ -184,11 +186,18 @@ export class FlatIndexValidatorService {
(property) => property.isIncludedInUniqueConstraint,
);
// MORPH_RELATION resolves to multiple join columns and can't
// satisfy a single-column UNIQUE constraint. RELATION is only
// accepted as MANY_TO_ONE: that side owns a single join column
// (UUID) which Postgres can uniquely index.
const isUnindexableRelation =
isMorphOrRelationUniversalFlatFieldMetadata(relatedFlatField) &&
(relatedFlatField.type === FieldMetadataType.MORPH_RELATION ||
relatedFlatField.universalSettings?.relationType !==
RelationType.MANY_TO_ONE);
if (
[
FieldMetadataType.MORPH_RELATION,
FieldMetadataType.RELATION,
].includes(relatedFlatField.type) ||
isUnindexableRelation ||
isCompositeFieldWithNonIncludedUniqueConstraint
) {
const fieldType = relatedFlatField.type;
@@ -138,7 +138,21 @@ export class UpdateFieldActionHandlerService extends WorkspaceMigrationRunnerAct
const { entityId, update } = flatAction;
await fieldMetadataRepository.update({ id: entityId, workspaceId }, update);
// isUnique is derived from IndexMetadata at cache build time and has
// no underlying column on fieldMetadata. It travels in the update
// payload only so per-type validators (e.g. FILES rejection) can run
// — the actual state change is handled by the side-effect index
// create/delete in handleIndexChangesDuringFieldUpdate.
const { isUnique: _droppedIsUnique, ...persistedUpdate } = update;
if (Object.keys(persistedUpdate).length === 0) {
return;
}
await fieldMetadataRepository.update(
{ id: entityId, workspaceId },
persistedUpdate,
);
}
async executeForWorkspaceSchema(
@@ -59,6 +59,7 @@ export const fromUniversalFlatIndexToFlatIndex = ({
indexMetadataId,
fieldMetadataId: fieldMetadata.id,
order: universalFlatIndexFieldMetadata.order,
subFieldName: universalFlatIndexFieldMetadata.subFieldName,
createdAt: now,
updatedAt: now,
workspaceId,
@@ -28,57 +28,81 @@ export const computeFlatIndexFieldColumnNames = ({
flatIndexFieldMetadatas: FlatIndexFieldMetadata[];
flatFieldMetadataMaps: MetadataFlatEntityMaps<'fieldMetadata'>;
}): string[] => {
return flatIndexFieldMetadatas.flatMap(({ fieldMetadataId }) => {
const flatFieldMetadata = findFlatEntityByIdInFlatEntityMapsOrThrow({
flatEntityId: fieldMetadataId,
flatEntityMaps: flatFieldMetadataMaps,
});
return flatIndexFieldMetadatas.flatMap(
({ fieldMetadataId, subFieldName }) => {
const flatFieldMetadata = findFlatEntityByIdInFlatEntityMapsOrThrow({
flatEntityId: fieldMetadataId,
flatEntityMaps: flatFieldMetadataMaps,
});
if (!isDefined(flatFieldMetadata)) {
throw new FlatEntityMapsException(
'Index field related field metadata not found',
FlatEntityMapsExceptionCode.ENTITY_NOT_FOUND,
);
}
if (isMorphOrRelationFlatFieldMetadata(flatFieldMetadata)) {
if (
flatFieldMetadata.settings?.relationType !== RelationType.MANY_TO_ONE
) {
if (!isDefined(flatFieldMetadata)) {
throw new FlatEntityMapsException(
'Cannot index a relation field that has no join column',
'Index field related field metadata not found',
FlatEntityMapsExceptionCode.ENTITY_NOT_FOUND,
);
}
return computeMorphOrRelationFieldJoinColumnName({
name: flatFieldMetadata.name,
});
}
if (isMorphOrRelationFlatFieldMetadata(flatFieldMetadata)) {
if (
flatFieldMetadata.settings?.relationType !== RelationType.MANY_TO_ONE
) {
throw new FlatEntityMapsException(
'Cannot index a relation field that has no join column',
FlatEntityMapsExceptionCode.ENTITY_NOT_FOUND,
);
}
if (isCompositeFieldMetadataType(flatFieldMetadata.type)) {
const compositeType = compositeTypeDefinitions.get(
flatFieldMetadata.type,
);
return computeMorphOrRelationFieldJoinColumnName({
name: flatFieldMetadata.name,
});
}
if (!compositeType) {
throw new FlatEntityMapsException(
'Composite type not found',
FlatEntityMapsExceptionCode.INTERNAL_SERVER_ERROR,
if (isCompositeFieldMetadataType(flatFieldMetadata.type)) {
const compositeType = compositeTypeDefinitions.get(
flatFieldMetadata.type,
);
if (!compositeType) {
throw new FlatEntityMapsException(
'Composite type not found',
FlatEntityMapsExceptionCode.INTERNAL_SERVER_ERROR,
);
}
if (isDefined(subFieldName)) {
const property = compositeType.properties.find(
(compositeProperty) => compositeProperty.name === subFieldName,
);
if (!isDefined(property)) {
throw new FlatEntityMapsException(
`Composite sub-field "${subFieldName}" not found on ${flatFieldMetadata.name}`,
FlatEntityMapsExceptionCode.ENTITY_NOT_FOUND,
);
}
return [
computeCompositeColumnName(
{ name: flatFieldMetadata.name, type: flatFieldMetadata.type },
property,
),
];
}
// System indexes (no subFieldName) project the composite parent onto
// every property flagged isIncludedInUniqueConstraint.
const uniqueCompositeProperties = compositeType.properties.filter(
(property) => property.isIncludedInUniqueConstraint,
);
return uniqueCompositeProperties.map((subField) =>
computeCompositeColumnName(flatFieldMetadata.name, subField),
);
}
const uniqueCompositeProperties = compositeType.properties.filter(
(property) => property.isIncludedInUniqueConstraint,
);
return uniqueCompositeProperties.map((subField) =>
computeCompositeColumnName(flatFieldMetadata.name, subField),
);
}
return flatFieldMetadata.name;
});
return flatFieldMetadata.name;
},
);
};
export const deleteIndexMetadata = async ({
@@ -59,7 +59,6 @@ describe('Generate Column Definitions', () => {
isArray: false,
isNullable: true,
isPrimary: false,
isUnique: false,
default: 'NULL',
});
});
@@ -104,7 +103,6 @@ describe('Generate Column Definitions', () => {
isArray: true,
isNullable: true,
isPrimary: false,
isUnique: false,
default: 'NULL',
});
});
@@ -157,7 +155,6 @@ describe('Generate Column Definitions', () => {
type: 'uuid',
isNullable: true,
isPrimary: false,
isUnique: false,
default: null,
isArray: false,
});
@@ -199,7 +196,6 @@ describe('Generate Column Definitions', () => {
columns.forEach((column) => {
expect(column.isNullable).toBe(true);
expect(column.isPrimary).toBe(false);
expect(column.isUnique).toBe(false);
expect(column.default).toBe('NULL');
});
});
@@ -233,7 +229,6 @@ describe('Generate Column Definitions', () => {
type: 'numeric',
isNullable: true,
isPrimary: false,
isUnique: false,
default: "'100000000'::numeric",
});
@@ -242,7 +237,6 @@ describe('Generate Column Definitions', () => {
type: 'text',
isNullable: true,
isPrimary: false,
isUnique: false,
default: "'USD'::text",
});
});
@@ -354,7 +348,6 @@ describe('Generate Column Definitions', () => {
type: 'text',
isNullable: true,
isPrimary: false,
isUnique: false,
default: 'NULL',
isArray: false,
},
@@ -382,7 +375,6 @@ describe('Generate Column Definitions', () => {
type: 'boolean',
isNullable: true,
isPrimary: false,
isUnique: false,
default: "'true'::boolean",
isArray: false,
},
@@ -411,7 +403,6 @@ describe('Generate Column Definitions', () => {
type: 'text',
isNullable: true,
isPrimary: false,
isUnique: false,
default: 'NULL',
isArray: false,
},
@@ -438,7 +429,6 @@ describe('Generate Column Definitions', () => {
type: 'uuid',
isNullable: true,
isPrimary: false,
isUnique: false,
default: 'NULL',
isArray: false,
},
@@ -89,7 +89,6 @@ export const generateCompositeColumnDefinition = ({
: columnType,
isNullable:
parentFlatFieldMetadata.isNullable || !compositeProperty.isRequired,
isUnique: parentFlatFieldMetadata.isUnique ?? false,
default: serializedDefaultValue,
isArray: isArrayFlag,
isPrimary: false,
@@ -108,7 +107,6 @@ const generateTsVectorColumnDefinition = (
type: fieldMetadataTypeToColumnType(flatFieldMetadata.type),
isNullable: true,
isArray: false,
isUnique: false,
default: null,
asExpression: flatFieldMetadata.settings?.asExpression ?? undefined,
generatedType: flatFieldMetadata.settings?.generatedType ?? undefined,
@@ -134,7 +132,6 @@ const generateRelationColumnDefinition = (
type: fieldMetadataTypeToColumnType(FieldMetadataType.UUID),
isNullable: true,
isArray: false,
isUnique: false,
default: null,
isPrimary: false,
};
@@ -171,7 +168,6 @@ const generateColumnDefinition = ({
isArray:
flatFieldMetadata.type === FieldMetadataType.ARRAY ||
flatFieldMetadata.type === FieldMetadataType.MULTI_SELECT,
isUnique: flatFieldMetadata.isUnique ?? false,
default: serializedDefaultValue,
isPrimary: flatFieldMetadata.name === 'id',
};