refactor(server): unify the two metadata override mechanisms into one (#22417)
## Unify the two metadata override mechanisms into one Twenty had **two** override mechanisms: - **`standardOverrides`** — a bespoke JSONB column on `objectMetadata`/`fieldMetadata` with typed DTOs and a per-locale `translations` map, resolved by two i18n-aware resolvers. - **`OverridableEntity.overrides`** — a flat, registry-driven JSONB blob on view / view-field / view-field-group / command-menu-item / page-layout-tab / page-layout-widget, resolved by a plain spread. This PR collapses them into **one** concept: a single `overrides` blob, one registry-driven overridable set, one i18n-aware read path, and one write path (`computeMetadataOverridesBlob`, extracted in #22404). Object/field **stay on `SyncableEntity`** (not reparented to `OverridableEntity`) so their `isActive` default stays **FALSE** — this sidesteps the `isActive` default conflict entirely. ### GraphQL breaking change (accepted) The `standardOverrides` field is **removed** with no deprecation alias — `overrides` (a `JSON` scalar) is exposed instead on `Object` and `Field`. Product confirmed negligible external usage; the front-end has no hand-written consumer (only generated types), which are regenerated here. ### Commit structure (reviewable commit-by-commit) 1. **Unified resolver + parity harness** — `resolveEffectiveEntityProperty` is a strict superset of the three legacy resolvers; a corpus parity spec compares it against a *frozen reference* of the old logic across every locale, `isStandardApp` branch and override shape. 2. **Registry-driven** — object/field presentation props tagged `isOverridable` + `translatable`; the overridable/translatable sets are derived from the registry (a test asserts they equal the legacy hardcoded lists). 3. **Rename + swap + delete** — `standardOverrides` → `overrides` across entities, DTOs, flat/universal types, producers, the ~12 resolve/write/create/sync call sites, mocks and specs; the reconciler's two compare entries collapse to one; the three legacy resolvers, both DTOs and the hardcoded constants/types are deleted. 4. **Migration (zero-downtime, two-phase)** — split across two releases so a rolling deploy never drops a column a previous-release pod still `SELECT`s: - **2.19 fast** — add the `overrides` column (schema only). - **2.19 slow** — backfill `overrides` from `standardOverrides` in `runDataMigration` (kept out of the schema transaction so the bulk write doesn't hold the ACCESS EXCLUSIVE lock; skipped on fresh installs, which have no data to copy). - **2.20 fast** — drop the legacy `standardOverrides` column (gated by `TWENTY_NEXT_VERSIONS`, so it stays dormant until the instance reaches 2.20). 5. **Front/client-SDK regen** — regenerated metadata GraphQL types. 6. **Integration specs + i18n** — updated the standard object/field update integration specs + snapshots, and the reworded validator message catalog entry. ### Rolling-deploy safety `standardOverrides` is retained through 2.19 and only dropped in 2.20, mirroring the codebase's deferred-drop convention (`isUIReadOnly`/`isCustom`). During the 2.19 rollout both columns exist, so old and new pods coexist without "column does not exist" errors. The backfill lives in a slow `runDataMigration` (per the `no-data-mutation-in-fast-instance-command` rule) so it doesn't stall reads. ### `isActive` guard The migration never reads or writes `isActive`; the backfill asserts the active-row count is unchanged and aborts otherwise. Verified on a real DB: apply + revert preserves the blob **and** the nested `translations` map, with `isActive` counts identical before/after. ### Verification (local) - `nx typecheck twenty-server` + `nx typecheck twenty-front` — green - `nx lint:diff-with-main twenty-server` (oxlint `--type-aware` + oxfmt) — green - `nx test twenty-server` — green (unit + parity + registry + migration tests) - `nx run twenty-server:test:integration:with-db-reset` — green - `database:reset` applies the 2.19 phases and leaves **both** columns present (2.20 drop stays dormant); backfill + revert round-trip verified on a real DB - Metadata integration suites (standard object/field update, application sync) pass end-to-end against the two-column schema - Metadata GraphQL types regenerated against a booted server; zero `standardOverrides` references remain in application code (only the migration commands + the legacy schema baseline) --------- Co-authored-by: prastoin <paul@twenty.com>
This commit is contained in:
Félix Malfait
+1
-1
@@ -127,7 +127,7 @@ const buildLegacyCalendarEventRecordingPreferenceFieldMetadata = ({
|
||||
isUnique: false,
|
||||
isUIEditable: true,
|
||||
isLabelSyncedWithName: false,
|
||||
standardOverrides: null,
|
||||
overrides: null,
|
||||
defaultValue: "'AUTO'",
|
||||
settings: null,
|
||||
options: [
|
||||
|
||||
+32
@@ -0,0 +1,32 @@
|
||||
import { QueryRunner } from 'typeorm';
|
||||
|
||||
import { RegisteredInstanceCommand } from 'src/engine/core-modules/upgrade/decorators/registered-instance-command.decorator';
|
||||
import { FastInstanceCommand } from 'src/engine/core-modules/upgrade/interfaces/fast-instance-command.interface';
|
||||
|
||||
const TABLES = ['objectMetadata', 'fieldMetadata'] as const;
|
||||
|
||||
@RegisteredInstanceCommand('2.19.0', 1820000100000)
|
||||
export class AddMetadataOverridesColumnFastInstanceCommand
|
||||
implements FastInstanceCommand
|
||||
{
|
||||
public async up(queryRunner: QueryRunner): Promise<void> {
|
||||
for (const table of TABLES) {
|
||||
await queryRunner.query(
|
||||
`ALTER TABLE "core"."${table}" ADD COLUMN IF NOT EXISTS "overrides" jsonb`,
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
public async down(queryRunner: QueryRunner): Promise<void> {
|
||||
for (const table of TABLES) {
|
||||
// Unconditional copy: a WHERE "overrides" IS NOT NULL guard would leave a
|
||||
// stale value in standardOverrides and resurrect a cleared override.
|
||||
await queryRunner.query(
|
||||
`UPDATE "core"."${table}" SET "standardOverrides" = "overrides"`,
|
||||
);
|
||||
await queryRunner.query(
|
||||
`ALTER TABLE "core"."${table}" DROP COLUMN IF EXISTS "overrides"`,
|
||||
);
|
||||
}
|
||||
}
|
||||
}
|
||||
+48
@@ -0,0 +1,48 @@
|
||||
import { DataSource, QueryRunner } from 'typeorm';
|
||||
|
||||
import { RegisteredInstanceCommand } from 'src/engine/core-modules/upgrade/decorators/registered-instance-command.decorator';
|
||||
import { SlowInstanceCommand } from 'src/engine/core-modules/upgrade/interfaces/slow-instance-command.interface';
|
||||
|
||||
const TABLES = ['objectMetadata', 'fieldMetadata'] as const;
|
||||
|
||||
@RegisteredInstanceCommand('2.19.0', 1820000110000, { type: 'slow' })
|
||||
export class BackfillMetadataOverridesSlowInstanceCommand
|
||||
implements SlowInstanceCommand
|
||||
{
|
||||
async runDataMigration(dataSource: DataSource): Promise<void> {
|
||||
for (const table of TABLES) {
|
||||
const activeCountBefore = await this.getActiveCount(dataSource, table);
|
||||
|
||||
await dataSource.query(
|
||||
`UPDATE "core"."${table}" SET "overrides" = "standardOverrides" WHERE "standardOverrides" IS NOT NULL AND "overrides" IS NULL`,
|
||||
);
|
||||
|
||||
const activeCountAfter = await this.getActiveCount(dataSource, table);
|
||||
|
||||
if (activeCountBefore !== activeCountAfter) {
|
||||
throw new Error(
|
||||
`BackfillMetadataOverrides: "isActive" changed on "core"."${table}" (${activeCountBefore} -> ${activeCountAfter}), aborting.`,
|
||||
);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
public async up(_queryRunner: QueryRunner): Promise<void> {
|
||||
return;
|
||||
}
|
||||
|
||||
public async down(_queryRunner: QueryRunner): Promise<void> {
|
||||
return;
|
||||
}
|
||||
|
||||
private async getActiveCount(
|
||||
dataSource: DataSource,
|
||||
table: (typeof TABLES)[number],
|
||||
): Promise<number> {
|
||||
const [{ count }] = await dataSource.query(
|
||||
`SELECT count(*)::int AS count FROM "core"."${table}" WHERE "isActive" = true`,
|
||||
);
|
||||
|
||||
return count;
|
||||
}
|
||||
}
|
||||
+51
@@ -0,0 +1,51 @@
|
||||
import { type QueryRunner } from 'typeorm';
|
||||
|
||||
import { AddMetadataOverridesColumnFastInstanceCommand } from 'src/database/commands/upgrade-version-command/2-19/2-19-instance-command-fast-1820000100000-add-metadata-overrides-column';
|
||||
|
||||
describe('AddMetadataOverridesColumnFastInstanceCommand', () => {
|
||||
let command: AddMetadataOverridesColumnFastInstanceCommand;
|
||||
|
||||
beforeEach(() => {
|
||||
command = new AddMetadataOverridesColumnFastInstanceCommand();
|
||||
});
|
||||
|
||||
describe('up', () => {
|
||||
it('adds the overrides column to both tables without mutating data or dropping standardOverrides', async () => {
|
||||
const query = jest.fn().mockResolvedValue(undefined);
|
||||
const queryRunner = { query } as unknown as QueryRunner;
|
||||
|
||||
await command.up(queryRunner);
|
||||
|
||||
const statements = query.mock.calls.map((call) => call[0] as string);
|
||||
|
||||
expect(statements).toEqual([
|
||||
'ALTER TABLE "core"."objectMetadata" ADD COLUMN IF NOT EXISTS "overrides" jsonb',
|
||||
'ALTER TABLE "core"."fieldMetadata" ADD COLUMN IF NOT EXISTS "overrides" jsonb',
|
||||
]);
|
||||
expect(
|
||||
statements.some(
|
||||
(statement) =>
|
||||
statement.includes('UPDATE') || statement.includes('DROP COLUMN'),
|
||||
),
|
||||
).toBe(false);
|
||||
});
|
||||
});
|
||||
|
||||
describe('down', () => {
|
||||
it('copies overrides back into standardOverrides before dropping the column', async () => {
|
||||
const query = jest.fn().mockResolvedValue(undefined);
|
||||
const queryRunner = { query } as unknown as QueryRunner;
|
||||
|
||||
await command.down(queryRunner);
|
||||
|
||||
const statements = query.mock.calls.map((call) => call[0] as string);
|
||||
|
||||
expect(statements).toEqual([
|
||||
'UPDATE "core"."objectMetadata" SET "standardOverrides" = "overrides"',
|
||||
'ALTER TABLE "core"."objectMetadata" DROP COLUMN IF EXISTS "overrides"',
|
||||
'UPDATE "core"."fieldMetadata" SET "standardOverrides" = "overrides"',
|
||||
'ALTER TABLE "core"."fieldMetadata" DROP COLUMN IF EXISTS "overrides"',
|
||||
]);
|
||||
});
|
||||
});
|
||||
});
|
||||
+45
@@ -0,0 +1,45 @@
|
||||
import { type DataSource } from 'typeorm';
|
||||
|
||||
import { BackfillMetadataOverridesSlowInstanceCommand } from 'src/database/commands/upgrade-version-command/2-19/2-19-instance-command-slow-1820000110000-backfill-metadata-overrides';
|
||||
|
||||
describe('BackfillMetadataOverridesSlowInstanceCommand', () => {
|
||||
let command: BackfillMetadataOverridesSlowInstanceCommand;
|
||||
|
||||
beforeEach(() => {
|
||||
command = new BackfillMetadataOverridesSlowInstanceCommand();
|
||||
});
|
||||
|
||||
describe('runDataMigration', () => {
|
||||
it('copies standardOverrides into overrides for both tables', async () => {
|
||||
const query = jest.fn().mockResolvedValue([{ count: 3 }]);
|
||||
const dataSource = { query } as unknown as DataSource;
|
||||
|
||||
await command.runDataMigration(dataSource);
|
||||
|
||||
const statements = query.mock.calls.map((call) => call[0] as string);
|
||||
|
||||
for (const table of ['objectMetadata', 'fieldMetadata']) {
|
||||
expect(statements).toEqual(
|
||||
expect.arrayContaining([
|
||||
expect.stringContaining(
|
||||
`UPDATE "core"."${table}" SET "overrides" = "standardOverrides"`,
|
||||
),
|
||||
]),
|
||||
);
|
||||
}
|
||||
});
|
||||
|
||||
it('aborts when the isActive row count changes', async () => {
|
||||
const query = jest
|
||||
.fn()
|
||||
.mockResolvedValueOnce([{ count: 5 }]) // objectMetadata before
|
||||
.mockResolvedValueOnce(undefined) // UPDATE backfill
|
||||
.mockResolvedValueOnce([{ count: 4 }]); // objectMetadata after
|
||||
const dataSource = { query } as unknown as DataSource;
|
||||
|
||||
await expect(command.runDataMigration(dataSource)).rejects.toThrow(
|
||||
/"isActive" changed on "core"\."objectMetadata"/,
|
||||
);
|
||||
});
|
||||
});
|
||||
});
|
||||
+4
@@ -0,0 +1,4 @@
|
||||
// Referenced by @WasIntroducedInUpgrade on the "overrides" column so pre-2.19
|
||||
// upgrade steps don't SELECT it before this command adds it.
|
||||
export const ADD_METADATA_OVERRIDES_COLUMN_UPGRADE_COMMAND_NAME =
|
||||
'2.19.0_AddMetadataOverridesColumnFastInstanceCommand_1820000100000';
|
||||
@@ -0,0 +1,134 @@
|
||||
# 2.20 — pending metadata-override cleanup
|
||||
|
||||
The `standardOverrides` column on `objectMetadata` and `fieldMetadata` is a
|
||||
**deferred drop**. As of 2.19 the column is:
|
||||
|
||||
- superseded by `overrides` (backfilled by
|
||||
`../2-19/2-19-instance-command-slow-1820000110000-backfill-metadata-overrides.ts`),
|
||||
- excluded from the flat-entity / registry via the `WasRemovedInUpgrade<T>` type
|
||||
wrapper on both entities,
|
||||
- still physically present in the database, so a 2.18 → 2.19 rolling deploy never
|
||||
breaks the previous release's pods, which still `SELECT "standardOverrides"`.
|
||||
|
||||
The physical `DROP COLUMN` is intentionally deferred to the next release (2.20):
|
||||
by the time it runs, every pod is on a release that reads/writes `overrides`
|
||||
only. This folder holds the ready-to-wire drop command so whoever opens the 2.20
|
||||
upgrade work can drop it in.
|
||||
|
||||
## Why it isn't wired up yet
|
||||
|
||||
`2.20.0` is not in `TWENTY_CROSS_UPGRADE_SUPPORTED_VERSIONS` (currently
|
||||
`[...TWENTY_PREVIOUS_VERSIONS, TWENTY_CURRENT_VERSION]`, and current is 2.19.0).
|
||||
Registering a 2.20 instance command — or pointing the entity's
|
||||
`@WasRemovedInUpgrade` decorator at a 2.20 command name — would fail boot
|
||||
validation (`unknown-step-name`) until 2.20 is the current version. So the
|
||||
command lives here as documentation, not as compiled code.
|
||||
|
||||
## Activation checklist (when 2.20 becomes the current version)
|
||||
|
||||
1. Create `2-20-instance-command-fast-<timestamp>-drop-metadata-standard-overrides-column.ts`
|
||||
in this folder with the command below. Regenerate `<timestamp>` (the value
|
||||
here is a placeholder) so ordering against other 2.20 commands is correct.
|
||||
2. Register it in
|
||||
`../instance-commands.constant.ts` (import + add to the array).
|
||||
3. Add the drop decorator to `standardOverrides` on **both**
|
||||
`object-metadata.entity.ts` and `field-metadata.entity.ts`:
|
||||
`@WasRemovedInUpgrade({ upgradeCommandName: DROP_METADATA_STANDARD_OVERRIDES_COLUMN_UPGRADE_COMMAND_NAME })`,
|
||||
and add the matching constant
|
||||
(`'2.20.0_DropMetadataStandardOverridesColumnFastInstanceCommand_<timestamp>'`),
|
||||
mirroring how `isCustom` pairs its decorator with a command name.
|
||||
4. Add the test below under `2-20/__tests__/`.
|
||||
5. Run `database:migrate:generate --name pending-migration-check` and confirm no
|
||||
drift, then run the integration suite.
|
||||
|
||||
## Command
|
||||
|
||||
```ts
|
||||
import { QueryRunner } from 'typeorm';
|
||||
|
||||
import { RegisteredInstanceCommand } from 'src/engine/core-modules/upgrade/decorators/registered-instance-command.decorator';
|
||||
import { FastInstanceCommand } from 'src/engine/core-modules/upgrade/interfaces/fast-instance-command.interface';
|
||||
|
||||
// Phase 2 of unifying the metadata override mechanisms. Drops the legacy
|
||||
// "standardOverrides" column now that every pod runs a release which reads and
|
||||
// writes "overrides" (backfilled in 2.19). Runs only once the instance reaches
|
||||
// 2.20, so the 2.19 rolling deploy keeps the column available for older pods.
|
||||
// down() restores the column and copies the blob back from "overrides".
|
||||
const TABLES = ['objectMetadata', 'fieldMetadata'] as const;
|
||||
|
||||
@RegisteredInstanceCommand('2.20.0', 1825000000000)
|
||||
export class DropMetadataStandardOverridesColumnFastInstanceCommand
|
||||
implements FastInstanceCommand
|
||||
{
|
||||
public async up(queryRunner: QueryRunner): Promise<void> {
|
||||
for (const table of TABLES) {
|
||||
await queryRunner.query(
|
||||
`ALTER TABLE "core"."${table}" DROP COLUMN IF EXISTS "standardOverrides"`,
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
public async down(queryRunner: QueryRunner): Promise<void> {
|
||||
for (const table of TABLES) {
|
||||
await queryRunner.query(
|
||||
`ALTER TABLE "core"."${table}" ADD COLUMN IF NOT EXISTS "standardOverrides" jsonb`,
|
||||
);
|
||||
await queryRunner.query(
|
||||
`UPDATE "core"."${table}" SET "standardOverrides" = "overrides"`,
|
||||
);
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Test
|
||||
|
||||
```ts
|
||||
import { type QueryRunner } from 'typeorm';
|
||||
|
||||
import { DropMetadataStandardOverridesColumnFastInstanceCommand } from 'src/database/commands/upgrade-version-command/2-20/2-20-instance-command-fast-1825000000000-drop-metadata-standard-overrides-column';
|
||||
|
||||
describe('DropMetadataStandardOverridesColumnFastInstanceCommand', () => {
|
||||
let command: DropMetadataStandardOverridesColumnFastInstanceCommand;
|
||||
|
||||
beforeEach(() => {
|
||||
command = new DropMetadataStandardOverridesColumnFastInstanceCommand();
|
||||
});
|
||||
|
||||
describe('up', () => {
|
||||
it('drops standardOverrides from both tables', async () => {
|
||||
const query = jest.fn().mockResolvedValue(undefined);
|
||||
const queryRunner = { query } as unknown as QueryRunner;
|
||||
|
||||
await command.up(queryRunner);
|
||||
|
||||
const statements = query.mock.calls.map((call) => call[0] as string);
|
||||
|
||||
expect(statements).toEqual([
|
||||
'ALTER TABLE "core"."objectMetadata" DROP COLUMN IF EXISTS "standardOverrides"',
|
||||
'ALTER TABLE "core"."fieldMetadata" DROP COLUMN IF EXISTS "standardOverrides"',
|
||||
]);
|
||||
});
|
||||
});
|
||||
|
||||
describe('down', () => {
|
||||
it('recreates and backfills standardOverrides from overrides', async () => {
|
||||
const query = jest.fn().mockResolvedValue(undefined);
|
||||
const queryRunner = { query } as unknown as QueryRunner;
|
||||
|
||||
await command.down(queryRunner);
|
||||
|
||||
const statements = query.mock.calls.map((call) => call[0] as string);
|
||||
|
||||
for (const table of ['objectMetadata', 'fieldMetadata']) {
|
||||
expect(statements).toEqual(
|
||||
expect.arrayContaining([
|
||||
`ALTER TABLE "core"."${table}" ADD COLUMN IF NOT EXISTS "standardOverrides" jsonb`,
|
||||
`UPDATE "core"."${table}" SET "standardOverrides" = "overrides"`,
|
||||
]),
|
||||
);
|
||||
}
|
||||
});
|
||||
});
|
||||
});
|
||||
```
|
||||
+4
@@ -41,6 +41,8 @@ import { CreateDpaAgreementCoreTableFastInstanceCommand } from 'src/database/com
|
||||
import { CreateApplicationTranslationCoreTableFastInstanceCommand } from 'src/database/commands/upgrade-version-command/2-17/2-17-instance-command-fast-1801000100000-create-application-translation-core-table';
|
||||
import { AddTsVectorFieldMetadataIdToSearchFieldMetadataFastInstanceCommand } from 'src/database/commands/upgrade-version-command/2-18/2-18-instance-command-fast-1810000001000-add-ts-vector-field-metadata-id-to-search-field-metadata';
|
||||
import { BackfillTsVectorFieldMetadataIdOnSearchFieldMetadataSlowInstanceCommand } from 'src/database/commands/upgrade-version-command/2-18/2-18-instance-command-slow-1810000003000-backfill-ts-vector-field-metadata-id-on-search-field-metadata';
|
||||
import { AddMetadataOverridesColumnFastInstanceCommand } from 'src/database/commands/upgrade-version-command/2-19/2-19-instance-command-fast-1820000100000-add-metadata-overrides-column';
|
||||
import { BackfillMetadataOverridesSlowInstanceCommand } from 'src/database/commands/upgrade-version-command/2-19/2-19-instance-command-slow-1820000110000-backfill-metadata-overrides';
|
||||
import { AddCacheTokensToAgentChatThreadFastInstanceCommand } from 'src/database/commands/upgrade-version-command/2-2/2-2-instance-command-fast-1777455269302-add-cache-tokens-to-agent-chat-thread';
|
||||
import { AddLogoToApplicationFastInstanceCommand } from 'src/database/commands/upgrade-version-command/2-2/2-2-instance-command-fast-1777539664664-add-logo-to-application';
|
||||
import { AddSubFieldNameToViewSortEarlyFastInstanceCommand } from 'src/database/commands/upgrade-version-command/2-3/2-3-instance-command-fast-1747234200000-add-sub-field-name-to-view-sort';
|
||||
@@ -170,4 +172,6 @@ export const INSTANCE_COMMANDS = [
|
||||
CreateApplicationTranslationCoreTableFastInstanceCommand,
|
||||
AddTsVectorFieldMetadataIdToSearchFieldMetadataFastInstanceCommand,
|
||||
BackfillTsVectorFieldMetadataIdOnSearchFieldMetadataSlowInstanceCommand,
|
||||
AddMetadataOverridesColumnFastInstanceCommand,
|
||||
BackfillMetadataOverridesSlowInstanceCommand,
|
||||
];
|
||||
|
||||
Reference in New Issue
Block a user