Implement cross version upgrade (#19559)

# Introduction
Refactoring the upgrade engine to handle cross version upgrade,
completely getting rid of the semver `version` at db and runtime level
It remains a visual a listing indicator for or CD process but also
during devenv in order to prepare next release
Will write a release process runbook documentation on how to handle
upgrade step patch, command insertion etc as it needs to be cascaded
across all the involved supported version

**The upgrade sequence model:**

The sequence is a flat, ordered array of upgrade steps
(`UpgradeStep[]`), built from the registry by chaining all versions in
order, each version contributing its fast-instance → slow-instance →
workspace commands sorted by timestamp. Version is metadata for logging,
not used in the algorithm.

**Segments:**

The sequence naturally splits into alternating segments of contiguous
instance steps and contiguous workspace steps. The runner processes
segments in order:

- **Instance segment:** Run sequentially from the instance cursor. Each
step runs once globally.
- **Workspace segment:** Each workspace independently walks from its own
cursor through the end of the segment. Workspaces are independent within
a segment — they can be at different positions.
- **Synchronization (workspace → instance):** The runner blocks before
entering an instance segment. All active/suspended workspaces must have
completed the last workspace step of the preceding workspace segment. If
any workspace failed, abort. This is the only explicit synchronization
point.
- Instance → workspace ordering is implicit — the runner processes
segments sequentially, so the instance segment naturally completes
before the workspace segment begins.


full docs
https://gist.github.com/prastoin/e62106d455fd72d6b6ebada8351e5492

## Version constants & type-level deprecation

Version management is split into three atomic constants:
`TWENTY_PREVIOUS_VERSIONS`, `TWENTY_CURRENT_VERSION`, and
`TWENTY_NEXT_VERSIONS`. Two derived constants compose them:
`CROSS_UPGRADE_SUPPORTED_VERSIONS` (previous + current — what the engine
runs) and `ALL_TWENTY_VERSIONS` (the full ordered tuple including next).
The registry service validates at module init that no version is
duplicated across constants and that at least one previous version
exists.

A `DeprecatedSinceVersion<RemoveAtVersion, T>` type utility resolves to
`T` while `TWENTY_CURRENT_VERSION` is below `RemoveAtVersion`, and to
`never` once it reaches it — turning deprecation into a compile-time
guarantee via `IndexOf` and `IsGreaterOrEqual` generics in
`twenty-shared`.

### `workspace.version` column deprecation

The column is replaced by cursor-based state inference from
`UpgradeMigration` records, but cannot be dropped in 1.22: workspaces
activated during 1.21 predate the cursor system and need their initial
cursor backfilled first (`backfillWorkspaceCreatedIn1_21_0Cursors`).
This backfill itself depends on a new `isInitial` column on
`UpgradeMigration`, bootstrapped via a targeted TypeORM migration before
the upgrade sequence runs.

Both functions and the entity field are typed with
`DeprecatedSinceVersion<'1.23.0', ...>`. When `TWENTY_CURRENT_VERSION`
reaches `1.23.0`, compile errors force their removal — and the
pre-declared `DropWorkspaceVersionColumnFastInstanceCommand` takes over
to drop the column.

## What's next
- ci cross version upgrade ( wip )
- banner asking to contact twenty administrator if workspace is outdated
- upgrade healthcheck cli 

## New unit/integ test pattern
Create a dedicated `createNestApp` that consumes a real database in
order not to have to mack any database interaction to the
`upgradeMigrations` allowing full coverage of the whole
`upgradeRunnerService.run` core logic
This commit is contained in:
Paul Rastoin
2026-04-13 11:42:27 +02:00
committed by GitHub
parent 7091561489
commit 21142d98fe
52 changed files with 2585 additions and 1359 deletions
@@ -0,0 +1,21 @@
import { QueryRunner } from 'typeorm';
import { RegisteredInstanceCommand } from 'src/engine/core-modules/upgrade/decorators/registered-instance-command.decorator';
import { FastInstanceCommand } from 'src/engine/core-modules/upgrade/interfaces/fast-instance-command.interface';
@RegisteredInstanceCommand('1.23.0', 1785000000000)
export class DropWorkspaceVersionColumnFastInstanceCommand
implements FastInstanceCommand
{
public async up(queryRunner: QueryRunner): Promise<void> {
await queryRunner.query(
`ALTER TABLE "core"."workspace" DROP COLUMN IF EXISTS "version"`,
);
}
public async down(queryRunner: QueryRunner): Promise<void> {
await queryRunner.query(
`ALTER TABLE "core"."workspace" ADD "version" character varying`,
);
}
}
@@ -0,0 +1,4 @@
import { Module } from '@nestjs/common';
@Module({})
export class V1_23_UpgradeVersionCommandModule {}
@@ -1,656 +0,0 @@
import { Test, type TestingModule } from '@nestjs/testing';
import {
eachTestingContextFilter,
type EachTestingContext,
} from 'twenty-shared/testing';
import { type DataSource, type QueryRunner } from 'typeorm';
import { type FastInstanceCommand } from 'src/engine/core-modules/upgrade/interfaces/fast-instance-command.interface';
import { type SlowInstanceCommand } from 'src/engine/core-modules/upgrade/interfaces/slow-instance-command.interface';
import { getDataSourceToken } from '@nestjs/typeorm';
import { WorkspaceIteratorService } from 'src/database/commands/command-runners/workspace-iterator.service';
import { RegisteredInstanceCommand } from 'src/engine/core-modules/upgrade/decorators/registered-instance-command.decorator';
import {
UpgradeCommand,
UpgradeCommandOptions,
} from 'src/database/commands/upgrade-version-command/upgrade.command';
import { UPGRADE_COMMAND_SUPPORTED_VERSIONS } from 'src/engine/constants/upgrade-command-supported-versions.constant';
import { CoreEngineVersionService } from 'src/engine/core-engine-version/services/core-engine-version.service';
import { type ConfigVariables } from 'src/engine/core-modules/twenty-config/config-variables';
import { TwentyConfigService } from 'src/engine/core-modules/twenty-config/twenty-config.service';
import { InstanceUpgradeService } from 'src/engine/core-modules/upgrade/services/instance-upgrade.service';
import { UpgradeCommandRegistryService } from 'src/engine/core-modules/upgrade/services/upgrade-command-registry.service';
import { WorkspaceUpgradeService } from 'src/engine/core-modules/upgrade/services/workspace-upgrade.service';
import { WorkspaceEntity } from 'src/engine/core-modules/workspace/workspace.entity';
import { WorkspaceVersionService } from 'src/engine/workspace-manager/workspace-version/services/workspace-version.service';
import { compareVersionMajorAndMinor } from 'src/utils/version/compare-version-minor-and-major';
const CURRENT_VERSION =
UPGRADE_COMMAND_SUPPORTED_VERSIONS[
UPGRADE_COMMAND_SUPPORTED_VERSIONS.length - 1
];
const PREVIOUS_VERSION =
UPGRADE_COMMAND_SUPPORTED_VERSIONS[
UPGRADE_COMMAND_SUPPORTED_VERSIONS.length - 2
];
type CommandRunnerValues = typeof UpgradeCommand;
const generateMockWorkspace = (overrides?: Partial<WorkspaceEntity>) =>
({
id: 'workspace-id',
version: PREVIOUS_VERSION,
createdAt: new Date(),
updatedAt: new Date(),
allowImpersonation: false,
isPublicInviteLinkEnabled: false,
displayName: 'Test Workspace',
domainName: 'test',
inviteHash: 'hash',
logo: null,
deletedAt: null,
activationStatus: 'active',
workspaceMembersCount: 1,
...overrides,
}) as WorkspaceEntity;
type BuildUpgradeCommandModuleArgs = {
workspaces: WorkspaceEntity[];
appVersion: string | null;
commandRunner: CommandRunnerValues;
migrations?: FastInstanceCommand[];
};
const buildUpgradeCommandModule = async ({
workspaces,
appVersion,
commandRunner,
migrations,
}: BuildUpgradeCommandModuleArgs) => {
const registryProvider = migrations
? {
provide: UpgradeCommandRegistryService,
useFactory: () => {
const fakeDiscoveryService = {
getProviders: () =>
migrations.map((migration) => ({
instance: migration,
metatype: migration.constructor,
})),
} as unknown as import('@nestjs/core').DiscoveryService;
const registry = new UpgradeCommandRegistryService(
fakeDiscoveryService,
);
registry.onModuleInit();
return registry;
},
}
: {
provide: UpgradeCommandRegistryService,
useValue: {
getBundleForVersion: jest.fn().mockReturnValue({
fastInstanceCommands: [],
slowInstanceCommands: [],
workspaceCommands: [],
}),
},
};
const module: TestingModule = await Test.createTestingModule({
providers: [
{
provide: getDataSourceToken(),
useValue: {
runMigrations: jest.fn().mockResolvedValue([]),
},
},
{
provide: commandRunner,
useFactory: (
coreEngineVersionService: CoreEngineVersionService,
workspaceVersionService: WorkspaceVersionService,
upgradeCommandRegistryService: UpgradeCommandRegistryService,
instanceUpgradeService: InstanceUpgradeService,
workspaceIteratorService: WorkspaceIteratorService,
workspaceUpgradeService: WorkspaceUpgradeService,
dataSource: DataSource,
) => {
return new commandRunner(
coreEngineVersionService,
workspaceVersionService,
upgradeCommandRegistryService,
instanceUpgradeService,
workspaceIteratorService,
workspaceUpgradeService,
dataSource,
);
},
inject: [
CoreEngineVersionService,
WorkspaceVersionService,
UpgradeCommandRegistryService,
InstanceUpgradeService,
WorkspaceIteratorService,
WorkspaceUpgradeService,
getDataSourceToken(),
],
},
{
provide: TwentyConfigService,
useValue: {
get: jest.fn().mockImplementation((key: keyof ConfigVariables) => {
switch (key) {
case 'APP_VERSION': {
return appVersion;
}
default: {
return;
}
}
}),
},
},
CoreEngineVersionService,
{
provide: WorkspaceVersionService,
useValue: {
hasActiveOrSuspendedWorkspaces: jest
.fn()
.mockResolvedValue(workspaces.length > 0),
getWorkspacesBelowVersion: jest
.fn()
.mockImplementation((version: string) => {
return workspaces.filter((workspace) => {
if (
workspace.version === null ||
workspace.version === undefined
) {
return true;
}
try {
return (
compareVersionMajorAndMinor(workspace.version, version) ===
'lower'
);
} catch {
return true;
}
});
}),
},
},
{
provide: InstanceUpgradeService,
useValue: {
runFastInstanceCommand: jest
.fn()
.mockResolvedValue({ status: 'success' }),
runSlowInstanceCommand: jest
.fn()
.mockResolvedValue({ status: 'success' }),
},
},
{
provide: WorkspaceUpgradeService,
useValue: {
upgradeWorkspace: jest.fn().mockResolvedValue(undefined),
},
},
registryProvider,
{
provide: WorkspaceIteratorService,
useValue: {
iterate: jest.fn().mockImplementation(async (args: any) => {
const { callback, ...options } = args;
const workspaceIds =
options.workspaceIds ??
workspaces.map((workspace) => workspace.id);
const report = {
fail: [] as any[],
success: [] as any[],
};
for (const [index, workspaceId] of workspaceIds.entries()) {
try {
await callback({
workspaceId,
index,
total: workspaceIds.length,
});
report.success.push({ workspaceId });
} catch (error) {
report.fail.push({ error, workspaceId });
}
}
return report;
}),
},
},
],
}).compile();
return module;
};
describe('UpgradeCommandRunner', () => {
let upgradeCommandRunner: UpgradeCommand;
type BuildModuleAndSetupSpiesArgs = {
numberOfWorkspace?: number;
workspaceOverride?: Partial<WorkspaceEntity>;
workspaces?: WorkspaceEntity[];
appVersion?: string | null;
commandRunner?: CommandRunnerValues;
migrations?: FastInstanceCommand[];
};
const buildModuleAndSetupSpies = async ({
numberOfWorkspace = 1,
workspaceOverride,
workspaces,
commandRunner = UpgradeCommand,
appVersion = CURRENT_VERSION,
migrations,
}: BuildModuleAndSetupSpiesArgs) => {
const generatedWorkspaces = Array.from(
{ length: numberOfWorkspace },
(_v, index) =>
generateMockWorkspace({
id: `workspace_${index}`,
...workspaceOverride,
}),
);
const module = await buildUpgradeCommandModule({
commandRunner,
appVersion,
workspaces: [...generatedWorkspaces, ...(workspaces ?? [])],
migrations,
});
upgradeCommandRunner = module.get(commandRunner);
jest.spyOn(upgradeCommandRunner['logger'], 'log').mockImplementation();
jest.spyOn(upgradeCommandRunner['logger'], 'error').mockImplementation();
jest.spyOn(upgradeCommandRunner['logger'], 'warn').mockImplementation();
return module;
};
it('should delegate workspace upgrade to WorkspaceUpgradeService', async () => {
const module = await buildModuleAndSetupSpies({});
const workspaceUpgradeService = module.get(WorkspaceUpgradeService);
const passedParams: string[] = [];
const options: UpgradeCommandOptions = {};
await upgradeCommandRunner.run(passedParams, options);
expect(workspaceUpgradeService.upgradeWorkspace).toHaveBeenCalledTimes(1);
});
it('should call upgradeWorkspace for each workspace', async () => {
const numberOfWorkspace = 42;
const module = await buildModuleAndSetupSpies({
numberOfWorkspace,
});
const workspaceUpgradeService = module.get(WorkspaceUpgradeService);
const passedParams: string[] = [];
const options: UpgradeCommandOptions = {};
await upgradeCommandRunner.run(passedParams, options);
expect(workspaceUpgradeService.upgradeWorkspace).toHaveBeenCalledTimes(
numberOfWorkspace,
);
});
describe('Workspace upgrade should succeed ', () => {
const successfulTestUseCases: EachTestingContext<{
input: Omit<BuildModuleAndSetupSpiesArgs, 'numberOfWorkspace'>;
}>[] = [
{
title: 'even if workspace version and app version differ in patch',
context: {
input: {
appVersion: `v${CURRENT_VERSION}`,
workspaceOverride: {
version: `v${PREVIOUS_VERSION.replace('.0', '.12')}`,
},
},
},
},
{
title:
'even if workspace version and app version differ in patch and semantic',
context: {
input: {
appVersion: `v${CURRENT_VERSION}`,
workspaceOverride: {
version: PREVIOUS_VERSION.replace('.0', '.12'),
},
},
},
},
{
title: 'even if app version contains a patch value',
context: {
input: {
appVersion: CURRENT_VERSION.replace('.0', '.24'),
workspaceOverride: {
version: PREVIOUS_VERSION.replace('.0', '.12'),
},
},
},
},
];
it.each(eachTestingContextFilter(successfulTestUseCases))(
'$title',
async ({ context: { input } }) => {
const module = await buildModuleAndSetupSpies(input);
const workspaceUpgradeService = module.get(WorkspaceUpgradeService);
const passedParams: string[] = [];
const options: UpgradeCommandOptions = {};
await upgradeCommandRunner.run(passedParams, options);
expect(workspaceUpgradeService.upgradeWorkspace).toHaveBeenCalled();
},
);
});
it('should call runFastInstanceCommand for each current-version instance command', async () => {
@RegisteredInstanceCommand(CURRENT_VERSION, 1770000000000)
class AddIndexToUsers1770000000000 implements FastInstanceCommand {
async up(_queryRunner: QueryRunner) {}
async down(_queryRunner: QueryRunner) {}
}
@RegisteredInstanceCommand(CURRENT_VERSION, 1771000000000)
class AddColumnToAccounts1771000000000 implements FastInstanceCommand {
async up(_queryRunner: QueryRunner) {}
async down(_queryRunner: QueryRunner) {}
}
@RegisteredInstanceCommand(PREVIOUS_VERSION, 1769000000000)
class DropLegacyTable1769000000000 implements FastInstanceCommand {
async up(_queryRunner: QueryRunner) {}
async down(_queryRunner: QueryRunner) {}
}
class UndecoratedMigration1768000000000 implements FastInstanceCommand {
async up(_queryRunner: QueryRunner) {}
async down(_queryRunner: QueryRunner) {}
}
const addIndex = new AddIndexToUsers1770000000000();
const addColumn = new AddColumnToAccounts1771000000000();
const dropLegacy = new DropLegacyTable1769000000000();
const undecorated = new UndecoratedMigration1768000000000();
const module = await buildModuleAndSetupSpies({
migrations: [undecorated, dropLegacy, addIndex, addColumn],
});
const instanceUpgradeService = module.get(InstanceUpgradeService);
await upgradeCommandRunner.run([], {});
expect(instanceUpgradeService.runFastInstanceCommand).toHaveBeenCalledTimes(
2,
);
expect(
instanceUpgradeService.runFastInstanceCommand,
).toHaveBeenNthCalledWith(1, {
command: addIndex,
name: `${CURRENT_VERSION}_AddIndexToUsers1770000000000_1770000000000`,
});
expect(
instanceUpgradeService.runFastInstanceCommand,
).toHaveBeenNthCalledWith(2, {
command: addColumn,
name: `${CURRENT_VERSION}_AddColumnToAccounts1771000000000_1771000000000`,
});
});
it('should propagate errors from runFastInstanceCommand', async () => {
@RegisteredInstanceCommand(CURRENT_VERSION, 1770000000000)
class FailingMigration1770000000000 implements FastInstanceCommand {
async up(_queryRunner: QueryRunner) {}
async down(_queryRunner: QueryRunner) {}
}
const module = await buildModuleAndSetupSpies({
migrations: [new FailingMigration1770000000000()],
});
const instanceUpgradeService = module.get(InstanceUpgradeService);
(
instanceUpgradeService.runFastInstanceCommand as jest.Mock
).mockResolvedValue({
status: 'failed',
error: new Error('SQL error'),
});
await expect(upgradeCommandRunner.run([], {})).rejects.toThrow('SQL error');
});
describe('Workspace upgrade should fail', () => {
const failingTestUseCases: EachTestingContext<{
input: Omit<BuildModuleAndSetupSpiesArgs, 'numberOfWorkspace'>;
expectedErrorMessage: string;
}>[] = [
{
title: 'when workspace version is not equal to fromVersion',
context: {
input: {
workspaceOverride: {
version: '0.1.0',
},
},
expectedErrorMessage:
'Unable to run the upgrade command. Aborting the upgrade process.',
},
},
{
title: 'when workspace version is not defined',
context: {
input: {
workspaceOverride: {
version: null,
},
},
expectedErrorMessage:
'Unable to run the upgrade command. Aborting the upgrade process.',
},
},
{
title: 'when APP_VERSION is not defined',
context: {
input: {
appVersion: null,
},
expectedErrorMessage:
'APP_VERSION is not defined, please double check your env variables',
},
},
{
title: 'when previous version is not found',
context: {
input: {
appVersion: UPGRADE_COMMAND_SUPPORTED_VERSIONS[0],
},
expectedErrorMessage: `No previous version found for version ${UPGRADE_COMMAND_SUPPORTED_VERSIONS[0]}`,
},
},
];
it.each(eachTestingContextFilter(failingTestUseCases))(
'$title',
async ({ context: { input, expectedErrorMessage } }) => {
await buildModuleAndSetupSpies(input);
const passedParams: string[] = [];
const options = {};
await expect(
upgradeCommandRunner.run(passedParams, options),
).rejects.toThrow(expectedErrorMessage);
},
);
});
it('should call runSlowInstanceCommand for each current-version slow command', async () => {
@RegisteredInstanceCommand(CURRENT_VERSION, 1780000000000, {
type: 'slow',
})
class SlowMigration1780000000000 implements SlowInstanceCommand {
async runDataMigration(_dataSource: DataSource): Promise<void> {}
async up(_queryRunner: QueryRunner) {}
async down(_queryRunner: QueryRunner) {}
}
const slowMigration = new SlowMigration1780000000000();
const module = await buildModuleAndSetupSpies({
migrations: [slowMigration],
});
const instanceUpgradeService = module.get(InstanceUpgradeService);
await upgradeCommandRunner.run([], {});
expect(instanceUpgradeService.runSlowInstanceCommand).toHaveBeenCalledTimes(
1,
);
expect(instanceUpgradeService.runSlowInstanceCommand).toHaveBeenCalledWith({
command: slowMigration,
name: `${CURRENT_VERSION}_SlowMigration1780000000000_1780000000000`,
skipDataMigration: false,
});
});
it('should run slow commands after fast commands but before workspace commands', async () => {
const executionOrder: string[] = [];
@RegisteredInstanceCommand(CURRENT_VERSION, 1770000000000)
class FastMigration1770000000000 implements FastInstanceCommand {
async up(_queryRunner: QueryRunner) {}
async down(_queryRunner: QueryRunner) {}
}
@RegisteredInstanceCommand(CURRENT_VERSION, 1780000000000, {
type: 'slow',
})
class SlowMigration1780000000000 implements SlowInstanceCommand {
async runDataMigration(_dataSource: DataSource): Promise<void> {}
async up(_queryRunner: QueryRunner) {}
async down(_queryRunner: QueryRunner) {}
}
const module = await buildModuleAndSetupSpies({
migrations: [
new FastMigration1770000000000(),
new SlowMigration1780000000000(),
],
});
const instanceUpgradeService = module.get(InstanceUpgradeService);
(
instanceUpgradeService.runFastInstanceCommand as jest.Mock
).mockImplementation(async () => {
executionOrder.push('fast');
return { status: 'success' };
});
(
instanceUpgradeService.runSlowInstanceCommand as jest.Mock
).mockImplementation(async () => {
executionOrder.push('slow');
return { status: 'success' };
});
const workspaceIteratorService = module.get(WorkspaceIteratorService);
(workspaceIteratorService.iterate as jest.Mock).mockImplementation(
async () => {
executionOrder.push('workspace');
return { success: [], fail: [] };
},
);
await upgradeCommandRunner.run([], {});
expect(executionOrder).toStrictEqual(['fast', 'slow', 'workspace']);
});
it('should pass skipDataMigration: true on fresh install (no workspaces)', async () => {
@RegisteredInstanceCommand(CURRENT_VERSION, 1780000000000, {
type: 'slow',
})
class SlowMigrationFreshInstall implements SlowInstanceCommand {
async runDataMigration(_dataSource: DataSource): Promise<void> {}
async up(_queryRunner: QueryRunner) {}
async down(_queryRunner: QueryRunner) {}
}
const module = await buildModuleAndSetupSpies({
numberOfWorkspace: 0,
migrations: [new SlowMigrationFreshInstall()],
});
const instanceUpgradeService = module.get(InstanceUpgradeService);
await upgradeCommandRunner.run([], {});
expect(instanceUpgradeService.runSlowInstanceCommand).toHaveBeenCalledWith({
command: expect.any(SlowMigrationFreshInstall),
name: `${CURRENT_VERSION}_SlowMigrationFreshInstall_1780000000000`,
skipDataMigration: true,
});
});
it('should propagate errors from runSlowInstanceCommand', async () => {
@RegisteredInstanceCommand(CURRENT_VERSION, 1780000000000, {
type: 'slow',
})
class FailingSlowMigration implements SlowInstanceCommand {
async runDataMigration(_dataSource: DataSource): Promise<void> {}
async up(_queryRunner: QueryRunner) {}
async down(_queryRunner: QueryRunner) {}
}
const module = await buildModuleAndSetupSpies({
migrations: [new FailingSlowMigration()],
});
const instanceUpgradeService = module.get(InstanceUpgradeService);
(
instanceUpgradeService.runSlowInstanceCommand as jest.Mock
).mockResolvedValue({
status: 'failed',
error: new Error('Data migration error'),
});
await expect(upgradeCommandRunner.run([], {})).rejects.toThrow(
'Data migration error',
);
});
});
@@ -5,9 +5,10 @@ import { MigrateMessagingCalendarToCoreFastInstanceCommand } from 'src/database/
import { AddEmailThreadWidgetTypeFastInstanceCommand } from 'src/database/commands/upgrade-version-command/1-21/1-21-instance-command-fast-1775200000000-add-email-thread-widget-type';
import { AddPermissionFlagRoleIdIndexFastInstanceCommand } from 'src/database/commands/upgrade-version-command/1-22/1-22-instance-command-fast-1775749486425-add-permission-flag-role-id-index';
import { AddWorkspaceIdToIndirectEntitiesFastInstanceCommand } from 'src/database/commands/upgrade-version-command/1-22/1-22-instance-command-fast-1775758621017-add-workspace-id-to-indirect-entities';
import { BackfillWorkspaceIdOnIndirectEntitiesSlowInstanceCommand } from 'src/database/commands/upgrade-version-command/1-22/1-22-instance-command-slow-1775758621018-backfill-workspace-id-on-indirect-entities';
import { AddWorkspaceIdIndexesAndFksFastInstanceCommand } from 'src/database/commands/upgrade-version-command/1-22/1-22-instance-command-fast-1775761294897-add-workspace-id-indexes-and-fks-to-indirect-entities';
import { DropObjectMetadataDataSourceFkFastInstanceCommand } from 'src/database/commands/upgrade-version-command/1-22/1-22-instance-command-fast-1775804361516-drop-object-metadata-data-source-fk';
import { BackfillWorkspaceIdOnIndirectEntitiesSlowInstanceCommand } from 'src/database/commands/upgrade-version-command/1-22/1-22-instance-command-slow-1775758621018-backfill-workspace-id-on-indirect-entities';
import { DropWorkspaceVersionColumnFastInstanceCommand } from 'src/database/commands/upgrade-version-command/1-23/1-23-instance-command-fast-1785000000000-drop-workspace-version-column';
export const INSTANCE_COMMANDS = [
AddViewFieldGroupIdIndexOnViewFieldFastInstanceCommand,
@@ -18,4 +19,5 @@ export const INSTANCE_COMMANDS = [
BackfillWorkspaceIdOnIndirectEntitiesSlowInstanceCommand,
AddWorkspaceIdIndexesAndFksFastInstanceCommand,
DropObjectMetadataDataSourceFkFastInstanceCommand,
DropWorkspaceVersionColumnFastInstanceCommand,
];
@@ -3,8 +3,8 @@ import { Module } from '@nestjs/common';
import { WorkspaceIteratorModule } from 'src/database/commands/command-runners/workspace-iterator.module';
import { V1_21_UpgradeVersionCommandModule } from 'src/database/commands/upgrade-version-command/1-21/1-21-upgrade-version-command.module';
import { V1_22_UpgradeVersionCommandModule } from 'src/database/commands/upgrade-version-command/1-22/1-22-upgrade-version-command.module';
import { V1_23_UpgradeVersionCommandModule } from 'src/database/commands/upgrade-version-command/1-23/1-23-upgrade-version-command.module';
import { UpgradeCommand } from 'src/database/commands/upgrade-version-command/upgrade.command';
import { CoreEngineVersionModule } from 'src/engine/core-engine-version/core-engine-version.module';
import { UpgradeModule } from 'src/engine/core-modules/upgrade/upgrade.module';
import { WorkspaceVersionModule } from 'src/engine/workspace-manager/workspace-version/workspace-version.module';
@@ -12,7 +12,7 @@ import { WorkspaceVersionModule } from 'src/engine/workspace-manager/workspace-v
imports: [
V1_21_UpgradeVersionCommandModule,
V1_22_UpgradeVersionCommandModule,
CoreEngineVersionModule,
V1_23_UpgradeVersionCommandModule,
UpgradeModule,
WorkspaceVersionModule,
WorkspaceIteratorModule,
@@ -2,25 +2,18 @@ import { InjectDataSource } from '@nestjs/typeorm';
import chalk from 'chalk';
import { Command, CommandRunner, Option } from 'nest-commander';
import { SemVer } from 'semver';
import { DataSource } from 'typeorm';
import { WorkspaceIteratorService } from 'src/database/commands/command-runners/workspace-iterator.service';
import { CommandLogger } from 'src/database/commands/logger';
import { type UpgradeCommandVersion } from 'src/engine/constants/upgrade-command-supported-versions.constant';
import { CoreEngineVersionService } from 'src/engine/core-engine-version/services/core-engine-version.service';
import { InstanceUpgradeService } from 'src/engine/core-modules/upgrade/services/instance-upgrade.service';
import {
UpgradeCommandRegistryService,
type RegisteredWorkspaceCommand,
type VersionBundle,
} from 'src/engine/core-modules/upgrade/services/upgrade-command-registry.service';
import { WorkspaceUpgradeService } from 'src/engine/core-modules/upgrade/services/workspace-upgrade.service';
import { UpgradeCommandRegistryService } from 'src/engine/core-modules/upgrade/services/upgrade-command-registry.service';
import { UpgradeMigrationService } from 'src/engine/core-modules/upgrade/services/upgrade-migration.service';
import { UpgradeSequenceReaderService } from 'src/engine/core-modules/upgrade/services/upgrade-sequence-reader.service';
import { UpgradeSequenceRunnerService } from 'src/engine/core-modules/upgrade/services/upgrade-sequence-runner.service';
import { RemovedSinceVersion } from 'src/engine/core-modules/upgrade/types/removed-since-version.type';
import { WorkspaceVersionService } from 'src/engine/workspace-manager/workspace-version/services/workspace-version.service';
import { isDefined } from 'twenty-shared/utils';
export type VersionCommands = RegisteredWorkspaceCommand[];
export type UpgradeCommandOptions = {
type RawUpgradeCommandOptions = {
workspaceId?: Set<string>;
startFromWorkspaceId?: string;
workspaceCountLimit?: number;
@@ -28,10 +21,12 @@ export type UpgradeCommandOptions = {
verbose?: boolean;
};
type VersionContext = VersionBundle & {
fromWorkspaceVersion: SemVer;
currentAppVersion: SemVer;
currentVersionMajorMinor: UpgradeCommandVersion;
export type ParsedUpgradeCommandOptions = {
workspaceIds?: string[];
startFromWorkspaceId?: string;
workspaceCountLimit?: number;
dryRun?: boolean;
verbose?: boolean;
};
@Command({
@@ -42,12 +37,11 @@ export class UpgradeCommand extends CommandRunner {
protected logger: CommandLogger;
constructor(
protected readonly coreEngineVersionService: CoreEngineVersionService,
protected readonly workspaceVersionService: WorkspaceVersionService,
protected readonly upgradeCommandRegistryService: UpgradeCommandRegistryService,
protected readonly instanceUpgradeService: InstanceUpgradeService,
protected readonly workspaceIteratorService: WorkspaceIteratorService,
protected readonly workspaceUpgradeService: WorkspaceUpgradeService,
protected readonly upgradeSequenceReaderService: UpgradeSequenceReaderService,
protected readonly upgradeSequenceRunnerService: UpgradeSequenceRunnerService,
protected readonly upgradeMigrationService: UpgradeMigrationService,
protected readonly workspaceVersionService: WorkspaceVersionService,
@InjectDataSource()
protected readonly dataSource: DataSource,
) {
@@ -122,7 +116,7 @@ export class UpgradeCommand extends CommandRunner {
override async run(
_passedParams: string[],
options: UpgradeCommandOptions,
options: RawUpgradeCommandOptions,
): Promise<void> {
if (options.verbose) {
this.logger = new CommandLogger({
@@ -132,93 +126,44 @@ export class UpgradeCommand extends CommandRunner {
}
try {
const versionContext = this.resolveVersionContext();
await this.runBootstrapMigrations();
await this.backfillWorkspaceCreatedIn1_21_0Cursors();
const sequence = this.upgradeSequenceReaderService.getUpgradeSequence();
this.logger.log(
chalk.blue(
[
'Initialized upgrade context with:',
`- currentVersion (migrating to): ${versionContext.currentAppVersion}`,
`- fromWorkspaceVersion: ${versionContext.fromWorkspaceVersion}`,
`- ${versionContext.fastInstanceCommands.length} fast instance commands (from registry)`,
`- ${versionContext.slowInstanceCommands.length} slow instance commands (from registry)`,
`- ${versionContext.workspaceCommands.length} workspace commands`,
'Initialized upgrade sequence:',
`- ${sequence.length} step(s)`,
...sequence.map(
(step, index) =>
` [${index}] ${step.kind}${step.name} (${step.version})`,
),
].join('\n '),
),
);
const workspacesBelowMinimumVersion =
await this.workspaceVersionService.getWorkspacesBelowVersion(
versionContext.fromWorkspaceVersion.version,
);
if (workspacesBelowMinimumVersion.length > 0) {
const ineligibleIds = workspacesBelowMinimumVersion
.map((workspace) => workspace.id)
.join(', ');
throw new Error(
`Unable to run the upgrade command. Aborting the upgrade process.
Workspaces below minimum version (${versionContext.fromWorkspaceVersion.version}): ${ineligibleIds}.
Please roll back to that version and run the upgrade command again.`,
);
}
await this.runLegacyPendingTypeOrmMigrations();
for (const { command, name } of versionContext.fastInstanceCommands) {
const result = await this.instanceUpgradeService.runFastInstanceCommand(
{
command,
name,
const { totalSuccesses, totalFailures } =
await this.upgradeSequenceRunnerService.run({
sequence,
options: {
...options,
workspaceIds: isDefined(options.workspaceId)
? Array.from(options.workspaceId)
: undefined,
},
);
if (result.status === 'failed') {
throw result.error;
}
}
const hasWorkspaces =
await this.workspaceVersionService.hasActiveOrSuspendedWorkspaces();
for (const { command, name } of versionContext.slowInstanceCommands) {
const result = await this.instanceUpgradeService.runSlowInstanceCommand(
{
command,
name,
skipDataMigration: !hasWorkspaces,
},
);
if (result.status === 'failed') {
throw result.error;
}
}
if (!hasWorkspaces) {
this.logger.log(
chalk.blue(
'Fresh installation detected, skipping workspace commands',
),
);
return;
}
const iteratorReport = await this.runWorkspaceCommands(
options,
versionContext,
);
});
this.logger.log(
chalk.blue(
`Upgrade summary: ${iteratorReport.success.length} succeeded, ${iteratorReport.fail.length} failed`,
`Upgrade summary: ${totalSuccesses} workspace(s) succeeded, ${totalFailures} workspace(s) failed`,
),
);
if (iteratorReport.fail.length > 0) {
if (totalFailures > 0) {
throw new Error(
`Upgrade completed with ${iteratorReport.fail.length} workspace failure(s)`,
`Upgrade completed with ${totalFailures} workspace failure(s)`,
);
}
} catch (error) {
@@ -227,70 +172,110 @@ Please roll back to that version and run the upgrade command again.`,
}
}
private async runLegacyPendingTypeOrmMigrations(): Promise<void> {
this.logger.log('Running legacy TypeORM migrations...');
// Workspaces created during 1.21 were activated before the cursor-based
// upgrade system existed. They have no upgradeMigration record yet.
// Stamp them with the last 1.21 workspace command as their initial cursor.
private async backfillWorkspaceCreatedIn1_21_0Cursors(): RemovedSinceVersion<
'1.23.0',
Promise<void>
> {
const allWorkspaceIds =
await this.workspaceVersionService.getActiveOrSuspendedWorkspaceIds();
const migrations = await this.dataSource.runMigrations({
transaction: 'each',
});
if (allWorkspaceIds.length === 0) {
return;
}
if (migrations.length === 0) {
this.logger.log('No pending legacy migrations');
} else {
this.logger.log(
`Executed ${migrations.length} legacy migration(s): ${migrations.map((migration) => migration.name).join(', ')}`,
const existingCursorWorkspaceIds: { workspaceId: string }[] =
await this.dataSource.query(
`SELECT DISTINCT "workspaceId" FROM "core"."upgradeMigration" WHERE "workspaceId" IS NOT NULL`,
);
const existingCursorSet = new Set(
existingCursorWorkspaceIds.map((row) => row.workspaceId),
);
const workspacesWithoutCursor = allWorkspaceIds.filter(
(workspaceId) => !existingCursorSet.has(workspaceId),
);
if (workspacesWithoutCursor.length === 0) {
return;
}
const lastWorkspaceCommand =
this.upgradeCommandRegistryService.getLastWorkspaceCommandForVersion(
'1.21.0',
);
if (!lastWorkspaceCommand) {
throw new Error(
`Cannot backfill workspace cursors: no workspace commands found for version 1.21.0`,
);
}
this.logger.log(
chalk.blue(
`Backfilling initial cursor for ${workspacesWithoutCursor.length} workspace(s) → "${lastWorkspaceCommand.name}"`,
),
);
for (const workspaceId of workspacesWithoutCursor) {
await this.upgradeMigrationService.markAsInitial({
name: lastWorkspaceCommand.name,
workspaceId,
executedByVersion: '1.21.0',
});
}
}
private resolveVersionContext(): VersionContext {
const currentAppVersion = this.coreEngineVersionService.getCurrentVersion();
const currentVersionMajorMinor =
`${currentAppVersion.major}.${currentAppVersion.minor}.0` as UpgradeCommandVersion;
// Schema changes required by the upgrade engine itself (e.g. new columns
// on upgradeMigration) must be applied before the sequence runs.
private async runBootstrapMigrations(): RemovedSinceVersion<
'1.23.0',
Promise<void>
> {
const BOOTSTRAP_MIGRATION = 'AddIsInitialToUpgradeMigration1775909335324';
const fromWorkspaceVersion =
this.coreEngineVersionService.getPreviousVersion();
const alreadyExecuted = await this.dataSource.query(
`SELECT 1 FROM "core"."_typeorm_migrations" WHERE "name" = $1`,
[BOOTSTRAP_MIGRATION],
);
const { fastInstanceCommands, slowInstanceCommands, workspaceCommands } =
this.upgradeCommandRegistryService.getBundleForVersion(
currentVersionMajorMinor,
if (alreadyExecuted.length > 0) {
return;
}
const migration = this.dataSource.migrations.find(
(migration) => migration.name === BOOTSTRAP_MIGRATION,
);
if (!migration) {
throw new Error(
`Bootstrap migration "${BOOTSTRAP_MIGRATION}" not found in registered migrations`,
);
}
const queryRunner = this.dataSource.createQueryRunner();
await queryRunner.connect();
await queryRunner.startTransaction();
try {
await migration.up(queryRunner);
await queryRunner.query(
`INSERT INTO "core"."_typeorm_migrations" ("timestamp", "name") VALUES ($1, $2)`,
[1775909335324, BOOTSTRAP_MIGRATION],
);
return {
fromWorkspaceVersion,
currentAppVersion,
currentVersionMajorMinor,
fastInstanceCommands,
slowInstanceCommands,
workspaceCommands,
};
}
await queryRunner.commitTransaction();
} catch (error) {
await queryRunner.rollbackTransaction();
private async runWorkspaceCommands(
options: UpgradeCommandOptions,
{
currentAppVersion,
fromWorkspaceVersion,
workspaceCommands,
}: VersionContext,
) {
return await this.workspaceIteratorService.iterate({
workspaceIds:
options.workspaceId && options.workspaceId.size > 0
? Array.from(options.workspaceId)
: undefined,
startFromWorkspaceId: options.startFromWorkspaceId,
workspaceCountLimit: options.workspaceCountLimit,
dryRun: options.dryRun,
callback: async (context) => {
await this.workspaceUpgradeService.upgradeWorkspace({
iteratorContext: context,
options,
fromWorkspaceVersion,
currentAppVersion,
workspaceCommands,
});
},
});
throw error;
} finally {
await queryRunner.release();
}
}
}