Implement cross version upgrade (#19559)
# Introduction Refactoring the upgrade engine to handle cross version upgrade, completely getting rid of the semver `version` at db and runtime level It remains a visual a listing indicator for or CD process but also during devenv in order to prepare next release Will write a release process runbook documentation on how to handle upgrade step patch, command insertion etc as it needs to be cascaded across all the involved supported version **The upgrade sequence model:** The sequence is a flat, ordered array of upgrade steps (`UpgradeStep[]`), built from the registry by chaining all versions in order, each version contributing its fast-instance → slow-instance → workspace commands sorted by timestamp. Version is metadata for logging, not used in the algorithm. **Segments:** The sequence naturally splits into alternating segments of contiguous instance steps and contiguous workspace steps. The runner processes segments in order: - **Instance segment:** Run sequentially from the instance cursor. Each step runs once globally. - **Workspace segment:** Each workspace independently walks from its own cursor through the end of the segment. Workspaces are independent within a segment — they can be at different positions. - **Synchronization (workspace → instance):** The runner blocks before entering an instance segment. All active/suspended workspaces must have completed the last workspace step of the preceding workspace segment. If any workspace failed, abort. This is the only explicit synchronization point. - Instance → workspace ordering is implicit — the runner processes segments sequentially, so the instance segment naturally completes before the workspace segment begins. full docs https://gist.github.com/prastoin/e62106d455fd72d6b6ebada8351e5492 ## Version constants & type-level deprecation Version management is split into three atomic constants: `TWENTY_PREVIOUS_VERSIONS`, `TWENTY_CURRENT_VERSION`, and `TWENTY_NEXT_VERSIONS`. Two derived constants compose them: `CROSS_UPGRADE_SUPPORTED_VERSIONS` (previous + current — what the engine runs) and `ALL_TWENTY_VERSIONS` (the full ordered tuple including next). The registry service validates at module init that no version is duplicated across constants and that at least one previous version exists. A `DeprecatedSinceVersion<RemoveAtVersion, T>` type utility resolves to `T` while `TWENTY_CURRENT_VERSION` is below `RemoveAtVersion`, and to `never` once it reaches it — turning deprecation into a compile-time guarantee via `IndexOf` and `IsGreaterOrEqual` generics in `twenty-shared`. ### `workspace.version` column deprecation The column is replaced by cursor-based state inference from `UpgradeMigration` records, but cannot be dropped in 1.22: workspaces activated during 1.21 predate the cursor system and need their initial cursor backfilled first (`backfillWorkspaceCreatedIn1_21_0Cursors`). This backfill itself depends on a new `isInitial` column on `UpgradeMigration`, bootstrapped via a targeted TypeORM migration before the upgrade sequence runs. Both functions and the entity field are typed with `DeprecatedSinceVersion<'1.23.0', ...>`. When `TWENTY_CURRENT_VERSION` reaches `1.23.0`, compile errors force their removal — and the pre-declared `DropWorkspaceVersionColumnFastInstanceCommand` takes over to drop the column. ## What's next - ci cross version upgrade ( wip ) - banner asking to contact twenty administrator if workspace is outdated - upgrade healthcheck cli ## New unit/integ test pattern Create a dedicated `createNestApp` that consumes a real database in order not to have to mack any database interaction to the `upgradeMigrations` allowing full coverage of the whole `upgradeRunnerService.run` core logic
This commit is contained in:
Paul Rastoin
+135
-150
@@ -2,25 +2,18 @@ import { InjectDataSource } from '@nestjs/typeorm';
|
||||
|
||||
import chalk from 'chalk';
|
||||
import { Command, CommandRunner, Option } from 'nest-commander';
|
||||
import { SemVer } from 'semver';
|
||||
import { DataSource } from 'typeorm';
|
||||
|
||||
import { WorkspaceIteratorService } from 'src/database/commands/command-runners/workspace-iterator.service';
|
||||
import { CommandLogger } from 'src/database/commands/logger';
|
||||
import { type UpgradeCommandVersion } from 'src/engine/constants/upgrade-command-supported-versions.constant';
|
||||
import { CoreEngineVersionService } from 'src/engine/core-engine-version/services/core-engine-version.service';
|
||||
import { InstanceUpgradeService } from 'src/engine/core-modules/upgrade/services/instance-upgrade.service';
|
||||
import {
|
||||
UpgradeCommandRegistryService,
|
||||
type RegisteredWorkspaceCommand,
|
||||
type VersionBundle,
|
||||
} from 'src/engine/core-modules/upgrade/services/upgrade-command-registry.service';
|
||||
import { WorkspaceUpgradeService } from 'src/engine/core-modules/upgrade/services/workspace-upgrade.service';
|
||||
import { UpgradeCommandRegistryService } from 'src/engine/core-modules/upgrade/services/upgrade-command-registry.service';
|
||||
import { UpgradeMigrationService } from 'src/engine/core-modules/upgrade/services/upgrade-migration.service';
|
||||
import { UpgradeSequenceReaderService } from 'src/engine/core-modules/upgrade/services/upgrade-sequence-reader.service';
|
||||
import { UpgradeSequenceRunnerService } from 'src/engine/core-modules/upgrade/services/upgrade-sequence-runner.service';
|
||||
import { RemovedSinceVersion } from 'src/engine/core-modules/upgrade/types/removed-since-version.type';
|
||||
import { WorkspaceVersionService } from 'src/engine/workspace-manager/workspace-version/services/workspace-version.service';
|
||||
import { isDefined } from 'twenty-shared/utils';
|
||||
|
||||
export type VersionCommands = RegisteredWorkspaceCommand[];
|
||||
|
||||
export type UpgradeCommandOptions = {
|
||||
type RawUpgradeCommandOptions = {
|
||||
workspaceId?: Set<string>;
|
||||
startFromWorkspaceId?: string;
|
||||
workspaceCountLimit?: number;
|
||||
@@ -28,10 +21,12 @@ export type UpgradeCommandOptions = {
|
||||
verbose?: boolean;
|
||||
};
|
||||
|
||||
type VersionContext = VersionBundle & {
|
||||
fromWorkspaceVersion: SemVer;
|
||||
currentAppVersion: SemVer;
|
||||
currentVersionMajorMinor: UpgradeCommandVersion;
|
||||
export type ParsedUpgradeCommandOptions = {
|
||||
workspaceIds?: string[];
|
||||
startFromWorkspaceId?: string;
|
||||
workspaceCountLimit?: number;
|
||||
dryRun?: boolean;
|
||||
verbose?: boolean;
|
||||
};
|
||||
|
||||
@Command({
|
||||
@@ -42,12 +37,11 @@ export class UpgradeCommand extends CommandRunner {
|
||||
protected logger: CommandLogger;
|
||||
|
||||
constructor(
|
||||
protected readonly coreEngineVersionService: CoreEngineVersionService,
|
||||
protected readonly workspaceVersionService: WorkspaceVersionService,
|
||||
protected readonly upgradeCommandRegistryService: UpgradeCommandRegistryService,
|
||||
protected readonly instanceUpgradeService: InstanceUpgradeService,
|
||||
protected readonly workspaceIteratorService: WorkspaceIteratorService,
|
||||
protected readonly workspaceUpgradeService: WorkspaceUpgradeService,
|
||||
protected readonly upgradeSequenceReaderService: UpgradeSequenceReaderService,
|
||||
protected readonly upgradeSequenceRunnerService: UpgradeSequenceRunnerService,
|
||||
protected readonly upgradeMigrationService: UpgradeMigrationService,
|
||||
protected readonly workspaceVersionService: WorkspaceVersionService,
|
||||
@InjectDataSource()
|
||||
protected readonly dataSource: DataSource,
|
||||
) {
|
||||
@@ -122,7 +116,7 @@ export class UpgradeCommand extends CommandRunner {
|
||||
|
||||
override async run(
|
||||
_passedParams: string[],
|
||||
options: UpgradeCommandOptions,
|
||||
options: RawUpgradeCommandOptions,
|
||||
): Promise<void> {
|
||||
if (options.verbose) {
|
||||
this.logger = new CommandLogger({
|
||||
@@ -132,93 +126,44 @@ export class UpgradeCommand extends CommandRunner {
|
||||
}
|
||||
|
||||
try {
|
||||
const versionContext = this.resolveVersionContext();
|
||||
await this.runBootstrapMigrations();
|
||||
await this.backfillWorkspaceCreatedIn1_21_0Cursors();
|
||||
|
||||
const sequence = this.upgradeSequenceReaderService.getUpgradeSequence();
|
||||
|
||||
this.logger.log(
|
||||
chalk.blue(
|
||||
[
|
||||
'Initialized upgrade context with:',
|
||||
`- currentVersion (migrating to): ${versionContext.currentAppVersion}`,
|
||||
`- fromWorkspaceVersion: ${versionContext.fromWorkspaceVersion}`,
|
||||
`- ${versionContext.fastInstanceCommands.length} fast instance commands (from registry)`,
|
||||
`- ${versionContext.slowInstanceCommands.length} slow instance commands (from registry)`,
|
||||
`- ${versionContext.workspaceCommands.length} workspace commands`,
|
||||
'Initialized upgrade sequence:',
|
||||
`- ${sequence.length} step(s)`,
|
||||
...sequence.map(
|
||||
(step, index) =>
|
||||
` [${index}] ${step.kind} — ${step.name} (${step.version})`,
|
||||
),
|
||||
].join('\n '),
|
||||
),
|
||||
);
|
||||
|
||||
const workspacesBelowMinimumVersion =
|
||||
await this.workspaceVersionService.getWorkspacesBelowVersion(
|
||||
versionContext.fromWorkspaceVersion.version,
|
||||
);
|
||||
|
||||
if (workspacesBelowMinimumVersion.length > 0) {
|
||||
const ineligibleIds = workspacesBelowMinimumVersion
|
||||
.map((workspace) => workspace.id)
|
||||
.join(', ');
|
||||
|
||||
throw new Error(
|
||||
`Unable to run the upgrade command. Aborting the upgrade process.
|
||||
Workspaces below minimum version (${versionContext.fromWorkspaceVersion.version}): ${ineligibleIds}.
|
||||
Please roll back to that version and run the upgrade command again.`,
|
||||
);
|
||||
}
|
||||
|
||||
await this.runLegacyPendingTypeOrmMigrations();
|
||||
|
||||
for (const { command, name } of versionContext.fastInstanceCommands) {
|
||||
const result = await this.instanceUpgradeService.runFastInstanceCommand(
|
||||
{
|
||||
command,
|
||||
name,
|
||||
const { totalSuccesses, totalFailures } =
|
||||
await this.upgradeSequenceRunnerService.run({
|
||||
sequence,
|
||||
options: {
|
||||
...options,
|
||||
workspaceIds: isDefined(options.workspaceId)
|
||||
? Array.from(options.workspaceId)
|
||||
: undefined,
|
||||
},
|
||||
);
|
||||
|
||||
if (result.status === 'failed') {
|
||||
throw result.error;
|
||||
}
|
||||
}
|
||||
|
||||
const hasWorkspaces =
|
||||
await this.workspaceVersionService.hasActiveOrSuspendedWorkspaces();
|
||||
|
||||
for (const { command, name } of versionContext.slowInstanceCommands) {
|
||||
const result = await this.instanceUpgradeService.runSlowInstanceCommand(
|
||||
{
|
||||
command,
|
||||
name,
|
||||
skipDataMigration: !hasWorkspaces,
|
||||
},
|
||||
);
|
||||
|
||||
if (result.status === 'failed') {
|
||||
throw result.error;
|
||||
}
|
||||
}
|
||||
|
||||
if (!hasWorkspaces) {
|
||||
this.logger.log(
|
||||
chalk.blue(
|
||||
'Fresh installation detected, skipping workspace commands',
|
||||
),
|
||||
);
|
||||
|
||||
return;
|
||||
}
|
||||
|
||||
const iteratorReport = await this.runWorkspaceCommands(
|
||||
options,
|
||||
versionContext,
|
||||
);
|
||||
});
|
||||
|
||||
this.logger.log(
|
||||
chalk.blue(
|
||||
`Upgrade summary: ${iteratorReport.success.length} succeeded, ${iteratorReport.fail.length} failed`,
|
||||
`Upgrade summary: ${totalSuccesses} workspace(s) succeeded, ${totalFailures} workspace(s) failed`,
|
||||
),
|
||||
);
|
||||
|
||||
if (iteratorReport.fail.length > 0) {
|
||||
if (totalFailures > 0) {
|
||||
throw new Error(
|
||||
`Upgrade completed with ${iteratorReport.fail.length} workspace failure(s)`,
|
||||
`Upgrade completed with ${totalFailures} workspace failure(s)`,
|
||||
);
|
||||
}
|
||||
} catch (error) {
|
||||
@@ -227,70 +172,110 @@ Please roll back to that version and run the upgrade command again.`,
|
||||
}
|
||||
}
|
||||
|
||||
private async runLegacyPendingTypeOrmMigrations(): Promise<void> {
|
||||
this.logger.log('Running legacy TypeORM migrations...');
|
||||
// Workspaces created during 1.21 were activated before the cursor-based
|
||||
// upgrade system existed. They have no upgradeMigration record yet.
|
||||
// Stamp them with the last 1.21 workspace command as their initial cursor.
|
||||
private async backfillWorkspaceCreatedIn1_21_0Cursors(): RemovedSinceVersion<
|
||||
'1.23.0',
|
||||
Promise<void>
|
||||
> {
|
||||
const allWorkspaceIds =
|
||||
await this.workspaceVersionService.getActiveOrSuspendedWorkspaceIds();
|
||||
|
||||
const migrations = await this.dataSource.runMigrations({
|
||||
transaction: 'each',
|
||||
});
|
||||
if (allWorkspaceIds.length === 0) {
|
||||
return;
|
||||
}
|
||||
|
||||
if (migrations.length === 0) {
|
||||
this.logger.log('No pending legacy migrations');
|
||||
} else {
|
||||
this.logger.log(
|
||||
`Executed ${migrations.length} legacy migration(s): ${migrations.map((migration) => migration.name).join(', ')}`,
|
||||
const existingCursorWorkspaceIds: { workspaceId: string }[] =
|
||||
await this.dataSource.query(
|
||||
`SELECT DISTINCT "workspaceId" FROM "core"."upgradeMigration" WHERE "workspaceId" IS NOT NULL`,
|
||||
);
|
||||
|
||||
const existingCursorSet = new Set(
|
||||
existingCursorWorkspaceIds.map((row) => row.workspaceId),
|
||||
);
|
||||
|
||||
const workspacesWithoutCursor = allWorkspaceIds.filter(
|
||||
(workspaceId) => !existingCursorSet.has(workspaceId),
|
||||
);
|
||||
|
||||
if (workspacesWithoutCursor.length === 0) {
|
||||
return;
|
||||
}
|
||||
|
||||
const lastWorkspaceCommand =
|
||||
this.upgradeCommandRegistryService.getLastWorkspaceCommandForVersion(
|
||||
'1.21.0',
|
||||
);
|
||||
|
||||
if (!lastWorkspaceCommand) {
|
||||
throw new Error(
|
||||
`Cannot backfill workspace cursors: no workspace commands found for version 1.21.0`,
|
||||
);
|
||||
}
|
||||
|
||||
this.logger.log(
|
||||
chalk.blue(
|
||||
`Backfilling initial cursor for ${workspacesWithoutCursor.length} workspace(s) → "${lastWorkspaceCommand.name}"`,
|
||||
),
|
||||
);
|
||||
|
||||
for (const workspaceId of workspacesWithoutCursor) {
|
||||
await this.upgradeMigrationService.markAsInitial({
|
||||
name: lastWorkspaceCommand.name,
|
||||
workspaceId,
|
||||
executedByVersion: '1.21.0',
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
private resolveVersionContext(): VersionContext {
|
||||
const currentAppVersion = this.coreEngineVersionService.getCurrentVersion();
|
||||
const currentVersionMajorMinor =
|
||||
`${currentAppVersion.major}.${currentAppVersion.minor}.0` as UpgradeCommandVersion;
|
||||
// Schema changes required by the upgrade engine itself (e.g. new columns
|
||||
// on upgradeMigration) must be applied before the sequence runs.
|
||||
private async runBootstrapMigrations(): RemovedSinceVersion<
|
||||
'1.23.0',
|
||||
Promise<void>
|
||||
> {
|
||||
const BOOTSTRAP_MIGRATION = 'AddIsInitialToUpgradeMigration1775909335324';
|
||||
|
||||
const fromWorkspaceVersion =
|
||||
this.coreEngineVersionService.getPreviousVersion();
|
||||
const alreadyExecuted = await this.dataSource.query(
|
||||
`SELECT 1 FROM "core"."_typeorm_migrations" WHERE "name" = $1`,
|
||||
[BOOTSTRAP_MIGRATION],
|
||||
);
|
||||
|
||||
const { fastInstanceCommands, slowInstanceCommands, workspaceCommands } =
|
||||
this.upgradeCommandRegistryService.getBundleForVersion(
|
||||
currentVersionMajorMinor,
|
||||
if (alreadyExecuted.length > 0) {
|
||||
return;
|
||||
}
|
||||
|
||||
const migration = this.dataSource.migrations.find(
|
||||
(migration) => migration.name === BOOTSTRAP_MIGRATION,
|
||||
);
|
||||
|
||||
if (!migration) {
|
||||
throw new Error(
|
||||
`Bootstrap migration "${BOOTSTRAP_MIGRATION}" not found in registered migrations`,
|
||||
);
|
||||
}
|
||||
|
||||
const queryRunner = this.dataSource.createQueryRunner();
|
||||
|
||||
await queryRunner.connect();
|
||||
await queryRunner.startTransaction();
|
||||
|
||||
try {
|
||||
await migration.up(queryRunner);
|
||||
|
||||
await queryRunner.query(
|
||||
`INSERT INTO "core"."_typeorm_migrations" ("timestamp", "name") VALUES ($1, $2)`,
|
||||
[1775909335324, BOOTSTRAP_MIGRATION],
|
||||
);
|
||||
|
||||
return {
|
||||
fromWorkspaceVersion,
|
||||
currentAppVersion,
|
||||
currentVersionMajorMinor,
|
||||
fastInstanceCommands,
|
||||
slowInstanceCommands,
|
||||
workspaceCommands,
|
||||
};
|
||||
}
|
||||
await queryRunner.commitTransaction();
|
||||
} catch (error) {
|
||||
await queryRunner.rollbackTransaction();
|
||||
|
||||
private async runWorkspaceCommands(
|
||||
options: UpgradeCommandOptions,
|
||||
{
|
||||
currentAppVersion,
|
||||
fromWorkspaceVersion,
|
||||
workspaceCommands,
|
||||
}: VersionContext,
|
||||
) {
|
||||
return await this.workspaceIteratorService.iterate({
|
||||
workspaceIds:
|
||||
options.workspaceId && options.workspaceId.size > 0
|
||||
? Array.from(options.workspaceId)
|
||||
: undefined,
|
||||
startFromWorkspaceId: options.startFromWorkspaceId,
|
||||
workspaceCountLimit: options.workspaceCountLimit,
|
||||
dryRun: options.dryRun,
|
||||
callback: async (context) => {
|
||||
await this.workspaceUpgradeService.upgradeWorkspace({
|
||||
iteratorContext: context,
|
||||
options,
|
||||
fromWorkspaceVersion,
|
||||
currentAppVersion,
|
||||
workspaceCommands,
|
||||
});
|
||||
},
|
||||
});
|
||||
throw error;
|
||||
} finally {
|
||||
await queryRunner.release();
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user