[Upgrade] Stop command gracefully on SIGINT/SIGTERM (#23481)

Ctrl+C on `upgrade` used to kill the process wherever it happened to be, potentially in the middle of a workspace command. It now stops at the next iteration boundary instead. ## Behavior - **First SIGINT/SIGTERM** — the runner finishes what it started, then stops instead of starting new work. Exits with `130` (SIGINT) or `143` (SIGTERM), following the 128+signal convention, so orchestrators can tell an interruption apart from a failure. - **Second signal** — immediate exit, leaving the command in progress unfinished. - **SIGKILL** — untrappable, same outcome as a second signal. Nothing is rolled back on stop: the run resumes from the last command recorded in `upgradeMigration`. ## Opt-in per command Registering a `SIGINT` listener removes Node's default kill-on-signal behavior, so a command that installs a handler without honoring the flag would ignore the first Ctrl+C entirely. Handlers are therefore opt-in via `CommandShutdownService.listenToShutdownSignals()`, called by the two commands that stop at a boundary: - `UpgradeCommand` - `WorkspaceCommandRunner`, the base for standalone workspace commands Everything else keeps today's behavior and dies on the first signal, `run-instance-commands` included: instance commands are transactional and cursor-guarded, so a hard kill rolls back and a rerun skips what completed. `install-application`, `rebuild-application-default-deps` and `install-pre-installed-apps` iterate over workspaces without going through `WorkspaceCommandRunner`, so they are not armed either; they are one call away if we want them. The server and worker processes share these services and never arm anything, so their shutdown semantics are unchanged. ## Where the flag is checked `CommandShutdownService` exposes a single boolean, `isShutdownRequested()`, read only by the iteration runners: - `UpgradeSequenceRunnerService.runInner` — before each sequence step - `WorkspaceIteratorService.iterate` — before each workspace There is deliberately no `AbortSignal`: in-flight work is never cancelled, it is allowed to finish. Individual commands know nothing about shutdown, so a workspace that has started runs its whole pending segment before the run stops. Each workspace ends up either fully done with the segment or untouched, never scattered at some cursor inside it. That keeps resume state coarse and the change out of the command layer, at the cost of a longer stop latency, which the second Ctrl+C covers. `WorkspaceIteratorReport` gained an `interrupted` flag. The sequence runner needs it: stopping partway through the workspace list and then advancing the cursor would run an instance step against workspaces that are not aligned yet, so it returns instead. ## Deployment note Under Kubernetes, `terminationGracePeriodSeconds` must exceed the time for one workspace to finish its segment, otherwise the SIGTERM path degrades into a SIGKILL. Documented in `docs/UPGRADE_COMMANDS.md`. ## Testing - New unit test for `CommandShutdownService` (7 cases); 293 tests pass across `database/commands` and `core-modules/upgrade` - `tsgo -p tsconfig.json` clean - oxlint and oxfmt clean on all touched files
2026-07-29 16:49:47 +02:00
parent 742f76e318
commit 133b3375b6
14 changed files with 395 additions and 5 deletions
@@ -180,7 +180,7 @@ export class ApplicationUpgradeService {
    // provisioned workspace, which would upgrade workspaces that were
    // filtered out.
    if (!isNonEmptyArray(applications)) {
-      return { success: [], fail: [] };
+      return { success: [], fail: [], interrupted: false };
    }

    return this.workspaceIteratorService.iterate({
@@ -0,0 +1,139 @@
+import { type WorkspaceIteratorService } from 'src/database/commands/command-runners/workspace-iterator.service';
+import { type CommandShutdownService } from 'src/database/commands/command-runners/command-shutdown.service';
+import { type InstanceCommandRunnerService } from 'src/engine/core-modules/upgrade/services/instance-command-runner.service';
+import { type UpgradeMigrationService } from 'src/engine/core-modules/upgrade/services/upgrade-migration.service';
+import {
+  type UpgradeStep,
+  type UpgradeSequenceReaderService,
+} from 'src/engine/core-modules/upgrade/services/upgrade-sequence-reader.service';
+import { UpgradeSequenceRunnerService } from 'src/engine/core-modules/upgrade/services/upgrade-sequence-runner.service';
+import { type WorkspaceCommandRunnerService } from 'src/engine/core-modules/upgrade/services/workspace-command-runner.service';
+import { type UpgradeAwareEntityMetadataAdapter } from 'src/engine/twenty-orm/upgrade-aware/upgrade-aware-entity-metadata.adapter';
+import { type WorkspaceVersionService } from 'src/engine/workspace-manager/workspace-version/services/workspace-version.service';
+
+const WORKSPACE_ID = 'workspace-1';
+
+const makeStep = (kind: UpgradeStep['kind'], name: string) =>
+  ({
+    kind,
+    name,
+    command: {},
+    version: '1.0.0',
+    timestamp: 0,
+  }) as unknown as UpgradeStep;
+
+describe('UpgradeSequenceRunnerService shutdown handling', () => {
+  let service: UpgradeSequenceRunnerService;
+  let isShutdownRequested: jest.Mock<boolean, []>;
+  let runFastInstanceCommand: jest.Mock;
+  let iterate: jest.Mock;
+
+  beforeEach(() => {
+    isShutdownRequested = jest.fn().mockReturnValue(false);
+    runFastInstanceCommand = jest.fn().mockResolvedValue({ status: 'success' });
+    iterate = jest.fn().mockResolvedValue({
+      success: [{ workspaceId: WORKSPACE_ID }],
+      fail: [],
+      interrupted: false,
+    });
+
+    const upgradeMigrationService = {
+      getLastAttemptedCommandNameOrThrow: jest
+        .fn()
+        .mockResolvedValue({ name: 'step-1', status: 'failed' }),
+      getWorkspaceLastAttemptedCommandNameOrThrow: jest
+        .fn()
+        .mockResolvedValue(new Map()),
+    } as unknown as UpgradeMigrationService;
+
+    const upgradeSequenceReaderService = {
+      locateStepInSequenceOrThrow: jest.fn().mockReturnValue(0),
+      getWorkspaceSegmentBounds: jest
+        .fn()
+        .mockReturnValue({ startCursor: 0, endCursor: 0 }),
+      collectWorkspaceCommandsStartingFrom: jest
+        .fn()
+        .mockImplementation(({ fromWorkspaceCommand }) => [
+          fromWorkspaceCommand,
+        ]),
+      getPendingWorkspaceCommands: jest.fn().mockReturnValue([]),
+    } as unknown as UpgradeSequenceReaderService;
+
+    service = new UpgradeSequenceRunnerService(
+      upgradeMigrationService,
+      { runFastInstanceCommand } as unknown as InstanceCommandRunnerService,
+      {
+        runWorkspaceCommands: jest.fn(),
+      } as unknown as WorkspaceCommandRunnerService,
+      upgradeSequenceReaderService,
+      {
+        refresh: jest.fn().mockResolvedValue(undefined),
+      } as unknown as UpgradeAwareEntityMetadataAdapter,
+      { iterate } as unknown as WorkspaceIteratorService,
+      {
+        getProvisionedWorkspaceIds: jest.fn().mockResolvedValue([WORKSPACE_ID]),
+      } as unknown as WorkspaceVersionService,
+      { isShutdownRequested } as unknown as CommandShutdownService,
+    );
+
+    jest.spyOn(service['logger'], 'log').mockImplementation();
+    jest.spyOn(service['logger'], 'warn').mockImplementation();
+    jest.spyOn(service['logger'], 'error').mockImplementation();
+  });
+
+  it('should run every step when no shutdown is requested', async () => {
+    const sequence = [
+      makeStep('fast-instance', 'step-1'),
+      makeStep('fast-instance', 'step-2'),
+    ];
+
+    await service.run({ sequence, options: {} });
+
+    expect(runFastInstanceCommand).toHaveBeenCalledTimes(2);
+  });
+
+  it('should stop before the next step when a shutdown is requested', async () => {
+    const sequence = [
+      makeStep('fast-instance', 'step-1'),
+      makeStep('fast-instance', 'step-2'),
+    ];
+
+    isShutdownRequested.mockReturnValueOnce(false).mockReturnValue(true);
+
+    await service.run({ sequence, options: {} });
+
+    expect(runFastInstanceCommand).toHaveBeenCalledTimes(1);
+    expect(runFastInstanceCommand).toHaveBeenCalledWith(
+      expect.objectContaining({ name: 'step-1' }),
+    );
+  });
+
+  it('should not advance to the next instance step when a workspace segment is interrupted', async () => {
+    const sequence = [
+      makeStep('workspace', 'workspace-step-1'),
+      makeStep('fast-instance', 'instance-step-1'),
+    ];
+
+    iterate.mockResolvedValue({
+      success: [{ workspaceId: WORKSPACE_ID }],
+      fail: [],
+      interrupted: true,
+    });
+
+    const report = await service.run({ sequence, options: {} });
+
+    expect(runFastInstanceCommand).not.toHaveBeenCalled();
+    expect(report).toEqual({ totalSuccesses: 1, totalFailures: 0 });
+  });
+
+  it('should advance to the next instance step when a workspace segment completes', async () => {
+    const sequence = [
+      makeStep('workspace', 'workspace-step-1'),
+      makeStep('fast-instance', 'instance-step-1'),
+    ];
+
+    await service.run({ sequence, options: {} });
+
+    expect(runFastInstanceCommand).toHaveBeenCalledTimes(1);
+  });
+});
@@ -1,5 +1,6 @@
 import { Injectable, Logger } from '@nestjs/common';

+import { CommandShutdownService } from 'src/database/commands/command-runners/command-shutdown.service';
 import {
  type WorkspaceIteratorReport,
  WorkspaceIteratorService,
@@ -39,6 +40,7 @@ export class UpgradeSequenceRunnerService {
    private readonly upgradeAwareEntityMetadataAdapter: UpgradeAwareEntityMetadataAdapter,
    private readonly workspaceIteratorService: WorkspaceIteratorService,
    private readonly workspaceVersionService: WorkspaceVersionService,
+    private readonly commandShutdownService: CommandShutdownService,
  ) {}

  async run({
@@ -95,6 +97,23 @@ export class UpgradeSequenceRunnerService {
    while (cursor < sequence.length) {
      const step = sequence[cursor];

+      if (this.commandShutdownService.isShutdownRequested()) {
+        this.logger.warn(
+          formatUpgradeLog({
+            humanMessage:
+              `Stopping before step "${step.name}": shutdown requested. ` +
+              'Rerun the upgrade to resume from this step.',
+            event: 'sequence.stopped',
+            logFields: {
+              before: step.name,
+              reason: 'shutdown-requested',
+            },
+          }),
+        );
+
+        break;
+      }
+
      if (step.kind === 'fast-instance' || step.kind === 'slow-instance') {
        if (
          (isDefined(options.workspaceIds) &&
@@ -173,6 +192,23 @@ export class UpgradeSequenceRunnerService {
        return { totalSuccesses, totalFailures };
      }

+      if (report.interrupted) {
+        this.logger.warn(
+          formatUpgradeLog({
+            humanMessage:
+              'Stopped during workspace steps: shutdown requested. ' +
+              'Rerun the upgrade to process the remaining workspaces.',
+            event: 'sequence.stopped',
+            logFields: {
+              reason: 'shutdown-requested',
+              processedWorkspaces: report.success.length,
+            },
+          }),
+        );
+
+        return { totalSuccesses, totalFailures };
+      }
+
      cursor += workspaceCommandsSegment.length;

      workspaceCursors = await this.fetchWorkspaceCursors(
@@ -1,5 +1,6 @@
 import { Module } from '@nestjs/common';

+import { CommandShutdownModule } from 'src/database/commands/command-runners/command-shutdown.module';
 import { WorkspaceIteratorModule } from 'src/database/commands/command-runners/workspace-iterator.module';
 import { InstanceCommandProviderModule } from 'src/database/commands/upgrade-version-command/instance-command-provider.module';
 import { WorkspaceCommandProviderModule } from 'src/database/commands/upgrade-version-command/workspace-command-provider.module';
@@ -14,6 +15,7 @@ import { WorkspaceVersionModule } from 'src/engine/workspace-manager/workspace-v

@Module({
  imports: [
+    CommandShutdownModule,
    InstanceCommandProviderModule,
    MetricsModule,
    UpgradeStatusModule,