Add workspace and server level stop commands for applications (#23183)

## Context When an installed application misbehaves (e.g. a logic function loop DDoSing the server or the database), we currently have no targeted way to shut it down in production: the only kill switch is `LOGIC_FUNCTION_TYPE=DISABLED`, which disables logic functions for the whole instance. This PR adds an emergency stop mechanism at two levels: - **Workspace level**: stop one installed application in its workspace. - **Server level**: stop every application installed from an `applicationRegistration`, across all workspaces. ## How it works **New nullable `stoppedAt` columns** on `core.application` and `core.applicationRegistration` (fast instance command `2.24.0`, with `up`/`down` and `@WasIntroducedInUpgrade` decorators on the entities). **Enforcement in a single choke point**: `LogicFunctionExecutorService.execute()` is the funnel behind every execution path (public route triggers, server route triggers, cron triggers, database event triggers, workflow actions, agent tool calls, manual GraphQL execution, install hooks). A new `assertApplicationNotStopped` guard runs right after the flat entities are resolved and throws `LOGIC_FUNCTION_DISABLED` (already mapped to a 403 on route triggers and handled by the GraphQL exception handler) when: - `flatApplication.stoppedAt` is set (workspace-level stop, read from the cached flat application maps: zero extra runtime cost), or - the linked registration is stopped (one indexed PK lookup, same pattern as the existing per-execution server-variable query). **Propagation**: the workspace-level stop invalidates and recomputes `flatApplicationMaps` for the workspace, so all server instances pick the flag up within the local cache TTL (100ms). The registration-level flag is read live, so it is effective immediately. ## Ops commands ```bash # Workspace level yarn command:prod application:stop -a <application-id> yarn command:prod application:start -a <application-id> # Server level (all applications of the registration, all workspaces) yarn command:prod application-registration:stop -r <application-registration-id> yarn command:prod application-registration:start -r <application-registration-id> ``` Each command logs what was stopped/started and, for registrations, how many installed applications are affected. ## Notes - Stopped executions fail fast at the guard, so queued trigger jobs (cron/db-event) burn a negligible amount of work while stopped. - The two flags are independent: lifting a registration-level stop does not clear workspace-level stops that were set individually, and vice versa. - Unit tests added for `ApplicationStopService`. --- _Generated by [Claude Code](https://claude.ai/code/session_01CjEnKUACn89aSgK1wEMH2d)_  <a href="https://cubic.dev/pr/twentyhq/twenty/pull/23183?utm_source=github" target="_blank" rel="noopener noreferrer" data-no-image-dialog="true"><picture><source media="(prefers-color-scheme: dark)" srcset="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"><source media="(prefers-color-scheme: light)" srcset="https://www.cubic.dev/buttons/review-in-cubic-light.svg"><img alt="Review in cubic" src="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"></picture></a>
2026-07-22 20:30:56 +02:00
parent a65da48591
commit a3f4acadb1
21 changed files with 799 additions and 8 deletions
@@ -12,6 +12,10 @@ import { ListOrphanedWorkspaceEntitiesCommand } from 'src/database/commands/list
 import { ConfirmationQuestion } from 'src/database/commands/questions/confirmation.question';
 import { RebuildApplicationDefaultDepsCommand } from 'src/database/commands/rebuild-application-default-deps.command';
 import { RunInstanceCommandsCommand } from 'src/database/commands/run-instance-commands.command';
+import { StartApplicationRegistrationCommand } from 'src/database/commands/start-application-registration.command';
+import { StartApplicationCommand } from 'src/database/commands/start-application.command';
+import { StopApplicationRegistrationCommand } from 'src/database/commands/stop-application-registration.command';
+import { StopApplicationCommand } from 'src/database/commands/stop-application.command';
 import { UpgradeVersionCommandModule } from 'src/database/commands/upgrade-version-command/upgrade-version-command.module';
 import { WorkspaceExportModule } from 'src/database/commands/workspace-export/workspace-export.module';
 import { TypeORMModule } from 'src/database/typeorm/typeorm.module';
@@ -110,6 +114,10 @@ import { AutomatedTriggerModule } from 'src/modules/workflow/workflow-trigger/au
    UpgradeStatusCommand,
    RebuildApplicationDefaultDepsCommand,
    InstallPreInstalledAppsCommand,
+    StopApplicationCommand,
+    StartApplicationCommand,
+    StopApplicationRegistrationCommand,
+    StartApplicationRegistrationCommand,
    provideWorkspaceScopedRepository(RoleEntity),
  ],
 })
@@ -0,0 +1,46 @@
+import { Logger } from '@nestjs/common';
+
+import { Command, CommandRunner, Option } from 'nest-commander';
+
+import { ApplicationStopService } from 'src/engine/core-modules/application/application-stop.service';
+
+@Command({
+  name: 'application-registration:start',
+  description:
+    'Lift the server-level kill switch set by application-registration:stop and resume logic function executions of all applications installed from the registration.',
+})
+export class StartApplicationRegistrationCommand extends CommandRunner {
+  private readonly logger = new Logger(
+    StartApplicationRegistrationCommand.name,
+  );
+
+  constructor(private readonly applicationStopService: ApplicationStopService) {
+    super();
+  }
+
+  @Option({
+    flags:
+      '-r, --application-registration-universal-identifier <universal_identifier>',
+    description:
+      'universal identifier of the application registration to start',
+    required: true,
+  })
+  parseApplicationRegistrationUniversalIdentifier(value: string): string {
+    return value;
+  }
+
+  override async run(
+    _passedParams: string[],
+    options: { applicationRegistrationUniversalIdentifier: string },
+  ): Promise<void> {
+    const { applicationRegistration, installedApplicationCount } =
+      await this.applicationStopService.startApplicationRegistration({
+        applicationRegistrationUniversalIdentifier:
+          options.applicationRegistrationUniversalIdentifier,
+      });
+
+    this.logger.log(
+      `Started application registration "${applicationRegistration.name}" (universalIdentifier ${applicationRegistration.universalIdentifier}): the server-level stop is lifted for its ${installedApplicationCount} installed application(s). Workspace-level stops set with application:stop remain in effect.`,
+    );
+  }
+}
@@ -0,0 +1,67 @@
+import { Logger } from '@nestjs/common';
+
+import { Command, CommandRunner, Option } from 'nest-commander';
+import { isDefined } from 'twenty-shared/utils';
+
+import { ApplicationStopService } from 'src/engine/core-modules/application/application-stop.service';
+
+type StartApplicationCommandOptions = {
+  workspaceId: string;
+  applicationUniversalIdentifier: string;
+};
+
+@Command({
+  name: 'application:start',
+  description:
+    'Lift the workspace-level kill switch set by application:stop and resume logic function executions of the application.',
+})
+export class StartApplicationCommand extends CommandRunner {
+  private readonly logger = new Logger(StartApplicationCommand.name);
+
+  constructor(private readonly applicationStopService: ApplicationStopService) {
+    super();
+  }
+
+  @Option({
+    flags: '-w, --workspace-id <workspace_id>',
+    description: 'id of the workspace the application is installed in',
+    required: true,
+  })
+  parseWorkspaceId(value: string): string {
+    return value;
+  }
+
+  @Option({
+    flags: '-a, --application-universal-identifier <universal_identifier>',
+    description: 'universal identifier of the application to start',
+    required: true,
+  })
+  parseApplicationUniversalIdentifier(value: string): string {
+    return value;
+  }
+
+  override async run(
+    _passedParams: string[],
+    options: StartApplicationCommandOptions,
+  ): Promise<void> {
+    const application = await this.applicationStopService.startApplication({
+      workspaceId: options.workspaceId,
+      applicationUniversalIdentifier: options.applicationUniversalIdentifier,
+    });
+
+    this.logger.log(
+      `Started application "${application.name}" (universalIdentifier ${application.universalIdentifier}) in workspace ${application.workspaceId}: the workspace-level stop is lifted.`,
+    );
+
+    if (
+      isDefined(application.applicationRegistrationId) &&
+      (await this.applicationStopService.isApplicationRegistrationStopped(
+        application.applicationRegistrationId,
+      ))
+    ) {
+      this.logger.warn(
+        `Application registration ${application.applicationRegistrationId} is still stopped server-wide: executions of this application remain blocked until application-registration:start is run.`,
+      );
+    }
+  }
+}
@@ -0,0 +1,43 @@
+import { Logger } from '@nestjs/common';
+
+import { Command, CommandRunner, Option } from 'nest-commander';
+
+import { ApplicationStopService } from 'src/engine/core-modules/application/application-stop.service';
+
+@Command({
+  name: 'application-registration:stop',
+  description:
+    'Server-level kill switch: block all logic function executions of every application installed from a registration, across all workspaces. Reverse with application-registration:start.',
+})
+export class StopApplicationRegistrationCommand extends CommandRunner {
+  private readonly logger = new Logger(StopApplicationRegistrationCommand.name);
+
+  constructor(private readonly applicationStopService: ApplicationStopService) {
+    super();
+  }
+
+  @Option({
+    flags:
+      '-r, --application-registration-universal-identifier <universal_identifier>',
+    description: 'universal identifier of the application registration to stop',
+    required: true,
+  })
+  parseApplicationRegistrationUniversalIdentifier(value: string): string {
+    return value;
+  }
+
+  override async run(
+    _passedParams: string[],
+    options: { applicationRegistrationUniversalIdentifier: string },
+  ): Promise<void> {
+    const { applicationRegistration, installedApplicationCount } =
+      await this.applicationStopService.stopApplicationRegistration({
+        applicationRegistrationUniversalIdentifier:
+          options.applicationRegistrationUniversalIdentifier,
+      });
+
+    this.logger.log(
+      `Stopped application registration "${applicationRegistration.name}" (universalIdentifier ${applicationRegistration.universalIdentifier}). All logic function executions of its ${installedApplicationCount} installed application(s) are now blocked, across all workspaces.`,
+    );
+  }
+}
@@ -0,0 +1,55 @@
+import { Logger } from '@nestjs/common';
+
+import { Command, CommandRunner, Option } from 'nest-commander';
+
+import { ApplicationStopService } from 'src/engine/core-modules/application/application-stop.service';
+
+type StopApplicationCommandOptions = {
+  workspaceId: string;
+  applicationUniversalIdentifier: string;
+};
+
+@Command({
+  name: 'application:stop',
+  description:
+    'Workspace-level kill switch: block all logic function executions of one installed application. Reverse with application:start.',
+})
+export class StopApplicationCommand extends CommandRunner {
+  private readonly logger = new Logger(StopApplicationCommand.name);
+
+  constructor(private readonly applicationStopService: ApplicationStopService) {
+    super();
+  }
+
+  @Option({
+    flags: '-w, --workspace-id <workspace_id>',
+    description: 'id of the workspace the application is installed in',
+    required: true,
+  })
+  parseWorkspaceId(value: string): string {
+    return value;
+  }
+
+  @Option({
+    flags: '-a, --application-universal-identifier <universal_identifier>',
+    description: 'universal identifier of the application to stop',
+    required: true,
+  })
+  parseApplicationUniversalIdentifier(value: string): string {
+    return value;
+  }
+
+  override async run(
+    _passedParams: string[],
+    options: StopApplicationCommandOptions,
+  ): Promise<void> {
+    const application = await this.applicationStopService.stopApplication({
+      workspaceId: options.workspaceId,
+      applicationUniversalIdentifier: options.applicationUniversalIdentifier,
+    });
+
+    this.logger.log(
+      `Stopped application "${application.name}" (universalIdentifier ${application.universalIdentifier}) in workspace ${application.workspaceId}. All its logic function executions are now blocked.`,
+    );
+  }
+}
@@ -0,0 +1,27 @@
+import { QueryRunner } from 'typeorm';
+
+import { RegisteredInstanceCommand } from 'src/engine/core-modules/upgrade/decorators/registered-instance-command.decorator';
+import { FastInstanceCommand } from 'src/engine/core-modules/upgrade/interfaces/fast-instance-command.interface';
+
+@RegisteredInstanceCommand('2.24.0', 1784734278506)
+export class AddStoppedAtToApplicationAndApplicationRegistrationFastInstanceCommand
+  implements FastInstanceCommand
+{
+  public async up(queryRunner: QueryRunner): Promise<void> {
+    await queryRunner.query(
+      'ALTER TABLE "core"."application" ADD COLUMN IF NOT EXISTS "stoppedAt" TIMESTAMP WITH TIME ZONE',
+    );
+    await queryRunner.query(
+      'ALTER TABLE "core"."applicationRegistration" ADD COLUMN IF NOT EXISTS "stoppedAt" TIMESTAMP WITH TIME ZONE',
+    );
+  }
+
+  public async down(queryRunner: QueryRunner): Promise<void> {
+    await queryRunner.query(
+      'ALTER TABLE "core"."applicationRegistration" DROP COLUMN IF EXISTS "stoppedAt"',
+    );
+    await queryRunner.query(
+      'ALTER TABLE "core"."application" DROP COLUMN IF EXISTS "stoppedAt"',
+    );
+  }
+}
@@ -53,6 +53,7 @@ import { AllowServerScopedFileFastInstanceCommand } from 'src/database/commands/
 import { AddCalendarEndFieldMetadataIdToViewFastInstanceCommand } from 'src/database/commands/upgrade-version-command/2-22/2-22-instance-command-fast-1783956795000-add-calendar-end-field-metadata-id-to-view';
 import { AddSdkClientCoreChecksumToApplicationFastInstanceCommand } from 'src/database/commands/upgrade-version-command/2-23/2-23-instance-command-fast-1784625638000-add-sdk-client-core-checksum-to-application';
 import { AddAutoUpgradeToApplicationFastInstanceCommand } from 'src/database/commands/upgrade-version-command/2-23/2-23-instance-command-fast-1784297307235-add-auto-upgrade-to-application';
+import { AddStoppedAtToApplicationAndApplicationRegistrationFastInstanceCommand } from 'src/database/commands/upgrade-version-command/2-24/2-24-instance-command-fast-1784734278506-add-stopped-at-to-application-and-application-registration';
 import { AddSubFieldNameToViewSortEarlyFastInstanceCommand } from 'src/database/commands/upgrade-version-command/2-3/2-3-instance-command-fast-1747234200000-add-sub-field-name-to-view-sort';
 import { AddRelationTargetFieldMetadataIdToViewFilterEarlyFastInstanceCommand } from 'src/database/commands/upgrade-version-command/2-3/2-3-instance-command-fast-1747234300000-add-relation-target-field-metadata-id-to-view-filter';
 import { AddUpgradeMigrationWorkspaceIdIndexFastInstanceCommand } from 'src/database/commands/upgrade-version-command/2-3/2-3-instance-command-fast-1777308014234-add-upgrade-migration-workspace-id-index';
@@ -244,4 +245,5 @@ export const INSTANCE_COMMANDS = [
  AddSdkClientCoreChecksumToApplicationFastInstanceCommand,
  AddStatusesToBillingSubscriptionIndexSlowInstanceCommand,
  AddOnConnectLogicFunctionToConnectionProviderFastInstanceCommand,
+  AddStoppedAtToApplicationAndApplicationRegistrationFastInstanceCommand,
 ];