Add server upgrade status on admin panel (#20107)

## Summary

Adds an admin upgrade-status panel that surfaces per-instance and
per-workspace migration health, backed by a Redis-cached aggregate to
keep the page snappy on large fleets.

<img width="827" height="880" alt="Screenshot 2026-04-28 at 10 21 03"
src="https://github.com/user-attachments/assets/8f88baa9-7268-4eff-bf6a-906a7f06ca91"
/>
<img width="804" height="892" alt="Screenshot 2026-04-28 at 10 21 11"
src="https://github.com/user-attachments/assets/1e6decf8-766a-4d0e-96b1-03a9962bba3c"
/>


## Computed metrics

**Instance** (`InstanceUpgradeStatus`)
- `inferredVersion` — version derived from the latest non-initial
instance command name
- `health` — `upToDate` | `behind` | `failed`, derived from the latest
attempt vs. the last expected instance step in the upgrade sequence
- `latestCommand` — `{ name, status, executedByVersion, errorMessage,
createdAt }` from the most recent attempt

**Per-workspace** (`WorkspaceUpgradeStatus`)
- `workspaceId`, `displayName`
- `inferredVersion`, `health`, `latestCommand` (same shape as instance),
computed against the latest expected step in the sequence

**Aggregate** (`AllWorkspacesUpgradeStatus`, only across `ACTIVE` /
`SUSPENDED` workspaces)
- `instanceUpgradeStatus`
- `totalCount`, `upToDateCount`, `behindCount`, `failedCount`
- `workspacesBehindIds[]`, `workspacesFailedIds[]`
- `computedAt`

## Fetching strategy

All reads go through `UpgradeStatusCacheService` (cache namespace:
`EngineHealth`).

- **Aggregate read** (`getAllWorkspacesStatus` →
`getAllWorkspacesUpgradeStatus` query):
reads summary + behind-ids + failed-ids in parallel; if any of the three
keys is missing, full recompute (`recomputeAllWorkspaces`) is triggered,
which also primes per-workspace entries.
- **Per-workspace read** (`getWorkspacesStatus(ids)` →
`getUpgradeStatus(ids)` query):
`mget` on workspace keys; misses are recomputed individually
(`recomputeWorkspace`), and aggregates are reconciled in place (count +
id list deltas) without a full recompute.
- **Recompute on demand**: `refreshUpgradeStatus` mutation calls
`recomputeAllWorkspaces` to bypass cache and rewrite all keys.
- **Auto-invalidation**: `InstanceCommandRunnerService` (fast + slow
paths) and `WorkspaceCommandRunnerService` invalidate after every run
via `safeInvalidateUpgradeStatusCache()`
(`flushByPattern('upgrade-status:*')`). Failures in cache invalidation
are swallowed and logged so they never break the migration runner.
- **TTL**: `60 * 60 * 1000` ms (1 hour) on every key — protects against
stale data even if a runner crashes before invalidating.

## Introduced cache keys

All under the `EngineHealth` cache-storage namespace:

| Key | Type | Purpose |
| --- | --- | --- |
| `upgrade-status:all-workspaces:summary` |
`CachedAllWorkspacesStatusSummary` | Counts + instance status +
`computedAt` |
| `upgrade-status:all-workspaces:behind-ids` | `string[]` | Workspace
ids in `behind` state |
| `upgrade-status:all-workspaces:failed-ids` | `string[]` | Workspace
ids in `failed` state |
| `upgrade-status:workspace:<workspaceId>` |
`CachedWorkspaceUpgradeStatus` | Per-workspace status (one key per
workspace) |

Full invalidation uses the pattern `upgrade-status:*`.

## Index added on `upgradeMigration` (already added on prod)

Migration
`2-2-instance-command-fast-1777308014234-addUpgradeMigrationWorkspaceIdIndex.ts`:

```sql
CREATE INDEX "IDX_upgradeMigration_workspaceId_name_attempt"
  ON "core"."upgradeMigration" ("workspaceId", "name", "attempt")
  WHERE "workspaceId" IS NOT NULL;
This commit is contained in:
Marie
2026-05-04 17:06:07 +02:00
committed by GitHub
parent 1cd983a330
commit 4852ac401a
49 changed files with 2100 additions and 213 deletions
@@ -5,13 +5,18 @@ import { TypeOrmModule } from '@nestjs/typeorm';
import { AdminPanelHealthService } from 'src/engine/core-modules/admin-panel/admin-panel-health.service';
import { AdminPanelQueueService } from 'src/engine/core-modules/admin-panel/admin-panel-queue.service';
import { AdminPanelResolver } from 'src/engine/core-modules/admin-panel/admin-panel.resolver';
import { AppHealthIndicator } from 'src/engine/core-modules/admin-panel/indicators/app.health';
import { ConnectedAccountHealth } from 'src/engine/core-modules/admin-panel/indicators/connected-account.health';
import { DatabaseHealthIndicator } from 'src/engine/core-modules/admin-panel/indicators/database.health';
import { RedisHealthIndicator } from 'src/engine/core-modules/admin-panel/indicators/redis.health';
import { WorkerHealthIndicator } from 'src/engine/core-modules/admin-panel/indicators/worker.health';
import { MaintenanceModeService } from 'src/engine/core-modules/admin-panel/maintenance-mode.service';
import { AdminPanelBillingService } from 'src/engine/core-modules/admin-panel/services/admin-panel-billing.service';
import { AdminPanelChatService } from 'src/engine/core-modules/admin-panel/services/admin-panel-chat.service';
import { AdminPanelConfigService } from 'src/engine/core-modules/admin-panel/services/admin-panel-config.service';
import { AdminPanelStatisticsService } from 'src/engine/core-modules/admin-panel/services/admin-panel-statistics.service';
import { AdminPanelUserLookupService } from 'src/engine/core-modules/admin-panel/services/admin-panel-user-lookup.service';
import { AdminPanelVersionService } from 'src/engine/core-modules/admin-panel/services/admin-panel-version.service';
import { MaintenanceModeService } from 'src/engine/core-modules/admin-panel/maintenance-mode.service';
import { ApplicationRegistrationModule } from 'src/engine/core-modules/application/application-registration/application-registration.module';
import { AuditModule } from 'src/engine/core-modules/audit/audit.module';
import { AuthModule } from 'src/engine/core-modules/auth/auth.module';
@@ -19,28 +24,24 @@ import { BillingModule } from 'src/engine/core-modules/billing/billing.module';
import { BillingCustomerEntity } from 'src/engine/core-modules/billing/entities/billing-customer.entity';
import { BillingPriceEntity } from 'src/engine/core-modules/billing/entities/billing-price.entity';
import { WorkspaceDomainsModule } from 'src/engine/core-modules/domain/workspace-domains/workspace-domains.module';
import { FeatureFlagEntity } from 'src/engine/core-modules/feature-flag/feature-flag.entity';
import { FeatureFlagModule } from 'src/engine/core-modules/feature-flag/feature-flag.module';
import { FileModule } from 'src/engine/core-modules/file/file.module';
import { AppHealthIndicator } from 'src/engine/core-modules/admin-panel/indicators/app.health';
import { ConnectedAccountHealth } from 'src/engine/core-modules/admin-panel/indicators/connected-account.health';
import { DatabaseHealthIndicator } from 'src/engine/core-modules/admin-panel/indicators/database.health';
import { RedisHealthIndicator } from 'src/engine/core-modules/admin-panel/indicators/redis.health';
import { WorkerHealthIndicator } from 'src/engine/core-modules/admin-panel/indicators/worker.health';
import { ImpersonationModule } from 'src/engine/core-modules/impersonation/impersonation.module';
import { KeyValuePairModule } from 'src/engine/core-modules/key-value-pair/key-value-pair.module';
import { MetricsModule } from 'src/engine/core-modules/metrics/metrics.module';
import { RedisClientModule } from 'src/engine/core-modules/redis-client/redis-client.module';
import { SecureHttpClientModule } from 'src/engine/core-modules/secure-http-client/secure-http-client.module';
import { TelemetryModule } from 'src/engine/core-modules/telemetry/telemetry.module';
import { UpgradeModule } from 'src/engine/core-modules/upgrade/upgrade.module';
import { UsageModule } from 'src/engine/core-modules/usage/usage.module';
import { FeatureFlagEntity } from 'src/engine/core-modules/feature-flag/feature-flag.entity';
import { UserModule } from 'src/engine/core-modules/user/user.module';
import { UserEntity } from 'src/engine/core-modules/user/user.entity';
import { UserVarsModule } from 'src/engine/core-modules/user/user-vars/user-vars.module';
import { UserWorkspaceEntity } from 'src/engine/core-modules/user-workspace/user-workspace.entity';
import { UserVarsModule } from 'src/engine/core-modules/user/user-vars/user-vars.module';
import { UserEntity } from 'src/engine/core-modules/user/user.entity';
import { UserModule } from 'src/engine/core-modules/user/user.module';
import { WorkspaceEntity } from 'src/engine/core-modules/workspace/workspace.entity';
import { AgentChatThreadEntity } from 'src/engine/metadata-modules/ai/ai-chat/entities/agent-chat-thread.entity';
import { AgentMessageEntity } from 'src/engine/metadata-modules/ai/ai-agent-execution/entities/agent-message.entity';
import { KeyValuePairModule } from 'src/engine/core-modules/key-value-pair/key-value-pair.module';
import { AgentChatThreadEntity } from 'src/engine/metadata-modules/ai/ai-chat/entities/agent-chat-thread.entity';
import { PermissionsModule } from 'src/engine/metadata-modules/permissions/permissions.module';
@Module({
@@ -72,6 +73,7 @@ import { PermissionsModule } from 'src/engine/metadata-modules/permissions/permi
UsageModule,
KeyValuePairModule,
UserVarsModule,
UpgradeModule,
UserModule,
],
providers: [
@@ -3,32 +3,23 @@ import { Args, Int, Mutation, Query } from '@nestjs/graphql';
import { InjectRepository } from '@nestjs/typeorm';
import GraphQLJSON from 'graphql-type-json';
import { In, type Repository } from 'typeorm';
import { isDefined } from 'twenty-shared/utils';
import { PermissionFlagType } from 'twenty-shared/constants';
import { isDefined } from 'twenty-shared/utils';
import { In, type Repository } from 'typeorm';
import { UUIDScalarType } from 'src/engine/api/graphql/workspace-schema-builder/graphql-types/scalars';
import { InstanceAndAllWorkspacesUpgradeStatusDTO } from 'src/engine/core-modules/upgrade/dtos/instance-and-all-workspaces-upgrade-status.dto';
import { WorkspaceUpgradeStatusDTO } from 'src/engine/core-modules/upgrade/dtos/workspace-upgrade-status.dto';
import { UpgradeStatusService } from 'src/engine/core-modules/upgrade/services/upgrade-status.service';
import { AdminResolver } from 'src/engine/api/graphql/graphql-config/decorators/admin-resolver.decorator';
import { AdminPanelHealthService } from 'src/engine/core-modules/admin-panel/admin-panel-health.service';
import { AdminPanelQueueService } from 'src/engine/core-modules/admin-panel/admin-panel-queue.service';
import { AdminPanelBillingService } from 'src/engine/core-modules/admin-panel/services/admin-panel-billing.service';
import { AdminPanelChatService } from 'src/engine/core-modules/admin-panel/services/admin-panel-chat.service';
import { AdminPanelConfigService } from 'src/engine/core-modules/admin-panel/services/admin-panel-config.service';
import { AdminPanelStatisticsService } from 'src/engine/core-modules/admin-panel/services/admin-panel-statistics.service';
import { AdminPanelUserLookupService } from 'src/engine/core-modules/admin-panel/services/admin-panel-user-lookup.service';
import { AdminPanelVersionService } from 'src/engine/core-modules/admin-panel/services/admin-panel-version.service';
import { MaintenanceModeService } from 'src/engine/core-modules/admin-panel/maintenance-mode.service';
import { AdminChatThreadMessagesDTO } from 'src/engine/core-modules/admin-panel/dtos/admin-chat-thread-messages.dto';
import { AdminPanelRecentUserDTO } from 'src/engine/core-modules/admin-panel/dtos/admin-panel-recent-user.dto';
import { AdminPanelTopWorkspaceDTO } from 'src/engine/core-modules/admin-panel/dtos/admin-panel-top-workspace.dto';
import { AdminPanelWorkspaceBillingDTO } from 'src/engine/core-modules/admin-panel/dtos/admin-panel-workspace-billing.dto';
import { AdminWorkspaceChatThreadDTO } from 'src/engine/core-modules/admin-panel/dtos/admin-workspace-chat-thread.dto';
import { AdminChatThreadMessagesDTO } from 'src/engine/core-modules/admin-panel/dtos/admin-chat-thread-messages.dto';
import { ApplicationRegistrationEntity } from 'src/engine/core-modules/application/application-registration/application-registration.entity';
import { ApplicationRegistrationService } from 'src/engine/core-modules/application/application-registration/application-registration.service';
import { AdminAiModelsDTO } from 'src/engine/core-modules/client-config/client-config.entity';
import { UsageBreakdownItemDTO } from 'src/engine/core-modules/usage/dtos/usage-breakdown-item.dto';
import { UsageAnalyticsService } from 'src/engine/core-modules/usage/services/usage-analytics.service';
import { AiModelRole } from 'src/engine/metadata-modules/ai/ai-models/types/ai-model-role.enum';
import { ConfigVariableDTO } from 'src/engine/core-modules/admin-panel/dtos/config-variable.dto';
import { ConfigVariablesDTO } from 'src/engine/core-modules/admin-panel/dtos/config-variables.dto';
import { DeleteJobsResponseDTO } from 'src/engine/core-modules/admin-panel/dtos/delete-jobs-response.dto';
@@ -39,33 +30,45 @@ import { UpdateWorkspaceFeatureFlagInput } from 'src/engine/core-modules/admin-p
import { UserLookup } from 'src/engine/core-modules/admin-panel/dtos/user-lookup.dto';
import { UserLookupInput } from 'src/engine/core-modules/admin-panel/dtos/user-lookup.input';
import { VersionInfoDTO } from 'src/engine/core-modules/admin-panel/dtos/version-info.dto';
import { HealthIndicatorId } from 'src/engine/core-modules/admin-panel/enums/health-indicator-id.enum';
import { JobStateEnum } from 'src/engine/core-modules/admin-panel/enums/job-state.enum';
import { QueueMetricsTimeRange } from 'src/engine/core-modules/admin-panel/enums/queue-metrics-time-range.enum';
import { MaintenanceModeService } from 'src/engine/core-modules/admin-panel/maintenance-mode.service';
import { AdminPanelBillingService } from 'src/engine/core-modules/admin-panel/services/admin-panel-billing.service';
import { AdminPanelChatService } from 'src/engine/core-modules/admin-panel/services/admin-panel-chat.service';
import { AdminPanelConfigService } from 'src/engine/core-modules/admin-panel/services/admin-panel-config.service';
import { AdminPanelStatisticsService } from 'src/engine/core-modules/admin-panel/services/admin-panel-statistics.service';
import { AdminPanelUserLookupService } from 'src/engine/core-modules/admin-panel/services/admin-panel-user-lookup.service';
import { AdminPanelVersionService } from 'src/engine/core-modules/admin-panel/services/admin-panel-version.service';
import { ApplicationRegistrationEntity } from 'src/engine/core-modules/application/application-registration/application-registration.entity';
import { ApplicationRegistrationService } from 'src/engine/core-modules/application/application-registration/application-registration.service';
import { AuthGraphqlApiExceptionFilter } from 'src/engine/core-modules/auth/filters/auth-graphql-api-exception.filter';
import { AdminAiModelsDTO } from 'src/engine/core-modules/client-config/client-config.entity';
import { FeatureFlagException } from 'src/engine/core-modules/feature-flag/feature-flag.exception';
import { FeatureFlagService } from 'src/engine/core-modules/feature-flag/services/feature-flag.service';
import { PreventNestToAutoLogGraphqlErrorsFilter } from 'src/engine/core-modules/graphql/filters/prevent-nest-to-auto-log-graphql-errors.filter';
import { ResolverValidationPipe } from 'src/engine/core-modules/graphql/pipes/resolver-validation.pipe';
import { UserInputError } from 'src/engine/core-modules/graphql/utils/graphql-errors.util';
import { HealthIndicatorId } from 'src/engine/core-modules/admin-panel/enums/health-indicator-id.enum';
import { type MessageQueue } from 'src/engine/core-modules/message-queue/message-queue.constants';
import { type ConfigVariables } from 'src/engine/core-modules/twenty-config/config-variables';
import { ConfigVariableGraphqlApiExceptionFilter } from 'src/engine/core-modules/twenty-config/filters/config-variable-graphql-api-exception.filter';
import { TwentyConfigService } from 'src/engine/core-modules/twenty-config/twenty-config.service';
import { AiModelRegistryService } from 'src/engine/metadata-modules/ai/ai-models/services/ai-model-registry.service';
import { ModelsDevCatalogService } from 'src/engine/metadata-modules/ai/ai-models/services/models-dev-catalog.service';
import { MODEL_FAMILY_LABELS } from 'src/engine/metadata-modules/ai/ai-models/constants/model-family-labels.const';
import { type AiProviderConfig } from 'src/engine/metadata-modules/ai/ai-models/types/ai-provider-config.type';
import { type AiProviderModelConfig } from 'src/engine/metadata-modules/ai/ai-models/types/ai-provider-model-config.type';
import { extractConfigVariableName } from 'src/engine/metadata-modules/ai/ai-models/utils/extract-config-variable-name.util';
import { DefaultAiCatalogService } from 'src/engine/metadata-modules/ai/ai-models/services/default-ai-catalog.service';
import { UsageBreakdownItemDTO } from 'src/engine/core-modules/usage/dtos/usage-breakdown-item.dto';
import { UsageAnalyticsService } from 'src/engine/core-modules/usage/services/usage-analytics.service';
import { WorkspaceEntity } from 'src/engine/core-modules/workspace/workspace.entity';
import { AdminResolver } from 'src/engine/api/graphql/graphql-config/decorators/admin-resolver.decorator';
import { AdminPanelGuard } from 'src/engine/guards/admin-panel-guard';
import { ServerLevelImpersonateGuard } from 'src/engine/guards/server-level-impersonate.guard';
import { SettingsPermissionGuard } from 'src/engine/guards/settings-permission.guard';
import { UserAuthGuard } from 'src/engine/guards/user-auth.guard';
import { WorkspaceAuthGuard } from 'src/engine/guards/workspace-auth.guard';
import { MODEL_FAMILY_LABELS } from 'src/engine/metadata-modules/ai/ai-models/constants/model-family-labels.const';
import { AiModelRegistryService } from 'src/engine/metadata-modules/ai/ai-models/services/ai-model-registry.service';
import { DefaultAiCatalogService } from 'src/engine/metadata-modules/ai/ai-models/services/default-ai-catalog.service';
import { ModelsDevCatalogService } from 'src/engine/metadata-modules/ai/ai-models/services/models-dev-catalog.service';
import { AiModelRole } from 'src/engine/metadata-modules/ai/ai-models/types/ai-model-role.enum';
import { type AiProviderConfig } from 'src/engine/metadata-modules/ai/ai-models/types/ai-provider-config.type';
import { type AiProviderModelConfig } from 'src/engine/metadata-modules/ai/ai-models/types/ai-provider-model-config.type';
import { extractConfigVariableName } from 'src/engine/metadata-modules/ai/ai-models/utils/extract-config-variable-name.util';
import { AdminPanelHealthServiceDataDTO } from './dtos/admin-panel-health-service-data.dto';
import { MaintenanceModeDTO } from './dtos/maintenance-mode.dto';
@@ -73,7 +76,6 @@ import { ModelsDevModelSuggestionDTO } from './dtos/models-dev-model-suggestion.
import { ModelsDevProviderSuggestionDTO } from './dtos/models-dev-provider-suggestion.dto';
import { QueueMetricsDataDTO } from './dtos/queue-metrics-data.dto';
import { SetMaintenanceModeInput } from './dtos/set-maintenance-mode.input';
import { AuthWorkspace } from 'src/engine/decorators/auth/auth-workspace.decorator';
@UsePipes(ResolverValidationPipe)
@AdminResolver()
@@ -105,6 +107,7 @@ export class AdminPanelResolver {
private readonly modelsDevCatalogService: ModelsDevCatalogService,
private readonly usageAnalyticsService: UsageAnalyticsService,
private readonly maintenanceModeService: MaintenanceModeService,
private readonly upgradeStatusService: UpgradeStatusService,
@InjectRepository(WorkspaceEntity)
private readonly workspaceRepository: Repository<WorkspaceEntity>,
) {}
@@ -701,4 +704,29 @@ export class AdminPanelResolver {
): Promise<ApplicationRegistrationEntity> {
return this.applicationRegistrationService.findOneByIdGlobal(id);
}
@UseGuards(AdminPanelGuard)
@Query(() => InstanceAndAllWorkspacesUpgradeStatusDTO)
async getInstanceAndAllWorkspacesUpgradeStatus(): Promise<InstanceAndAllWorkspacesUpgradeStatusDTO> {
return this.upgradeStatusService.getInstanceAndAllWorkspacesStatus();
}
@UseGuards(AdminPanelGuard)
@Mutation(() => InstanceAndAllWorkspacesUpgradeStatusDTO)
async refreshUpgradeStatus(): Promise<InstanceAndAllWorkspacesUpgradeStatusDTO> {
return this.upgradeStatusService.refreshInstanceAndAllWorkspacesStatus();
}
@UseGuards(AdminPanelGuard)
@Query(() => [WorkspaceUpgradeStatusDTO])
async getUpgradeStatus(
@Args('workspaceIds', { type: () => [UUIDScalarType] })
workspaceIds: string[],
): Promise<WorkspaceUpgradeStatusDTO[]> {
if (workspaceIds.length === 0) {
return [];
}
return this.upgradeStatusService.getWorkspaceStatuses(workspaceIds);
}
}
@@ -62,8 +62,8 @@ class WorkspaceInfoDTO {
@ObjectType('UserLookup')
export class UserLookup {
@Field(() => UserInfoDTO)
user: UserInfoDTO;
@Field(() => UserInfoDTO, { nullable: true })
user?: UserInfoDTO | null;
@Field(() => [WorkspaceInfoDTO])
workspaces: WorkspaceInfoDTO[];
@@ -14,9 +14,9 @@ import {
import { WorkspaceDomainsService } from 'src/engine/core-modules/domain/workspace-domains/services/workspace-domains.service';
import { FeatureFlagEntity } from 'src/engine/core-modules/feature-flag/feature-flag.entity';
import { FileUrlService } from 'src/engine/core-modules/file/file-url/file-url.service';
import { UserEntity } from 'src/engine/core-modules/user/user.entity';
import { UserWorkspaceEntity } from 'src/engine/core-modules/user-workspace/user-workspace.entity';
import { UserService } from 'src/engine/core-modules/user/services/user.service';
import { UserEntity } from 'src/engine/core-modules/user/user.entity';
import { userValidator } from 'src/engine/core-modules/user/user.validate';
import { WorkspaceEntity } from 'src/engine/core-modules/workspace/workspace.entity';
@@ -208,16 +208,16 @@ export class AdminPanelUserLookupService {
const firstUser = workspaceUsers.find((wu) => isDefined(wu.user))?.user;
return {
user: {
id: firstUser?.id ?? '',
email: firstUser?.email ?? '',
firstName: firstUser?.firstName,
lastName: firstUser?.lastName,
avatarUrl: firstUser
? (avatarUrlsByUserId.get(firstUser.id) ?? null)
: null,
createdAt: firstUser?.createdAt ?? new Date(),
},
user: isDefined(firstUser)
? {
id: firstUser.id,
email: firstUser.email,
firstName: firstUser.firstName,
lastName: firstUser.lastName,
avatarUrl: avatarUrlsByUserId.get(firstUser.id) ?? null,
createdAt: firstUser.createdAt,
}
: null,
workspaces: [workspaceInfo],
};
}
@@ -4,11 +4,12 @@ import chalk from 'chalk';
import { Command, CommandRunner, Option } from 'nest-commander';
import { TwentyConfigService } from 'src/engine/core-modules/twenty-config/twenty-config.service';
import { UpgradeHealthEnum } from 'twenty-shared/types';
import { formatUpgradeCommandName } from 'twenty-shared/utils';
import {
type MigrationCursorStatus,
UpgradeHealth,
type InstanceUpgradeStatus,
UpgradeStatusService,
type WorkspaceStatus,
type WorkspaceUpgradeStatus,
} from 'src/engine/core-modules/upgrade/services/upgrade-status.service';
type UpgradeStatusOptions = {
@@ -16,16 +17,16 @@ type UpgradeStatusOptions = {
failedOnly?: boolean;
};
type GroupedWorkspaceStatuses = {
upToDate: WorkspaceStatus[];
behind: WorkspaceStatus[];
failed: WorkspaceStatus[];
type GroupedWorkspaceUpgradeStatuses = {
upToDate: WorkspaceUpgradeStatus[];
behind: WorkspaceUpgradeStatus[];
failed: WorkspaceUpgradeStatus[];
};
const HEALTH_LABELS: Record<UpgradeHealth, string> = {
'up-to-date': chalk.green('Up to date'),
behind: chalk.yellow('Behind'),
failed: chalk.red('Failed'),
const HEALTH_LABELS: Record<UpgradeHealthEnum, string> = {
[UpgradeHealthEnum.UP_TO_DATE]: chalk.green('Up to date'),
[UpgradeHealthEnum.BEHIND]: chalk.yellow('Behind'),
[UpgradeHealthEnum.FAILED]: chalk.red('Failed'),
};
@Command({
@@ -87,18 +88,18 @@ export class UpgradeStatusCommand extends CommandRunner {
requestedWorkspaceIds,
);
const groupedWorkspaceStatuses =
this.groupWorkspaceStatusesByHealth(workspaceStatuses);
const groupedWorkspaceUpgradeStatuses =
this.groupWorkspaceUpgradeStatusesByHealth(workspaceStatuses);
lines.push(
...this.formatWorkspaceStatuses(
groupedWorkspaceStatuses,
...this.formatWorkspaceUpgradeStatuses(
groupedWorkspaceUpgradeStatuses,
options.failedOnly,
),
);
lines.push(
...this.formatSummary(instanceStatus, groupedWorkspaceStatuses),
...this.formatSummary(instanceStatus, groupedWorkspaceUpgradeStatuses),
);
console.log(lines.join('\n'));
@@ -115,7 +116,7 @@ export class UpgradeStatusCommand extends CommandRunner {
return ['', chalk.bold(`APP_VERSION: ${appVersion}`), ''];
}
private formatInstanceStatus(status: MigrationCursorStatus): string[] {
private formatInstanceStatus(status: InstanceUpgradeStatus): string[] {
return [
chalk.bold.underline('Instance'),
...this.formatCursorStatus(status),
@@ -123,8 +124,8 @@ export class UpgradeStatusCommand extends CommandRunner {
];
}
private formatWorkspaceStatuses(
{ upToDate, behind, failed }: GroupedWorkspaceStatuses,
private formatWorkspaceUpgradeStatuses(
{ upToDate, behind, failed }: GroupedWorkspaceUpgradeStatuses,
failedOnly?: boolean,
): string[] {
const lines: string[] = [chalk.bold.underline('Workspace')];
@@ -137,19 +138,22 @@ export class UpgradeStatusCommand extends CommandRunner {
if (!failedOnly) {
for (const workspaceStatus of upToDate) {
lines.push(...this.formatWorkspaceStatus(workspaceStatus));
lines.push(...this.formatWorkspaceUpgradeStatus(workspaceStatus));
}
}
for (const workspaceStatus of behind) {
lines.push(...this.formatWorkspaceStatus(workspaceStatus));
lines.push(...this.formatWorkspaceUpgradeStatus(workspaceStatus));
}
if (failed.length > 0) {
const groupedByCommand = new Map<string, WorkspaceStatus[]>();
const groupedByCommand = new Map<
string | null,
WorkspaceUpgradeStatus[]
>();
for (const workspaceStatus of failed) {
const commandName = workspaceStatus.latestCommand?.name ?? 'unknown';
const commandName = workspaceStatus.latestCommand?.name ?? null;
if (!groupedByCommand.has(commandName)) {
groupedByCommand.set(commandName, []);
@@ -159,10 +163,16 @@ export class UpgradeStatusCommand extends CommandRunner {
}
for (const [commandName, statuses] of groupedByCommand) {
lines.push(chalk.red.bold(` Failed at: ${commandName}`));
const formattedCommandName = commandName
? formatUpgradeCommandName(commandName)
: 'unknown';
lines.push(chalk.red.bold(` Failed at: ${formattedCommandName}`));
for (const workspaceStatus of statuses) {
lines.push(...this.formatWorkspaceStatus(workspaceStatus, true));
lines.push(
...this.formatWorkspaceUpgradeStatus(workspaceStatus, true),
);
}
}
}
@@ -170,8 +180,8 @@ export class UpgradeStatusCommand extends CommandRunner {
return lines;
}
private formatWorkspaceStatus(
status: WorkspaceStatus,
private formatWorkspaceUpgradeStatus(
status: WorkspaceUpgradeStatus,
nested = false,
): string[] {
const baseIndent = nested ? ' ' : ' ';
@@ -188,7 +198,7 @@ export class UpgradeStatusCommand extends CommandRunner {
}
private formatCursorStatus(
status: MigrationCursorStatus,
status: InstanceUpgradeStatus,
indent = ' ',
): string[] {
if (!status.latestCommand) {
@@ -199,7 +209,7 @@ export class UpgradeStatusCommand extends CommandRunner {
const lines: string[] = [
`${indent}Inferred version: ${status.inferredVersion ?? chalk.dim('unknown')}`,
`${indent}Latest command: ${latestCommand.name}`,
`${indent}Latest command: ${formatUpgradeCommandName(latestCommand.name)}`,
`${indent}Status: ${HEALTH_LABELS[status.health]}`,
`${indent}Executed by: ${latestCommand.executedByVersion}`,
`${indent}At: ${latestCommand.createdAt.toISOString()}`,
@@ -215,8 +225,8 @@ export class UpgradeStatusCommand extends CommandRunner {
}
private formatSummary(
instanceStatus: MigrationCursorStatus,
{ upToDate, behind, failed }: GroupedWorkspaceStatuses,
instanceStatus: InstanceUpgradeStatus,
{ upToDate, behind, failed }: GroupedWorkspaceUpgradeStatuses,
): string[] {
const lines: string[] = [chalk.bold.underline('Summary')];
const totalCount = upToDate.length + behind.length + failed.length;
@@ -238,24 +248,30 @@ export class UpgradeStatusCommand extends CommandRunner {
lines.push(` Workspaces: ${parts.join(', ')} (${totalCount} total)`);
if (behind.length > 0) {
const behindCounts = new Map<string, number>();
const behindCounts = new Map<string | null, number>();
for (const status of behind) {
const commandName = status.latestCommand?.name ?? 'no commands';
const commandName = status.latestCommand?.name ?? null;
behindCounts.set(commandName, (behindCounts.get(commandName) ?? 0) + 1);
}
for (const [commandName, count] of behindCounts) {
lines.push(chalk.yellow(` ${count} behind at: ${commandName}`));
const formattedCommandName = commandName
? formatUpgradeCommandName(commandName)
: 'no commands';
lines.push(
chalk.yellow(` ${count} behind at: ${formattedCommandName}`),
);
}
}
if (failed.length > 0) {
const failureCounts = new Map<string, number>();
const failureCounts = new Map<string | null, number>();
for (const status of failed) {
const commandName = status.latestCommand?.name ?? 'unknown';
const commandName = status.latestCommand?.name ?? null;
failureCounts.set(
commandName,
@@ -264,7 +280,13 @@ export class UpgradeStatusCommand extends CommandRunner {
}
for (const [commandName, count] of failureCounts) {
lines.push(chalk.red(` ${count} failed at: ${commandName}`));
const formattedCommandName = commandName
? formatUpgradeCommandName(commandName)
: 'unknown';
lines.push(
chalk.red(` ${count} failed at: ${formattedCommandName}`),
);
}
}
@@ -273,22 +295,22 @@ export class UpgradeStatusCommand extends CommandRunner {
return lines;
}
private groupWorkspaceStatusesByHealth(
workspaceStatuses: WorkspaceStatus[],
): GroupedWorkspaceStatuses {
const upToDate: WorkspaceStatus[] = [];
const behind: WorkspaceStatus[] = [];
const failed: WorkspaceStatus[] = [];
private groupWorkspaceUpgradeStatusesByHealth(
workspaceStatuses: WorkspaceUpgradeStatus[],
): GroupedWorkspaceUpgradeStatuses {
const upToDate: WorkspaceUpgradeStatus[] = [];
const behind: WorkspaceUpgradeStatus[] = [];
const failed: WorkspaceUpgradeStatus[] = [];
for (const status of workspaceStatuses) {
switch (status.health) {
case 'up-to-date':
case UpgradeHealthEnum.UP_TO_DATE:
upToDate.push(status);
break;
case 'behind':
case UpgradeHealthEnum.BEHIND:
behind.push(status);
break;
case 'failed':
case UpgradeHealthEnum.FAILED:
failed.push(status);
break;
}
@@ -0,0 +1,19 @@
import { Field, ObjectType } from '@nestjs/graphql';
import { InstanceUpgradeStatusDTO } from 'src/engine/core-modules/upgrade/dtos/instance-upgrade-status.dto';
import { WorkspaceUpgradeRefDTO } from 'src/engine/core-modules/upgrade/dtos/workspace-upgrade-ref.dto';
@ObjectType('InstanceAndAllWorkspacesUpgradeStatus')
export class InstanceAndAllWorkspacesUpgradeStatusDTO {
@Field(() => InstanceUpgradeStatusDTO)
instanceUpgradeStatus: InstanceUpgradeStatusDTO;
@Field(() => [WorkspaceUpgradeRefDTO])
workspacesBehind: WorkspaceUpgradeRefDTO[];
@Field(() => [WorkspaceUpgradeRefDTO])
workspacesFailed: WorkspaceUpgradeRefDTO[];
@Field(() => Date)
computedAt: Date;
}
@@ -0,0 +1,34 @@
import { Field, ObjectType } from '@nestjs/graphql';
import { UpgradeHealthEnum } from 'src/engine/core-modules/upgrade/dtos/upgrade-health.enum';
import { type UpgradeMigrationStatus } from 'src/engine/core-modules/upgrade/upgrade-migration.entity';
@ObjectType('LatestUpgradeCommand')
export class LatestUpgradeCommandDTO {
@Field(() => String)
name: string;
@Field(() => String)
status: UpgradeMigrationStatus;
@Field(() => String)
executedByVersion: string;
@Field(() => String, { nullable: true })
errorMessage: string | null;
@Field(() => Date)
createdAt: Date;
}
@ObjectType('InstanceUpgradeStatus')
export class InstanceUpgradeStatusDTO {
@Field(() => String, { nullable: true })
inferredVersion: string | null;
@Field(() => UpgradeHealthEnum)
health: UpgradeHealthEnum;
@Field(() => LatestUpgradeCommandDTO, { nullable: true })
latestCommand: LatestUpgradeCommandDTO | null;
}
@@ -0,0 +1,9 @@
import { registerEnumType } from '@nestjs/graphql';
import { UpgradeHealthEnum } from 'twenty-shared/types';
export { UpgradeHealthEnum };
registerEnumType(UpgradeHealthEnum, {
name: 'UpgradeHealth',
});
@@ -0,0 +1,12 @@
import { Field, ObjectType } from '@nestjs/graphql';
import { UUIDScalarType } from 'src/engine/api/graphql/workspace-schema-builder/graphql-types/scalars';
@ObjectType('WorkspaceUpgradeRef')
export class WorkspaceUpgradeRefDTO {
@Field(() => UUIDScalarType)
id: string;
@Field(() => String, { nullable: true })
name: string | null;
}
@@ -0,0 +1,23 @@
import { Field, ObjectType } from '@nestjs/graphql';
import { UUIDScalarType } from 'src/engine/api/graphql/workspace-schema-builder/graphql-types/scalars';
import { LatestUpgradeCommandDTO } from 'src/engine/core-modules/upgrade/dtos/instance-upgrade-status.dto';
import { UpgradeHealthEnum } from 'src/engine/core-modules/upgrade/dtos/upgrade-health.enum';
@ObjectType('WorkspaceUpgradeStatus')
export class WorkspaceUpgradeStatusDTO {
@Field(() => UUIDScalarType)
workspaceId: string;
@Field(() => String, { nullable: true })
displayName: string | null;
@Field(() => String, { nullable: true })
inferredVersion: string | null;
@Field(() => UpgradeHealthEnum)
health: UpgradeHealthEnum;
@Field(() => LatestUpgradeCommandDTO, { nullable: true })
latestCommand: LatestUpgradeCommandDTO | null;
}
@@ -1,8 +1,13 @@
import { Test } from '@nestjs/testing';
import { getRepositoryToken } from '@nestjs/typeorm';
import { UpgradeHealthEnum } from 'twenty-shared/types';
import { WorkspaceActivationStatus } from 'twenty-shared/workspace';
import { CoreEntityCacheService } from 'src/engine/core-entity-cache/services/core-entity-cache.service';
import { UpgradeMigrationService } from 'src/engine/core-modules/upgrade/services/upgrade-migration.service';
import { UpgradeSequenceReaderService } from 'src/engine/core-modules/upgrade/services/upgrade-sequence-reader.service';
import { UpgradeStatusCacheService } from 'src/engine/core-modules/upgrade/services/upgrade-status-cache.service';
import { UpgradeStatusService } from 'src/engine/core-modules/upgrade/services/upgrade-status.service';
import { WorkspaceEntity } from 'src/engine/core-modules/workspace/workspace.entity';
@@ -16,16 +21,61 @@ const MOCK_SEQUENCE = [
{ kind: 'workspace', name: LAST_WORKSPACE_COMMAND },
];
type WorkspaceRecord = {
id: string;
displayName: string | null;
};
const buildWorkspaceCacheGetMock = (
workspaces: WorkspaceRecord[],
): jest.Mock => {
const byId = new Map(
workspaces.map((workspace) => [workspace.id, workspace]),
);
return jest.fn(async (_cacheKey: string, workspaceId: string) => {
const workspace = byId.get(workspaceId);
if (!workspace) {
return null;
}
return {
activationStatus: WorkspaceActivationStatus.ACTIVE,
...workspace,
};
});
};
describe('UpgradeStatusService', () => {
let service: UpgradeStatusService;
let getLastAttemptedInstanceCommand: jest.Mock;
let getWorkspaceLastAttemptedCommandName: jest.Mock;
let workspaceFind: jest.Mock;
let coreEntityCacheGet: jest.Mock;
let cacheGetComputedAt: jest.Mock;
let cacheGetBehindWorkspaceIds: jest.Mock;
let cacheGetFailedWorkspaceIds: jest.Mock;
let cacheWrite: jest.Mock;
let cacheInvalidate: jest.Mock;
const mockActiveWorkspaces = (workspaces: WorkspaceRecord[]) => {
workspaceFind.mockResolvedValue(workspaces);
coreEntityCacheGet.mockImplementation(
buildWorkspaceCacheGetMock(workspaces),
);
};
beforeEach(async () => {
getLastAttemptedInstanceCommand = jest.fn();
getWorkspaceLastAttemptedCommandName = jest.fn();
workspaceFind = jest.fn();
workspaceFind = jest.fn().mockResolvedValue([]);
coreEntityCacheGet = jest.fn().mockResolvedValue(null);
cacheGetComputedAt = jest.fn();
cacheGetBehindWorkspaceIds = jest.fn().mockResolvedValue([]);
cacheGetFailedWorkspaceIds = jest.fn().mockResolvedValue([]);
cacheWrite = jest.fn().mockResolvedValue(undefined);
cacheInvalidate = jest.fn().mockResolvedValue(undefined);
const module = await Test.createTestingModule({
providers: [
@@ -47,6 +97,20 @@ describe('UpgradeStatusService', () => {
provide: getRepositoryToken(WorkspaceEntity),
useValue: { find: workspaceFind },
},
{
provide: CoreEntityCacheService,
useValue: { get: coreEntityCacheGet },
},
{
provide: UpgradeStatusCacheService,
useValue: {
getComputedAt: cacheGetComputedAt,
getBehindWorkspaceIds: cacheGetBehindWorkspaceIds,
getFailedWorkspaceIds: cacheGetFailedWorkspaceIds,
write: cacheWrite,
invalidate: cacheInvalidate,
},
},
],
}).compile();
@@ -65,7 +129,7 @@ describe('UpgradeStatusService', () => {
const result = await service.getInstanceStatus();
expect(result.health).toBe('up-to-date');
expect(result.health).toBe(UpgradeHealthEnum.UP_TO_DATE);
expect(result.inferredVersion).toBe('1.23.0');
});
@@ -80,7 +144,7 @@ describe('UpgradeStatusService', () => {
const result = await service.getInstanceStatus();
expect(result.health).toBe('behind');
expect(result.health).toBe(UpgradeHealthEnum.BEHIND);
expect(result.inferredVersion).toBe('1.22.0');
});
@@ -95,7 +159,7 @@ describe('UpgradeStatusService', () => {
const result = await service.getInstanceStatus();
expect(result.health).toBe('failed');
expect(result.health).toBe(UpgradeHealthEnum.FAILED);
expect(result.latestCommand?.errorMessage).toBe('column does not exist');
});
@@ -104,7 +168,7 @@ describe('UpgradeStatusService', () => {
const result = await service.getInstanceStatus();
expect(result.health).toBe('behind');
expect(result.health).toBe(UpgradeHealthEnum.BEHIND);
expect(result.inferredVersion).toBeNull();
expect(result.latestCommand).toBeNull();
});
@@ -112,7 +176,7 @@ describe('UpgradeStatusService', () => {
describe('getWorkspaceStatuses', () => {
it('should return up-to-date for workspace at last command', async () => {
workspaceFind.mockResolvedValue([{ id: 'ws-1', displayName: 'Apple' }]);
mockActiveWorkspaces([{ id: 'ws-1', displayName: 'Apple' }]);
getWorkspaceLastAttemptedCommandName.mockResolvedValue(
new Map([
@@ -133,11 +197,11 @@ describe('UpgradeStatusService', () => {
const results = await service.getWorkspaceStatuses();
expect(results).toHaveLength(1);
expect(results[0].health).toBe('up-to-date');
expect(results[0].health).toBe(UpgradeHealthEnum.UP_TO_DATE);
});
it('should return behind for workspace not at last command', async () => {
workspaceFind.mockResolvedValue([
mockActiveWorkspaces([
{ id: 'ws-1', displayName: 'Apple' },
{ id: 'ws-2', displayName: 'Google' },
]);
@@ -172,24 +236,24 @@ describe('UpgradeStatusService', () => {
const results = await service.getWorkspaceStatuses();
expect(results).toHaveLength(2);
expect(results[0].health).toBe('up-to-date');
expect(results[1].health).toBe('behind');
expect(results[0].health).toBe(UpgradeHealthEnum.UP_TO_DATE);
expect(results[1].health).toBe(UpgradeHealthEnum.BEHIND);
});
it('should return behind for workspace with no migration history', async () => {
workspaceFind.mockResolvedValue([{ id: 'ws-1', displayName: 'Apple' }]);
mockActiveWorkspaces([{ id: 'ws-1', displayName: 'Apple' }]);
getWorkspaceLastAttemptedCommandName.mockResolvedValue(new Map());
const results = await service.getWorkspaceStatuses();
expect(results).toHaveLength(1);
expect(results[0].health).toBe('behind');
expect(results[0].health).toBe(UpgradeHealthEnum.BEHIND);
expect(results[0].latestCommand).toBeNull();
});
it('should return empty array when no workspaces exist', async () => {
workspaceFind.mockResolvedValue([]);
mockActiveWorkspaces([]);
getWorkspaceLastAttemptedCommandName.mockResolvedValue(new Map());
const results = await service.getWorkspaceStatuses();
@@ -197,4 +261,136 @@ describe('UpgradeStatusService', () => {
expect(results).toHaveLength(0);
});
});
describe('getInstanceAndAllWorkspacesStatus', () => {
it('should hydrate cached behind/failed ids with display names without calling getWorkspaceStatuses', async () => {
const computedAt = new Date('2025-06-02T10:00:00Z');
cacheGetComputedAt.mockResolvedValue(computedAt);
cacheGetBehindWorkspaceIds.mockResolvedValue(['ws-2']);
cacheGetFailedWorkspaceIds.mockResolvedValue(['ws-3']);
getLastAttemptedInstanceCommand.mockResolvedValue({
name: LAST_INSTANCE_COMMAND,
status: 'completed',
executedByVersion: '1.23.0',
errorMessage: null,
createdAt: new Date('2025-06-01T00:00:00Z'),
});
coreEntityCacheGet.mockImplementation(
buildWorkspaceCacheGetMock([
{ id: 'ws-2', displayName: 'Banana' },
{ id: 'ws-3', displayName: 'Cherry' },
]),
);
const result = await service.getInstanceAndAllWorkspacesStatus();
expect(result.workspacesBehind).toEqual([{ id: 'ws-2', name: 'Banana' }]);
expect(result.workspacesFailed).toEqual([{ id: 'ws-3', name: 'Cherry' }]);
expect(result.computedAt).toEqual(computedAt);
expect(getWorkspaceLastAttemptedCommandName).not.toHaveBeenCalled();
expect(cacheWrite).not.toHaveBeenCalled();
});
it('should fall back to a refresh when the cache marker is missing', async () => {
cacheGetComputedAt.mockResolvedValue(null);
getLastAttemptedInstanceCommand.mockResolvedValue(null);
mockActiveWorkspaces([{ id: 'ws-1', displayName: 'Apple' }]);
getWorkspaceLastAttemptedCommandName.mockResolvedValue(new Map());
const result = await service.getInstanceAndAllWorkspacesStatus();
expect(cacheWrite).toHaveBeenCalledTimes(1);
expect(result.workspacesBehind).toEqual([{ id: 'ws-1', name: 'Apple' }]);
});
it('should use null name when a cached id is missing from the cache', async () => {
cacheGetComputedAt.mockResolvedValue(new Date());
cacheGetBehindWorkspaceIds.mockResolvedValue(['ws-orphan']);
getLastAttemptedInstanceCommand.mockResolvedValue(null);
coreEntityCacheGet.mockResolvedValue(null);
const result = await service.getInstanceAndAllWorkspacesStatus();
expect(result.workspacesBehind).toEqual([
{ id: 'ws-orphan', name: null },
]);
});
it('should not query workspace names when both cached id sets are empty', async () => {
cacheGetComputedAt.mockResolvedValue(new Date());
getLastAttemptedInstanceCommand.mockResolvedValue(null);
await service.getInstanceAndAllWorkspacesStatus();
expect(coreEntityCacheGet).not.toHaveBeenCalled();
});
});
describe('refreshInstanceAndAllWorkspacesStatus', () => {
it('should partition workspaces by health, write to cache, and return the fresh payload', async () => {
getLastAttemptedInstanceCommand.mockResolvedValue(null);
mockActiveWorkspaces([
{ id: 'ws-1', displayName: 'Apple' },
{ id: 'ws-2', displayName: 'Banana' },
{ id: 'ws-3', displayName: 'Cherry' },
]);
getWorkspaceLastAttemptedCommandName.mockResolvedValue(
new Map([
[
'ws-1',
{
workspaceId: 'ws-1',
name: LAST_WORKSPACE_COMMAND,
status: 'completed',
executedByVersion: '1.23.0',
errorMessage: null,
createdAt: new Date('2025-06-01T00:00:00Z'),
},
],
[
'ws-2',
{
workspaceId: 'ws-2',
name: EARLIER_COMMAND,
status: 'completed',
executedByVersion: '1.22.0',
errorMessage: null,
createdAt: new Date('2025-05-01T00:00:00Z'),
},
],
[
'ws-3',
{
workspaceId: 'ws-3',
name: LAST_WORKSPACE_COMMAND,
status: 'failed',
executedByVersion: '1.23.0',
errorMessage: 'boom',
createdAt: new Date('2025-06-01T00:00:00Z'),
},
],
]),
);
const result = await service.refreshInstanceAndAllWorkspacesStatus();
expect(result.workspacesBehind).toEqual([{ id: 'ws-2', name: 'Banana' }]);
expect(result.workspacesFailed).toEqual([{ id: 'ws-3', name: 'Cherry' }]);
expect(cacheWrite).toHaveBeenCalledWith({
behindWorkspaceIds: ['ws-2'],
failedWorkspaceIds: ['ws-3'],
computedAt: expect.any(Date),
});
});
});
describe('invalidateInstanceAndAllWorkspacesStatus', () => {
it('should delegate to the cache service', async () => {
await service.invalidateInstanceAndAllWorkspacesStatus();
expect(cacheInvalidate).toHaveBeenCalledTimes(1);
});
});
});
@@ -7,6 +7,7 @@ import { TwentyConfigService } from 'src/engine/core-modules/twenty-config/twent
import { type FastInstanceCommand } from 'src/engine/core-modules/upgrade/interfaces/fast-instance-command.interface';
import { type SlowInstanceCommand } from 'src/engine/core-modules/upgrade/interfaces/slow-instance-command.interface';
import { UpgradeMigrationService } from 'src/engine/core-modules/upgrade/services/upgrade-migration.service';
import { UpgradeStatusService } from 'src/engine/core-modules/upgrade/services/upgrade-status.service';
import { WorkspaceVersionService } from 'src/engine/workspace-manager/workspace-version/services/workspace-version.service';
type RunSingleMigrationResult =
@@ -24,6 +25,7 @@ export class InstanceCommandRunnerService {
private readonly twentyConfigService: TwentyConfigService,
private readonly upgradeMigrationService: UpgradeMigrationService,
private readonly workspaceVersionService: WorkspaceVersionService,
private readonly upgradeStatusService: UpgradeStatusService,
) {}
async runFastInstanceCommand({
@@ -71,6 +73,10 @@ export class InstanceCommandRunnerService {
});
await queryRunner.commitTransaction();
this.logger.log(`${name} executed successfully`);
return { status: 'success' };
} catch (error) {
if (queryRunner.isTransactionActive) {
await queryRunner.rollbackTransaction();
@@ -96,11 +102,20 @@ export class InstanceCommandRunnerService {
return { status: 'failed', error };
} finally {
await queryRunner.release();
await this.safeInvalidateUpgradeStatusCache();
}
}
this.logger.log(`${name} executed successfully`);
return { status: 'success' };
private async safeInvalidateUpgradeStatusCache(): Promise<void> {
try {
await this.upgradeStatusService.invalidateInstanceAndAllWorkspacesStatus();
} catch (error) {
this.logger.warn(
`Failed to invalidate upgrade-status cache: ${
error instanceof Error ? error.message : String(error)
}`,
);
}
}
async runSlowInstanceCommand({
@@ -150,6 +165,8 @@ export class InstanceCommandRunnerService {
error instanceof Error ? error.stack : String(error),
);
await this.safeInvalidateUpgradeStatusCache();
return { status: 'failed', error };
}
}
@@ -6,7 +6,7 @@ import { In, IsNull, type QueryRunner, Repository } from 'typeorm';
import {
UpgradeMigrationEntity,
type UpgradeMigrationStatus,
UpgradeMigrationStatus,
} from 'src/engine/core-modules/upgrade/upgrade-migration.entity';
import { formatUpgradeErrorForStorage } from 'src/engine/core-modules/upgrade/utils/format-upgrade-error-for-storage.util';
@@ -0,0 +1,75 @@
import { Injectable } from '@nestjs/common';
import { isDefined } from 'twenty-shared/utils';
import { InjectCacheStorage } from 'src/engine/core-modules/cache-storage/decorators/cache-storage.decorator';
import { CacheStorageService } from 'src/engine/core-modules/cache-storage/services/cache-storage.service';
import { CacheStorageNamespace } from 'src/engine/core-modules/cache-storage/types/cache-storage-namespace.enum';
const BEHIND_IDS_KEY = 'upgrade-status:behind-workspace-ids';
const FAILED_IDS_KEY = 'upgrade-status:failed-workspace-ids';
const COMPUTED_AT_KEY = 'upgrade-status:computed-at';
const CACHE_TTL_MS = 60 * 60 * 1000;
@Injectable()
export class UpgradeStatusCacheService {
constructor(
@InjectCacheStorage(CacheStorageNamespace.EngineHealth)
private readonly cacheStorage: CacheStorageService,
) {}
async getComputedAt(): Promise<Date | null> {
const computedAt = await this.cacheStorage.get<string>(COMPUTED_AT_KEY);
return isDefined(computedAt) ? new Date(computedAt) : null;
}
async getBehindWorkspaceIds(): Promise<string[]> {
return this.cacheStorage.setMembers(BEHIND_IDS_KEY);
}
async getFailedWorkspaceIds(): Promise<string[]> {
return this.cacheStorage.setMembers(FAILED_IDS_KEY);
}
async write({
behindWorkspaceIds,
failedWorkspaceIds,
computedAt,
}: {
behindWorkspaceIds: string[];
failedWorkspaceIds: string[];
computedAt: Date;
}): Promise<void> {
await Promise.all([
this.cacheStorage.del(BEHIND_IDS_KEY),
this.cacheStorage.del(FAILED_IDS_KEY),
]);
await Promise.all([
this.cacheStorage.setAdd(
BEHIND_IDS_KEY,
behindWorkspaceIds,
CACHE_TTL_MS,
),
this.cacheStorage.setAdd(
FAILED_IDS_KEY,
failedWorkspaceIds,
CACHE_TTL_MS,
),
this.cacheStorage.set(
COMPUTED_AT_KEY,
computedAt.toISOString(),
CACHE_TTL_MS,
),
]);
}
async invalidate(): Promise<void> {
await Promise.all([
this.cacheStorage.del(BEHIND_IDS_KEY),
this.cacheStorage.del(FAILED_IDS_KEY),
this.cacheStorage.del(COMPUTED_AT_KEY),
]);
}
}
@@ -1,50 +1,69 @@
import { Injectable, Logger } from '@nestjs/common';
import { InjectRepository } from '@nestjs/typeorm';
import { UpgradeHealthEnum } from 'twenty-shared/types';
import { isDefined } from 'twenty-shared/utils';
import { WorkspaceActivationStatus } from 'twenty-shared/workspace';
import { In, Repository } from 'typeorm';
import { InjectRepository } from '@nestjs/typeorm';
import { CoreEntityCacheService } from 'src/engine/core-entity-cache/services/core-entity-cache.service';
import { UpgradeMigrationService } from 'src/engine/core-modules/upgrade/services/upgrade-migration.service';
import { UpgradeSequenceReaderService } from 'src/engine/core-modules/upgrade/services/upgrade-sequence-reader.service';
import { UpgradeStatusCacheService } from 'src/engine/core-modules/upgrade/services/upgrade-status-cache.service';
import { type UpgradeMigrationStatus } from 'src/engine/core-modules/upgrade/upgrade-migration.entity';
import { extractVersionFromCommandName } from 'src/engine/core-modules/upgrade/utils/extract-version-from-command-name.util';
import { WorkspaceEntity } from 'src/engine/core-modules/workspace/workspace.entity';
import { In, Repository } from 'typeorm';
export type UpgradeHealth = 'up-to-date' | 'behind' | 'failed';
export type MigrationCursorStatus = {
inferredVersion: string | null;
health: UpgradeHealth;
latestCommand: {
name: string;
status: UpgradeMigrationStatus;
executedByVersion: string;
errorMessage: string | null;
createdAt: Date;
} | null;
export type LatestUpgradeCommand = {
name: string;
status: UpgradeMigrationStatus;
executedByVersion: string;
errorMessage: string | null;
createdAt: Date;
};
export type WorkspaceStatus = MigrationCursorStatus & {
export type InstanceUpgradeStatus = {
inferredVersion: string | null;
health: UpgradeHealthEnum;
latestCommand: LatestUpgradeCommand | null;
};
export type WorkspaceUpgradeStatus = {
workspaceId: string;
displayName: string | null;
inferredVersion: string | null;
health: UpgradeHealthEnum;
latestCommand: LatestUpgradeCommand | null;
};
export type WorkspaceUpgradeRef = {
id: string;
name: string | null;
};
export type InstanceAndAllWorkspacesUpgradeStatus = {
instanceUpgradeStatus: InstanceUpgradeStatus;
workspacesBehind: WorkspaceUpgradeRef[];
workspacesFailed: WorkspaceUpgradeRef[];
computedAt: Date;
};
const deriveHealth = (
migration: { name: string; status: UpgradeMigrationStatus },
lastExpectedCommandName: string | null,
): UpgradeHealth => {
): UpgradeHealthEnum => {
if (migration.status === 'failed') {
return 'failed';
return UpgradeHealthEnum.FAILED;
}
if (
lastExpectedCommandName !== null &&
migration.name !== lastExpectedCommandName
) {
return 'behind';
return UpgradeHealthEnum.BEHIND;
}
return 'up-to-date';
return UpgradeHealthEnum.UP_TO_DATE;
};
@Injectable()
@@ -56,9 +75,11 @@ export class UpgradeStatusService {
private readonly upgradeSequenceReaderService: UpgradeSequenceReaderService,
@InjectRepository(WorkspaceEntity)
private readonly workspaceRepository: Repository<WorkspaceEntity>,
private readonly upgradeStatusCacheService: UpgradeStatusCacheService,
private readonly coreEntityCacheService: CoreEntityCacheService,
) {}
async getInstanceStatus(): Promise<MigrationCursorStatus> {
async getInstanceStatus(): Promise<InstanceUpgradeStatus> {
const migration =
await this.upgradeMigrationService.getLastAttemptedInstanceCommand();
@@ -75,8 +96,9 @@ export class UpgradeStatusService {
async getWorkspaceStatuses(
filterWorkspaceIds?: string[],
): Promise<WorkspaceStatus[]> {
const workspaces = await this.loadWorkspaces(filterWorkspaceIds);
): Promise<WorkspaceUpgradeStatus[]> {
const workspaces =
await this.loadActiveOrSuspendedWorkspaces(filterWorkspaceIds);
if (filterWorkspaceIds) {
const foundIds = new Set(workspaces.map((workspace) => workspace.id));
@@ -110,18 +132,93 @@ export class UpgradeStatusService {
}));
}
async getInstanceAndAllWorkspacesStatus(): Promise<InstanceAndAllWorkspacesUpgradeStatus> {
const computedAt = await this.upgradeStatusCacheService.getComputedAt();
if (!isDefined(computedAt)) {
return this.refreshInstanceAndAllWorkspacesStatus();
}
const [instanceUpgradeStatus, behindWorkspaceIds, failedWorkspaceIds] =
await Promise.all([
this.getInstanceStatus(),
this.upgradeStatusCacheService.getBehindWorkspaceIds(),
this.upgradeStatusCacheService.getFailedWorkspaceIds(),
]);
const workspaceNamesById = await this.loadWorkspaceNamesById([
...behindWorkspaceIds,
...failedWorkspaceIds,
]);
return {
instanceUpgradeStatus,
workspacesBehind: this.toWorkspaceRefs(
behindWorkspaceIds,
workspaceNamesById,
),
workspacesFailed: this.toWorkspaceRefs(
failedWorkspaceIds,
workspaceNamesById,
),
computedAt,
};
}
async refreshInstanceAndAllWorkspacesStatus(): Promise<InstanceAndAllWorkspacesUpgradeStatus> {
this.logger.log('Recomputing upgrade status for all workspaces');
const [instanceUpgradeStatus, workspaceStatuses] = await Promise.all([
this.getInstanceStatus(),
this.getWorkspaceStatuses(),
]);
const workspacesBehind: WorkspaceUpgradeRef[] = [];
const workspacesFailed: WorkspaceUpgradeRef[] = [];
for (const workspaceStatus of workspaceStatuses) {
const workspaceRef: WorkspaceUpgradeRef = {
id: workspaceStatus.workspaceId,
name: workspaceStatus.displayName,
};
if (workspaceStatus.health === UpgradeHealthEnum.BEHIND) {
workspacesBehind.push(workspaceRef);
} else if (workspaceStatus.health === UpgradeHealthEnum.FAILED) {
workspacesFailed.push(workspaceRef);
}
}
const computedAt = new Date();
await this.upgradeStatusCacheService.write({
behindWorkspaceIds: workspacesBehind.map((workspace) => workspace.id),
failedWorkspaceIds: workspacesFailed.map((workspace) => workspace.id),
computedAt,
});
return {
instanceUpgradeStatus,
workspacesBehind,
workspacesFailed,
computedAt,
};
}
async invalidateInstanceAndAllWorkspacesStatus(): Promise<void> {
await this.upgradeStatusCacheService.invalidate();
}
private buildCursorStatus(
migration: {
name: string;
status: UpgradeMigrationStatus;
executedByVersion: string;
errorMessage: string | null;
createdAt: Date;
} | null,
migration: LatestUpgradeCommand | null,
lastExpectedCommandName: string | null,
): MigrationCursorStatus {
): InstanceUpgradeStatus {
if (!migration) {
return { inferredVersion: null, health: 'behind', latestCommand: null };
return {
inferredVersion: null,
health: UpgradeHealthEnum.BEHIND,
latestCommand: null,
};
}
const health = deriveHealth(migration, lastExpectedCommandName);
@@ -139,7 +236,7 @@ export class UpgradeStatusService {
};
}
private async loadWorkspaces(
private async loadActiveOrSuspendedWorkspaces(
workspaceIds?: string[],
): Promise<Pick<WorkspaceEntity, 'id' | 'displayName'>[]> {
return this.workspaceRepository.find({
@@ -156,4 +253,38 @@ export class UpgradeStatusService {
order: { id: 'ASC' },
});
}
private async loadWorkspaceNamesById(
workspaceIds: string[],
): Promise<Map<string, string | null>> {
const namesById = new Map<string, string | null>();
if (workspaceIds.length === 0) {
return namesById;
}
const workspaces = await Promise.all(
workspaceIds.map((workspaceId) =>
this.coreEntityCacheService.get('workspaceEntity', workspaceId),
),
);
for (const workspace of workspaces) {
if (isDefined(workspace)) {
namesById.set(workspace.id, workspace.displayName ?? null);
}
}
return namesById;
}
private toWorkspaceRefs(
workspaceIds: string[],
workspaceNamesById: Map<string, string | null>,
): WorkspaceUpgradeRef[] {
return workspaceIds.map((workspaceId) => ({
id: workspaceId,
name: workspaceNamesById.get(workspaceId) ?? null,
}));
}
}
@@ -5,6 +5,7 @@ import { type ParsedUpgradeCommandOptions } from 'src/database/commands/upgrade-
import { TwentyConfigService } from 'src/engine/core-modules/twenty-config/twenty-config.service';
import { type RegisteredWorkspaceCommand } from 'src/engine/core-modules/upgrade/services/upgrade-command-registry.service';
import { UpgradeMigrationService } from 'src/engine/core-modules/upgrade/services/upgrade-migration.service';
import { UpgradeStatusService } from 'src/engine/core-modules/upgrade/services/upgrade-status.service';
type WorkspaceCommandEntry = Pick<
RegisteredWorkspaceCommand,
@@ -24,6 +25,7 @@ export class WorkspaceCommandRunnerService {
constructor(
private readonly twentyConfigService: TwentyConfigService,
private readonly upgradeMigrationService: UpgradeMigrationService,
private readonly upgradeStatusService: UpgradeStatusService,
) {}
async runWorkspaceCommands({
@@ -40,17 +42,35 @@ export class WorkspaceCommandRunnerService {
const executedByVersion =
this.twentyConfigService.get('APP_VERSION') ?? 'unknown';
for (const workspaceCommandEntry of workspaceCommands) {
await this.runSingleWorkspaceCommandOrThrow({
workspaceCommandEntry,
workspaceId,
executedByVersion,
options,
iteratorContext,
});
}
try {
for (const workspaceCommandEntry of workspaceCommands) {
await this.runSingleWorkspaceCommandOrThrow({
workspaceCommandEntry,
workspaceId,
executedByVersion,
options,
iteratorContext,
});
}
this.logger.log(`Upgrade for workspace ${workspaceId} completed.`);
this.logger.log(`Upgrade for workspace ${workspaceId} completed.`);
} finally {
if (!options.dryRun) {
await this.safeInvalidateWorkspace(workspaceId);
}
}
}
private async safeInvalidateWorkspace(workspaceId: string): Promise<void> {
try {
await this.upgradeStatusService.invalidateInstanceAndAllWorkspacesStatus();
} catch (error) {
this.logger.warn(
`Failed to invalidate upgrade-status cache for workspace ${workspaceId}: ${
error instanceof Error ? error.message : String(error)
}`,
);
}
}
private async runSingleWorkspaceCommandOrThrow({
@@ -22,6 +22,13 @@ export type UpgradeMigrationStatus = 'completed' | 'failed';
unique: true,
where: '"workspaceId" IS NOT NULL',
})
@Index(
'IDX_UPGRADE_MIGRATION_WORKSPACE_ID_NAME_ATTEMPT',
['workspaceId', 'name', 'attempt'],
{
where: '"workspaceId" IS NOT NULL',
},
)
export class UpgradeMigrationEntity {
@PrimaryGeneratedColumn('uuid')
id: string;
@@ -5,11 +5,13 @@ import { TypeOrmModule } from '@nestjs/typeorm';
import { WorkspaceIteratorModule } from 'src/database/commands/command-runners/workspace-iterator.module';
import { InstanceCommandProviderModule } from 'src/database/commands/upgrade-version-command/instance-command-provider.module';
import { WorkspaceCommandProviderModule } from 'src/database/commands/upgrade-version-command/workspace-command-provider.module';
import { CoreEntityCacheModule } from 'src/engine/core-entity-cache/core-entity-cache.module';
import { InstanceCommandRunnerService } from 'src/engine/core-modules/upgrade/services/instance-command-runner.service';
import { UpgradeCommandRegistryService } from 'src/engine/core-modules/upgrade/services/upgrade-command-registry.service';
import { UpgradeMigrationService } from 'src/engine/core-modules/upgrade/services/upgrade-migration.service';
import { UpgradeSequenceReaderService } from 'src/engine/core-modules/upgrade/services/upgrade-sequence-reader.service';
import { UpgradeSequenceRunnerService } from 'src/engine/core-modules/upgrade/services/upgrade-sequence-runner.service';
import { UpgradeStatusCacheService } from 'src/engine/core-modules/upgrade/services/upgrade-status-cache.service';
import { UpgradeStatusService } from 'src/engine/core-modules/upgrade/services/upgrade-status.service';
import { WorkspaceCommandRunnerService } from 'src/engine/core-modules/upgrade/services/workspace-command-runner.service';
import { UpgradeMigrationEntity } from 'src/engine/core-modules/upgrade/upgrade-migration.entity';
@@ -18,6 +20,7 @@ import { WorkspaceVersionModule } from 'src/engine/workspace-manager/workspace-v
@Module({
imports: [
CoreEntityCacheModule,
DiscoveryModule,
InstanceCommandProviderModule,
WorkspaceCommandProviderModule,
@@ -33,6 +36,7 @@ import { WorkspaceVersionModule } from 'src/engine/workspace-manager/workspace-v
UpgradeSequenceReaderService,
UpgradeSequenceRunnerService,
UpgradeStatusService,
UpgradeStatusCacheService,
],
exports: [
UpgradeMigrationService,
@@ -42,6 +46,7 @@ import { WorkspaceVersionModule } from 'src/engine/workspace-manager/workspace-v
UpgradeSequenceReaderService,
UpgradeSequenceRunnerService,
UpgradeStatusService,
UpgradeStatusCacheService,
],
})
export class UpgradeModule {}