## Summary
Adds the symmetric counterpart to `@WasIntroducedInUpgrade` for the
upgrade-aware ORM. Today the framework can describe "this column will
exist once upgrade X applies" but not "this
column will stop existing once upgrade X applies". Plain field deletion
only works when nothing writes to the table during the mid-state window
between the binary booting and the drop
migration completing for a given workspace — fine for sparse tables
(`DropWorkspaceVersionColumn`, `DropPostgresCredentialsTable`), risky
for hot-write tables.
This PR ships the primitive on its own so the upcoming
`rolePermissionFlag.flag` drop has the framework support it needs. No
in-tree consumer yet — coverage is via unit tests against
synthetic entities.
### What's in it
- **New `@WasRemovedInUpgrade({ upgradeCommandName })` decorator**
(class- or property-scope) — mirrors `@WasIntroducedInUpgrade`, uses the
shared
`defineUpgradeMetadataOnClassOrProperty` helper, exposes class +
property getters.
- **`resolveEntityShapeAtUpgradeCursor`** now folds applied-removals
into the existing `hiddenPropertyNames` set. Intro-pending and
removal-applied share one hide bucket — both ask
TypeORM for the same thing.
- **`UpgradeAwareEntityMetadataAdapter`** now disables `isSelect`,
`isInsert`, **and** `isUpdate` for any hidden column, restoring
canonical values when the column comes back.
Previously only `isSelect` was flipped, which left an
INSERT-into-nonexistent-column hole the intro path was tacitly relying
on application code to avoid; this PR closes that hole for
both directions.
- **`validateUpgradeAwareEntityDecorators`** validates
`@WasRemovedInUpgrade` `upgradeCommandName` references, and surfaces a
new `removal-before-introduction` problem when a property
has both decorators with the removal step preceding the introduction
step.
## Summary
- Moves current version to previous versions array
- Sets TWENTY_CURRENT_VERSION to the new version
- Updates TWENTY_NEXT_VERSIONS with the next minor version
## Checklist
- [ ] Verify version constants are correct
Co-authored-by: Github Action Deploy <github-action-deploy@twenty.com>
## What
When the same PR introduces a new core entity *and* adds a cache
provider that queries it, every workspace step from older versions that
runs before the introducing instance step hits `relation … does not
exist` — the cause of the failed [v2.6.0 staging-ci
run](https://github.com/twentyhq/twenty-infra/actions/runs/26042742000).
Same class of failure for renamed core entities and for new FK columns
hidden inside relation loads.
This PR adds **upgrade-aware entity decorators** + a runtime that adapts
TypeORM's view of the schema to the current `core.upgradeMigration`
cursor.
## Strategy
```
┌────────────────────────────────┐
│ @Entity classes (final shape) │
│ + @WasIntroducedInUpgrade │
│ + @WasRenamedInUpgrade │
└───────────────┬────────────────┘
│
UpgradeSequenceRunner.run()
┌─────────────────────┴─────────────────────┐
▼ ▼
step N+1 begins step N just completed
│ │
└────────► adapter.refresh() ◄──────────────┘
│
reads core.upgradeMigration via
UpgradeMigrationService.getLastAttemptedInstanceCommand
│
▼
┌────────────────────────────────────────────────────────┐
│ UpgradeAwareEntityMetadataAdapter │
│ • mutates EntityMetadata.tableName / tablePath │
│ -> historical name for renames not yet applied │
│ • flips column.isSelect = false for not-yet-introduced│
│ columns │
│ • tracks per-entity availability sidecar │
└─────────────────┬──────────────────────────────────────┘
│
▼
DataSource.getRepository wrapped at TypeOrmModule.forRoot:
repo.find() / findOne() / count() / …
┌─────────────────────────────────────────┐
│ wrapRepositoryWithUpgradeAwareProxy │
│ • entity unavailable -> short-circuit │
│ (find -> [], count -> 0, │
│ findOneOrFail -> EntityNotFound) │
│ • write -> Promise.reject( │
│ UpgradeUnavailableEntityWriteEx) │
│ • find({ relations: ['X'] }) with X │
│ unavailable -> X stripped │
└─────────────────────────────────────────┘
```
The decorator strings reference real `core.upgradeMigration.name` values
(`${version}_${className}_${timestamp}`). A boot-time validator walks
the actual `UpgradeSequenceReaderService.getUpgradeSequence()` and fails
fast on typos.
## Files
- New decorators:
`engine/core-modules/upgrade/decorators/was-introduced-in-upgrade.decorator.ts`,
`was-renamed-in-upgrade.decorator.ts`
- Runtime: `engine/twenty-orm/upgrade-aware/` (adapter, proxy, install
hook, state singleton, exceptions)
- Wired into `UpgradeSequenceRunnerService` (`refresh()` between steps)
and `TypeOrmModule.forRoot` (proxy install)
- 2-6 entity decorations: `RolePermissionFlagEntity` (rename history +
new `permissionFlagId` column), `PermissionFlagEntity` (new catalog)
## Validation
End-to-end local cross-version upgrade (v1.22 → HEAD): `28 workspace(s)
succeeded, 0 failed`; `upgrade:status → Instance: Up to date, 4 up to
date, 0 behind, 0 failed`. Full log excerpts and the
second-failure-found-and-fixed (`WorkspaceRolesPermissionsCacheService`
relation load) in [this
comment](https://github.com/twentyhq/twenty/pull/20686#issuecomment-4480036816).
## Test plan
- [x] Adapter spec covers rename mutation; proxy spec covers `find()`
short-circuit on unavailable entity. Resolver + validator + decorators
are covered by `resolve-entity-shape-at-upgrade-cursor.util.spec.ts`
(integration-level via real decorator application).
- [x] `nx lint:diff-with-main twenty-server` + `nx typecheck
twenty-server` clean
- [x] All 82 affected tests passing
- [ ] Cross-version-upgrade CI re-runs after this lands; v2.6.0 retag
once green
## Follow-ups deferred
- v2.7 `connectionProvider` rename repro as a permanent end-to-end test
artifact
- Extending the proxy to also cover `EntityManager.getRepository` and
`createQueryBuilder` if a non-`find()` upgrade-time consumer surfaces
## Summary
- The upgrade runner calls `getWorkspaceLastAttemptedCommandName` twice
per workspace step. Grafana showed it averaging ~4.4s and trending
upward as the `core.upgradeMigration` table grows during an in-flight
upgrade.
- The old query joined every outer row against a correlated subquery
(`attempt = (SELECT MAX(sub.attempt) ... WHERE sub.name = m.name AND
sub."workspaceId" = m."workspaceId")`). Even with the `(workspaceId,
name, attempt)` index added in 2.3, each outer row triggers an index
lookup — fine for a few rows, painful at production scale.
- Replaced with a two-level `DISTINCT ON`:
- Inner `DISTINCT ON ("workspaceId", name) ORDER BY "workspaceId", name,
attempt DESC` walks `IDX_UPGRADE_MIGRATION_WORKSPACE_ID_NAME_ATTEMPT`
directly and yields one row per `(workspaceId, name)` at max attempt.
- Outer `DISTINCT ON ("workspaceId") ORDER BY "workspaceId", "createdAt"
DESC` picks the most recent row per workspace.
- Semantically identical; planner now does a single index walk + one
sort instead of N correlated lookups.
The same correlated-subquery shape exists in
`getLastAttemptedCommandNameOrThrow`, `areAllWorkspacesAtCommand`, and
`getLastAttemptedInstanceCommand`. They run far less often during an
upgrade (per instance step, not per workspace step), so they're out of
scope for this hotfix — happy to follow up if we want them too.
## Benchmark (prod)
Run over all distinct workspaceIds in `core."upgradeMigration"`:
| Variant | Execution Time |
| --- | --- |
| Before (correlated subquery) | **2979.659 ms** |
| After (two-level DISTINCT ON) | **1225.690 ms** |
~2.4× faster, and the gap widens as the table grows over the course of
an upgrade.
Equivalence confirmed: the diff query below returned `0` divergent
workspaces on prod.
### Variant A — original (correlated subquery)
```sql
SELECT DISTINCT ON (m."workspaceId")
m."workspaceId", m.name, m.status, m."executedByVersion",
m."errorMessage", m."createdAt", m."isInitial"
FROM core."upgradeMigration" m
WHERE m."workspaceId" IN ($1, $2, ...)
AND m.attempt = (
SELECT MAX(sub.attempt)
FROM core."upgradeMigration" sub
WHERE sub.name = m.name
AND sub."workspaceId" = m."workspaceId"
)
ORDER BY m."workspaceId", m."createdAt" DESC;
```
### Variant B — new (two-level DISTINCT ON)
```sql
SELECT DISTINCT ON (latest_per_name."workspaceId")
latest_per_name."workspaceId",
latest_per_name.name,
latest_per_name.status,
latest_per_name."executedByVersion",
latest_per_name."errorMessage",
latest_per_name."createdAt",
latest_per_name."isInitial"
FROM (
SELECT DISTINCT ON ("workspaceId", name)
"workspaceId", name, status, "executedByVersion",
"errorMessage", "createdAt", "isInitial"
FROM core."upgradeMigration"
WHERE "workspaceId" = ANY($1)
ORDER BY "workspaceId", name, attempt DESC
) latest_per_name
ORDER BY latest_per_name."workspaceId", latest_per_name."createdAt" DESC;
```
### Equivalence check (returned 0 on prod)
```sql
WITH target_ids AS (
SELECT DISTINCT "workspaceId"
FROM core."upgradeMigration"
WHERE "workspaceId" IS NOT NULL
),
old_result AS (
SELECT DISTINCT ON (m."workspaceId")
m."workspaceId", m.name, m.status, m."executedByVersion",
m."errorMessage", m."createdAt", m."isInitial"
FROM core."upgradeMigration" m
WHERE m."workspaceId" IN (SELECT "workspaceId" FROM target_ids)
AND m.attempt = (
SELECT MAX(sub.attempt)
FROM core."upgradeMigration" sub
WHERE sub.name = m.name
AND sub."workspaceId" = m."workspaceId"
)
ORDER BY m."workspaceId", m."createdAt" DESC
),
new_result AS (
SELECT DISTINCT ON (latest_per_name."workspaceId")
latest_per_name."workspaceId", latest_per_name.name, latest_per_name.status,
latest_per_name."executedByVersion", latest_per_name."errorMessage",
latest_per_name."createdAt", latest_per_name."isInitial"
FROM (
SELECT DISTINCT ON ("workspaceId", name)
"workspaceId", name, status, "executedByVersion",
"errorMessage", "createdAt", "isInitial"
FROM core."upgradeMigration"
WHERE "workspaceId" IN (SELECT "workspaceId" FROM target_ids)
ORDER BY "workspaceId", name, attempt DESC
) latest_per_name
ORDER BY latest_per_name."workspaceId", latest_per_name."createdAt" DESC
),
diffs AS (
SELECT 'only_in_old' AS bucket, o."workspaceId", o.name, o.status, o."createdAt"
FROM old_result o
LEFT JOIN new_result n ON n."workspaceId" = o."workspaceId"
WHERE n."workspaceId" IS NULL OR n.name <> o.name OR n.status <> o.status
UNION ALL
SELECT 'only_in_new', n."workspaceId", n.name, n.status, n."createdAt"
FROM new_result n
LEFT JOIN old_result o ON o."workspaceId" = n."workspaceId"
WHERE o."workspaceId" IS NULL OR o.name <> n.name OR o.status <> n.status
)
SELECT COUNT(*) AS divergent_workspaces FROM diffs;
```
## Test plan
- [ ] `npx nx test twenty-server --testPathPattern upgrade-migration`
- [ ] Integration tests: `npx nx run
twenty-server:test:integration:with-db-reset --testPathPattern
sequence-runner`
- [ ] Verify on staging that the slow query disappears from the
PostgreSQL Grafana board during the next upgrade run
## Summary
Prod deploy of v2.5.0 fails with a query failure inserting into
`core.upgradeMigration`:
```
query failed: INSERT INTO "core"."upgradeMigration" ("id", "name", "status", "attempt", "executedByVersion", "errorMessage", "isInitial", "workspaceId", "createdAt")
VALUES (DEFAULT, $1, $2, $3, $4, $5, DEFAULT, $6, DEFAULT),
(DEFAULT, $7, $8, $9, $10, $11, DEFAULT, $12, DEFAULT),
... (continues past $2515) ...
```
### Root cause
`UpgradeMigrationService.recordUpgradeMigration` writes one row per
workspace via a single `repository.save([...rows])` call.
`UpgradeMigrationEntity` has **6 user-provided columns** per row
(`name`, `status`, `attempt`, `executedByVersion`, `errorMessage`,
`workspaceId`), so the multi-row INSERT binds `6 * (1 + N_workspaces)`
parameters.
Postgres' wire protocol caps a single statement at **65,535 bind
parameters** (16-bit count). That gives a hard ceiling of ~10,920 rows
per call. Production has enough workspaces to overflow.
## Summary
- Moves current version to previous versions array
- Sets TWENTY_CURRENT_VERSION to the new version
- Updates TWENTY_NEXT_VERSIONS with the next minor version
## Checklist
- [ ] Verify version constants are correct
Co-authored-by: Github Action Deploy <github-action-deploy@twenty.com>
## Summary
Adds a fourth gauge alongside the existing
`twenty_upgrade_workspaces_behind_total` /
`twenty_upgrade_workspaces_failed_total` so dashboards can show how many
workspaces are currently healthy, not just the ones that need attention.
- New gauge: `twenty_upgrade_workspaces_up_to_date_total`
- New count is computed during
`UpgradeStatusService.refreshInstanceAndAllWorkspacesStatus` (cheap — we
already iterate over every workspace), persisted in the existing
`UpgradeStatusCacheService` so the cache-hit path stays a single round
trip, and surfaced via `InstanceAndAllWorkspacesUpgradeStatusDTO` for
the admin panel.
## Files
-
`packages/twenty-server/src/engine/core-modules/upgrade/upgrade-gauge.service.ts`
— register the new ObservableGauge
-
`packages/twenty-server/src/engine/core-modules/upgrade/services/upgrade-status.service.ts`
— count UP_TO_DATE workspaces during refresh, propagate through cached
path
-
`packages/twenty-server/src/engine/core-modules/upgrade/services/upgrade-status-cache.service.ts`
— persist `upToDateWorkspaceCount` next to behind/failed sets
-
`packages/twenty-server/src/engine/core-modules/upgrade/dtos/instance-and-all-workspaces-upgrade-status.dto.ts`
— `Int` field on the admin DTO
- Tests: extended `upgrade-status.service.spec.ts` (14/14 green) —
cached and refresh paths both assert on `upToDateWorkspaceCount`
## Follow-up
A companion `twenty-infra` PR adds the new tile + line on the
upgrade-status Grafana dashboard.
## Summary
Adds a small helper that lets every log line in the upgrade flow stay
human-readable while emitting a structured tail that Loki / the
upgrade-status Grafana dashboard can filter on.
Output shape per `logger.log()` call:
```
<humanMessage as-is, may span multiple lines>
[upgrade] event=<event> key=value … ← always single line
```
Same call produces **one** structured Loki event regardless of how
chatty the human-readable part gets — the dashboard's `|= "[upgrade]"`
filter only matches the trailing line.
## Helper API
```ts
formatUpgradeLog({
humanMessage: string, // free-form, multi-line OK, for engineers scrolling raw pod logs
event: string, // required anchor for Loki filtering / dashboards
logFields?: Record<string, // short structured key=value tail
string | number | boolean | null | undefined
>,
});
```
- `humanMessage` is preserved as-is. A thrown `new Error('line one\nline
two')` surfacing through `humanMessage` stays human-readable across
multiple lines.
- `logFields` values are logfmt-escaped: whitespace / `=` / `"` get
quoted, embedded `\` / `"` / `\n` / `\r` / `\t` are escaped, `null` /
`undefined` emit literally (`key=null`, `key=undefined`) instead of
being silently dropped — caught via `isDefined` from
`twenty-shared/utils`.
- `event` itself runs through the same escape so an event name with
whitespace or `=` can't break parsing.
## Example output
```
Initialized upgrade sequence: 8 step(s)
[upgrade] event=sequence.initialized stepCount=8 dryRun=false
Upgrading workspace abc-123 1/10
[upgrade] event=workspace.start workspaceId=abc-123 index=1 total=10 dryRun=false
Upgrade for workspace abc-123 completed.
[upgrade] event=workspace.success workspaceId=abc-123 executedByVersion=1.4.0 dryRun=false
Upgrade summary: 42 workspace(s) succeeded, 1 workspace(s) failed
[upgrade] event=summary totalSuccesses=42 totalFailures=1 dryRun=false
Upgrade failed: Workspace migration runner failed:
- Option id is required
- Option id is invalid
[upgrade] event=aborted totalSuccesses=41 totalFailures=2 dryRun=false
```
Loki query for the dashboard: `{namespace="twenty"} |= "[upgrade]" |
logfmt event, workspaceId, command, executedByVersion`
## Scope
Only the **upgrade-specific** call sites carry the tag:
- `upgrade.command.ts` — `sequence.initialized`, `sequence.step`
(verbose), `summary`, `aborted`
- `upgrade-sequence-runner.service.ts` — `sequence.stopped`,
`sequence.aborted`
- `workspace-command-runner.service.ts` — `workspace.start`,
`workspace.success`, `cache.invalidate.failed`
`instance-command-runner.service.ts` is intentionally **not** tagged —
`runFastInstanceCommand` / `runSlowInstanceCommand` are also invoked
from `RunInstanceCommandsCommand` (DB init / `run-instance-commands`),
so an `[upgrade]` tag there would mislead at init time. Those lines stay
plain-text; stacks still flow on their own via NestJS
`logger.error(message, error.stack)`.
`chalk` is dropped from `upgrade.command.ts` — ANSI escapes break log
parsers and chalk is a no-op without a TTY anyway.
## Tests
9 inline-snapshot tests in `format-upgrade-log.util.spec.ts` surface the
actual output of every interesting shape (summary call site, multi-line
humanMessage, quoted / escaped / control-character logField values,
null/undefined fields, event name escaping). Snapshots double as
documentation of what a real upgrade log line looks like.
## Test plan
- [x] Unit tests green (`jest format-upgrade-log`)
- [x] oxlint + prettier clean
- [x] tsgo typecheck clean on the upgrade module
- [x] CI green
- [ ] Smoke test on staging: run `upgrade` command, confirm `[upgrade]`
structured lines surface in Loki and `| logfmt` extracts fields
## Summary
- Moves current version to previous versions array
- Sets TWENTY_CURRENT_VERSION to the new version
- Updates TWENTY_NEXT_VERSIONS with the next minor version
## Checklist
- [ ] Verify version constants are correct
---------
Co-authored-by: Github Action Deploy <github-action-deploy@twenty.com>
Co-authored-by: prastoin <paul@twenty.com>
## Summary
- Moves current version to previous versions array
- Sets TWENTY_CURRENT_VERSION to the new version
- Updates TWENTY_NEXT_VERSIONS with the next minor version
## Checklist
- [x] Verify version constants are correct
---------
Co-authored-by: Github Action Deploy <github-action-deploy@twenty.com>
Co-authored-by: prastoin <paul@twenty.com>
## Summary
- Converts applicationVariable from a bespoke sync path to a proper
SyncableEntity,
unifying it with the workspace migration pipeline used by all other
manifest-managed
entities (agent, skill, frontComponent, webhook, etc.)
- Removes the upsertManyApplicationVariableEntities method and its
direct-DB-mutation
approach in favor of the standard validate → build → run action handler
pipeline
- Adds universalIdentifier, deletedAt columns and makes applicationId
NOT NULL via an
instance command migration
## Motivation
Before this change, applicationVariable was the only manifest-managed
entity that bypassed
ApplicationManifestMigrationService.syncMetadataFromManifest(). It used
a bespoke service
method called directly from syncApplication(), creating two mental
models, two validation
styles, and two cache invalidation patterns. Now there's one unified
pipeline for all
manifest entities.
## What changed
### Entity refactor:
- ApplicationVariableEntity now extends SyncableEntity (gains
universalIdentifier,
non-nullable applicationId with CASCADE, soft-delete via deletedAt)
### New flat entity layer (flat-application-variable/):
- Type, maps type, editable properties constant, entity-to-flat
converter, cache service,
module
### New migration pipeline wiring:
- Manifest converter
(fromApplicationVariableManifestToUniversalFlatApplicationVariable)
- Validator service (FlatApplicationVariableValidatorService)
- Builder service
(WorkspaceMigrationApplicationVariableActionsBuilderService)
- Create/Update/Delete action handlers with secret encryption hooks
- Registered in orchestrator, builder module, runner module, and all
type registries
### Removed bespoke path:
- Deleted upsertManyApplicationVariableEntities from
ApplicationVariableEntityService
- Removed its call from ApplicationSyncService.syncApplication()
- Kept update() (operator-set value at runtime) and getDisplayValue()
(runtime display)
### Database migration:
- Instance command to add columns, backfill universalIdentifier, enforce
NOT NULL
constraints, and update indexes
## Test plan
- npx nx typecheck twenty-server passes (0 errors)
- Unit tests pass (application-variable.service.spec.ts,
build-env-var.spec.ts)
- Install an app with applicationVariables in its manifest → variables
appear with correct
universalIdentifier
- Update app manifest (add/remove/modify a variable) → migration
pipeline handles diff
correctly
- Operator-set value via update endpoint persists correctly with
encryption
- Uninstall app → variables cascade-deleted
- app dev --once on example app syncs without errors
# Introduction
This PR introduces a workflow and nx command that allow bumping to a
given version or incrementing the current `TWENTY_CURRENT_VERSION`
Combined with accurate on point cd triggered and CI upgrade sequence
guard mutation workflow the window where a PR can corrupt an already
released twenty version is mitigated
## Summary
- Adds `UpgradeGaugeService` that exposes three observable Prometheus
gauges based on the recently merged upgrade status service:
- `twenty_upgrade_instance_health` — 1 (up-to-date), 0 (behind), -1
(failed)
- `twenty_upgrade_workspaces_behind_total` — count of workspaces with
pending upgrade commands
- `twenty_upgrade_workspaces_failed_total` — count of workspaces with a
failed upgrade command
- Follows the existing gauge pattern (`WorkspaceGaugeService`,
`BillingGaugeService`, `DatabaseGaugeService`)
### Caching & QPS design
Prometheus scrapes every **15s** via `ServiceMonitor`. Each gauge uses
the `MetricsService.createObservableGauge({ cacheValue: true })` pattern
which caches the value in Redis for **60 seconds**. Under that,
`UpgradeStatusService.getInstanceAndAllWorkspacesStatus()` uses
`UpgradeStatusCacheService` with a **1-hour TTL** in Redis.
Result: at most 1 DB query per hour regardless of scrape frequency.
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
## Summary
Adds an admin upgrade-status panel that surfaces per-instance and
per-workspace migration health, backed by a Redis-cached aggregate to
keep the page snappy on large fleets.
<img width="827" height="880" alt="Screenshot 2026-04-28 at 10 21 03"
src="https://github.com/user-attachments/assets/8f88baa9-7268-4eff-bf6a-906a7f06ca91"
/>
<img width="804" height="892" alt="Screenshot 2026-04-28 at 10 21 11"
src="https://github.com/user-attachments/assets/1e6decf8-766a-4d0e-96b1-03a9962bba3c"
/>
## Computed metrics
**Instance** (`InstanceUpgradeStatus`)
- `inferredVersion` — version derived from the latest non-initial
instance command name
- `health` — `upToDate` | `behind` | `failed`, derived from the latest
attempt vs. the last expected instance step in the upgrade sequence
- `latestCommand` — `{ name, status, executedByVersion, errorMessage,
createdAt }` from the most recent attempt
**Per-workspace** (`WorkspaceUpgradeStatus`)
- `workspaceId`, `displayName`
- `inferredVersion`, `health`, `latestCommand` (same shape as instance),
computed against the latest expected step in the sequence
**Aggregate** (`AllWorkspacesUpgradeStatus`, only across `ACTIVE` /
`SUSPENDED` workspaces)
- `instanceUpgradeStatus`
- `totalCount`, `upToDateCount`, `behindCount`, `failedCount`
- `workspacesBehindIds[]`, `workspacesFailedIds[]`
- `computedAt`
## Fetching strategy
All reads go through `UpgradeStatusCacheService` (cache namespace:
`EngineHealth`).
- **Aggregate read** (`getAllWorkspacesStatus` →
`getAllWorkspacesUpgradeStatus` query):
reads summary + behind-ids + failed-ids in parallel; if any of the three
keys is missing, full recompute (`recomputeAllWorkspaces`) is triggered,
which also primes per-workspace entries.
- **Per-workspace read** (`getWorkspacesStatus(ids)` →
`getUpgradeStatus(ids)` query):
`mget` on workspace keys; misses are recomputed individually
(`recomputeWorkspace`), and aggregates are reconciled in place (count +
id list deltas) without a full recompute.
- **Recompute on demand**: `refreshUpgradeStatus` mutation calls
`recomputeAllWorkspaces` to bypass cache and rewrite all keys.
- **Auto-invalidation**: `InstanceCommandRunnerService` (fast + slow
paths) and `WorkspaceCommandRunnerService` invalidate after every run
via `safeInvalidateUpgradeStatusCache()`
(`flushByPattern('upgrade-status:*')`). Failures in cache invalidation
are swallowed and logged so they never break the migration runner.
- **TTL**: `60 * 60 * 1000` ms (1 hour) on every key — protects against
stale data even if a runner crashes before invalidating.
## Introduced cache keys
All under the `EngineHealth` cache-storage namespace:
| Key | Type | Purpose |
| --- | --- | --- |
| `upgrade-status:all-workspaces:summary` |
`CachedAllWorkspacesStatusSummary` | Counts + instance status +
`computedAt` |
| `upgrade-status:all-workspaces:behind-ids` | `string[]` | Workspace
ids in `behind` state |
| `upgrade-status:all-workspaces:failed-ids` | `string[]` | Workspace
ids in `failed` state |
| `upgrade-status:workspace:<workspaceId>` |
`CachedWorkspaceUpgradeStatus` | Per-workspace status (one key per
workspace) |
Full invalidation uses the pattern `upgrade-status:*`.
## Index added on `upgradeMigration` (already added on prod)
Migration
`2-2-instance-command-fast-1777308014234-addUpgradeMigrationWorkspaceIdIndex.ts`:
```sql
CREATE INDEX "IDX_upgradeMigration_workspaceId_name_attempt"
ON "core"."upgradeMigration" ("workspaceId", "name", "attempt")
WHERE "workspaceId" IS NOT NULL;
# Introduction
Prevent using both `--start-from-workspace-id` and `--workspace`
When any of the two are being passed we prevent passing to the next
instance segment, it would require an upgrade re run even if legit
When `--start-from-workspace-id` is passed we filter from all the
fetched active or suspended workspace ids and apply equivalent filter as
before
# Introduction
In a nutshell, added a more readable logs that relates that when
performing a workspace fitlered workspace you can only browser the
workspace commands sequence
This PR is not a fix, but a log improvement
As before when cross-upgrading a single workspace you would be facing a
`instance` sync barrier error as not all your workspaces would have been
updated
Now logging and early returning instead of letting the guard hard throw
## Tests
Created a dedicated test for instance prevention on filtered upgrade
Standardized `migrationRecordToKey` usage across all upgrade integration
tests suites
## Summary
We are releasing Twenty v2.2.0. This PR sets up the
upgrade-version-command machinery for the new release line:
- Promote `2.1.0` into `TWENTY_PREVIOUS_VERSIONS` (it just shipped)
- Set `TWENTY_CURRENT_VERSION` to `2.2.0`
- Reset `TWENTY_NEXT_VERSIONS` to `[]`
- Refresh the `InstanceCommandGenerationService` snapshots to reflect
the new current version (`2.2.0` / `2-2-` slug)
- Update the failing-sequence-runner snapshot to include `2.2.0` in the
covered versions list
The `2-2/` upgrade-version-command module is already in place and wired
into `WorkspaceCommandProviderModule`, so future upgrade commands
targeting `2.2.0` can land directly under `2-2/` (or be generated
against `--version 2.2.0`).
## Summary
Fixes Sentry errors of the form:
> \`messages.3: \`tool_use\` ids were found without \`tool_result\`
blocks immediately after: srvtoolu_…. Each \`tool_use\` block must have
a corresponding \`tool_result\` block in the next message.\`
### Root cause
When the model invokes a **provider-hosted tool** (e.g. Anthropic's
native \`web_search\` — note the \`srvtoolu_\` ID prefix), the AI SDK
marks the resulting \`UIMessagePart\` with \`providerExecuted: true\`.
\`convertToModelMessages\` uses that flag to emit the
tool_use/tool_result pair *inside the same assistant message* — the
format Anthropic requires for server-side tools.
Our \`AgentMessagePart\` persistence was dropping \`providerExecuted\`
on the way to the DB (and re-hydration didn't know to set it). On the
next turn, \`convertToModelMessages\` treated the rehydrated part as a
client-side tool call, splitting it into \`assistant(tool_use)\` +
\`user(tool_result)\` — which Anthropic then rejects with the error
above.
### Fix
- Add nullable \`providerExecuted BOOLEAN\` column on
\`core.agentMessagePart\` via a fast instance command.
- Surface the field on \`AgentMessagePartDTO\` (GraphQL).
- Preserve it through \`mapUIMessagePartsToDBParts\` (server) and both
\`mapDBPartToUIMessagePart\` mappers (server + frontend).
- Include it in \`GET_CHAT_MESSAGES\` and \`GET_AGENT_TURNS\`
selections.
- Regenerate \`generated-metadata/graphql.ts\`.
### Backwards compatibility
Existing rows have \`NULL providerExecuted\` and round-trip as the
omitted flag — which is exactly the pre-fix behaviour for tool parts
that were never provider-executed. Only *new* assistant messages using
\`web_search\` (or other provider-hosted tools) will write \`true\`, and
those are the only ones that were breaking.
## Test plan
- [x] \`npx tsgo\` typecheck — server + front clean
- [x] \`oxlint\` + \`prettier --check\` on all touched files — clean
- [x] \`npx nx run twenty-server:database:migrate:prod\` runs the new
instance command locally; \`providerExecuted\` column present on
\`core.agentMessagePart\`
- [x] Regenerated \`generated-metadata/graphql.ts\` —
\`providerExecuted\` wired into both queries and \`AgentMessagePart\`
type
- [ ] Manual: start a chat with Anthropic web_search enabled, invoke the
tool in turn 1, reply in turn 2 — should not throw the srvtoolu error
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
## Summary
We are releasing Twenty v2.0. This PR sets up the
upgrade-version-command machinery for the new release line:
- Move `1.23.0` into `TWENTY_PREVIOUS_VERSIONS` (it just shipped)
- Set `TWENTY_CURRENT_VERSION` to `2.0.0` (no specific upgrade commands
— this is just the major version cut)
- Set `TWENTY_NEXT_VERSIONS` to `['2.1.0']` so future PRs that
previously would have targeted `1.24.0` now target `2.1.0`
- Add empty `V2_0_UpgradeVersionCommandModule` and
`V2_1_UpgradeVersionCommandModule` and wire them into
`WorkspaceCommandProviderModule`
- Refresh the `InstanceCommandGenerationService` snapshots to reflect
the new current version (`2.0.0` / `2-0-` slug)
The `2-0/` directory is intentionally empty — there are no specific
upgrade commands for the v2.0 cut. New upgrade commands authored after
this merges should land in `2-1/` (or be generated against `--version
2.1.0`).
## Test plan
- [x] `npx jest` on the impacted upgrade test files
(`upgrade-sequence-reader`, `upgrade-command-registry`,
`instance-command-generation`) passes (41 tests, 8 snapshots)
- [x] `prettier --check` and `oxlint` clean on touched files
- [ ] Manual: open `nx run twenty-server:command -- upgrade --dry-run`
against a local stack with workspaces still on `1.23.0` and confirm the
sequence is computed without errors
Made with [Cursor](https://cursor.com)
## Summary
### Problem
The upgrade migration system required new workspaces to always start
from a workspace command, which was too rigid. When the system was
mid-upgrade within an instance command (IC) segment, workspace creation
would fail or produce inconsistent state.
### Solution
#### Workspace-scoped instance command rows
Instance commands now write upgrade migration rows for **all
active/suspended workspaces** alongside the global row. This means every
workspace has a complete migration history, including instance command
records.
- `InstanceCommandRunnerService` reloads `activeOrSuspendedWorkspaceIds`
immediately before writing records (both success and failure paths) to
mitigate race conditions with concurrent workspace creation.
- `recordUpgradeMigration` in `UpgradeMigrationService` accepts a
discriminated union over `status`, handles `error: unknown` formatting
internally, and writes global + workspace rows in batch.
#### Flexible initial cursor for new workspaces
`getInitialCursorForNewWorkspace` now accepts the last **attempted**
(not just completed) instance command with its status:
- If the IC is `completed` and the next step is a workspace segment →
cursor is set to the last WC of that segment (existing behavior).
- If the IC is `failed` or not the last of its segment → cursor is set
to that IC itself, preserving its status.
This allows workspaces to be created at any point during the upgrade
lifecycle, including mid-IC-segment and after IC failure.
#### Relaxed workspace segment validation
`validateWorkspaceCursorsAreInWorkspaceSegment` accepts workspaces whose
cursor is:
1. Within the current workspace segment, OR
2. At the immediately preceding instance command with `completed` status
(handles the `-w` single-workspace upgrade scenario).
Workspaces with cursors in a previous segment, ahead of the current
segment, or at a preceding IC with `failed` status are rejected.
### Test plan
created empty workspaces to allow testing upgrade with several active
workspaces
# Introduction
We were allowing the sequence to be empty in the worker context that was
facing an edge case importing the UpgradeModule through the
WorkspaceModule god module, no commands were discovered and it was
throwing as the sequence must have at least one workspace commands to
allow a workspace creation
Though the issue was also applicable to the twenty-server `AppModule`
too that was not discovering any commands
## Integration tests were passing
The integration test were importing the `CommandModule` at the nest
testing app creating leading to asymmetric testing context
It was a requirement for a legacy commands import and global assignation
## Fix
The `UpgradeModule` now import both `WorkspaceCommandsProviderModule`
and `InstanceCommandProviderModule` which ships the commands directly in
the module
We could consider moving the commands into the `engine/upgrade` folder
## Concern
Bootstrap could become more and more long to load at both server and
worker start
When this becomes a problem we will have to only import the latest
workspace command or whatever
For the moment this is not worth it the risk to import not the latest
workspace command
## Introduction
As the new upgrade sequence engine is released in `1.22` it requires all
workspaces to be in `1.21.0` which mean they will have a cursor on the
sequence
As if if someone upgrades from `1.20` to `1.22` no `upgradeMigration`
will exist and throw a pretty basic `Could not find any cursor, database
might not been initialized correctly`
Here we allow a meaningful error
# Introduction
Refactoring the upgrade engine to handle cross version upgrade,
completely getting rid of the semver `version` at db and runtime level
It remains a visual a listing indicator for or CD process but also
during devenv in order to prepare next release
Will write a release process runbook documentation on how to handle
upgrade step patch, command insertion etc as it needs to be cascaded
across all the involved supported version
**The upgrade sequence model:**
The sequence is a flat, ordered array of upgrade steps
(`UpgradeStep[]`), built from the registry by chaining all versions in
order, each version contributing its fast-instance → slow-instance →
workspace commands sorted by timestamp. Version is metadata for logging,
not used in the algorithm.
**Segments:**
The sequence naturally splits into alternating segments of contiguous
instance steps and contiguous workspace steps. The runner processes
segments in order:
- **Instance segment:** Run sequentially from the instance cursor. Each
step runs once globally.
- **Workspace segment:** Each workspace independently walks from its own
cursor through the end of the segment. Workspaces are independent within
a segment — they can be at different positions.
- **Synchronization (workspace → instance):** The runner blocks before
entering an instance segment. All active/suspended workspaces must have
completed the last workspace step of the preceding workspace segment. If
any workspace failed, abort. This is the only explicit synchronization
point.
- Instance → workspace ordering is implicit — the runner processes
segments sequentially, so the instance segment naturally completes
before the workspace segment begins.
full docs
https://gist.github.com/prastoin/e62106d455fd72d6b6ebada8351e5492
## Version constants & type-level deprecation
Version management is split into three atomic constants:
`TWENTY_PREVIOUS_VERSIONS`, `TWENTY_CURRENT_VERSION`, and
`TWENTY_NEXT_VERSIONS`. Two derived constants compose them:
`CROSS_UPGRADE_SUPPORTED_VERSIONS` (previous + current — what the engine
runs) and `ALL_TWENTY_VERSIONS` (the full ordered tuple including next).
The registry service validates at module init that no version is
duplicated across constants and that at least one previous version
exists.
A `DeprecatedSinceVersion<RemoveAtVersion, T>` type utility resolves to
`T` while `TWENTY_CURRENT_VERSION` is below `RemoveAtVersion`, and to
`never` once it reaches it — turning deprecation into a compile-time
guarantee via `IndexOf` and `IsGreaterOrEqual` generics in
`twenty-shared`.
### `workspace.version` column deprecation
The column is replaced by cursor-based state inference from
`UpgradeMigration` records, but cannot be dropped in 1.22: workspaces
activated during 1.21 predate the cursor system and need their initial
cursor backfilled first (`backfillWorkspaceCreatedIn1_21_0Cursors`).
This backfill itself depends on a new `isInitial` column on
`UpgradeMigration`, bootstrapped via a targeted TypeORM migration before
the upgrade sequence runs.
Both functions and the entity field are typed with
`DeprecatedSinceVersion<'1.23.0', ...>`. When `TWENTY_CURRENT_VERSION`
reaches `1.23.0`, compile errors force their removal — and the
pre-declared `DropWorkspaceVersionColumnFastInstanceCommand` takes over
to drop the column.
## What's next
- ci cross version upgrade ( wip )
- banner asking to contact twenty administrator if workspace is outdated
- upgrade healthcheck cli
## New unit/integ test pattern
Create a dedicated `createNestApp` that consumes a real database in
order not to have to mack any database interaction to the
`upgradeMigrations` allowing full coverage of the whole
`upgradeRunnerService.run` core logic
# Introduction
Persist in the db the registry computed name that contains
version_classname_timestamp to be consistent with the instance command
pattern too
# Introduction
This PR introduces the slow instance commands pattern, that allow
migrating data in prior of the schema migration, that would fail if not.
Slow instance commands runs after the fast instance commands and before
the workspace commands.
On twenty instance that do not has any active or suspended workspace the
data migration part is skipped but the migration still runs, especially
for fresh installs
We were previously hacking through typeorm transaction system to gain
such granularity using save points:
```ts
export class AddPayloadToCommandMenuItem1775129635528
implements MigrationInterface
{
name = 'AddPayloadToCommandMenuItem1775129635528';
public async up(queryRunner: QueryRunner): Promise<void> {
await queryRunner.query(
`ALTER TABLE "core"."commandMenuItem" ADD "payload" jsonb`,
);
const savepointName =
'sp_add_payload_check_constraint_to_command_menu_item';
try {
await queryRunner.query(`SAVEPOINT ${savepointName}`);
await addPayloadCheckConstraintToCommandMenuItem(queryRunner);
await queryRunner.query(`RELEASE SAVEPOINT ${savepointName}`);
} catch (e) {
try {
await queryRunner.query(`ROLLBACK TO SAVEPOINT ${savepointName}`);
await queryRunner.query(`RELEASE SAVEPOINT ${savepointName}`);
} catch (rollbackError) {
// oxlint-disable-next-line no-console
console.error(
'Failed to rollback to savepoint in AddPayloadToCommandMenuItem1775129635528',
rollbackError,
);
throw rollbackError;
}
// oxlint-disable-next-line no-console
console.error(
'Swallowing AddPayloadToCommandMenuItem1775129635528 error',
e,
);
}
}
```
It was afterwards re-applied within an workspace commands, it was hacky
and missleading for the self host having false positive in logs
## New pattern
Generate the slow instance command
```
npx nx database:migrate:generate twenty-server -- --name add-foo-bar-columns --type slow
```
```ts
import { DataSource, QueryRunner } from 'typeorm';
import { RegisteredInstanceCommand } from 'src/engine/core-modules/upgrade/decorators/registered-instance-command.decorator';
import { SlowInstanceCommand } from 'src/engine/core-modules/upgrade/interfaces/slow-instance-command.interface';
@RegisteredInstanceCommand('1.21.0', 1775640902366, { type: 'slow' })
export class AddPrastoinColToWorkspaceSlowInstanceCommand implements SlowInstanceCommand {
async runDataMigration(dataSource: DataSource): Promise<void> {
// TODO: implement data backfill before the DDL migration
}
public async up(queryRunner: QueryRunner): Promise<void> {
await queryRunner.query('ALTER TABLE "core"."workspace" ADD "prastoin" character varying');
}
public async down(queryRunner: QueryRunner): Promise<void> {
await queryRunner.query('ALTER TABLE "core"."workspace" DROP COLUMN "prastoin"');
}
}
```
### Why `run-instance-commands` and `upgrade` remain separate commands
These two commands serve fundamentally different purposes with
incompatible scoping semantics:
The run-instance-commands iterates over all the legacy typeorm and the
instance commands of all versions, used for database init and so on. we
could be centralizing both but the readability tradeoff isn't worth it
In the future thanks to the cross-version pattern we will be able to
centralize them but not right now
Please note that by default the `run-instance-commands` only run the
fast instance commands which is expected for our cloud prod CD
# Introduction
Migrating the workspace commands to the decorator version + timestamp
listing as for the instance commands
We've now been able to remove the upgrade command abstraction where we
needed to import all modules and order them
Now they're dynamically retrieved at upgrade runtime, sorted by
timestamp
## Instance and workspace commands name
The name is computed from the command metadata `version` `className` and
`timestamp` we have a duplicate validation at module init from the
unified registry
# Introduction
As for the instance commands we want to keep a track of what has been
run for the workspace commands
Note that the history will be updated only when the workspace command
has been run through the upgrade directly and not when run atomically
## What's next
Later we will use this history in order to determine the current
workspace's version and instance's version getting rid of the version in
database, that will be the last stone
# Introduction
Now only using typeorm to generate migrations up and down statement
We handle and maintain our own migration table history
## What's new
Now all the instance commands will live within the same module and
folder than the upgrade commands
Sequentiality comes from the timestamp located in the filename
Same sequentiality also applies to the workspace commands in the future,
for the moment still expected a as code explicit declaration
( below screen is an example see below section )
<img width="1382" height="634" alt="image"
src="https://github.com/user-attachments/assets/5610a246-4eae-485e-99f4-98fb89ad5ac8"
/>
## Existing 1.21 migrations
We won't start following this pattern in 1.21 yet at least not with the
migration that has already been released as typeorm migrations in cloud
production as they would rerun
## Small duplication
Duplicating the legacy typeorm and instance commands run in the
`run-instance-commands` to avoid any merge of interest for the moment
## Concurrency
Not handling any run in parrallel of the upgrade for the moment
Weiko