# Fix broken 2-19 upgrade command sequence (dev incident:
`lastStreamError` / `workspaceDiscoverability`)
## Incident
The dev environment (tracking main) throws:
- `Property "lastStreamError" was not found in "AgentChatThreadEntity"`
- `Cannot return null for non-nullable field
Workspace.workspaceDiscoverability`
## Root cause
The 2-19 upgrade commands were committed with **fabricated future
timestamps** (year-2027 epochs like `1820000000000`). The upgrade cursor
(`upgrade-aware-entity-metadata.adapter.ts`) tracks a **single**
most-recent applied step: it looks up the latest
`core."upgradeMigration"` row's name in the sequence (sorted by
timestamp within kind) and hides every `@WasIntroducedInUpgrade` column
at or past that index.
Two ways this breaks, and both were live on main:
1. **Cursor regression**: a command merged *later* with a *smaller*
timestamp (e.g. `pendingQuestion` at `1811…` after `metadata-overrides`
at `1820…` had run) sorts *before* already-applied steps. When migrate
runs, the "latest" row now points earlier in the sequence, re-hiding
columns that were already applied.
2. **Migrate not running at all** (Felix's hypothesis): if the deploy
pipeline skipped `database:migrate:prod`, none of the 2-19 rows exist
and every 2-19-gated column is hidden.
Both hypotheses have the same fix path; the discriminating query is in
the verification section below.
## Fix
**Real timestamps** (per maintainer direction — no more fabricated
epochs):
| Command | Old (fabricated) | New (real merge epoch) | Introduced by |
|---|---|---|---|
| workspace: backfill-workspace-custom-application-registration |
`1820000000000` | `1782853718000` | #22378 |
| fast: add-metadata-overrides-column | `1820000100000` |
`1782986475000` | #22417 |
| slow: backfill-metadata-overrides | `1820000110000` | `1782986476000`
(+1s to order after its fast pair) | #22417 |
| fast: add-last-stream-error-to-agent-chat-thread | `1821000000000` |
`1782996657000` | #22434 |
| fast: add-pending-question-to-agent-chat-thread | `1811000000000` |
`1782999138000` | #22346 |
| fast: add-workspace-discoverability-to-workspace | `1820000001000` |
`1783004140000` | #22423 |
Each value is the committer epoch of the squash-merge commit that
introduced the command on main (verified via `git log --diff-filter=A`).
Sorted by real time, the fast sequence is strictly increasing, so the
cursor can no longer regress.
**Idempotency**: renaming a command changes its step name, so every one
of these re-runs on any instance that already applied it under the old
name (dev cluster, edge self-hosters — 2.19 is unreleased, so tagged
releases are unaffected). All six are now safe to re-run:
- `lastStreamError`, `pendingQuestion`, `metadata-overrides` fast: `ADD
COLUMN IF NOT EXISTS` (already were)
- `metadata-overrides` slow backfill: `WHERE … IS NULL` guard (already
was)
- workspace command: skips when `applicationRegistrationId` is already
set (already did)
- `workspaceDiscoverability`: **made idempotent in this PR** — `CREATE
TYPE` wrapped in a `duplicate_object` handler, `ADD COLUMN IF NOT
EXISTS`
**CI guard** (replaces the append-only check added earlier on this
branch):
- Timestamps must be **real**: within `[now − 60 days, now + 2 days]`.
This is the check that would have prevented the original sin — 2027
epochs can never pass.
- Still **append-only** within the version directory, but computed from
`git diff --name-status --find-renames` so renamed/copied files are
checked too (cubic's P2), and files the PR deletes/renames away no
longer count toward the existing max (otherwise a re-slotting PR like
this one could never pass its own guard).
- Covers `workspace-command-<ts>-` filenames, not just
`instance-command-fast|slow-<ts>-`; skips `.spec.ts` files.
- Failure message documents the escape path (re-slot the fabricated
blocker to its real epoch + make it idempotent) and a bypass label
`ci:allow-upgrade-command-timestamp-exception` for deliberate
exceptions.
## Deploy sequencing (important)
After this merges and deploys, `database:migrate:prod` **must run**
before the API pods are relied on: the old step names no longer exist in
the sequence, so until the renamed commands run once, the cursor
resolves to 0 and *every* gated column is hidden. The commands are
idempotent, so the re-run is harmless. Running API/worker pods only
compute the cursor at boot — restart them after migrate.
## Verification / diagnosis on dev
```sql
SELECT name, status, "createdAt"
FROM core."upgradeMigration"
WHERE "workspaceId" IS NULL
ORDER BY "createdAt" DESC
LIMIT 15;
```
- Latest rows named `…_182xxxxxxxxxx` (fabricated) and completed →
migrate ran, cursor regressed (hypothesis 1).
- No 2-19 rows at all → migrate never ran for 2-19 (hypothesis 2).
- After the fix: latest row should be
`2.19.0_AddWorkspaceDiscoverabilityToWorkspaceFastInstanceCommand_1783004140000`,
status `completed`.
https://claude.ai/code/session_01Lyi6zTema2FMVVh8MD6c38
Félix Malfait
024a9b4d94