## Context
A workflow run only reaches STOPPED via the in-flight worker execution:
`stopWorkflowRun` just flips a RUNNING run to `STOPPING` (records
intent), and the `STOPPING -> STOPPED` transition is done later inside
`computeWorkflowRunStatus`, which only runs while a worker is executing
the run's steps.
If no worker is executing the run at that point, nothing ever finalizes
it:
- the worker that owned the run crashed / was killed mid-step (e.g.
under heavy load), or
- a step legitimately sits in RUNNING awaiting an external event that
never arrives (the user stopped it).
Only `ENQUEUED` runs had a staleness sweep, so `STOPPING` (and
`RUNNING`) had no recovery path and would stay stuck indefinitely. This
has been observed in production (~150 runs stuck in `STOPPING` after
manual stops during a migration).
## Change
Extend the existing staled-runs machinery to also finalize runs left in
`STOPPING`:
- New `stuck-stopping-runs-threshold` (1h) +
`getStuckStoppingRunsFindOptions` matching `status = STOPPING AND
updatedAt < now - 1h`. `updatedAt` is a TypeORM update-date column, so
it reliably marks when the run entered `STOPPING`, and 1h stays above
any legitimate in-flight step.
- `handleStuckStoppingRunsForWorkspace` finalizes each match to
`STOPPED` via `endWorkflowRun`, so `endedAt`, step infos and the
`WorkflowRunStopped` metric stay consistent. It pages the backlog with
keyset pagination on `(createdAt, id)`, so a page whose finalizations
all fail can't stay at the front of the query and starve later runs
(failed ones are retried on the next sweep).
- Wired into the same cron (`WorkflowHandleStaledRunsCronJob`, every 10
min), per-workspace job, and the manual `workflow:handle-staled-runs`
command — so ops can also clear an existing backlog immediately. The
staled-ENQUEUED and stuck-STOPPING handlers run independently
(`Promise.allSettled` in the job, separate try/catch in the command), so
a failure in one doesn't block the other.
Stop remains manual and unchanged; this only guarantees a stopped run
eventually reaches `STOPPED`.
## Notes / scope
- No schema change (reuses `updatedAt`), so no migration.
- `RUNNING` runs orphaned by a worker crash have the same missing-net
problem; left out of scope here (this covers the user-triggered STOPPING
case).
- The new detection query scans `status`/`updatedAt` like the existing
ENQUEUED sweep; at very high `workflowRun` volumes an index on `(status,
updatedAt)` would help — same pre-existing consideration as the ENQUEUED
path.
## Tests
Unit tests for `handleStuckStoppingRunsForWorkspace`: no-op when none,
finalizes each match to STOPPED, pages through a multi-page backlog, and
advances past a fully-failed page instead of starving later runs. Plus a
unit test for the `(createdAt, id)` keyset condition in
`getStuckStoppingRunsFindOptions`. Full suite green, lint + typecheck
clean on changed files.
Manually verified on a real instance (Postgres): seeded a `STOPPING` run
aged 2h and ran `workflow:handle-staled-runs` -> transitioned to
`STOPPED` with `endedAt` set; a freshly-`STOPPING` run (updatedAt now)
was correctly left untouched by the 1h threshold.
<!-- This is an auto-generated description by cubic. -->
<a
href="https://cubic.dev/pr/twentyhq/twenty/pull/22900?utm_source=github"
target="_blank" rel="noopener noreferrer"
data-no-image-dialog="true"><picture><source
media="(prefers-color-scheme: dark)"
srcset="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"><source
media="(prefers-color-scheme: light)"
srcset="https://www.cubic.dev/buttons/review-in-cubic-light.svg"><img
alt="Review in cubic"
src="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"></picture></a>
<!-- End of auto-generated description by cubic. -->
Thomas Trompette
f4b2968a74
