## Context
A workflow run can get permanently stuck in RUNNING when the queue job
executing a step dies without failing (worker crash/restart, lost BullMQ
job). The step stays `RUNNING` in the persisted state, nothing ever
recomputes the run status, and the run never terminates. If a user
clicks Stop, it wedges in STOPPING instead (the stuck-STOPPING sweeper
from #22900 then catches it after 1h, but only because of the manual
stop). Self-hosters also have no way to tell that a job was lost, or
when.
## What this PR does
### Detect runs stuck in RUNNING (monitoring only, no finalization yet)
New `handleStuckRunningRunsForWorkspace` in the staled-runs sweeper
(same cron/job/CLI wiring as the stuck-STOPPING recovery):
- Targets RUNNING runs with `updatedAt` older than 1h (`updatedAt`
refreshes on every step-info write, so staleness means zero progress).
- Skips any run that still has a job in the queue: new `getInFlightJobs`
on the message queue driver
(active/waiting/waiting-children/paused/prioritized/delayed), matched by
run-id-prefixed job id with a `job.data.workflowRunId` fallback for jobs
enqueued before this deploys.
- A truly orphaned run (orphaned RUNNING step, lost between two steps,
failed branch, or finished-but-never-finalized) is **flagged, not
finalized**: warn log + `WorkflowRunStuckRunningDetected` metric + entry
in a per-workspace cache.
- On every subsequent sweep, flagged runs are re-checked. One that ended
or got a new queue job on its own is recorded as
`WorkflowRunStuckRunningFalsePositive` (warn log with the status it
reached) and unflagged. The cron keeps sweeping a workspace as long as
it has flagged runs.
This validates the detection before it is allowed to act: if flagged
runs never resolve on their own (no false positives) while `Detected`
counts real incidents, a follow-up PR can turn the flag into an actual
finalization (fail with a clear "job lost" error so Retry works). Runs
waiting on PENDING steps (delay, form) are never flagged.
### Make queue jobs traceable to their run
All RunWorkflowJob dispatches now set the job id prefix to the workflow
run id, so BullMQ job ids become `<workflowRunId>-<uuid>`. Worker logs
(`Processing job <id>` / `processed`) and Redis job keys are now
greppable by run id. A new opt-in `allowDuplicatedPrefixes` queue option
bypasses the one-waiting-job-per-id dedup (which would otherwise drop
parallel-branch continuations); existing `id` users keep dedup by
default.
### Observability
- `stalled` worker event listener: warn log + new `JobStalled` metric —
emitted when BullMQ detects a job whose worker stopped renewing its lock
(i.e. died mid-job).
- `WorkflowRunStuckRunningDetected` /
`WorkflowRunStuckRunningFalsePositive` metrics as described above.
## Out of scope (follow-ups)
- Actually finalizing flagged runs once monitoring shows no false
positives.
- Recovering lost *delayed* resume jobs (PENDING delay step whose
scheduled job vanished).
- Persisting job ids on the run entity — unnecessary given derived ids.
## Testing
- 11 unit tests for the monitoring sweeper (flagging, id-prefix + data
fallback in-flight guards, pending skip, failed-branch precedence,
false-positive tracking, still-stuck retention, error isolation,
never-finalizes) plus find-options specs; 613 tests pass across workflow
and message-queue modules.
- Not covered: end-to-end kill-the-worker scenario against a real queue.
Thomas Trompette
98c71d7b3d