Thomas Trompette
98c71d7b3d
## Context A workflow run can get permanently stuck in RUNNING when the queue job executing a step dies without failing (worker crash/restart, lost BullMQ job). The step stays `RUNNING` in the persisted state, nothing ever recomputes the run status, and the run never terminates. If a user clicks Stop, it wedges in STOPPING instead (the stuck-STOPPING sweeper from #22900 then catches it after 1h, but only because of the manual stop). Self-hosters also have no way to tell that a job was lost, or when. ## What this PR does ### Detect runs stuck in RUNNING (monitoring only, no finalization yet) New `handleStuckRunningRunsForWorkspace` in the staled-runs sweeper (same cron/job/CLI wiring as the stuck-STOPPING recovery): - Targets RUNNING runs with `updatedAt` older than 1h (`updatedAt` refreshes on every step-info write, so staleness means zero progress). - Skips any run that still has a job in the queue: new `getInFlightJobs` on the message queue driver (active/waiting/waiting-children/paused/prioritized/delayed), matched by run-id-prefixed job id with a `job.data.workflowRunId` fallback for jobs enqueued before this deploys. - A truly orphaned run (orphaned RUNNING step, lost between two steps, failed branch, or finished-but-never-finalized) is **flagged, not finalized**: warn log + `WorkflowRunStuckRunningDetected` metric + entry in a per-workspace cache. - On every subsequent sweep, flagged runs are re-checked. One that ended or got a new queue job on its own is recorded as `WorkflowRunStuckRunningFalsePositive` (warn log with the status it reached) and unflagged. The cron keeps sweeping a workspace as long as it has flagged runs. This validates the detection before it is allowed to act: if flagged runs never resolve on their own (no false positives) while `Detected` counts real incidents, a follow-up PR can turn the flag into an actual finalization (fail with a clear "job lost" error so Retry works). Runs waiting on PENDING steps (delay, form) are never flagged. ### Make queue jobs traceable to their run All RunWorkflowJob dispatches now set the job id prefix to the workflow run id, so BullMQ job ids become `<workflowRunId>-<uuid>`. Worker logs (`Processing job <id>` / `processed`) and Redis job keys are now greppable by run id. A new opt-in `allowDuplicatedPrefixes` queue option bypasses the one-waiting-job-per-id dedup (which would otherwise drop parallel-branch continuations); existing `id` users keep dedup by default. ### Observability - `stalled` worker event listener: warn log + new `JobStalled` metric — emitted when BullMQ detects a job whose worker stopped renewing its lock (i.e. died mid-job). - `WorkflowRunStuckRunningDetected` / `WorkflowRunStuckRunningFalsePositive` metrics as described above. ## Out of scope (follow-ups) - Actually finalizing flagged runs once monitoring shows no false positives. - Recovering lost *delayed* resume jobs (PENDING delay step whose scheduled job vanished). - Persisting job ids on the run entity — unnecessary given derived ids. ## Testing - 11 unit tests for the monitoring sweeper (flagging, id-prefix + data fallback in-flight guards, pending skip, failed-branch precedence, false-positive tracking, still-stuck retention, error isolation, never-finalizes) plus find-options specs; 613 tests pass across workflow and message-queue modules. - Not covered: end-to-end kill-the-worker scenario against a real queue.
The #1 Open-Source CRM
Website · 
Documentation · 
Roadmap · 
Discord · 
Figma
Why Twenty
Twenty gives technical teams the building blocks for a custom CRM that meets complex business needs and quickly adapts as the business evolves. Twenty is the CRM you build, ship, and version like the rest of your stack.
Learn more about why we built Twenty
Installation
Cloud
The fastest way to get started. Sign up at twenty.com and spin up a workspace in under a minute, with no infrastructure to manage and always up to date.
Build an app
Scaffold a new app with the Twenty CLI:
npx create-twenty-app my-app
Define objects, fields, and views as code:
import { defineObject, FieldType } from 'twenty-sdk/define';
export default defineObject({
nameSingular: 'deal',
namePlural: 'deals',
labelSingular: 'Deal',
labelPlural: 'Deals',
fields: [
{ name: 'name', label: 'Name', type: FieldType.TEXT },
{ name: 'amount', label: 'Amount', type: FieldType.CURRENCY },
{ name: 'closeDate', label: 'Close Date', type: FieldType.DATE_TIME },
],
});
Then ship it to your workspace:
npx twenty app:publish --private
See the app development guide for objects, views, agents, and logic functions.
Self-hosting
Run Twenty on your own infrastructure with Docker Compose, or contribute locally via the local setup guide.
Everything you need
Twenty gives you the building blocks of a modern CRM (objects, views, workflows, and agents) and lets you extend them as code. Here's a tour of what's in the box.
Want to go deeper? Read the 
User Guide for product walkthroughs, or the Documentation for developer reference.
|
|
|
|
|
|
Learn more about version control in docStack
TypeScript
Nx
NestJS, with BullMQ, 
PostgreSQL, 
Redis
React, with Jotai, Linaria and Lingui
Thanks
Thanks to these amazing services that we use and recommend for code review (Greptile), catching bugs (Sentry) and translating (Crowdin).
Join the Community
Star the repo · Discord · 
Feature requests · Releases · 
X · 
LinkedIn · 
Crowdin · 
Contribute