Thomas Trompette
f4b2968a74
## Context A workflow run only reaches STOPPED via the in-flight worker execution: `stopWorkflowRun` just flips a RUNNING run to `STOPPING` (records intent), and the `STOPPING -> STOPPED` transition is done later inside `computeWorkflowRunStatus`, which only runs while a worker is executing the run's steps. If no worker is executing the run at that point, nothing ever finalizes it: - the worker that owned the run crashed / was killed mid-step (e.g. under heavy load), or - a step legitimately sits in RUNNING awaiting an external event that never arrives (the user stopped it). Only `ENQUEUED` runs had a staleness sweep, so `STOPPING` (and `RUNNING`) had no recovery path and would stay stuck indefinitely. This has been observed in production (~150 runs stuck in `STOPPING` after manual stops during a migration). ## Change Extend the existing staled-runs machinery to also finalize runs left in `STOPPING`: - New `stuck-stopping-runs-threshold` (1h) + `getStuckStoppingRunsFindOptions` matching `status = STOPPING AND updatedAt < now - 1h`. `updatedAt` is a TypeORM update-date column, so it reliably marks when the run entered `STOPPING`, and 1h stays above any legitimate in-flight step. - `handleStuckStoppingRunsForWorkspace` finalizes each match to `STOPPED` via `endWorkflowRun`, so `endedAt`, step infos and the `WorkflowRunStopped` metric stay consistent. It pages the backlog with keyset pagination on `(createdAt, id)`, so a page whose finalizations all fail can't stay at the front of the query and starve later runs (failed ones are retried on the next sweep). - Wired into the same cron (`WorkflowHandleStaledRunsCronJob`, every 10 min), per-workspace job, and the manual `workflow:handle-staled-runs` command — so ops can also clear an existing backlog immediately. The staled-ENQUEUED and stuck-STOPPING handlers run independently (`Promise.allSettled` in the job, separate try/catch in the command), so a failure in one doesn't block the other. Stop remains manual and unchanged; this only guarantees a stopped run eventually reaches `STOPPED`. ## Notes / scope - No schema change (reuses `updatedAt`), so no migration. - `RUNNING` runs orphaned by a worker crash have the same missing-net problem; left out of scope here (this covers the user-triggered STOPPING case). - The new detection query scans `status`/`updatedAt` like the existing ENQUEUED sweep; at very high `workflowRun` volumes an index on `(status, updatedAt)` would help — same pre-existing consideration as the ENQUEUED path. ## Tests Unit tests for `handleStuckStoppingRunsForWorkspace`: no-op when none, finalizes each match to STOPPED, pages through a multi-page backlog, and advances past a fully-failed page instead of starving later runs. Plus a unit test for the `(createdAt, id)` keyset condition in `getStuckStoppingRunsFindOptions`. Full suite green, lint + typecheck clean on changed files. Manually verified on a real instance (Postgres): seeded a `STOPPING` run aged 2h and ran `workflow:handle-staled-runs` -> transitioned to `STOPPED` with `endedAt` set; a freshly-`STOPPING` run (updatedAt now) was correctly left untouched by the 1h threshold. <!-- This is an auto-generated description by cubic. --> <a href="https://cubic.dev/pr/twentyhq/twenty/pull/22900?utm_source=github" target="_blank" rel="noopener noreferrer" data-no-image-dialog="true"><picture><source media="(prefers-color-scheme: dark)" srcset="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"><source media="(prefers-color-scheme: light)" srcset="https://www.cubic.dev/buttons/review-in-cubic-light.svg"><img alt="Review in cubic" src="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"></picture></a> <!-- End of auto-generated description by cubic. -->
The #1 Open-Source CRM
Website · 
Documentation · 
Roadmap · 
Discord · 
Figma
Why Twenty
Twenty gives technical teams the building blocks for a custom CRM that meets complex business needs and quickly adapts as the business evolves. Twenty is the CRM you build, ship, and version like the rest of your stack.
Learn more about why we built Twenty
Installation
Cloud
The fastest way to get started. Sign up at twenty.com and spin up a workspace in under a minute, with no infrastructure to manage and always up to date.
Build an app
Scaffold a new app with the Twenty CLI:
npx create-twenty-app my-app
Define objects, fields, and views as code:
import { defineObject, FieldType } from 'twenty-sdk/define';
export default defineObject({
nameSingular: 'deal',
namePlural: 'deals',
labelSingular: 'Deal',
labelPlural: 'Deals',
fields: [
{ name: 'name', label: 'Name', type: FieldType.TEXT },
{ name: 'amount', label: 'Amount', type: FieldType.CURRENCY },
{ name: 'closeDate', label: 'Close Date', type: FieldType.DATE_TIME },
],
});
Then ship it to your workspace:
npx twenty app:publish --private
See the app development guide for objects, views, agents, and logic functions.
Self-hosting
Run Twenty on your own infrastructure with Docker Compose, or contribute locally via the local setup guide.
Everything you need
Twenty gives you the building blocks of a modern CRM (objects, views, workflows, and agents) and lets you extend them as code. Here's a tour of what's in the box.
Want to go deeper? Read the 
User Guide for product walkthroughs, or the Documentation for developer reference.
|
|
|
|
|
|
Learn more about version control in doc
Stack
TypeScript
Nx
NestJS, with BullMQ, 
PostgreSQL, 
Redis
React, with Jotai, Linaria and Lingui
Thanks
Thanks to these amazing services that we use and recommend for code review (Greptile), catching bugs (Sentry) and translating (Crowdin).
Join the Community
Star the repo · Discord · 
Feature requests · Releases · 
X · 
LinkedIn · 
Crowdin · 
Contribute