- if >5000 workflows per hour, new ones should failed
- if >100 workflow per min, new ones should be set as not started.
Except manual trigger
- when enqueued, we check if there a not started workflows that may be
queued. If yes, we call the associated job
---------
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
- new status `ENQUEUED` added. With a command to backfill
- counter in cache per workspace, managed by a new service
[workflow-run-queue.workspace-service.ts](https://github.com/twentyhq/twenty/compare/tt-improve-workflow-run-queueing?expand=1#diff-1e2de2a48cd482a3bd7e8dedf1150a19d0b200afbd9282181a24ecddddb56927)
- cron added that will run every minute to look for not started
workflows
Here is the new flow:
- When executing a workflow, we check if the queue is not full. If not,
run is created as `ENQUEUED` and the run workflow job is triggered as
usual. If full, create the run as NOT_STARTED and do not trigger the job
- Cron will look for NOT_STARTED workflows and queue some if there is
some place again
- Only MANUAL and Form submit skip the queue limit
## Context
Added to the existing useGraphQLErrorHandlerHook yoga hook to increment
metrics after all query executions based on their error codes. I
originally wanted to create a new useMetrics hook but most of the error
handling was done in useGraphQLErrorHandlerHook so we decided to keep it
there for now.
<img width="1310" alt="Screenshot 2025-06-24 at 15 58 26"
src="https://github.com/user-attachments/assets/498d3754-851a-4051-a5c2-23ac8253aa6a"
/>
Done :
- move metrics and health cache services from health module to metrics
module
- refactor metrics counter from specific method to set up from enum keys
- add OpenTelemetry (Otel) instrumentation for metrics
- set up Otel SDK to send metrics to Otel collector
To do later :
- implement Otel instrumentation for traces + plug Sentry on top
Charles Bochet