Files
twenty/packages
martmull a99e62fbca Add application job enqueue limits guard on all message queue drivers (#23570)
## Context

Applications trigger logic functions from several paths (cron, database
events, HTTP routes, install/connect hooks). Without a cap, a single app
can flood the `logic-function-queue` and starve it. This adds enqueue
limits scoped to that queue, mirroring the existing per-application API
rate limiting.

## What changed

`JobEnqueueThrottlerGuard` reuses the `ThrottlerService` token bucket
(same primitive as the API rate limiter) with two tiers:

- **Per application installation**
(`enqueue:throttler:application:{applicationId}`) - lower ceiling,
`APPLICATION_JOB_ENQUEUE_RATE_LIMITING_LIMIT` (default 500).
- **Per application registration**
(`enqueue:throttler:application-registration:{applicationRegistrationId}`)
- higher ceiling shared across all workspaces that installed the same
app, `APPLICATION_REGISTRATION_JOB_ENQUEUE_RATE_LIMITING_LIMIT` (default
2000).

Both share `APPLICATION_JOB_ENQUEUE_RATE_LIMITING_TTL_IN_MS` (default
60s). Both buckets are checked before either is debited, so a rejection
on one tier never burns quota on the other.

**Per-queue guarding.** A `ThrottledMessageQueueDriver` decorator wraps
the concrete driver (BullMQ or Sync) in the `QUEUE_DRIVER` provider and
passes the `queueName` to the guard. The guard only acts on queues in
`GUARDED_ENQUEUE_QUEUES` (currently just `logic-function-queue`); every
other queue is untouched.

**Required application context.** The guard reads a dedicated
`applicationJobEnqueueContextStorage` (AsyncLocalStorage) carrying `{
applicationId, applicationRegistrationId }`, and throws if a
guarded-queue enqueue runs without both. Every logic-function-queue
enqueue site wraps its `add`/`bulkAdd` in
`withApplicationJobEnqueueContext`:

- cron trigger
- database-event trigger (groups logic functions by application, one
batch per application)
- server route trigger
- application post-install hook
- connection-provider on-connect hook

When the limit is reached the guard records a
`JobEnqueueApplicationRateLimited` metric and throws
`ThrottlerException` (mapped to 429 by the existing handlers).

## Files

- `message-queue/guards/job-enqueue-throttler.guard.ts` - the guard
(new)
- `message-queue/storage/application-job-enqueue-context.storage.ts` -
dedicated enqueue context (new)
- `message-queue/constants/guarded-enqueue-queues.constant.ts` -
guarded-queue set (new)
- `message-queue/drivers/throttled-message-queue.driver.ts` - decorator
driver wrapping any driver (new)
- `message-queue/message-queue-core.module.ts` - wires the guard into
the driver provider
- `twenty-config/config-variables.ts` - three tunable `RATE_LIMITING`
config variables
- `metrics/types/metrics-keys.type.ts` -
`JobEnqueueApplicationRateLimited` key
- the 5 enqueue sites above - inject the enqueue context

## Notes / trade-offs

- A logic function whose application has no `applicationRegistrationId`
is skipped at the trigger paths (the install hook throws), matching how
the server route trigger already treats "not linked to a registration".
- `addCron` is left ungated (idempotent upsert).
- Default limits are placeholders and tunable per instance.

## Testing

- `JobEnqueueThrottlerGuard` unit tests (7 cases): non-guarded queue
skip, throw on missing/partial context, two-tier throttling with
distinct limits, per-item token consumption on bulk, no partial debit
when either tier is exhausted.
- Updated `connection-provider-oauth-flow.service.spec.ts` for the new
cache key.
- `npx nx typecheck twenty-server` passes; oxlint + oxfmt clean on
changed files.
2026-07-30 16:40:22 +00:00
..
2026-07-30 17:17:05 +02:00