Add per-application API rate limiter (#23100)
## Context
Application-token API requests are not rate limited today:
`throttleQueryExecution` in the common query runner only throttles
API-key requests, per workspace. An application installed on hundreds of
workspaces (Call Recorder is on 700+) can therefore hit the API with
large synchronized bursts, as seen with the recovery crons impacting
production.
## What changed
- New rate limiter in `CommonBaseQueryRunnerService`, applied when the
auth context is an application context, using the existing
`ThrottlerService.tokenBucketThrottleOrThrow` like the per-workspace
API-key throttle. Both REST and GraphQL record operations go through
this path.
- The limiter key is `api:throttler:application:{universalIdentifier}`
with no workspace component: the budget is shared by every installation
of the application on the instance, which is what protects production
from install-count-proportional load. `universalIdentifier` was chosen
over `applicationRegistrationId` because the latter is null for
unpublished/local applications.
- Two new config variables in the `RATE_LIMITING` group:
`APPLICATION_API_RATE_LIMITING_LIMIT` (default 500) and
`APPLICATION_API_RATE_LIMITING_TTL_IN_MS` (default 60000), i.e. 500
requests/min per application across all workspaces.
- Rejections raise the existing `ThrottlerException`, already mapped by
both the REST and GraphQL exception handlers, and increment a new
`common-api-query/application-rate-limited` metric (only for throttler
rejections, so cache infrastructure failures are not reported as rate
limiting).
- Cron trigger dispatch `retryLimit` raised from 3 to 10 so throttled
logic function executions eventually run once the budget refills.
The existing per-workspace API-key throttle is unchanged (extracted to
its own method).
## Notes
- The limit is tunable per environment without a deploy pipeline change.
## Test
- `npx nx lint:diff-with-main twenty-server` clean.
- `npx nx typecheck twenty-server` clean.
- Throttler spec passes (5 tests).
---------
Co-authored-by: martmull <martin@twenty.com>
This commit is contained in:
+1
-1
@@ -94,7 +94,7 @@ export class CronTriggerCronJob {
|
||||
payload: {},
|
||||
},
|
||||
],
|
||||
{ retryLimit: 3 },
|
||||
{ retryLimit: 10 },
|
||||
);
|
||||
}
|
||||
} catch (error) {
|
||||
|
||||
@@ -52,6 +52,7 @@ export enum MetricsKeys {
|
||||
JobWebhookCallCompleted = 'job/webhook-call-completed',
|
||||
SignUpSuccess = 'sign-up/success',
|
||||
CommonApiQueryRateLimited = 'common-api-query/rate-limited',
|
||||
CommonApiApplicationQueryRateLimited = 'common-api-query/application-rate-limited',
|
||||
JobCompleted = 'job/completed',
|
||||
JobFailed = 'job/failed',
|
||||
JobStalled = 'job/stalled',
|
||||
|
||||
@@ -1483,6 +1483,24 @@ export class ConfigVariables {
|
||||
@CastToPositiveNumber()
|
||||
API_RATE_LIMITING_LONG_LIMIT = 100;
|
||||
|
||||
@ConfigVariablesMetadata({
|
||||
group: ConfigVariablesGroup.RATE_LIMITING,
|
||||
description:
|
||||
'Time-to-live for per-application API rate limiting in milliseconds',
|
||||
type: ConfigVariableType.NUMBER,
|
||||
})
|
||||
@CastToPositiveNumber()
|
||||
APPLICATION_API_RATE_LIMITING_TTL_IN_MS = 60_000;
|
||||
|
||||
@ConfigVariablesMetadata({
|
||||
group: ConfigVariablesGroup.RATE_LIMITING,
|
||||
description:
|
||||
'Maximum number of API requests allowed per application across all workspaces in the rate limiting window',
|
||||
type: ConfigVariableType.NUMBER,
|
||||
})
|
||||
@CastToPositiveNumber()
|
||||
APPLICATION_API_RATE_LIMITING_LIMIT = 500;
|
||||
|
||||
@CastToPositiveNumber()
|
||||
@ConfigVariablesMetadata({
|
||||
group: ConfigVariablesGroup.RATE_LIMITING,
|
||||
|
||||
Reference in New Issue
Block a user