fix(server): pace lambda control-plane calls to avoid 'Rate exceeded' on release (#22569)
## Problem Logic functions intermittently fail with: ``` Lambda invocation failed for function '<id>' during build: Rate exceeded ``` `Rate exceeded` is AWS Lambda's control-plane throttling (`TooManyRequestsException`), thrown during the **build** phase — before invoke — inside `buildExecutor`. ### Why it spikes on release A build is skipped (`canSkip = true`, zero control-plane calls) unless the executor is missing/inactive **or** `flatApplication.isSdkLayerStale` is true. `isSdkLayerStale` is flipped to `true` for the **whole application at once** whenever the SDK client regenerates (app install / development / schema change). So on release, every logic function in the app goes stale simultaneously → each enters `ensureExecutor` in its own per-function lock → a burst of `Create`/`Update`/`PublishLayer`/`GetFunction` calls across many functions at once → the low, account-region-wide control-plane quota is exceeded → `Rate exceeded`. Between releases everything is warm and no control-plane calls happen — hence "spikes on release, silent otherwise". The Lambda client was created with no retry override, so it used the SDK default (`standard` mode, `maxAttempts = 3`): a few retries with backoff, but no client-side pacing. ## Change Configure the shared Lambda client with: - `retryMode: 'adaptive'` — adds a client-side token-bucket rate limiter that slows outgoing requests when it sees throttling, instead of fire-then-backoff. - `maxAttempts: 8` — rides out the burst. Applied after the options spread so it always takes effect, and covers **every** control-plane call including the `waitUntilFunctionActive/UpdatedV2` pollers (same client). ## Scope / follow-up This is the cheap, high-leverage mitigation and dampens the burst per process. It does **not** add a cross-function/cross-pod concurrency cap, so a large enough release across multiple replicas could still exceed the account quota. A follow-up could add a limiter (in-process semaphore, or a distributed token bucket via the existing Redis cache-lock) around `ensureExecutor`. ## Testing - `tsc --noEmit` on twenty-server: clean. - Not runtime-tested — AWS control-plane throttling can't be reproduced locally. Worth confirming against a real release-time CloudWatch window after deploy. <!-- This is an auto-generated description by cubic. --> <a href="https://cubic.dev/pr/twentyhq/twenty/pull/22569?utm_source=github" target="_blank" rel="noopener noreferrer" data-no-image-dialog="true"><picture><source media="(prefers-color-scheme: dark)" srcset="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"><source media="(prefers-color-scheme: light)" srcset="https://www.cubic.dev/buttons/review-in-cubic-light.svg"><img alt="Review in cubic" src="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"></picture></a> <!-- End of auto-generated description by cubic. -->
This commit is contained in:
+3
@@ -5,6 +5,9 @@ import { ASSET_PATH } from 'src/constants/assets-path';
|
||||
export const UPDATE_FUNCTION_DURATION_TIMEOUT_IN_SECONDS = 60;
|
||||
export const CREDENTIALS_DURATION_IN_SECONDS = 60 * 60; // 1h
|
||||
|
||||
export const LAMBDA_CLIENT_MAX_ATTEMPTS = 8;
|
||||
export const LAMBDA_CLIENT_RETRY_MODE = 'adaptive' as const;
|
||||
|
||||
export const YARN_INSTALL_LAMBDA_TIMEOUT_SECONDS = 300;
|
||||
export const YARN_INSTALL_LAMBDA_MEMORY_MB = 1024;
|
||||
export const BUILDER_LAMBDA_TIMEOUT_SECONDS = 60;
|
||||
|
||||
+4
@@ -11,6 +11,8 @@ import { isDefined } from 'twenty-shared/utils';
|
||||
|
||||
import {
|
||||
CREDENTIALS_DURATION_IN_SECONDS,
|
||||
LAMBDA_CLIENT_MAX_ATTEMPTS,
|
||||
LAMBDA_CLIENT_RETRY_MODE,
|
||||
UPDATE_FUNCTION_DURATION_TIMEOUT_IN_SECONDS,
|
||||
} from 'src/engine/core-modules/logic-function/logic-function-drivers/drivers/lambda/constants/lambda-driver.constant';
|
||||
import { type LambdaDriverOptions } from 'src/engine/core-modules/logic-function/logic-function-drivers/drivers/lambda/types/lambda-driver.type';
|
||||
@@ -35,6 +37,8 @@ export class LambdaAwsClientService {
|
||||
...(isDefined(this.options.subhostingRole) && {
|
||||
credentials: await this.getAssumeRoleCredentials(),
|
||||
}),
|
||||
maxAttempts: LAMBDA_CLIENT_MAX_ATTEMPTS,
|
||||
retryMode: LAMBDA_CLIENT_RETRY_MODE,
|
||||
});
|
||||
}
|
||||
|
||||
|
||||
Reference in New Issue
Block a user