fix(server): pace lambda control-plane calls to avoid 'Rate exceeded' on release (#22569)

## Problem

Logic functions intermittently fail with:

```
Lambda invocation failed for function '<id>' during build: Rate exceeded
```

`Rate exceeded` is AWS Lambda's control-plane throttling
(`TooManyRequestsException`), thrown during the **build** phase — before
invoke — inside `buildExecutor`.

### Why it spikes on release
A build is skipped (`canSkip = true`, zero control-plane calls) unless
the executor is missing/inactive **or**
`flatApplication.isSdkLayerStale` is true. `isSdkLayerStale` is flipped
to `true` for the **whole application at once** whenever the SDK client
regenerates (app install / development / schema change).

So on release, every logic function in the app goes stale simultaneously
→ each enters `ensureExecutor` in its own per-function lock → a burst of
`Create`/`Update`/`PublishLayer`/`GetFunction` calls across many
functions at once → the low, account-region-wide control-plane quota is
exceeded → `Rate exceeded`. Between releases everything is warm and no
control-plane calls happen — hence "spikes on release, silent
otherwise".

The Lambda client was created with no retry override, so it used the SDK
default (`standard` mode, `maxAttempts = 3`): a few retries with
backoff, but no client-side pacing.

## Change

Configure the shared Lambda client with:
- `retryMode: 'adaptive'` — adds a client-side token-bucket rate limiter
that slows outgoing requests when it sees throttling, instead of
fire-then-backoff.
- `maxAttempts: 8` — rides out the burst.

Applied after the options spread so it always takes effect, and covers
**every** control-plane call including the
`waitUntilFunctionActive/UpdatedV2` pollers (same client).

## Scope / follow-up

This is the cheap, high-leverage mitigation and dampens the burst per
process. It does **not** add a cross-function/cross-pod concurrency cap,
so a large enough release across multiple replicas could still exceed
the account quota. A follow-up could add a limiter (in-process
semaphore, or a distributed token bucket via the existing Redis
cache-lock) around `ensureExecutor`.

## Testing

- `tsc --noEmit` on twenty-server: clean.
- Not runtime-tested — AWS control-plane throttling can't be reproduced
locally. Worth confirming against a real release-time CloudWatch window
after deploy.

<!-- This is an auto-generated description by cubic. -->
<a
href="https://cubic.dev/pr/twentyhq/twenty/pull/22569?utm_source=github"
target="_blank" rel="noopener noreferrer"
data-no-image-dialog="true"><picture><source
media="(prefers-color-scheme: dark)"
srcset="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"><source
media="(prefers-color-scheme: light)"
srcset="https://www.cubic.dev/buttons/review-in-cubic-light.svg"><img
alt="Review in cubic"
src="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"></picture></a>
<!-- End of auto-generated description by cubic. -->
This commit is contained in:
Thomas Trompette
2026-07-06 13:44:33 +02:00
committed by GitHub
parent 0baf213fa4
commit fb0a54c73a
2 changed files with 7 additions and 0 deletions
@@ -5,6 +5,9 @@ import { ASSET_PATH } from 'src/constants/assets-path';
export const UPDATE_FUNCTION_DURATION_TIMEOUT_IN_SECONDS = 60;
export const CREDENTIALS_DURATION_IN_SECONDS = 60 * 60; // 1h
export const LAMBDA_CLIENT_MAX_ATTEMPTS = 8;
export const LAMBDA_CLIENT_RETRY_MODE = 'adaptive' as const;
export const YARN_INSTALL_LAMBDA_TIMEOUT_SECONDS = 300;
export const YARN_INSTALL_LAMBDA_MEMORY_MB = 1024;
export const BUILDER_LAMBDA_TIMEOUT_SECONDS = 60;
@@ -11,6 +11,8 @@ import { isDefined } from 'twenty-shared/utils';
import {
CREDENTIALS_DURATION_IN_SECONDS,
LAMBDA_CLIENT_MAX_ATTEMPTS,
LAMBDA_CLIENT_RETRY_MODE,
UPDATE_FUNCTION_DURATION_TIMEOUT_IN_SECONDS,
} from 'src/engine/core-modules/logic-function/logic-function-drivers/drivers/lambda/constants/lambda-driver.constant';
import { type LambdaDriverOptions } from 'src/engine/core-modules/logic-function/logic-function-drivers/drivers/lambda/types/lambda-driver.type';
@@ -35,6 +37,8 @@ export class LambdaAwsClientService {
...(isDefined(this.options.subhostingRole) && {
credentials: await this.getAssumeRoleCredentials(),
}),
maxAttempts: LAMBDA_CLIENT_MAX_ATTEMPTS,
retryMode: LAMBDA_CLIENT_RETRY_MODE,
});
}