Files
twenty/packages/twenty-server
Thomas Trompette fb0a54c73a fix(server): pace lambda control-plane calls to avoid 'Rate exceeded' on release (#22569)
## Problem

Logic functions intermittently fail with:

```
Lambda invocation failed for function '<id>' during build: Rate exceeded
```

`Rate exceeded` is AWS Lambda's control-plane throttling
(`TooManyRequestsException`), thrown during the **build** phase — before
invoke — inside `buildExecutor`.

### Why it spikes on release
A build is skipped (`canSkip = true`, zero control-plane calls) unless
the executor is missing/inactive **or**
`flatApplication.isSdkLayerStale` is true. `isSdkLayerStale` is flipped
to `true` for the **whole application at once** whenever the SDK client
regenerates (app install / development / schema change).

So on release, every logic function in the app goes stale simultaneously
→ each enters `ensureExecutor` in its own per-function lock → a burst of
`Create`/`Update`/`PublishLayer`/`GetFunction` calls across many
functions at once → the low, account-region-wide control-plane quota is
exceeded → `Rate exceeded`. Between releases everything is warm and no
control-plane calls happen — hence "spikes on release, silent
otherwise".

The Lambda client was created with no retry override, so it used the SDK
default (`standard` mode, `maxAttempts = 3`): a few retries with
backoff, but no client-side pacing.

## Change

Configure the shared Lambda client with:
- `retryMode: 'adaptive'` — adds a client-side token-bucket rate limiter
that slows outgoing requests when it sees throttling, instead of
fire-then-backoff.
- `maxAttempts: 8` — rides out the burst.

Applied after the options spread so it always takes effect, and covers
**every** control-plane call including the
`waitUntilFunctionActive/UpdatedV2` pollers (same client).

## Scope / follow-up

This is the cheap, high-leverage mitigation and dampens the burst per
process. It does **not** add a cross-function/cross-pod concurrency cap,
so a large enough release across multiple replicas could still exceed
the account quota. A follow-up could add a limiter (in-process
semaphore, or a distributed token bucket via the existing Redis
cache-lock) around `ensureExecutor`.

## Testing

- `tsc --noEmit` on twenty-server: clean.
- Not runtime-tested — AWS control-plane throttling can't be reproduced
locally. Worth confirming against a real release-time CloudWatch window
after deploy.

<!-- This is an auto-generated description by cubic. -->
<a
href="https://cubic.dev/pr/twentyhq/twenty/pull/22569?utm_source=github"
target="_blank" rel="noopener noreferrer"
data-no-image-dialog="true"><picture><source
media="(prefers-color-scheme: dark)"
srcset="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"><source
media="(prefers-color-scheme: light)"
srcset="https://www.cubic.dev/buttons/review-in-cubic-light.svg"><img
alt="Review in cubic"
src="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"></picture></a>
<!-- End of auto-generated description by cubic. -->
2026-07-06 13:44:33 +02:00
..
2026-04-02 11:20:08 +00:00