fb0a54c73a
## Problem Logic functions intermittently fail with: ``` Lambda invocation failed for function '<id>' during build: Rate exceeded ``` `Rate exceeded` is AWS Lambda's control-plane throttling (`TooManyRequestsException`), thrown during the **build** phase — before invoke — inside `buildExecutor`. ### Why it spikes on release A build is skipped (`canSkip = true`, zero control-plane calls) unless the executor is missing/inactive **or** `flatApplication.isSdkLayerStale` is true. `isSdkLayerStale` is flipped to `true` for the **whole application at once** whenever the SDK client regenerates (app install / development / schema change). So on release, every logic function in the app goes stale simultaneously → each enters `ensureExecutor` in its own per-function lock → a burst of `Create`/`Update`/`PublishLayer`/`GetFunction` calls across many functions at once → the low, account-region-wide control-plane quota is exceeded → `Rate exceeded`. Between releases everything is warm and no control-plane calls happen — hence "spikes on release, silent otherwise". The Lambda client was created with no retry override, so it used the SDK default (`standard` mode, `maxAttempts = 3`): a few retries with backoff, but no client-side pacing. ## Change Configure the shared Lambda client with: - `retryMode: 'adaptive'` — adds a client-side token-bucket rate limiter that slows outgoing requests when it sees throttling, instead of fire-then-backoff. - `maxAttempts: 8` — rides out the burst. Applied after the options spread so it always takes effect, and covers **every** control-plane call including the `waitUntilFunctionActive/UpdatedV2` pollers (same client). ## Scope / follow-up This is the cheap, high-leverage mitigation and dampens the burst per process. It does **not** add a cross-function/cross-pod concurrency cap, so a large enough release across multiple replicas could still exceed the account quota. A follow-up could add a limiter (in-process semaphore, or a distributed token bucket via the existing Redis cache-lock) around `ensureExecutor`. ## Testing - `tsc --noEmit` on twenty-server: clean. - Not runtime-tested — AWS control-plane throttling can't be reproduced locally. Worth confirming against a real release-time CloudWatch window after deploy. <!-- This is an auto-generated description by cubic. --> <a href="https://cubic.dev/pr/twentyhq/twenty/pull/22569?utm_source=github" target="_blank" rel="noopener noreferrer" data-no-image-dialog="true"><picture><source media="(prefers-color-scheme: dark)" srcset="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"><source media="(prefers-color-scheme: light)" srcset="https://www.cubic.dev/buttons/review-in-cubic-light.svg"><img alt="Review in cubic" src="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"></picture></a> <!-- End of auto-generated description by cubic. -->