Give AWS SDK clients an explicit request timeout (#23857)

## Why The AWS SDK defaults `requestTimeout` to `0`, which means *no timeout* — see `DEFAULT_REQUEST_TIMEOUT` in `@smithy/node-http-handler`. None of our clients overrode it, so a request that never completes holds its socket forever. Once that happens to `maxSockets` requests (default 50), the client's connection pool is permanently exhausted and every subsequent call on that process queues indefinitely rather than failing. To the caller it is indistinguishable from a hang. This caused a production incident on 2026-08-06. A single `twenty-server` pod reached: ``` @smithy/node-http-handler:WARN - socket usage at capacity=50 and 1004 additional requests are enqueued. ``` and never recovered — the queue grew monotonically and the pod completed **zero** Lambda invocations over 12 hours while its six siblings completed dozens each. Egress was fine (a direct HTTPS call to the Lambda API returned in 53ms) and the pod was never OOM-killed or restarted, so nothing surfaced as an error anywhere. The user-visible effect was that workspace creation hung. Activation reached `synchronizeTwentyStandardApplicationOrThrow`, blocked on a Lambda call, and never returned or threw — so the `catch` in `activateWorkspace` that resets a workspace to `PENDING_CREATION` never ran, and the workspace was stranded in `ONGOING_CREATION`. Retrying didn't help because the pod was still poisoned. With one bad pod out of seven, roughly one signup in seven failed: | pod | activations started | completed | |---|---|---| | healthy × 5 | 13 | 13 | | poisoned | 3 | **0** | Nothing appeared in Sentry, because nothing ever threw. ## What this changes - **Every AWS SDK client now sets `connectionTimeout` and `requestTimeout`** via a shared `buildAwsRequestHandlerOptions()` helper. A saturated pool now surfaces as an ordinary error the caller can catch and retry instead of hanging forever. This is the fix that matters. - **The Lambda client gets a higher ceiling and a larger socket pool.** Synchronous invocations legitimately hold a socket for as long as the function runs, so its `requestTimeout` clears `EXECUTOR_LAMBDA_TIMEOUT_SECONDS` (900s) with a minute to spare, and `maxSockets` goes to 200 so long invocations cannot starve the control-plane calls (layer lookups, waiters) sharing that client. - **`S3Client` and `STSClient` in `LambdaAwsClientService` are now reused.** Both were constructed on every call and never destroyed, so each leaked its own agent and socket pool. They are invalidated alongside `lambdaClient` when assume-role credentials refresh. No new dependency: `requestHandler` already accepts a plain `NodeHttpHandlerOptions` object. ## Deliberately not in scope - **A timeout around `activateWorkspace` itself.** A hung activation still strands a workspace in `ONGOING_CREATION` until the 5-minute stale-lock reclaim, and that only fires if the user happens to retry. Worth fixing separately. - **Alerting on `socket usage at capacity`.** That warning was the only signal this was happening and nobody was watching it — an infra change rather than a code one. ## Testing - Unit tests for the helper, including that the timeout is always non-zero. - `tsc --noEmit` clean on the touched files; `oxlint` reports 0 warnings and 0 errors. - Not reproducible in a test environment — the leak needs a saturated pool — so the mechanism above is evidenced from production logs and the SDK's own defaults rather than from a regression test.  <a href="https://cubic.dev/pr/twentyhq/twenty/pull/23857?utm_source=github" target="_blank" rel="noopener noreferrer" data-no-image-dialog="true"><picture><source media="(prefers-color-scheme: dark)" srcset="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"><source media="(prefers-color-scheme: light)" srcset="https://www.cubic.dev/buttons/review-in-cubic-light.svg"><img alt="Review in cubic" src="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"></picture></a>
2026-08-06 17:02:05 +02:00
parent 577cd68cb5
commit f5a42cdbae
8 changed files with 138 additions and 10 deletions
@@ -6,6 +6,7 @@ import {
 } from '@aws-sdk/client-sesv2';

 import { TwentyConfigService } from 'src/engine/core-modules/twenty-config/twenty-config.service';
+import { buildAwsRequestHandlerOptions } from 'src/utils/aws-request-handler.util';

@Injectable()
 export class AwsSesClientProvider {
@@ -17,6 +18,7 @@ export class AwsSesClientProvider {
    if (!this.sesClient) {
      const config: SESClientConfig = {
        region: this.twentyConfigService.get('AWS_SES_REGION'),
+        requestHandler: buildAwsRequestHandlerOptions(),
      };

      const accessKeyId = this.twentyConfigService.get('AWS_SES_ACCESS_KEY_ID');
@@ -0,0 +1,3 @@
+export const FILE_STORAGE_S3_REQUEST_TIMEOUT_MS = 5 * 60 * 1000;
+export const FILE_STORAGE_S3_CONNECTION_TIMEOUT_MS = 30 * 1000;
+export const FILE_STORAGE_S3_MAX_SOCKETS = 200;
@@ -24,11 +24,17 @@ import { Upload } from '@aws-sdk/lib-storage';
 import { getSignedUrl } from '@aws-sdk/s3-request-presigner';
 import { isDefined } from 'twenty-shared/utils';

+import {
+  FILE_STORAGE_S3_CONNECTION_TIMEOUT_MS,
+  FILE_STORAGE_S3_MAX_SOCKETS,
+  FILE_STORAGE_S3_REQUEST_TIMEOUT_MS,
+} from 'src/engine/core-modules/file-storage/constants/s3-client-timeouts.constant';
 import { type StorageDriver } from 'src/engine/core-modules/file-storage/drivers/interfaces/storage-driver.interface';
 import {
  FileStorageException,
  FileStorageExceptionCode,
 } from 'src/engine/core-modules/file-storage/interfaces/file-storage-exception';
+import { buildAwsRequestHandlerOptions } from 'src/utils/aws-request-handler.util';

 export interface S3DriverOptions extends S3ClientConfig {
  bucketName: string;
@@ -58,12 +64,23 @@ export class S3Driver implements StorageDriver {
      return;
    }

-    this.s3Client = new S3({ ...s3Options, region, endpoint });
+    const requestHandler = buildAwsRequestHandlerOptions({
+      requestTimeoutMs: FILE_STORAGE_S3_REQUEST_TIMEOUT_MS,
+      connectionTimeoutMs: FILE_STORAGE_S3_CONNECTION_TIMEOUT_MS,
+      maxSockets: FILE_STORAGE_S3_MAX_SOCKETS,
+    });
+
+    this.s3Client = new S3({ ...s3Options, region, endpoint, requestHandler });
    this.bucketName = bucketName;

    if (presignEnabled) {
      this.presignClient = presignEndpoint
-        ? new S3({ ...s3Options, region, endpoint: presignEndpoint })
+        ? new S3({
+            ...s3Options,
+            region,
+            endpoint: presignEndpoint,
+            requestHandler,
+          })
        : this.s3Client;
    }
  }
@@ -16,6 +16,11 @@ export const EXECUTOR_LAMBDA_MEMORY_MB = 512;
 export const EXECUTOR_LAMBDA_TIMEOUT_SECONDS = 900;
 export const LAMBDA_EPHEMERAL_STORAGE_MB = 4096;

+export const LAMBDA_CLIENT_REQUEST_TIMEOUT_MS =
+  (EXECUTOR_LAMBDA_TIMEOUT_SECONDS + 60) * 1000;
+export const LAMBDA_CLIENT_MAX_SOCKETS = 500;
+export const LAMBDA_CLIENT_CONNECTION_TIMEOUT_MS = 60_000;
+
 export const COMMON_LAYER_NAME_PREFIX = 'twenty-common-layer';
 export const YARN_INSTALL_FUNCTION_NAME_PREFIX = 'twenty-yarn-install';
 export const BUILDER_FUNCTION_NAME_PREFIX = 'twenty-builder';
@@ -11,14 +11,20 @@ import { isDefined } from 'twenty-shared/utils';

 import {
  CREDENTIALS_DURATION_IN_SECONDS,
+  LAMBDA_CLIENT_CONNECTION_TIMEOUT_MS,
  LAMBDA_CLIENT_MAX_ATTEMPTS,
+  LAMBDA_CLIENT_MAX_SOCKETS,
+  LAMBDA_CLIENT_REQUEST_TIMEOUT_MS,
  LAMBDA_CLIENT_RETRY_MODE,
  UPDATE_FUNCTION_DURATION_TIMEOUT_IN_SECONDS,
 } from 'src/engine/core-modules/logic-function/logic-function-drivers/drivers/lambda/constants/lambda-driver.constant';
 import { type LambdaDriverOptions } from 'src/engine/core-modules/logic-function/logic-function-drivers/drivers/lambda/types/lambda-driver.type';
+import { buildAwsRequestHandlerOptions } from 'src/utils/aws-request-handler.util';

 export class LambdaAwsClientService {
  private lambdaClient: Lambda | undefined;
+  private s3Client: S3Client | undefined;
+  private stsClient: STSClient | undefined;
  private assumeRoleCredentials:
    | { accessKeyId: string; secretAccessKey: string; sessionToken: string }
    | undefined;
@@ -39,22 +45,40 @@ export class LambdaAwsClientService {
        }),
        maxAttempts: LAMBDA_CLIENT_MAX_ATTEMPTS,
        retryMode: LAMBDA_CLIENT_RETRY_MODE,
+        requestHandler: buildAwsRequestHandlerOptions({
+          requestTimeoutMs: LAMBDA_CLIENT_REQUEST_TIMEOUT_MS,
+          connectionTimeoutMs: LAMBDA_CLIENT_CONNECTION_TIMEOUT_MS,
+          maxSockets: LAMBDA_CLIENT_MAX_SOCKETS,
+        }),
      });
    }

    return this.lambdaClient;
  }

+  private async getS3Client(): Promise<S3Client> {
+    if (
+      !isDefined(this.s3Client) ||
+      (isDefined(this.options.subhostingRole) &&
+        this.areAssumeRoleCredentialsExpired())
+    ) {
+      this.s3Client = new S3Client({
+        region: this.options.layerBucketRegion,
+        credentials: isDefined(this.options.subhostingRole)
+          ? await this.getAssumeRoleCredentials()
+          : this.options.credentials,
+        requestHandler: buildAwsRequestHandlerOptions(),
+      });
+    }
+
+    return this.s3Client;
+  }
+
  async generatePresignedUploadUrl(
    s3Key: string,
    expiresIn: number = 300,
  ): Promise<string> {
-    const s3Client = new S3Client({
-      region: this.options.layerBucketRegion,
-      credentials: isDefined(this.options.subhostingRole)
-        ? await this.getAssumeRoleCredentials()
-        : this.options.credentials,
-    });
+    const s3Client = await this.getS3Client();

    const putCommand = new PutObjectCommand({
      Bucket: this.options.layerBucket,
@@ -107,7 +131,12 @@ export class LambdaAwsClientService {
  }

  private async refreshAssumeRoleCredentials() {
-    const stsClient = new STSClient({ region: this.options.region });
+    this.stsClient ??= new STSClient({
+      region: this.options.region,
+      requestHandler: buildAwsRequestHandlerOptions(),
+    });
+
+    const stsClient = this.stsClient;

    const assumeRoleCommand = new AssumeRoleCommand({
      RoleArn: this.options.subhostingRole,
@@ -137,6 +166,7 @@ export class LambdaAwsClientService {
    );

    this.lambdaClient = undefined;
+    this.s3Client = undefined;
  }

  private async getAssumeRoleCredentials() {
@@ -5,6 +5,7 @@ import { isNonEmptyString } from '@sniptt/guards';

 import { StorageDriverType } from 'src/engine/core-modules/file-storage/interfaces/file-storage.interface';
 import { TwentyConfigService } from 'src/engine/core-modules/twenty-config/twenty-config.service';
+import { buildAwsRequestHandlerOptions } from 'src/utils/aws-request-handler.util';

@Injectable()
 export class InboundEmailS3ClientProvider {
@@ -54,7 +55,10 @@ export class InboundEmailS3ClientProvider {
      throw new Error('STORAGE_S3_REGION must be set to use email group.');
    }

-    const config: S3ClientConfig = { region };
+    const config: S3ClientConfig = {
+      region,
+      requestHandler: buildAwsRequestHandlerOptions(),
+    };

    const endpoint = this.twentyConfigService.get('STORAGE_S3_ENDPOINT');

@@ -0,0 +1,47 @@
+import {
+  AWS_DEFAULT_CONNECTION_TIMEOUT_MS,
+  AWS_DEFAULT_MAX_SOCKETS,
+  AWS_DEFAULT_REQUEST_TIMEOUT_MS,
+  buildAwsRequestHandlerOptions,
+} from 'src/utils/aws-request-handler.util';
+
+describe('buildAwsRequestHandlerOptions', () => {
+  it('always sets a non-zero request timeout', () => {
+    expect(buildAwsRequestHandlerOptions().requestTimeout).toBeGreaterThan(0);
+  });
+
+  it('always throws on request timeout rather than only warning', () => {
+    expect(buildAwsRequestHandlerOptions().throwOnRequestTimeout).toBe(true);
+  });
+
+  it('always sets a non-zero connection timeout', () => {
+    expect(buildAwsRequestHandlerOptions().connectionTimeout).toBeGreaterThan(
+      0,
+    );
+  });
+
+  it('applies the defaults when called with no arguments', () => {
+    expect(buildAwsRequestHandlerOptions()).toEqual({
+      connectionTimeout: AWS_DEFAULT_CONNECTION_TIMEOUT_MS,
+      requestTimeout: AWS_DEFAULT_REQUEST_TIMEOUT_MS,
+      throwOnRequestTimeout: true,
+      httpsAgent: { maxSockets: AWS_DEFAULT_MAX_SOCKETS },
+    });
+  });
+
+  it('allows callers with long-running requests to raise the timeouts', () => {
+    const options = buildAwsRequestHandlerOptions({
+      requestTimeoutMs: 960_000,
+      connectionTimeoutMs: 60_000,
+    });
+
+    expect(options.requestTimeout).toBe(960_000);
+    expect(options.connectionTimeout).toBe(60_000);
+  });
+
+  it('allows the socket pool size to be raised above the SDK default', () => {
+    expect(
+      buildAwsRequestHandlerOptions({ maxSockets: 500 }).httpsAgent.maxSockets,
+    ).toBe(500);
+  });
+});
@@ -0,0 +1,20 @@
+export const AWS_DEFAULT_CONNECTION_TIMEOUT_MS = 10_000;
+export const AWS_DEFAULT_REQUEST_TIMEOUT_MS = 30_000;
+export const AWS_DEFAULT_MAX_SOCKETS = 100;
+
+type AwsRequestHandlerOptions = {
+  requestTimeoutMs?: number;
+  connectionTimeoutMs?: number;
+  maxSockets?: number;
+};
+
+export const buildAwsRequestHandlerOptions = ({
+  requestTimeoutMs = AWS_DEFAULT_REQUEST_TIMEOUT_MS,
+  connectionTimeoutMs = AWS_DEFAULT_CONNECTION_TIMEOUT_MS,
+  maxSockets = AWS_DEFAULT_MAX_SOCKETS,
+}: AwsRequestHandlerOptions = {}) => ({
+  connectionTimeout: connectionTimeoutMs,
+  requestTimeout: requestTimeoutMs,
+  throwOnRequestTimeout: true,
+  httpsAgent: { maxSockets },
+});