fix(server): stop redundant lambda rebuilds causing build-lock acquisition failures (#21442)

## Context

`Lambda invocation failed for function '<id>' during build: Failed to
acquire lock for key: lambda-build:<id>` fires ~1000 times/day in
production.

## Root cause

`LambdaExecutorManagerService.buildExecutor` re-checks `canSkip` inside
the `lambda-build:<functionId>` lock, but the re-check reuses the
`flatApplication` snapshot captured when the request started. `canSkip`
depends on `!flatApplication.isSdkLayerStale`, so:

1. An app sync/install regenerates the SDK client and sets
`isSdkLayerStale = true`
2. All in-flight executions of the function fail `canSkip` and queue on
the lock
3. The first holder rebuilds and `markSdkLayerFresh` clears the flag in
DB + workspace cache
4. Queued waiters can't see that fix — their in-memory snapshot still
says stale — so **each waiter redoes the full rebuild serially**
(download SDK archive, delete + republish layer, update function config,
wait for update)
5. The lock is held back-to-back for minutes; everyone deeper in the
queue exhausts the 120s retry budget and throws

The local driver already handles this correctly
(`LocalLayerManagerService` refreshes the flat application from the
workspace cache inside its lock); the lambda driver missed it.

## Fix

- Refresh `flatApplication` from the workspace cache inside the lock
before re-checking `canSkip`, so waiters skip in ~100ms once the first
holder finishes
- Degrade gracefully on lock-acquisition timeout: re-check build status
with fresh data and proceed with the invocation if the executor is
already usable, instead of failing the run (introduces a typed
`CacheLockAcquisitionError` so only that case is caught)

## Test plan

- [x] `cache-lock.service.spec.ts` passes
- [x] `lint:diff-with-main` + typecheck pass
- [ ] Monitor `Failed to acquire lock for key: lambda-build:*` error
rate in production after deploy
This commit is contained in:
Thomas Trompette
2026-06-11 15:26:38 +02:00
committed by GitHub
parent 796f763bb7
commit 0ac4f237c0
6 changed files with 109 additions and 18 deletions
@@ -1,5 +1,9 @@
import { Injectable, Logger } from '@nestjs/common';
import {
CacheLockException,
CacheLockExceptionCode,
} from 'src/engine/core-modules/cache-lock/exceptions/cache-lock.exception';
import { InjectCacheStorage } from 'src/engine/core-modules/cache-storage/decorators/cache-storage.decorator';
import { CacheStorageNamespace } from 'src/engine/core-modules/cache-storage/types/cache-storage-namespace.enum';
import { CacheStorageService } from 'src/engine/core-modules/cache-storage/services/cache-storage.service';
@@ -50,6 +54,9 @@ export class CacheLockService {
await this.delay(ms);
}
throw new Error(`Failed to acquire lock for key: ${key}`);
throw new CacheLockException(
`Failed to acquire lock for key: ${key}`,
CacheLockExceptionCode.LOCK_ACQUISITION_TIMEOUT,
);
}
}
@@ -0,0 +1,26 @@
import { type MessageDescriptor } from '@lingui/core';
import { msg } from '@lingui/core/macro';
import {
appendCommonExceptionCode,
CustomException,
} from 'src/utils/custom-exception';
export class CacheLockException extends CustomException<
keyof typeof CacheLockExceptionCode
> {
constructor(
message: string,
code: keyof typeof CacheLockExceptionCode,
{ userFriendlyMessage }: { userFriendlyMessage?: MessageDescriptor } = {},
) {
super(message, code, {
userFriendlyMessage:
userFriendlyMessage ?? msg`A cache lock error occurred.`,
});
}
}
export const CacheLockExceptionCode = appendCommonExceptionCode({
LOCK_ACQUISITION_TIMEOUT: 'LOCK_ACQUISITION_TIMEOUT',
} as const);