fix(server): stop redundant lambda rebuilds causing build-lock acquisition failures (#21442)
## Context `Lambda invocation failed for function '<id>' during build: Failed to acquire lock for key: lambda-build:<id>` fires ~1000 times/day in production. ## Root cause `LambdaExecutorManagerService.buildExecutor` re-checks `canSkip` inside the `lambda-build:<functionId>` lock, but the re-check reuses the `flatApplication` snapshot captured when the request started. `canSkip` depends on `!flatApplication.isSdkLayerStale`, so: 1. An app sync/install regenerates the SDK client and sets `isSdkLayerStale = true` 2. All in-flight executions of the function fail `canSkip` and queue on the lock 3. The first holder rebuilds and `markSdkLayerFresh` clears the flag in DB + workspace cache 4. Queued waiters can't see that fix — their in-memory snapshot still says stale — so **each waiter redoes the full rebuild serially** (download SDK archive, delete + republish layer, update function config, wait for update) 5. The lock is held back-to-back for minutes; everyone deeper in the queue exhausts the 120s retry budget and throws The local driver already handles this correctly (`LocalLayerManagerService` refreshes the flat application from the workspace cache inside its lock); the lambda driver missed it. ## Fix - Refresh `flatApplication` from the workspace cache inside the lock before re-checking `canSkip`, so waiters skip in ~100ms once the first holder finishes - Degrade gracefully on lock-acquisition timeout: re-check build status with fresh data and proceed with the invocation if the executor is already usable, instead of failing the run (introduces a typed `CacheLockAcquisitionError` so only that case is caught) ## Test plan - [x] `cache-lock.service.spec.ts` passes - [x] `lint:diff-with-main` + typecheck pass - [ ] Monitor `Failed to acquire lock for key: lambda-build:*` error rate in production after deploy
This commit is contained in:
@@ -1,5 +1,9 @@
|
||||
import { Injectable, Logger } from '@nestjs/common';
|
||||
|
||||
import {
|
||||
CacheLockException,
|
||||
CacheLockExceptionCode,
|
||||
} from 'src/engine/core-modules/cache-lock/exceptions/cache-lock.exception';
|
||||
import { InjectCacheStorage } from 'src/engine/core-modules/cache-storage/decorators/cache-storage.decorator';
|
||||
import { CacheStorageNamespace } from 'src/engine/core-modules/cache-storage/types/cache-storage-namespace.enum';
|
||||
import { CacheStorageService } from 'src/engine/core-modules/cache-storage/services/cache-storage.service';
|
||||
@@ -50,6 +54,9 @@ export class CacheLockService {
|
||||
await this.delay(ms);
|
||||
}
|
||||
|
||||
throw new Error(`Failed to acquire lock for key: ${key}`);
|
||||
throw new CacheLockException(
|
||||
`Failed to acquire lock for key: ${key}`,
|
||||
CacheLockExceptionCode.LOCK_ACQUISITION_TIMEOUT,
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
+26
@@ -0,0 +1,26 @@
|
||||
import { type MessageDescriptor } from '@lingui/core';
|
||||
import { msg } from '@lingui/core/macro';
|
||||
|
||||
import {
|
||||
appendCommonExceptionCode,
|
||||
CustomException,
|
||||
} from 'src/utils/custom-exception';
|
||||
|
||||
export class CacheLockException extends CustomException<
|
||||
keyof typeof CacheLockExceptionCode
|
||||
> {
|
||||
constructor(
|
||||
message: string,
|
||||
code: keyof typeof CacheLockExceptionCode,
|
||||
{ userFriendlyMessage }: { userFriendlyMessage?: MessageDescriptor } = {},
|
||||
) {
|
||||
super(message, code, {
|
||||
userFriendlyMessage:
|
||||
userFriendlyMessage ?? msg`A cache lock error occurred.`,
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
export const CacheLockExceptionCode = appendCommonExceptionCode({
|
||||
LOCK_ACQUISITION_TIMEOUT: 'LOCK_ACQUISITION_TIMEOUT',
|
||||
} as const);
|
||||
Reference in New Issue
Block a user