feat(logic-function): namespace shared Lambda resources per instance (#22509)

## Problem When several Twenty instances run in the **same AWS account + region** (our customer's setup), their logic-function Lambda resources collide. Four shared resources are named purely from a content hash: | Resource | Name | Derived from | |---|---|---| | Builder fn | `twenty-builder-<sha256(handler)>` | handler code (⇒ version) | | Yarn-install fn | `twenty-yarn-install-<sha256(handler)>` | handler code | | Common layer | `twenty-common-layer-<sha256(pkg+lock)>` | dependencies | | Deps layer | `deps-<yarnLockChecksum>` | app yarn.lock | Because the hash is identical across instances of the same version, every instance computes the **same name**. `ensureBuilderLambdaExists` / `ensureYarnInstallLambdaExists` do `GetFunction → if it exists, return` with **no role check**, so the first instance to create the function binds it to *its* `LOGIC_FUNCTION_LAMBDA_ROLE` and every other instance silently reuses it. When that role is later deleted or belongs to a different account, invokes fail with: > The role defined for the function cannot be assumed by Lambda. Nothing tears these shared resources down, so a poisoned function persists indefinitely. (Executors are UUID-named and SDK layers are workspace-scoped, so they don't collide.) ## Fix Namespace the four shared resources by a per-instance segment. - New optional config var **`LOGIC_FUNCTION_LAMBDA_RESOURCE_NAMESPACE`** (LOGIC_FUNCTION_CONFIG group). - When unset it defaults to `sha256(LOGIC_FUNCTION_LAMBDA_ROLE).slice(0, 10)`. Keying the namespace on the execution role makes the sharing boundary correct: - **same role → same names →** instances still dedupe (original intent preserved), - **different role → different names →** full isolation, invoke can never hit a role it can't assume, - **role change →** resources are recreated fresh under a new name (self-healing). Names become `twenty-builder-<namespace>-<checksum>`, `deps-<namespace>-<checksum>`, etc. — the owning instance stays legible in the AWS console. ### Defensive role-heal As a safety net (and to heal already-poisoned functions), after `GetFunction` succeeds we compare `Configuration.Role` to the configured role; on mismatch we delete and recreate the tool function. (`waitFunctionDeleted` polls `GetFunction` until `ResourceNotFoundException` — this SDK version has no `waitUntilFunctionNotExists` waiter.) ## Scope / compatibility - Executor functions (`<logicFunctionId>`) and SDK layers (`sdk-<workspaceId>-<appUUID>`) are unchanged — already unique. - On upgrade, shared-resource names change once (role-hash namespace), so each instance recreates its builder/yarn-install/common-layer/deps on first use; old ones are orphaned (harmless, unreferenced). - Operators who want explicit control can set `LOGIC_FUNCTION_LAMBDA_RESOURCE_NAMESPACE`. ## Ops note (immediate unblock, independent of this PR) Delete the poisoned `twenty-builder-<hash>` (and sibling `twenty-yarn-install-*`) in the affected region; it is recreated with the correct role on next use. ## Tests - `compute-hashed-lambda-resource-name.util.spec.ts` — namespace segment behavior - `get-lambda-deps-layer-name.util.spec.ts` — namespaced deps layer name - `get-lambda-resource-namespace.util.spec.ts` — stable, role-distinct namespace All pass; `lint:diff-with-main` clean; typecheck clean for changed files.  <a href="https://cubic.dev/pr/twentyhq/twenty/pull/22509?utm_source=github" target="_blank" rel="noopener noreferrer" data-no-image-dialog="true"><picture><source media="(prefers-color-scheme: dark)" srcset="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"><source media="(prefers-color-scheme: light)" srcset="https://www.cubic.dev/buttons/review-in-cubic-light.svg"><img alt="Review in cubic" src="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"></picture></a>
2026-07-03 13:18:26 +02:00
parent 8b191d6fcc
commit 203da78b05
12 changed files with 208 additions and 24 deletions
@@ -32,7 +32,10 @@ export class LambdaLayerManagerService {
  private readonly logger = new Logger(LambdaLayerManagerService.name);

  constructor(
-    private readonly options: Pick<LambdaDriverOptions, 'layerBucket'>,
+    private readonly options: Pick<
+      LambdaDriverOptions,
+      'layerBucket' | 'resourceNamespace'
+    >,
    private readonly awsClient: LambdaAwsClientService,
    private readonly toolFunctions: LambdaToolFunctionsService,
    private readonly logicFunctionResourceService: LogicFunctionResourceService,
@@ -40,7 +43,10 @@ export class LambdaLayerManagerService {
  ) {}

  async ensureDepsLayer(context: LayerAppContext): Promise<string> {
-    const layerName = getLambdaDepsLayerName(context.flatApplication);
+    const layerName = getLambdaDepsLayerName({
+      flatApplication: context.flatApplication,
+      namespace: this.options.resourceNamespace,
+    });

    const existingArn = await this.awsClient.getExistingLayerArn(layerName);

@@ -128,7 +134,10 @@ export class LambdaLayerManagerService {
      return false;
    }

-    const depsLayerName = getLambdaDepsLayerName(flatApplication);
+    const depsLayerName = getLambdaDepsLayerName({
+      flatApplication,
+      namespace: this.options.resourceNamespace,
+    });
    const sdkLayerName = getLambdaSdkLayerName({
      workspaceId: flatApplication.workspaceId,
      applicationUniversalIdentifier,
@@ -52,7 +52,10 @@ export class LambdaToolFunctionsService {
  private builderFunctionName: string | undefined;

  constructor(
-    private readonly options: Pick<LambdaDriverOptions, 'lambdaRole'>,
+    private readonly options: Pick<
+      LambdaDriverOptions,
+      'lambdaRole' | 'resourceNamespace'
+    >,
    private readonly awsClient: LambdaAwsClientService,
  ) {}

@@ -322,7 +325,8 @@ export class LambdaToolFunctionsService {
    ]);

    this.commonLayerName = computeHashedLambdaResourceName({
-      prefix: COMMON_LAYER_NAME_PREFIX,
+      resourceNamePrefix: COMMON_LAYER_NAME_PREFIX,
+      namespace: this.options.resourceNamespace,
      contents: [packageJson, yarnLock],
    });

@@ -340,7 +344,8 @@ export class LambdaToolFunctionsService {
    );

    this.yarnInstallFunctionName = computeHashedLambdaResourceName({
-      prefix: YARN_INSTALL_FUNCTION_NAME_PREFIX,
+      resourceNamePrefix: YARN_INSTALL_FUNCTION_NAME_PREFIX,
+      namespace: this.options.resourceNamespace,
      contents: [handlerContent],
    });

@@ -355,7 +360,8 @@ export class LambdaToolFunctionsService {
    const handlerContent = await fs.readFile(BUILDER_HANDLER_PATH, 'utf-8');

    this.builderFunctionName = computeHashedLambdaResourceName({
-      prefix: BUILDER_FUNCTION_NAME_PREFIX,
+      resourceNamePrefix: BUILDER_FUNCTION_NAME_PREFIX,
+      namespace: this.options.resourceNamespace,
      contents: [handlerContent],
    });

@@ -44,6 +44,7 @@ export interface LambdaDriverOptions extends LambdaClientConfig {
  subhostingRole?: string;
  layerBucket: string;
  layerBucketRegion: string;
+  resourceNamespace: string;
 }

 export enum LambdaExecutionPhase {
@@ -0,0 +1,32 @@
+import { buildLambdaResourceName } from 'src/engine/core-modules/logic-function/logic-function-drivers/drivers/lambda/utils/build-lambda-resource-name.util';
+
+describe('buildLambdaResourceName', () => {
+  it('joins prefix, namespace and checksum', () => {
+    expect(
+      buildLambdaResourceName({
+        resourceNamePrefix: 'twenty-builder',
+        namespace: 'abc123',
+        checksum: 'def456',
+      }),
+    ).toBe('twenty-builder-abc123-def456');
+  });
+
+  it('omits the namespace segment when it is undefined', () => {
+    expect(
+      buildLambdaResourceName({
+        resourceNamePrefix: 'deps',
+        checksum: 'def456',
+      }),
+    ).toBe('deps-def456');
+  });
+
+  it('omits the namespace segment when it is an empty string', () => {
+    expect(
+      buildLambdaResourceName({
+        resourceNamePrefix: 'deps',
+        namespace: '',
+        checksum: 'def456',
+      }),
+    ).toBe('deps-def456');
+  });
+});
@@ -3,7 +3,43 @@ import { computeHashedLambdaResourceName } from 'src/engine/core-modules/logic-f
 describe('computeHashedLambdaResourceName', () => {
  it('returns prefix concatenated with a 12-character sha256 hex checksum of the contents', () => {
    const name = computeHashedLambdaResourceName({
-      prefix: 'twenty-builder',
+      resourceNamePrefix: 'twenty-builder',
+      contents: ['handler-code'],
+    });
+
+    expect(name).toMatch(/^twenty-builder-[0-9a-f]{12}$/);
+  });
+
+  it('inserts the namespace as a distinct segment before the checksum', () => {
+    const name = computeHashedLambdaResourceName({
+      resourceNamePrefix: 'twenty-builder',
+      namespace: 'abc123def0',
+      contents: ['handler-code'],
+    });
+
+    expect(name).toMatch(/^twenty-builder-abc123def0-[0-9a-f]{12}$/);
+  });
+
+  it('keeps the checksum stable and only prepends the namespace', () => {
+    const withoutNamespace = computeHashedLambdaResourceName({
+      resourceNamePrefix: 'twenty-builder',
+      contents: ['handler-code'],
+    });
+    const withNamespace = computeHashedLambdaResourceName({
+      resourceNamePrefix: 'twenty-builder',
+      namespace: 'abc123def0',
+      contents: ['handler-code'],
+    });
+
+    const checksum = withoutNamespace.replace('twenty-builder-', '');
+
+    expect(withNamespace).toBe(`twenty-builder-abc123def0-${checksum}`);
+  });
+
+  it('omits the namespace segment when it is an empty string', () => {
+    const name = computeHashedLambdaResourceName({
+      resourceNamePrefix: 'twenty-builder',
+      namespace: '',
      contents: ['handler-code'],
    });

@@ -11,24 +11,44 @@ const buildFlatApplication = (

 describe('getLambdaDepsLayerName', () => {
  it('returns deps-<checksum> when yarnLockChecksum is set', () => {
-    expect(getLambdaDepsLayerName(buildFlatApplication())).toBe('deps-abc123');
+    expect(
+      getLambdaDepsLayerName({ flatApplication: buildFlatApplication() }),
+    ).toBe('deps-abc123');
  });

  it('falls back to deps-default when yarnLockChecksum is undefined', () => {
    expect(
-      getLambdaDepsLayerName(
-        buildFlatApplication({ yarnLockChecksum: undefined }),
-      ),
+      getLambdaDepsLayerName({
+        flatApplication: buildFlatApplication({ yarnLockChecksum: undefined }),
+      }),
    ).toBe('deps-default');
  });

  it('falls back to deps-default when yarnLockChecksum is null', () => {
    expect(
-      getLambdaDepsLayerName(
-        buildFlatApplication({
+      getLambdaDepsLayerName({
+        flatApplication: buildFlatApplication({
          yarnLockChecksum: null as unknown as string,
        }),
-      ),
+      }),
    ).toBe('deps-default');
  });
+
+  it('inserts the namespace segment when provided', () => {
+    expect(
+      getLambdaDepsLayerName({
+        flatApplication: buildFlatApplication(),
+        namespace: 'ns123',
+      }),
+    ).toBe('deps-ns123-abc123');
+  });
+
+  it('omits the namespace segment when it is an empty string', () => {
+    expect(
+      getLambdaDepsLayerName({
+        flatApplication: buildFlatApplication(),
+        namespace: '',
+      }),
+    ).toBe('deps-abc123');
+  });
 });
@@ -0,0 +1,33 @@
+import { getLambdaResourceNamespace } from 'src/engine/core-modules/logic-function/logic-function-drivers/drivers/lambda/utils/get-lambda-resource-namespace.util';
+
+describe('getLambdaResourceNamespace', () => {
+  it('returns a stable 10-character hex namespace for a role ARN', () => {
+    const lambdaRoleArn = 'arn:aws:iam::123456789012:role/twenty-lambda';
+
+    const namespace = getLambdaResourceNamespace({ lambdaRoleArn });
+
+    expect(namespace).toMatch(/^[0-9a-f]{10}$/);
+    expect(getLambdaResourceNamespace({ lambdaRoleArn })).toBe(namespace);
+  });
+
+  it('produces different namespaces for different role ARNs', () => {
+    const namespaceA = getLambdaResourceNamespace({
+      lambdaRoleArn: 'arn:aws:iam::111111111111:role/twenty-lambda',
+    });
+    const namespaceB = getLambdaResourceNamespace({
+      lambdaRoleArn: 'arn:aws:iam::222222222222:role/twenty-lambda',
+    });
+
+    expect(namespaceA).not.toBe(namespaceB);
+  });
+
+  it.each([undefined, ''])(
+    'falls back to a stable sentinel namespace when the role is missing (%p)',
+    (lambdaRoleArn) => {
+      const namespace = getLambdaResourceNamespace({ lambdaRoleArn });
+
+      expect(namespace).toMatch(/^[0-9a-f]{10}$/);
+      expect(getLambdaResourceNamespace({ lambdaRoleArn })).toBe(namespace);
+    },
+  );
+});
@@ -0,0 +1,14 @@
+import { isNonEmptyString } from '@sniptt/guards';
+
+export const buildLambdaResourceName = ({
+  resourceNamePrefix,
+  namespace,
+  checksum,
+}: {
+  resourceNamePrefix: string;
+  namespace?: string;
+  checksum: string;
+}): string =>
+  isNonEmptyString(namespace)
+    ? `${resourceNamePrefix}-${namespace}-${checksum}`
+    : `${resourceNamePrefix}-${checksum}`;
@@ -1,12 +1,16 @@
 import { createHash } from 'crypto';

+import { buildLambdaResourceName } from 'src/engine/core-modules/logic-function/logic-function-drivers/drivers/lambda/utils/build-lambda-resource-name.util';
+
 const RESOURCE_NAME_CHECKSUM_LENGTH = 12;

 export const computeHashedLambdaResourceName = ({
-  prefix,
+  resourceNamePrefix,
+  namespace,
  contents,
 }: {
-  prefix: string;
+  resourceNamePrefix: string;
+  namespace?: string;
  contents: ReadonlyArray<string>;
 }): string => {
  const hash = createHash('sha256');
@@ -17,5 +21,5 @@ export const computeHashedLambdaResourceName = ({

  const checksum = hash.digest('hex').slice(0, RESOURCE_NAME_CHECKSUM_LENGTH);

-  return `${prefix}-${checksum}`;
+  return buildLambdaResourceName({ resourceNamePrefix, namespace, checksum });
 };
@@ -1,9 +1,17 @@
 import { type FlatApplication } from 'src/engine/core-modules/application/types/flat-application.type';
+import { buildLambdaResourceName } from 'src/engine/core-modules/logic-function/logic-function-drivers/drivers/lambda/utils/build-lambda-resource-name.util';

-export const getLambdaDepsLayerName = (
-  flatApplication: FlatApplication,
-): string => {
-  const checksum = flatApplication.yarnLockChecksum ?? 'default';
+const DEPS_LAYER_NAME_PREFIX = 'deps';

-  return `deps-${checksum}`;
-};
+export const getLambdaDepsLayerName = ({
+  flatApplication,
+  namespace,
+}: {
+  flatApplication: FlatApplication;
+  namespace?: string;
+}): string =>
+  buildLambdaResourceName({
+    resourceNamePrefix: DEPS_LAYER_NAME_PREFIX,
+    namespace,
+    checksum: flatApplication.yarnLockChecksum ?? 'default',
+  });
@@ -0,0 +1,16 @@
+import { createHash } from 'crypto';
+
+import { isNonEmptyString } from '@sniptt/guards';
+
+const RESOURCE_NAMESPACE_LENGTH = 10;
+const NO_ROLE_SENTINEL = 'no-role';
+
+export const getLambdaResourceNamespace = ({
+  lambdaRoleArn,
+}: {
+  lambdaRoleArn?: string;
+}): string =>
+  createHash('sha256')
+    .update(isNonEmptyString(lambdaRoleArn) ? lambdaRoleArn : NO_ROLE_SENTINEL)
+    .digest('hex')
+    .slice(0, RESOURCE_NAMESPACE_LENGTH);
@@ -11,6 +11,7 @@ import { CacheLockService } from 'src/engine/core-modules/cache-lock/cache-lock.
 import { DisabledDriver } from 'src/engine/core-modules/logic-function/logic-function-drivers/drivers/disabled.driver';
 import { LambdaDriver } from 'src/engine/core-modules/logic-function/logic-function-drivers/drivers/lambda.driver';
 import { LocalDriver } from 'src/engine/core-modules/logic-function/logic-function-drivers/drivers/local.driver';
+import { getLambdaResourceNamespace } from 'src/engine/core-modules/logic-function/logic-function-drivers/drivers/lambda/utils/get-lambda-resource-namespace.util';
 import { LogicFunctionResourceService } from 'src/engine/core-modules/logic-function/logic-function-resource/logic-function-resource.service';
 import { SdkClientArchiveService } from 'src/engine/core-modules/sdk-client/sdk-client-archive.service';
 import { DriverFactoryBase } from 'src/engine/core-modules/twenty-config/dynamic-factory.base';
@@ -82,6 +83,9 @@ export class LogicFunctionDriverFactory extends DriverFactoryBase<LogicFunctionD
          this.twentyConfigService.get(
            'LOGIC_FUNCTION_LAMBDA_LAYER_BUCKET_REGION',
          ) ?? region;
+        const resourceNamespace = getLambdaResourceNamespace({
+          lambdaRoleArn: lambdaRole,
+        });

        return new LambdaDriver({
          logicFunctionResourceService: this.logicFunctionResourceService,
@@ -95,6 +99,7 @@ export class LogicFunctionDriverFactory extends DriverFactoryBase<LogicFunctionD
          subhostingRole,
          layerBucket,
          layerBucketRegion,
+          resourceNamespace,
          sdkClientArchiveService: this.sdkClientArchiveService,
        });
      }