feat(logic-function): add LIVE / PREBUILT execution modes (#20873)

## Summary ### Why 1. Sending the code to the lambda (~1Mb usually) is heavy on network and results to a constant traffic of ~30Mb/s on AWS which results into TB of network data every month 2. eval(1MB of code) is not that fast, it's heavy on memory and CPU on lambda side ### High level Adds two execution modes for logic functions, gated behind the new `IS_LOGIC_FUNCTION_PREBUILT_MODE_ENABLED` workspace feature flag (off everywhere by default): - **LIVE** (current behavior, preserved bit-for-bit): the compiled bundle is read from object storage and shipped in every Lambda invoke payload. Used for fast iteration in the workflow editor / Settings test runs. - **PREBUILT** (new): the bundle is installed onto the per-function Lambda alongside the unified executor, and invocations carry only `{ params, env, handlerName }` — saving JSON payload egress and warm-start `import()` cost on every call. ### Key design choices - **Unified Lambda handler** (`constants/executor/index.mjs`) dispatches at runtime: `event.code` present ? LIVE (write to `/tmp`, dynamic import) : `import('./prebuilt-logic-function.mjs')`. Both code paths always coexist on the deployment package, so the same Lambda can serve either mode without redeploying. - **Install runs inside the `validateBuildAndRun` migration pipeline**, not at execute time. `Create/UpdateLogicFunctionActionHandlerService` calls `driver.installPrebuiltBundle` when `executionMode` flips LIVE?PREBUILT or `checksum` changes while PREBUILT, gated on `isBuildUpToDate=true` and a fresh checksum. - **Strict execute, no reconciliation**: `LogicFunctionExecutorService.execute` resolves `effectiveExecutionMode` (caller override > feature flag > entity column). For PREBUILT it asks the driver `getInstalledBundleChecksum` (Lambda `twenty:bundle-checksum` tag for AWS, sidecar file locally) and throws `LOGIC_FUNCTION_PREBUILT_BUNDLE_NOT_INSTALLED` on mismatch. - **Feature flag gates every side effect**: with the flag off the executor forces LIVE, the action-handler install hooks bail before AWS, and workflow activation does not flip the mode. Rollback is just turning the flag off. ### Lifecycle - New workflow CODE step ? `LIVE`, no install. - Workflow activated ? build + activation flips `executionMode=PREBUILT` ? action-handler installs the bundle + sets the Lambda tag. - Draft from active version ? duplicated logic function reset to `LIVE`. - App install ? manifest converter sets `PREBUILT`, create-action handler installs. - Test runs (`executeOneFromSource`, workflow editor) pass `executionMode=LIVE` explicitly. ### Observability `[lambda-timing]` log lines now include `effectiveExecutionMode` and `payloadBytes`; the action handler logs `install_duration_ms` for each install. ## Test plan - [x] `npx nx typecheck twenty-server` ? passes - [x] `npx oxlint --type-aware` on all changed files ? 0 warnings, 0 errors - [x] `npx nx test twenty-server` ? 588 suites / 5009 tests pass (no regressions vs main) - [x] New unit suite `flat-logic-function-validator.service.spec.ts` ? 9/9 - [x] Existing `workflow-version-step-operations.workspace-service.spec.ts` ? 8/8 (verified the new token-based DI avoids a circular-import regression) - [x] Snapshot for `ALL_UNIVERSAL_FLAT_ENTITY_PROPERTIES_TO_COMPARE_AND_STRINGIFY` updated to include `executionMode` - [x] Integration suite `logic-function-execution.integration-spec.ts` extended to assert `executionMode=LIVE` on newly-created functions and continues to exercise the LIVE happy path - [ ] Manual staging rollout: flip `IS_LOGIC_FUNCTION_PREBUILT_MODE_ENABLED` per workspace, observe `[lambda-timing]` `payloadBytes` drop + `install_duration_ms`, then ramp in prod.
2026-06-02 13:14:39 +02:00
parent 1ae00d6753
commit 58907b733c
44 changed files with 2212 additions and 951 deletions
@@ -7,7 +7,9 @@ import { isDefined } from 'twenty-shared/utils';

 import { validateFilePath } from 'src/engine/core-modules/file-storage/utils/validate-file-path.util';
 import { findFlatEntityByUniversalIdentifier } from 'src/engine/metadata-modules/flat-entity/utils/find-flat-entity-by-universal-identifier.util';
+import { LogicFunctionExecutionMode } from 'src/engine/metadata-modules/logic-function/logic-function.entity';
 import { LogicFunctionExceptionCode } from 'src/engine/metadata-modules/logic-function/logic-function.exception';
+import { isLogicFunctionReadyForPrebuiltInstall } from 'src/engine/metadata-modules/logic-function/utils/is-logic-function-ready-for-prebuilt-install.util';
 import { type FailedFlatEntityValidation } from 'src/engine/workspace-manager/workspace-migration/workspace-migration-builder/builders/types/failed-flat-entity-validation.type';
 import { getEmptyFlatEntityValidationError } from 'src/engine/workspace-manager/workspace-migration/workspace-migration-builder/builders/utils/get-flat-entity-validation-error.util';
 import { type FlatEntityUpdateValidationArgs } from 'src/engine/workspace-manager/workspace-migration/workspace-migration-builder/types/universal-flat-entity-update-validation-args.type';
@@ -79,6 +81,31 @@ export class FlatLogicFunctionValidatorService {
      }
    }

+    const mergedPrebuiltState = {
+      executionMode:
+        flatEntityUpdate.executionMode ??
+        existingFlatLogicFunction.executionMode,
+      isBuildUpToDate:
+        flatEntityUpdate.isBuildUpToDate ??
+        existingFlatLogicFunction.isBuildUpToDate,
+      checksum:
+        flatEntityUpdate.checksum !== undefined
+          ? flatEntityUpdate.checksum
+          : existingFlatLogicFunction.checksum,
+    };
+
+    if (
+      mergedPrebuiltState.executionMode ===
+        LogicFunctionExecutionMode.PREBUILT &&
+      !isLogicFunctionReadyForPrebuiltInstall(mergedPrebuiltState)
+    ) {
+      validationResult.errors.push({
+        code: LogicFunctionExceptionCode.INVALID_LOGIC_FUNCTION_INPUT,
+        message: t`Logic function cannot be in PREBUILT mode without a fresh build and a checksum`,
+        userFriendlyMessage: msg`Logic function cannot be in PREBUILT mode without a fresh build and a checksum`,
+      });
+    }
+
    return validationResult;
  }

@@ -175,6 +202,18 @@ export class FlatLogicFunctionValidatorService {
      }
    }

+    if (
+      flatLogicFunctionToValidate.executionMode ===
+        LogicFunctionExecutionMode.PREBUILT &&
+      !isLogicFunctionReadyForPrebuiltInstall(flatLogicFunctionToValidate)
+    ) {
+      validationResult.errors.push({
+        code: LogicFunctionExceptionCode.INVALID_LOGIC_FUNCTION_INPUT,
+        message: t`Logic function cannot be in PREBUILT mode without a fresh build and a checksum`,
+        userFriendlyMessage: msg`Logic function cannot be in PREBUILT mode without a fresh build and a checksum`,
+      });
+    }
+
    return validationResult;
  }
 }
@@ -1,9 +1,13 @@
-import { Injectable } from '@nestjs/common';
+import { Inject, Injectable } from '@nestjs/common';

 import { v4 } from 'uuid';

 import { WorkspaceMigrationRunnerActionHandler } from 'src/engine/workspace-manager/workspace-migration/workspace-migration-runner/interfaces/workspace-migration-runner-action-handler-service.interface';

+import { LOGIC_FUNCTION_DRIVER_FACTORY_TOKEN } from 'src/engine/core-modules/logic-function/logic-function-drivers/constants/logic-function-driver-factory.token';
+import { isLogicFunctionReadyForPrebuiltInstall } from 'src/engine/metadata-modules/logic-function/utils/is-logic-function-ready-for-prebuilt-install.util';
+
+import type { LogicFunctionDriverFactory } from 'src/engine/core-modules/logic-function/logic-function-drivers/logic-function-driver.factory';
 import { getUniversalFlatEntityEmptyForeignKeyAggregators } from 'src/engine/workspace-manager/workspace-migration/universal-flat-entity/utils/reset-universal-flat-entity-foreign-key-aggregators.util';
 import {
  FlatCreateLogicFunctionAction,
@@ -19,6 +23,13 @@ export class CreateLogicFunctionActionHandlerService extends WorkspaceMigrationR
  'create',
  'logicFunction',
 ) {
+  constructor(
+    @Inject(LOGIC_FUNCTION_DRIVER_FACTORY_TOKEN)
+    private readonly logicFunctionDriverFactory: LogicFunctionDriverFactory,
+  ) {
+    super();
+  }
+
  override async transpileUniversalActionToFlatAction({
    action,
    flatApplication,
@@ -44,13 +55,42 @@ export class CreateLogicFunctionActionHandlerService extends WorkspaceMigrationR
  async executeForMetadata(
    context: WorkspaceMigrationActionRunnerContext<FlatCreateLogicFunctionAction>,
  ): Promise<void> {
-    const { flatAction, queryRunner } = context;
+    const { flatAction, queryRunner, flatApplication } = context;
    const { flatEntity: logicFunction } = flatAction;

    await this.insertFlatEntitiesInRepository({
      queryRunner,
      flatEntities: [logicFunction],
    });
+
+    if (isLogicFunctionReadyForPrebuiltInstall(logicFunction)) {
+      const driver = this.logicFunctionDriverFactory.getCurrentDriver();
+
+      const installStart = Date.now();
+
+      try {
+        await driver.installPrebuiltBundle({
+          flatLogicFunction: logicFunction,
+          flatApplication,
+          applicationUniversalIdentifier: flatApplication.universalIdentifier,
+        });
+
+        this.logger.log(
+          `[lambda-timing] event=install_prebuilt fnId=${logicFunction.id} ` +
+            `reason=app_install install_duration_ms=${Date.now() - installStart}`,
+          CreateLogicFunctionActionHandlerService.name,
+        );
+      } catch (error) {
+        this.logger.error(
+          `Failed to install prebuilt bundle on app-install for function ${logicFunction.id} ` +
+            `after ${Date.now() - installStart}ms: ` +
+            `${error instanceof Error ? error.message : String(error)}`,
+          CreateLogicFunctionActionHandlerService.name,
+          error instanceof Error ? error.stack : undefined,
+        );
+        throw error;
+      }
+    }
  }

  async rollbackForMetadata(
@@ -1,4 +1,4 @@
-import { Injectable } from '@nestjs/common';
+import { Inject, Injectable } from '@nestjs/common';

 import { FileFolder } from 'twenty-shared/types';
 import { isDefined } from 'twenty-shared/utils';
@@ -6,9 +6,17 @@ import { isDefined } from 'twenty-shared/utils';
 import { WorkspaceMigrationRunnerActionHandler } from 'src/engine/workspace-manager/workspace-migration/workspace-migration-runner/interfaces/workspace-migration-runner-action-handler-service.interface';

 import { FileStorageService } from 'src/engine/core-modules/file-storage/file-storage.service';
+import { LOGIC_FUNCTION_DRIVER_FACTORY_TOKEN } from 'src/engine/core-modules/logic-function/logic-function-drivers/constants/logic-function-driver-factory.token';
 import { findFlatEntityByIdInFlatEntityMapsOrThrow } from 'src/engine/metadata-modules/flat-entity/utils/find-flat-entity-by-id-in-flat-entity-maps-or-throw.util';
+
+import type { LogicFunctionDriverFactory } from 'src/engine/core-modules/logic-function/logic-function-drivers/logic-function-driver.factory';
 import { findFlatEntityByUniversalIdentifierOrThrow } from 'src/engine/metadata-modules/flat-entity/utils/find-flat-entity-by-universal-identifier-or-throw.util';
-import { LogicFunctionEntity } from 'src/engine/metadata-modules/logic-function/logic-function.entity';
+import {
+  LogicFunctionEntity,
+  LogicFunctionExecutionMode,
+} from 'src/engine/metadata-modules/logic-function/logic-function.entity';
+import { type FlatLogicFunction } from 'src/engine/metadata-modules/logic-function/types/flat-logic-function.type';
+import { shouldReinstallLogicFunctionPrebuiltBundle } from 'src/engine/metadata-modules/logic-function/utils/should-reinstall-logic-function-prebuilt-bundle.util';
 import { resolveUniversalUpdateRelationIdentifiersToIds } from 'src/engine/workspace-manager/workspace-migration/universal-flat-entity/utils/resolve-universal-update-relation-identifiers-to-ids.util';
 import {
  FlatUpdateLogicFunctionAction,
@@ -24,7 +32,11 @@ export class UpdateLogicFunctionActionHandlerService extends WorkspaceMigrationR
  'update',
  'logicFunction',
 ) {
-  constructor(private readonly fileStorageService: FileStorageService) {
+  constructor(
+    private readonly fileStorageService: FileStorageService,
+    @Inject(LOGIC_FUNCTION_DRIVER_FACTORY_TOKEN)
+    private readonly logicFunctionDriverFactory: LogicFunctionDriverFactory,
+  ) {
    super();
  }

@@ -64,10 +76,11 @@ export class UpdateLogicFunctionActionHandlerService extends WorkspaceMigrationR
    } = context;
    const { entityId, update } = flatAction;

-    const existingLogicFunction = findFlatEntityByIdInFlatEntityMapsOrThrow({
-      flatEntityMaps: allFlatEntityMaps.flatLogicFunctionMaps,
-      flatEntityId: entityId,
-    });
+    const existingLogicFunction =
+      findFlatEntityByIdInFlatEntityMapsOrThrow<FlatLogicFunction>({
+        flatEntityMaps: allFlatEntityMaps.flatLogicFunctionMaps,
+        flatEntityId: entityId,
+      });

    const applicationUniversalIdentifier = flatApplication.universalIdentifier;

@@ -85,6 +98,13 @@ export class UpdateLogicFunctionActionHandlerService extends WorkspaceMigrationR
      update as Parameters<typeof logicFunctionRepository.update>[1],
    );

+    await this.installPrebuiltBundleIfNeeded({
+      existingLogicFunction,
+      update,
+      applicationUniversalIdentifier,
+      context,
+    });
+
    if (builtPathChanged) {
      await this.fileStorageService.deleteFile({
        workspaceId,
@@ -94,4 +114,71 @@ export class UpdateLogicFunctionActionHandlerService extends WorkspaceMigrationR
      });
    }
  }
+
+  private async installPrebuiltBundleIfNeeded({
+    existingLogicFunction,
+    update,
+    applicationUniversalIdentifier,
+    context,
+  }: {
+    existingLogicFunction: FlatLogicFunction;
+    update: FlatUpdateLogicFunctionAction['update'];
+    applicationUniversalIdentifier: string;
+    context: WorkspaceMigrationActionRunnerContext<FlatUpdateLogicFunctionAction>;
+  }): Promise<void> {
+    const newLogicFunction: FlatLogicFunction = {
+      ...existingLogicFunction,
+      ...update,
+      executionMode:
+        update.executionMode ?? existingLogicFunction.executionMode,
+      isBuildUpToDate:
+        update.isBuildUpToDate ?? existingLogicFunction.isBuildUpToDate,
+      checksum: update.checksum ?? existingLogicFunction.checksum,
+    };
+
+    if (
+      !shouldReinstallLogicFunctionPrebuiltBundle({
+        existingLogicFunction,
+        newLogicFunction,
+      })
+    ) {
+      return;
+    }
+
+    const becamePrebuilt =
+      existingLogicFunction.executionMode !==
+      LogicFunctionExecutionMode.PREBUILT;
+
+    const driver = this.logicFunctionDriverFactory.getCurrentDriver();
+
+    const installStart = Date.now();
+
+    try {
+      await driver.installPrebuiltBundle({
+        flatLogicFunction: {
+          ...newLogicFunction,
+          executionMode: LogicFunctionExecutionMode.PREBUILT,
+          isBuildUpToDate: true,
+        },
+        flatApplication: context.flatApplication,
+        applicationUniversalIdentifier,
+      });
+
+      this.logger.log(
+        `[lambda-timing] event=install_prebuilt fnId=${existingLogicFunction.id} ` +
+          `reason=${becamePrebuilt ? 'mode_flip' : 'checksum_change'} ` +
+          `install_duration_ms=${Date.now() - installStart}`,
+        UpdateLogicFunctionActionHandlerService.name,
+      );
+    } catch (error) {
+      this.logger.error(
+        `Failed to install prebuilt bundle for function ${existingLogicFunction.id} ` +
+          `after ${Date.now() - installStart}ms: ` +
+          `${error instanceof Error ? error.message : String(error)}`,
+        UpdateLogicFunctionActionHandlerService.name,
+        error instanceof Error ? error.stack : undefined,
+      );
+      throw error;
+    }
+  }
 }