Refactor twenty client sdk provisioning for logic function and front-component (#18544)
## 1. The `twenty-client-sdk` Package (Source of Truth)
The monorepo package at `packages/twenty-client-sdk` ships with:
- A **pre-built metadata client** (static, generated from a fixed
schema)
- A **stub core client** that throws at runtime (`CoreApiClient was not
generated...`)
- Both ESM (`.mjs`) and CJS (`.cjs`) bundles in `dist/`
- A `package.json` with proper `exports` map for
`twenty-client-sdk/core`, `twenty-client-sdk/metadata`, and
`twenty-client-sdk/generate`
## 2. Generation & Upload (Server-Side, at Migration Time)
**When**: `WorkspaceMigrationRunnerService.run()` executes after a
metadata schema change.
**What happens in `SdkClientGenerationService.generateAndStore()`**:
1. Copies the stub `twenty-client-sdk` package from the server's assets
(resolved via `SDK_CLIENT_PACKAGE_DIRNAME` — from
`dist/assets/twenty-client-sdk/` in production, or from `node_modules`
in dev)
2. Filters out `node_modules/` and `src/` during copy — only
`package.json` + `dist/` are kept (like an npm publish)
3. Calls `replaceCoreClient()` which uses `@genql/cli` to introspect the
**application-scoped** GraphQL schema and generates a real
`CoreApiClient`, then compiles it to ESM+CJS and overwrites
`dist/core.mjs` and `dist/core.cjs`
4. Archives the **entire package** (with `package.json` + `dist/`) into
`twenty-client-sdk.zip`
5. Uploads the single archive to S3 under
`FileFolder.GeneratedSdkClient`
6. Sets `isSdkLayerStale = true` on the `ApplicationEntity` in the
database
## 3. Invalidation Signal
The `isSdkLayerStale` boolean column on `ApplicationEntity` is the
invalidation mechanism:
- **Set to `true`** by `generateAndStore()` after uploading a new client
archive
- **Checked** by both logic function drivers before execution — if
`true`, they rebuild their local layer
- **Set back to `false`** by `markSdkLayerFresh()` after the driver has
successfully consumed the new archive
Default is `false` so existing applications without a generated client
aren't affected.
## 4a. Logic Functions — Local Driver
**`ensureSdkLayer()`** is called before every execution:
1. Checks if the local SDK layer directory exists AND `isSdkLayerStale`
is `false` → early return
2. Otherwise, cleans the local layer directory
3. Calls `downloadAndExtractToPackage()` which streams the zip from S3
directly to disk and extracts the full package into
`<tmpdir>/sdk/<workspaceId>-<appId>/node_modules/twenty-client-sdk/`
4. Calls `markSdkLayerFresh()` to set `isSdkLayerStale = false`
**At execution time**, `assembleNodeModules()` symlinks everything from
the deps layer's `node_modules/` **except** `twenty-client-sdk`, which
is symlinked from the SDK layer instead. This ensures the logic
function's `import ... from 'twenty-client-sdk/core'` resolves to the
generated client.
## 4b. Logic Functions — Lambda Driver
**`ensureSdkLayer()`** is called during `build()`:
1. Checks if `isSdkLayerStale` is `false` and an existing Lambda layer
ARN exists → early return
2. Otherwise, deletes all existing layer versions for this SDK layer
name
3. Calls `downloadArchiveBuffer()` to get the raw zip from S3 (no disk
extraction)
4. Calls `reprefixZipEntries()` which streams the zip entries into a
**new zip** with the path prefix
`nodejs/node_modules/twenty-client-sdk/` — this is the Lambda layer
convention path. All done in memory, no disk round-trip
5. Publishes the re-prefixed zip as a new Lambda layer via
`publishLayer()`
6. Calls `markSdkLayerFresh()`
**At function creation**, the Lambda is created with **two layers**:
`[depsLayerArn, sdkLayerArn]`. The SDK layer is listed last so it
overwrites the stub `twenty-client-sdk` from the deps layer (later
layers take precedence in Lambda's `/opt` merge).
## 5. Front Components
Front components are built by `app:build` with `twenty-client-sdk/core`
and `twenty-client-sdk/metadata` as **esbuild externals**. The stored
`.mjs` in S3 has unresolved bare import specifiers like `import {
CoreApiClient } from 'twenty-client-sdk/core'`.
SDK import resolution is split between the **frontend host** (fetching &
caching SDK modules) and the **Web Worker** (rewriting imports):
**Server endpoints**:
- `GET /rest/front-components/:id` —
`FrontComponentService.getBuiltComponentStream()` returns the **raw
`.mjs`** directly from file storage. No bundling, no SDK injection.
- `GET /rest/sdk-client/:applicationId/:moduleName` —
`SdkClientController` reads a single file (e.g. `dist/core.mjs`) from
the generated SDK archive via
`SdkClientGenerationService.readFileFromArchive()` and serves it as
JavaScript.
**Frontend host** (`FrontComponentRenderer` in `twenty-front`):
1. Queries `FindOneFrontComponent` which returns `applicationId`,
`builtComponentChecksum`, `usesSdkClient`, and `applicationTokenPair`
2. If `usesSdkClient` is `true`, renders
`FrontComponentRendererWithSdkClient` which calls the
`useApplicationSdkClient` hook
3. `useApplicationSdkClient({ applicationId, accessToken })` checks the
Jotai atom family cache for existing blob URLs. On cache miss, fetches
both SDK modules from `GET /rest/sdk-client/:applicationId/core` and
`/metadata`, creates **blob URLs** for each, and stores them in the atom
family
4. Once the blob URLs are cached, passes them as `sdkClientUrls`
(already blob URLs, not server URLs) to `SharedFrontComponentRenderer` →
`FrontComponentWorkerEffect` → worker's `render()` call via
`HostToWorkerRenderContext`
**Worker** (`remote-worker.ts` in `twenty-sdk`):
1. Fetches the raw component `.mjs` source as text
2. If `sdkClientUrls` are provided and the source contains SDK import
specifiers (`twenty-client-sdk/core`, `twenty-client-sdk/metadata`),
**rewrites** the bare specifiers to the blob URLs received from the host
(e.g. `'twenty-client-sdk/core'` → `'blob:...'`)
3. Creates a blob URL for the rewritten source and `import()`s it
4. Revokes only the component blob URL after the module is loaded — the
SDK blob URLs are owned and managed by the host's Jotai cache
This approach eliminates server-side esbuild bundling on every request,
caches SDK modules per application in the frontend, and keeps the
worker's job to a simple string rewrite.
## Summary Diagram
```
app:build (SDK)
└─ twenty-client-sdk stub (metadata=real, core=stub)
│
▼
WorkspaceMigrationRunnerService.run()
└─ SdkClientGenerationService.generateAndStore()
├─ Copy stub package (package.json + dist/)
├─ replaceCoreClient() → regenerate core.mjs/core.cjs
├─ Zip entire package → upload to S3
└─ Set isSdkLayerStale = true
│
┌────────┴────────────────────┐
▼ ▼
Logic Functions Front Components
│ │
├─ Local Driver ├─ GET /rest/sdk-client/:appId/core
│ └─ downloadAndExtract │ → core.mjs from archive
│ → symlink into │
│ node_modules ├─ Host (useApplicationSdkClient)
│ │ ├─ Fetch SDK modules
└─ Lambda Driver │ ├─ Create blob URLs
└─ downloadArchiveBuffer │ └─ Cache in Jotai atom family
→ reprefixZipEntries │
→ publish as Lambda ├─ GET /rest/front-components/:id
layer │ → raw .mjs (no bundling)
│
└─ Worker (browser)
├─ Fetch component .mjs
├─ Rewrite imports → blob URLs
└─ import() rewritten source
```
## Next PR
- Estimate perf improvement by implementing a redis caching for front
component client storage ( we don't even cache front comp initially )
- Implem frontent blob invalidation sse event from server
---------
Co-authored-by: Charles Bochet <charlesBochet@users.noreply.github.com>
This commit is contained in:
+226
-33
@@ -6,6 +6,7 @@ import {
|
||||
CreateFunctionCommand,
|
||||
type CreateFunctionCommandInput,
|
||||
DeleteFunctionCommand,
|
||||
DeleteLayerVersionCommand,
|
||||
GetFunctionCommand,
|
||||
InvokeCommand,
|
||||
type InvokeCommandInput,
|
||||
@@ -24,9 +25,9 @@ import { AssumeRoleCommand, STSClient } from '@aws-sdk/client-sts';
|
||||
import { isDefined } from 'twenty-shared/utils';
|
||||
|
||||
import {
|
||||
type LogicFunctionDriver,
|
||||
type LogicFunctionExecuteParams,
|
||||
type LogicFunctionExecuteResult,
|
||||
type LogicFunctionDriver,
|
||||
type LogicFunctionTranspileParams,
|
||||
type LogicFunctionTranspileResult,
|
||||
} from 'src/engine/core-modules/logic-function/logic-function-drivers/interfaces/logic-function-driver.interface';
|
||||
@@ -40,6 +41,9 @@ import { copyExecutor } from 'src/engine/core-modules/logic-function/logic-funct
|
||||
import { copyYarnInstall } from 'src/engine/core-modules/logic-function/logic-function-drivers/utils/copy-yarn-install';
|
||||
import { createZipFile } from 'src/engine/core-modules/logic-function/logic-function-drivers/utils/create-zip-file';
|
||||
import { TemporaryDirManager } from 'src/engine/core-modules/logic-function/logic-function-drivers/utils/temporary-dir-manager';
|
||||
import { type LogicFunctionResourceService } from 'src/engine/core-modules/logic-function/logic-function-resource/logic-function-resource.service';
|
||||
import { type SdkClientArchiveService } from 'src/engine/core-modules/sdk-client/sdk-client-archive.service';
|
||||
import { callWithTimeout } from 'src/engine/core-modules/logic-function/logic-function-drivers/utils/call-with-timeout';
|
||||
import { LogicFunctionExecutionStatus } from 'src/engine/metadata-modules/logic-function/dtos/logic-function-execution-result.dto';
|
||||
import { LogicFunctionRuntime } from 'src/engine/metadata-modules/logic-function/logic-function.entity';
|
||||
import {
|
||||
@@ -47,8 +51,6 @@ import {
|
||||
LogicFunctionExceptionCode,
|
||||
} from 'src/engine/metadata-modules/logic-function/logic-function.exception';
|
||||
import { type FlatLogicFunction } from 'src/engine/metadata-modules/logic-function/types/flat-logic-function.type';
|
||||
import { type LogicFunctionResourceService } from 'src/engine/core-modules/logic-function/logic-function-resource/logic-function-resource.service';
|
||||
import { callWithTimeout } from 'src/engine/core-modules/logic-function/logic-function-drivers/utils/call-with-timeout';
|
||||
|
||||
const UPDATE_FUNCTION_DURATION_TIMEOUT_IN_SECONDS = 60;
|
||||
const CREDENTIALS_DURATION_IN_SECONDS = 60 * 60; // 1h
|
||||
@@ -105,6 +107,7 @@ export type BuilderLambdaResult = {
|
||||
|
||||
export interface LambdaDriverOptions extends LambdaClientConfig {
|
||||
logicFunctionResourceService: LogicFunctionResourceService;
|
||||
sdkClientArchiveService: SdkClientArchiveService;
|
||||
region: string;
|
||||
lambdaRole: string;
|
||||
subhostingRole?: string;
|
||||
@@ -120,11 +123,13 @@ export class LambdaDriver implements LogicFunctionDriver {
|
||||
private credentialsExpiry: Date | null = null;
|
||||
private readonly options: LambdaDriverOptions;
|
||||
private readonly logicFunctionResourceService: LogicFunctionResourceService;
|
||||
private readonly sdkClientArchiveService: SdkClientArchiveService;
|
||||
|
||||
constructor(options: LambdaDriverOptions) {
|
||||
this.options = options;
|
||||
this.lambdaClient = undefined;
|
||||
this.logicFunctionResourceService = options.logicFunctionResourceService;
|
||||
this.sdkClientArchiveService = options.sdkClientArchiveService;
|
||||
}
|
||||
|
||||
private areAssumeRoleCredentialsExpired(): boolean {
|
||||
@@ -223,8 +228,20 @@ export class LambdaDriver implements LogicFunctionDriver {
|
||||
);
|
||||
}
|
||||
|
||||
private getLayerName(flatApplication: FlatApplication) {
|
||||
return flatApplication.yarnLockChecksum ?? 'default';
|
||||
private getDepsLayerName(flatApplication: FlatApplication): string {
|
||||
const checksum = flatApplication.yarnLockChecksum ?? 'default';
|
||||
|
||||
return `deps-${checksum}`;
|
||||
}
|
||||
|
||||
private getSdkLayerName({
|
||||
workspaceId,
|
||||
applicationUniversalIdentifier,
|
||||
}: {
|
||||
workspaceId: string;
|
||||
applicationUniversalIdentifier: string;
|
||||
}): string {
|
||||
return `sdk-${workspaceId}-${applicationUniversalIdentifier}`;
|
||||
}
|
||||
|
||||
private yarnInstallFunctionName: string | undefined;
|
||||
@@ -538,6 +555,33 @@ export class LambdaDriver implements LogicFunctionDriver {
|
||||
return listLayerResult.LayerVersions?.[0]?.LayerVersionArn;
|
||||
}
|
||||
|
||||
private async publishLayer({
|
||||
layerName,
|
||||
zipBuffer,
|
||||
}: {
|
||||
layerName: string;
|
||||
zipBuffer: Buffer;
|
||||
}): Promise<string> {
|
||||
const result = await (
|
||||
await this.getLambdaClient()
|
||||
).send(
|
||||
new PublishLayerVersionCommand({
|
||||
LayerName: layerName,
|
||||
Content: { ZipFile: zipBuffer },
|
||||
CompatibleRuntimes: [
|
||||
LogicFunctionRuntime.NODE18,
|
||||
LogicFunctionRuntime.NODE22,
|
||||
],
|
||||
}),
|
||||
);
|
||||
|
||||
if (!isDefined(result.LayerVersionArn)) {
|
||||
throw new Error('New layer version ARN is undefined');
|
||||
}
|
||||
|
||||
return result.LayerVersionArn;
|
||||
}
|
||||
|
||||
private async getDependencyContents(
|
||||
flatApplication: FlatApplication,
|
||||
applicationUniversalIdentifier: string,
|
||||
@@ -570,7 +614,7 @@ export class LambdaDriver implements LogicFunctionDriver {
|
||||
flatApplication: FlatApplication;
|
||||
applicationUniversalIdentifier: string;
|
||||
}): Promise<void> {
|
||||
const layerName = this.getLayerName(flatApplication);
|
||||
const layerName = this.getDepsLayerName(flatApplication);
|
||||
|
||||
const existingArn = await this.getExistingLayerArn(layerName);
|
||||
|
||||
@@ -625,7 +669,7 @@ export class LambdaDriver implements LogicFunctionDriver {
|
||||
flatApplication: FlatApplication;
|
||||
applicationUniversalIdentifier: string;
|
||||
}): Promise<string> {
|
||||
const layerName = this.getLayerName(flatApplication);
|
||||
const layerName = this.getDepsLayerName(flatApplication);
|
||||
|
||||
const existingArn = await this.getExistingLayerArn(layerName);
|
||||
|
||||
@@ -649,6 +693,89 @@ export class LambdaDriver implements LogicFunctionDriver {
|
||||
return newArn;
|
||||
}
|
||||
|
||||
private async ensureSdkLayer({
|
||||
flatApplication,
|
||||
applicationUniversalIdentifier,
|
||||
}: {
|
||||
flatApplication: FlatApplication;
|
||||
applicationUniversalIdentifier: string;
|
||||
}): Promise<string> {
|
||||
const layerName = this.getSdkLayerName({
|
||||
workspaceId: flatApplication.workspaceId,
|
||||
applicationUniversalIdentifier,
|
||||
});
|
||||
|
||||
if (!flatApplication.isSdkLayerStale) {
|
||||
const existingArn = await this.getExistingLayerArn(layerName);
|
||||
|
||||
if (isDefined(existingArn)) {
|
||||
return existingArn;
|
||||
}
|
||||
}
|
||||
|
||||
await this.deleteAllLayerVersions({
|
||||
lambdaClient: await this.getLambdaClient(),
|
||||
layerName,
|
||||
});
|
||||
|
||||
const sdkArchiveBuffer =
|
||||
await this.sdkClientArchiveService.downloadArchiveBuffer({
|
||||
workspaceId: flatApplication.workspaceId,
|
||||
applicationUniversalIdentifier,
|
||||
});
|
||||
|
||||
const zipBuffer = await this.reprefixZipEntries({
|
||||
sourceBuffer: sdkArchiveBuffer,
|
||||
prefix: 'nodejs/node_modules/twenty-client-sdk',
|
||||
});
|
||||
|
||||
const arn = await this.publishLayer({ layerName, zipBuffer });
|
||||
|
||||
await this.sdkClientArchiveService.markSdkLayerFresh({
|
||||
applicationId: flatApplication.id,
|
||||
workspaceId: flatApplication.workspaceId,
|
||||
});
|
||||
|
||||
return arn;
|
||||
}
|
||||
|
||||
// Re-wraps zip entries under a new prefix path without extracting to disk.
|
||||
private async reprefixZipEntries({
|
||||
sourceBuffer,
|
||||
prefix,
|
||||
}: {
|
||||
sourceBuffer: Buffer;
|
||||
prefix: string;
|
||||
}): Promise<Buffer> {
|
||||
const { default: unzipper } = await import('unzipper');
|
||||
const archiver = (await import('archiver')).default;
|
||||
|
||||
const directory = await unzipper.Open.buffer(sourceBuffer);
|
||||
const archive = archiver('zip', { zlib: { level: 9 } });
|
||||
|
||||
const chunks: Buffer[] = [];
|
||||
|
||||
archive.on('data', (chunk: Buffer) => chunks.push(chunk));
|
||||
|
||||
for (const entry of directory.files) {
|
||||
if (entry.type === 'Directory') {
|
||||
continue;
|
||||
}
|
||||
|
||||
archive.append(entry.stream(), {
|
||||
name: `${prefix}/${entry.path}`,
|
||||
});
|
||||
}
|
||||
|
||||
await new Promise<void>((resolve, reject) => {
|
||||
archive.on('end', resolve);
|
||||
archive.on('error', reject);
|
||||
archive.finalize();
|
||||
});
|
||||
|
||||
return Buffer.concat(chunks);
|
||||
}
|
||||
|
||||
private async getLambdaExecutor(flatLogicFunction: FlatLogicFunction) {
|
||||
try {
|
||||
const getFunctionCommand: GetFunctionCommand = new GetFunctionCommand({
|
||||
@@ -675,10 +802,50 @@ export class LambdaDriver implements LogicFunctionDriver {
|
||||
}
|
||||
}
|
||||
|
||||
private async isAlreadyBuilt(
|
||||
flatLogicFunction: FlatLogicFunction,
|
||||
flatApplication: FlatApplication,
|
||||
) {
|
||||
private async deleteAllLayerVersions({
|
||||
lambdaClient,
|
||||
layerName,
|
||||
}: {
|
||||
lambdaClient: Lambda;
|
||||
layerName: string;
|
||||
}): Promise<void> {
|
||||
let marker: string | undefined;
|
||||
|
||||
do {
|
||||
const listResult = await lambdaClient.send(
|
||||
new ListLayerVersionsCommand({
|
||||
LayerName: layerName,
|
||||
MaxItems: 50,
|
||||
Marker: marker,
|
||||
}),
|
||||
);
|
||||
|
||||
const versions = listResult.LayerVersions ?? [];
|
||||
|
||||
await Promise.all(
|
||||
versions.map((version) =>
|
||||
lambdaClient.send(
|
||||
new DeleteLayerVersionCommand({
|
||||
LayerName: layerName,
|
||||
VersionNumber: version.Version,
|
||||
}),
|
||||
),
|
||||
),
|
||||
);
|
||||
|
||||
marker = listResult.NextMarker;
|
||||
} while (isDefined(marker));
|
||||
}
|
||||
|
||||
private async isAlreadyBuilt({
|
||||
flatLogicFunction,
|
||||
flatApplication,
|
||||
applicationUniversalIdentifier,
|
||||
}: {
|
||||
flatLogicFunction: FlatLogicFunction;
|
||||
flatApplication: FlatApplication;
|
||||
applicationUniversalIdentifier: string;
|
||||
}) {
|
||||
const lambdaExecutor = await this.getLambdaExecutor(flatLogicFunction);
|
||||
|
||||
if (!isDefined(lambdaExecutor)) {
|
||||
@@ -687,15 +854,23 @@ export class LambdaDriver implements LogicFunctionDriver {
|
||||
|
||||
const layers = lambdaExecutor.Configuration?.Layers;
|
||||
|
||||
if (!isDefined(layers) || layers.length !== 1) {
|
||||
if (!isDefined(layers) || layers.length !== 2) {
|
||||
await this.delete(flatLogicFunction);
|
||||
|
||||
return false;
|
||||
}
|
||||
|
||||
const layerName = this.getLayerName(flatApplication);
|
||||
const depsLayerName = this.getDepsLayerName(flatApplication);
|
||||
const sdkLayerName = this.getSdkLayerName({
|
||||
workspaceId: flatApplication.workspaceId,
|
||||
applicationUniversalIdentifier,
|
||||
});
|
||||
|
||||
if (layers[0].Arn?.includes(layerName)) {
|
||||
const hasExpectedLayers =
|
||||
layers.some((layer) => layer.Arn?.includes(depsLayerName)) &&
|
||||
layers.some((layer) => layer.Arn?.includes(sdkLayerName));
|
||||
|
||||
if (hasExpectedLayers) {
|
||||
return true;
|
||||
}
|
||||
|
||||
@@ -713,11 +888,25 @@ export class LambdaDriver implements LogicFunctionDriver {
|
||||
flatApplication: FlatApplication;
|
||||
applicationUniversalIdentifier: string;
|
||||
}) {
|
||||
if (await this.isAlreadyBuilt(flatLogicFunction, flatApplication)) {
|
||||
if (
|
||||
!flatApplication.isSdkLayerStale &&
|
||||
(await this.isAlreadyBuilt({
|
||||
flatLogicFunction,
|
||||
flatApplication,
|
||||
applicationUniversalIdentifier,
|
||||
}))
|
||||
) {
|
||||
return;
|
||||
}
|
||||
|
||||
const layerArn = await this.getLayerArn({
|
||||
await this.delete(flatLogicFunction);
|
||||
|
||||
const depsLayerArn = await this.getLayerArn({
|
||||
flatApplication,
|
||||
applicationUniversalIdentifier,
|
||||
});
|
||||
|
||||
const sdkLayerArn = await this.ensureSdkLayer({
|
||||
flatApplication,
|
||||
applicationUniversalIdentifier,
|
||||
});
|
||||
@@ -727,27 +916,31 @@ export class LambdaDriver implements LogicFunctionDriver {
|
||||
const { sourceTemporaryDir, lambdaZipPath } =
|
||||
await temporaryDirManager.init();
|
||||
|
||||
await copyExecutor(sourceTemporaryDir);
|
||||
try {
|
||||
await copyExecutor(sourceTemporaryDir);
|
||||
|
||||
await createZipFile(sourceTemporaryDir, lambdaZipPath);
|
||||
await createZipFile(sourceTemporaryDir, lambdaZipPath);
|
||||
|
||||
const params: CreateFunctionCommandInput = {
|
||||
Code: {
|
||||
ZipFile: await fs.readFile(lambdaZipPath),
|
||||
},
|
||||
FunctionName: flatLogicFunction.id,
|
||||
Layers: [layerArn],
|
||||
Handler: 'index.handler',
|
||||
Role: this.options.lambdaRole,
|
||||
Runtime: flatLogicFunction.runtime,
|
||||
Timeout: 900, // timeout is handled by the logic function service
|
||||
};
|
||||
// SDK layer listed last so it overwrites the stub twenty-client-sdk
|
||||
// from the deps layer (later layers take precedence in /opt merge).
|
||||
const params: CreateFunctionCommandInput = {
|
||||
Code: {
|
||||
ZipFile: await fs.readFile(lambdaZipPath),
|
||||
},
|
||||
FunctionName: flatLogicFunction.id,
|
||||
Layers: [depsLayerArn, sdkLayerArn],
|
||||
Handler: 'index.handler',
|
||||
Role: this.options.lambdaRole,
|
||||
Runtime: flatLogicFunction.runtime,
|
||||
Timeout: 900,
|
||||
};
|
||||
|
||||
const command = new CreateFunctionCommand(params);
|
||||
const command = new CreateFunctionCommand(params);
|
||||
|
||||
await (await this.getLambdaClient()).send(command);
|
||||
|
||||
await temporaryDirManager.clean();
|
||||
await (await this.getLambdaClient()).send(command);
|
||||
} finally {
|
||||
await temporaryDirManager.clean();
|
||||
}
|
||||
}
|
||||
|
||||
private extractLogs(logString: string): string {
|
||||
|
||||
+138
-30
@@ -21,23 +21,41 @@ import { HANDLER_NAME_REGEX } from 'src/engine/metadata-modules/logic-function/c
|
||||
import { LogicFunctionExecutionStatus } from 'src/engine/metadata-modules/logic-function/dtos/logic-function-execution-result.dto';
|
||||
import { copyYarnEngineAndBuildDependencies } from 'src/engine/core-modules/application/application-package/utils/copy-yarn-engine-and-build-dependencies';
|
||||
import type { LogicFunctionResourceService } from 'src/engine/core-modules/logic-function/logic-function-resource/logic-function-resource.service';
|
||||
import type { SdkClientArchiveService } from 'src/engine/core-modules/sdk-client/sdk-client-archive.service';
|
||||
|
||||
export interface LocalDriverOptions {
|
||||
logicFunctionResourceService: LogicFunctionResourceService;
|
||||
sdkClientArchiveService: SdkClientArchiveService;
|
||||
}
|
||||
|
||||
export class LocalDriver implements LogicFunctionDriver {
|
||||
private readonly logicFunctionResourceService: LogicFunctionResourceService;
|
||||
private readonly sdkClientArchiveService: SdkClientArchiveService;
|
||||
|
||||
constructor(options: LocalDriverOptions) {
|
||||
this.logicFunctionResourceService = options.logicFunctionResourceService;
|
||||
this.sdkClientArchiveService = options.sdkClientArchiveService;
|
||||
}
|
||||
|
||||
private getInMemoryLayerFolderPath = (flatApplication: FlatApplication) => {
|
||||
private getDepsLayerPath(flatApplication: FlatApplication): string {
|
||||
const checksum = flatApplication.yarnLockChecksum ?? 'default';
|
||||
|
||||
return join(LOGIC_FUNCTION_EXECUTOR_TMPDIR_FOLDER, checksum);
|
||||
};
|
||||
return join(LOGIC_FUNCTION_EXECUTOR_TMPDIR_FOLDER, 'deps', checksum);
|
||||
}
|
||||
|
||||
private getSdkLayerPath({
|
||||
workspaceId,
|
||||
applicationUniversalIdentifier,
|
||||
}: {
|
||||
workspaceId: string;
|
||||
applicationUniversalIdentifier: string;
|
||||
}): string {
|
||||
return join(
|
||||
LOGIC_FUNCTION_EXECUTOR_TMPDIR_FOLDER,
|
||||
'sdk',
|
||||
`${workspaceId}-${applicationUniversalIdentifier}`,
|
||||
);
|
||||
}
|
||||
|
||||
private async createLayerIfNotExist({
|
||||
flatApplication,
|
||||
@@ -45,20 +63,64 @@ export class LocalDriver implements LogicFunctionDriver {
|
||||
}: {
|
||||
flatApplication: FlatApplication;
|
||||
applicationUniversalIdentifier: string;
|
||||
}) {
|
||||
const inMemoryLayerFolderPath =
|
||||
this.getInMemoryLayerFolderPath(flatApplication);
|
||||
}): Promise<void> {
|
||||
const depsLayerPath = this.getDepsLayerPath(flatApplication);
|
||||
|
||||
try {
|
||||
await fs.access(inMemoryLayerFolderPath);
|
||||
await fs.access(depsLayerPath);
|
||||
|
||||
return;
|
||||
} catch {
|
||||
await this.logicFunctionResourceService.copyDependenciesInMemory({
|
||||
applicationUniversalIdentifier,
|
||||
workspaceId: flatApplication.workspaceId,
|
||||
inMemoryFolderPath: inMemoryLayerFolderPath,
|
||||
});
|
||||
await copyYarnEngineAndBuildDependencies(inMemoryLayerFolderPath);
|
||||
// Layer doesn't exist yet
|
||||
}
|
||||
|
||||
await this.logicFunctionResourceService.copyDependenciesInMemory({
|
||||
applicationUniversalIdentifier,
|
||||
workspaceId: flatApplication.workspaceId,
|
||||
inMemoryFolderPath: depsLayerPath,
|
||||
});
|
||||
await copyYarnEngineAndBuildDependencies(depsLayerPath);
|
||||
}
|
||||
|
||||
private async ensureSdkLayer({
|
||||
flatApplication,
|
||||
applicationUniversalIdentifier,
|
||||
}: {
|
||||
flatApplication: FlatApplication;
|
||||
applicationUniversalIdentifier: string;
|
||||
}): Promise<void> {
|
||||
const sdkLayerPath = this.getSdkLayerPath({
|
||||
workspaceId: flatApplication.workspaceId,
|
||||
applicationUniversalIdentifier,
|
||||
});
|
||||
|
||||
const layerExists = await fs
|
||||
.access(sdkLayerPath)
|
||||
.then(() => true)
|
||||
.catch(() => false);
|
||||
|
||||
if (layerExists && !flatApplication.isSdkLayerStale) {
|
||||
return;
|
||||
}
|
||||
|
||||
await fs.rm(sdkLayerPath, { recursive: true, force: true });
|
||||
|
||||
const sdkPackagePath = join(
|
||||
sdkLayerPath,
|
||||
'node_modules',
|
||||
'twenty-client-sdk',
|
||||
);
|
||||
|
||||
await this.sdkClientArchiveService.downloadAndExtractToPackage({
|
||||
workspaceId: flatApplication.workspaceId,
|
||||
applicationUniversalIdentifier,
|
||||
targetPackagePath: sdkPackagePath,
|
||||
});
|
||||
|
||||
await this.sdkClientArchiveService.markSdkLayerFresh({
|
||||
applicationId: flatApplication.id,
|
||||
workspaceId: flatApplication.workspaceId,
|
||||
});
|
||||
}
|
||||
|
||||
async transpile({
|
||||
@@ -110,6 +172,59 @@ export class LocalDriver implements LogicFunctionDriver {
|
||||
flatApplication,
|
||||
applicationUniversalIdentifier,
|
||||
});
|
||||
await this.ensureSdkLayer({
|
||||
flatApplication,
|
||||
applicationUniversalIdentifier,
|
||||
});
|
||||
}
|
||||
|
||||
// Symlinks everything from the deps layer except twenty-client-sdk,
|
||||
// which comes from the SDK layer (workspace-specific generated client).
|
||||
private async assembleNodeModules({
|
||||
sourceTemporaryDir,
|
||||
flatApplication,
|
||||
applicationUniversalIdentifier,
|
||||
}: {
|
||||
sourceTemporaryDir: string;
|
||||
flatApplication: FlatApplication;
|
||||
applicationUniversalIdentifier: string;
|
||||
}): Promise<void> {
|
||||
const depsNodeModules = join(
|
||||
this.getDepsLayerPath(flatApplication),
|
||||
'node_modules',
|
||||
);
|
||||
const sdkNodeModules = join(
|
||||
this.getSdkLayerPath({
|
||||
workspaceId: flatApplication.workspaceId,
|
||||
applicationUniversalIdentifier,
|
||||
}),
|
||||
'node_modules',
|
||||
);
|
||||
const execNodeModules = join(sourceTemporaryDir, 'node_modules');
|
||||
|
||||
await fs.mkdir(execNodeModules, { recursive: true });
|
||||
|
||||
const entries = await fs.readdir(depsNodeModules, {
|
||||
withFileTypes: true,
|
||||
});
|
||||
|
||||
const symlinkPromises = entries
|
||||
.filter((entry) => entry.name !== 'twenty-client-sdk')
|
||||
.map((entry) =>
|
||||
fs.symlink(
|
||||
join(depsNodeModules, entry.name),
|
||||
join(execNodeModules, entry.name),
|
||||
entry.isDirectory() ? 'dir' : 'file',
|
||||
),
|
||||
);
|
||||
|
||||
await Promise.all(symlinkPromises);
|
||||
|
||||
await fs.symlink(
|
||||
join(sdkNodeModules, 'twenty-client-sdk'),
|
||||
join(execNodeModules, 'twenty-client-sdk'),
|
||||
'dir',
|
||||
);
|
||||
}
|
||||
|
||||
async execute({
|
||||
@@ -124,6 +239,10 @@ export class LocalDriver implements LogicFunctionDriver {
|
||||
flatApplication,
|
||||
applicationUniversalIdentifier,
|
||||
});
|
||||
await this.ensureSdkLayer({
|
||||
flatApplication,
|
||||
applicationUniversalIdentifier,
|
||||
});
|
||||
|
||||
const startTime = Date.now();
|
||||
|
||||
@@ -140,20 +259,11 @@ export class LocalDriver implements LogicFunctionDriver {
|
||||
inMemoryDestinationPath: sourceTemporaryDir,
|
||||
});
|
||||
|
||||
try {
|
||||
await fs.symlink(
|
||||
join(
|
||||
this.getInMemoryLayerFolderPath(flatApplication),
|
||||
'node_modules',
|
||||
),
|
||||
join(sourceTemporaryDir, 'node_modules'),
|
||||
'dir',
|
||||
);
|
||||
} catch (err) {
|
||||
if (err.code !== 'EEXIST') {
|
||||
throw err;
|
||||
}
|
||||
}
|
||||
await this.assembleNodeModules({
|
||||
sourceTemporaryDir,
|
||||
flatApplication,
|
||||
applicationUniversalIdentifier,
|
||||
});
|
||||
|
||||
let logs = '';
|
||||
|
||||
@@ -169,7 +279,7 @@ export class LocalDriver implements LogicFunctionDriver {
|
||||
(_key, value) => {
|
||||
if (typeof value === 'object' && value !== null) {
|
||||
if (seen.has(value)) {
|
||||
return '[Circular]'; // Handle circular references
|
||||
return '[Circular]';
|
||||
}
|
||||
seen.add(value);
|
||||
}
|
||||
@@ -374,7 +484,6 @@ export class LocalDriver implements LogicFunctionDriver {
|
||||
if (settled) return;
|
||||
settled = true;
|
||||
if (code === 0) {
|
||||
// Fallback path if no IPC (shouldn't happen with our stdio)
|
||||
resolve({ ok: true, stdout, stderr });
|
||||
} else {
|
||||
resolve({
|
||||
@@ -398,7 +507,6 @@ export class LocalDriver implements LogicFunctionDriver {
|
||||
});
|
||||
}, timeoutMs);
|
||||
|
||||
// Kick it off
|
||||
child.send?.({ type: 'run', payload });
|
||||
|
||||
child.on('close', () => clearTimeout(t));
|
||||
|
||||
+4
@@ -11,6 +11,7 @@ import { DisabledDriver } from 'src/engine/core-modules/logic-function/logic-fun
|
||||
import { LambdaDriver } from 'src/engine/core-modules/logic-function/logic-function-drivers/drivers/lambda.driver';
|
||||
import { LocalDriver } from 'src/engine/core-modules/logic-function/logic-function-drivers/drivers/local.driver';
|
||||
import { LogicFunctionResourceService } from 'src/engine/core-modules/logic-function/logic-function-resource/logic-function-resource.service';
|
||||
import { SdkClientArchiveService } from 'src/engine/core-modules/sdk-client/sdk-client-archive.service';
|
||||
import { DriverFactoryBase } from 'src/engine/core-modules/twenty-config/dynamic-factory.base';
|
||||
import { ConfigVariablesGroup } from 'src/engine/core-modules/twenty-config/enums/config-variables-group.enum';
|
||||
import { TwentyConfigService } from 'src/engine/core-modules/twenty-config/twenty-config.service';
|
||||
@@ -20,6 +21,7 @@ export class LogicFunctionDriverFactory extends DriverFactoryBase<LogicFunctionD
|
||||
constructor(
|
||||
twentyConfigService: TwentyConfigService,
|
||||
private readonly logicFunctionResourceService: LogicFunctionResourceService,
|
||||
private readonly sdkClientArchiveService: SdkClientArchiveService,
|
||||
) {
|
||||
super(twentyConfigService);
|
||||
}
|
||||
@@ -44,6 +46,7 @@ export class LogicFunctionDriverFactory extends DriverFactoryBase<LogicFunctionD
|
||||
case LogicFunctionDriverType.LOCAL:
|
||||
return new LocalDriver({
|
||||
logicFunctionResourceService: this.logicFunctionResourceService,
|
||||
sdkClientArchiveService: this.sdkClientArchiveService,
|
||||
});
|
||||
|
||||
case LogicFunctionDriverType.LAMBDA: {
|
||||
@@ -82,6 +85,7 @@ export class LogicFunctionDriverFactory extends DriverFactoryBase<LogicFunctionD
|
||||
subhostingRole,
|
||||
layerBucket,
|
||||
layerBucketRegion,
|
||||
sdkClientArchiveService: this.sdkClientArchiveService,
|
||||
});
|
||||
}
|
||||
|
||||
|
||||
@@ -4,6 +4,7 @@ import { LogicFunctionDriverFactory } from 'src/engine/core-modules/logic-functi
|
||||
import { LogicFunctionResourceModule } from 'src/engine/core-modules/logic-function/logic-function-resource/logic-function-resource.module';
|
||||
import { LogicFunctionTriggerModule } from 'src/engine/core-modules/logic-function/logic-function-trigger/logic-function-trigger.module';
|
||||
import { LogicFunctionExecutorModule } from 'src/engine/core-modules/logic-function/logic-function-executor/logic-function-executor.module';
|
||||
import { SdkClientModule } from 'src/engine/core-modules/sdk-client/sdk-client.module';
|
||||
import { TwentyConfigModule } from 'src/engine/core-modules/twenty-config/twenty-config.module';
|
||||
|
||||
@Global()
|
||||
@@ -17,6 +18,7 @@ export class LogicFunctionModule {
|
||||
LogicFunctionResourceModule,
|
||||
LogicFunctionTriggerModule,
|
||||
LogicFunctionExecutorModule,
|
||||
SdkClientModule,
|
||||
],
|
||||
providers: [LogicFunctionDriverFactory],
|
||||
exports: [
|
||||
|
||||
Reference in New Issue
Block a user