Refactor twenty client sdk provisioning for logic function and front-component (#18544)

## 1. The `twenty-client-sdk` Package (Source of Truth)

The monorepo package at `packages/twenty-client-sdk` ships with:
- A **pre-built metadata client** (static, generated from a fixed
schema)
- A **stub core client** that throws at runtime (`CoreApiClient was not
generated...`)
- Both ESM (`.mjs`) and CJS (`.cjs`) bundles in `dist/`
- A `package.json` with proper `exports` map for
`twenty-client-sdk/core`, `twenty-client-sdk/metadata`, and
`twenty-client-sdk/generate`

## 2. Generation & Upload (Server-Side, at Migration Time)

**When**: `WorkspaceMigrationRunnerService.run()` executes after a
metadata schema change.

**What happens in `SdkClientGenerationService.generateAndStore()`**:
1. Copies the stub `twenty-client-sdk` package from the server's assets
(resolved via `SDK_CLIENT_PACKAGE_DIRNAME` — from
`dist/assets/twenty-client-sdk/` in production, or from `node_modules`
in dev)
2. Filters out `node_modules/` and `src/` during copy — only
`package.json` + `dist/` are kept (like an npm publish)
3. Calls `replaceCoreClient()` which uses `@genql/cli` to introspect the
**application-scoped** GraphQL schema and generates a real
`CoreApiClient`, then compiles it to ESM+CJS and overwrites
`dist/core.mjs` and `dist/core.cjs`
4. Archives the **entire package** (with `package.json` + `dist/`) into
`twenty-client-sdk.zip`
5. Uploads the single archive to S3 under
`FileFolder.GeneratedSdkClient`
6. Sets `isSdkLayerStale = true` on the `ApplicationEntity` in the
database

## 3. Invalidation Signal

The `isSdkLayerStale` boolean column on `ApplicationEntity` is the
invalidation mechanism:
- **Set to `true`** by `generateAndStore()` after uploading a new client
archive
- **Checked** by both logic function drivers before execution — if
`true`, they rebuild their local layer
- **Set back to `false`** by `markSdkLayerFresh()` after the driver has
successfully consumed the new archive

Default is `false` so existing applications without a generated client
aren't affected.

## 4a. Logic Functions — Local Driver

**`ensureSdkLayer()`** is called before every execution:
1. Checks if the local SDK layer directory exists AND `isSdkLayerStale`
is `false` → early return
2. Otherwise, cleans the local layer directory
3. Calls `downloadAndExtractToPackage()` which streams the zip from S3
directly to disk and extracts the full package into
`<tmpdir>/sdk/<workspaceId>-<appId>/node_modules/twenty-client-sdk/`
4. Calls `markSdkLayerFresh()` to set `isSdkLayerStale = false`

**At execution time**, `assembleNodeModules()` symlinks everything from
the deps layer's `node_modules/` **except** `twenty-client-sdk`, which
is symlinked from the SDK layer instead. This ensures the logic
function's `import ... from 'twenty-client-sdk/core'` resolves to the
generated client.

## 4b. Logic Functions — Lambda Driver

**`ensureSdkLayer()`** is called during `build()`:
1. Checks if `isSdkLayerStale` is `false` and an existing Lambda layer
ARN exists → early return
2. Otherwise, deletes all existing layer versions for this SDK layer
name
3. Calls `downloadArchiveBuffer()` to get the raw zip from S3 (no disk
extraction)
4. Calls `reprefixZipEntries()` which streams the zip entries into a
**new zip** with the path prefix
`nodejs/node_modules/twenty-client-sdk/` — this is the Lambda layer
convention path. All done in memory, no disk round-trip
5. Publishes the re-prefixed zip as a new Lambda layer via
`publishLayer()`
6. Calls `markSdkLayerFresh()`

**At function creation**, the Lambda is created with **two layers**:
`[depsLayerArn, sdkLayerArn]`. The SDK layer is listed last so it
overwrites the stub `twenty-client-sdk` from the deps layer (later
layers take precedence in Lambda's `/opt` merge).

## 5. Front Components

Front components are built by `app:build` with `twenty-client-sdk/core`
and `twenty-client-sdk/metadata` as **esbuild externals**. The stored
`.mjs` in S3 has unresolved bare import specifiers like `import {
CoreApiClient } from 'twenty-client-sdk/core'`.

SDK import resolution is split between the **frontend host** (fetching &
caching SDK modules) and the **Web Worker** (rewriting imports):

**Server endpoints**:
- `GET /rest/front-components/:id` —
`FrontComponentService.getBuiltComponentStream()` returns the **raw
`.mjs`** directly from file storage. No bundling, no SDK injection.
- `GET /rest/sdk-client/:applicationId/:moduleName` —
`SdkClientController` reads a single file (e.g. `dist/core.mjs`) from
the generated SDK archive via
`SdkClientGenerationService.readFileFromArchive()` and serves it as
JavaScript.

**Frontend host** (`FrontComponentRenderer` in `twenty-front`):
1. Queries `FindOneFrontComponent` which returns `applicationId`,
`builtComponentChecksum`, `usesSdkClient`, and `applicationTokenPair`
2. If `usesSdkClient` is `true`, renders
`FrontComponentRendererWithSdkClient` which calls the
`useApplicationSdkClient` hook
3. `useApplicationSdkClient({ applicationId, accessToken })` checks the
Jotai atom family cache for existing blob URLs. On cache miss, fetches
both SDK modules from `GET /rest/sdk-client/:applicationId/core` and
`/metadata`, creates **blob URLs** for each, and stores them in the atom
family
4. Once the blob URLs are cached, passes them as `sdkClientUrls`
(already blob URLs, not server URLs) to `SharedFrontComponentRenderer` →
`FrontComponentWorkerEffect` → worker's `render()` call via
`HostToWorkerRenderContext`

**Worker** (`remote-worker.ts` in `twenty-sdk`):
1. Fetches the raw component `.mjs` source as text
2. If `sdkClientUrls` are provided and the source contains SDK import
specifiers (`twenty-client-sdk/core`, `twenty-client-sdk/metadata`),
**rewrites** the bare specifiers to the blob URLs received from the host
(e.g. `'twenty-client-sdk/core'` → `'blob:...'`)
3. Creates a blob URL for the rewritten source and `import()`s it
4. Revokes only the component blob URL after the module is loaded — the
SDK blob URLs are owned and managed by the host's Jotai cache

This approach eliminates server-side esbuild bundling on every request,
caches SDK modules per application in the frontend, and keeps the
worker's job to a simple string rewrite.

## Summary Diagram

```
app:build (SDK)
  └─ twenty-client-sdk stub (metadata=real, core=stub)
       │
       ▼
WorkspaceMigrationRunnerService.run()
  └─ SdkClientGenerationService.generateAndStore()
       ├─ Copy stub package (package.json + dist/)
       ├─ replaceCoreClient() → regenerate core.mjs/core.cjs
       ├─ Zip entire package → upload to S3
       └─ Set isSdkLayerStale = true
              │
     ┌────────┴────────────────────┐
     ▼                             ▼
Logic Functions               Front Components
     │                             │
     ├─ Local Driver               ├─ GET /rest/sdk-client/:appId/core
     │   └─ downloadAndExtract     │    → core.mjs from archive
     │      → symlink into         │
     │        node_modules         ├─ Host (useApplicationSdkClient)
     │                             │    ├─ Fetch SDK modules
     └─ Lambda Driver              │    ├─ Create blob URLs
         └─ downloadArchiveBuffer  │    └─ Cache in Jotai atom family
            → reprefixZipEntries   │
            → publish as Lambda    ├─ GET /rest/front-components/:id
              layer                │    → raw .mjs (no bundling)
                                   │
                                   └─ Worker (browser)
                                        ├─ Fetch component .mjs
                                        ├─ Rewrite imports → blob URLs
                                        └─ import() rewritten source
```

## Next PR
- Estimate perf improvement by implementing a redis caching for front
component client storage ( we don't even cache front comp initially )
- Implem frontent blob invalidation sse event from server

---------

Co-authored-by: Charles Bochet <charlesBochet@users.noreply.github.com>
This commit is contained in:
Paul Rastoin
2026-03-24 19:10:25 +01:00
committed by GitHub
parent 16451ee2ee
commit 4ea2e32366
189 changed files with 12487 additions and 9856 deletions
@@ -7,6 +7,7 @@ import { ApplicationDevelopmentResolver } from 'src/engine/core-modules/applicat
import { TokenModule } from 'src/engine/core-modules/auth/token/token.module';
import { FeatureFlagModule } from 'src/engine/core-modules/feature-flag/feature-flag.module';
import { FileStorageModule } from 'src/engine/core-modules/file-storage/file-storage.module';
import { SdkClientModule } from 'src/engine/core-modules/sdk-client/sdk-client.module';
import { PermissionsModule } from 'src/engine/metadata-modules/permissions/permissions.module';
import { WorkspaceMigrationGraphqlApiExceptionInterceptor } from 'src/engine/workspace-manager/workspace-migration/interceptors/workspace-migration-graphql-api-exception.interceptor';
@@ -16,6 +17,7 @@ import { WorkspaceMigrationGraphqlApiExceptionInterceptor } from 'src/engine/wor
ApplicationManifestModule,
ApplicationRegistrationModule,
FeatureFlagModule,
SdkClientModule,
TokenModule,
FileStorageModule,
PermissionsModule,
@@ -8,32 +8,34 @@ import { Args, Mutation } from '@nestjs/graphql';
import GraphQLUpload from 'graphql-upload/GraphQLUpload.mjs';
import { PermissionFlagType } from 'twenty-shared/constants';
import { FileFolder, FeatureFlagKey } from 'twenty-shared/types';
import { FeatureFlagKey, FileFolder } from 'twenty-shared/types';
import { isDefined } from 'twenty-shared/utils';
import type { FileUpload } from 'graphql-upload/processRequest.mjs';
import { ApplicationRegistrationVariableService } from 'src/engine/core-modules/application/application-registration-variable/application-registration-variable.service';
import { ApplicationRegistrationService } from 'src/engine/core-modules/application/application-registration/application-registration.service';
import { MetadataResolver } from 'src/engine/api/graphql/graphql-config/decorators/metadata-resolver.decorator';
import { ApplicationExceptionFilter } from 'src/engine/core-modules/application/application-exception-filter';
import {
ApplicationException,
ApplicationExceptionCode,
} from 'src/engine/core-modules/application/application.exception';
import { ApplicationRegistrationSourceType } from 'src/engine/core-modules/application/application-registration/enums/application-registration-source-type.enum';
import { ApplicationSyncService } from 'src/engine/core-modules/application/application-manifest/application-sync.service';
import { ApplicationService } from 'src/engine/core-modules/application/application.service';
import { ApplicationInput } from 'src/engine/core-modules/application/application-development/dtos/application.input';
import { CreateDevelopmentApplicationInput } from 'src/engine/core-modules/application/application-development/dtos/create-development-application.input';
import { DevelopmentApplicationDTO } from 'src/engine/core-modules/application/application-development/dtos/development-application.dto';
import { GenerateApplicationTokenInput } from 'src/engine/core-modules/application/application-development/dtos/generate-application-token.input';
import { UploadApplicationFileInput } from 'src/engine/core-modules/application/application-development/dtos/upload-application-file.input';
import { WorkspaceMigrationDTO } from 'src/engine/core-modules/application/application-development/dtos/workspace-migration.dto';
import { ApplicationExceptionFilter } from 'src/engine/core-modules/application/application-exception-filter';
import { ApplicationSyncService } from 'src/engine/core-modules/application/application-manifest/application-sync.service';
import { ApplicationTokenPairDTO } from 'src/engine/core-modules/application/application-oauth/dtos/application-token-pair.dto';
import { ApplicationRegistrationVariableService } from 'src/engine/core-modules/application/application-registration-variable/application-registration-variable.service';
import { ApplicationRegistrationService } from 'src/engine/core-modules/application/application-registration/application-registration.service';
import { ApplicationRegistrationSourceType } from 'src/engine/core-modules/application/application-registration/enums/application-registration-source-type.enum';
import {
ApplicationException,
ApplicationExceptionCode,
} from 'src/engine/core-modules/application/application.exception';
import { ApplicationService } from 'src/engine/core-modules/application/application.service';
import { ApplicationTokenService } from 'src/engine/core-modules/auth/token/services/application-token.service';
import { FileStorageService } from 'src/engine/core-modules/file-storage/file-storage.service';
import { FileDTO } from 'src/engine/core-modules/file/dtos/file.dto';
import { ResolverValidationPipe } from 'src/engine/core-modules/graphql/pipes/resolver-validation.pipe';
import { SdkClientGenerationService } from 'src/engine/core-modules/sdk-client/sdk-client-generation.service';
import { type WorkspaceEntity } from 'src/engine/core-modules/workspace/workspace.entity';
import { AuthUser } from 'src/engine/decorators/auth/auth-user.decorator';
import { AuthUserWorkspaceId } from 'src/engine/decorators/auth/auth-user-workspace-id.decorator';
@@ -66,6 +68,7 @@ export class ApplicationDevelopmentResolver {
private readonly applicationRegistrationService: ApplicationRegistrationService,
private readonly applicationRegistrationVariableService: ApplicationRegistrationVariableService,
private readonly fileStorageService: FileStorageService,
private readonly sdkClientGenerationService: SdkClientGenerationService,
) {}
@Mutation(() => DevelopmentApplicationDTO)
@@ -133,13 +136,38 @@ export class ApplicationDevelopmentResolver {
workspaceId,
);
const workspaceMigration =
const application = await this.applicationService.findByUniversalIdentifier(
{
universalIdentifier: manifest.application.universalIdentifier,
workspaceId,
},
);
if (!isDefined(application)) {
throw new ApplicationException(
`Application "${manifest.application.universalIdentifier}" not found in workspace "${workspaceId}". Run createDevelopmentApplication first.`,
ApplicationExceptionCode.APPLICATION_NOT_FOUND,
);
}
const isFirstSync = !isDefined(application.version);
const { workspaceMigration, hasSchemaMetadataChanged } =
await this.applicationSyncService.synchronizeFromManifest({
workspaceId,
manifest,
applicationRegistrationId,
});
if (isFirstSync || hasSchemaMetadataChanged) {
await this.sdkClientGenerationService.generateSdkClientForApplication({
workspaceId,
applicationId: application.id,
applicationUniversalIdentifier:
manifest.application.universalIdentifier,
});
}
await this.syncRegistrationMetadata(
applicationRegistrationId,
manifest,
@@ -10,6 +10,7 @@ import { ApplicationPackageModule } from 'src/engine/core-modules/application/ap
import { ApplicationInstallResolver } from 'src/engine/core-modules/application/application-install/application-install.resolver';
import { ApplicationInstallService } from 'src/engine/core-modules/application/application-install/application-install.service';
import { FileStorageModule } from 'src/engine/core-modules/file-storage/file-storage.module';
import { SdkClientModule } from 'src/engine/core-modules/sdk-client/sdk-client.module';
import { PermissionsModule } from 'src/engine/metadata-modules/permissions/permissions.module';
@Module({
@@ -20,6 +21,7 @@ import { PermissionsModule } from 'src/engine/metadata-modules/permissions/permi
ApplicationPackageModule,
CacheLockModule,
FeatureFlagModule,
SdkClientModule,
PermissionsModule,
FileStorageModule,
],
@@ -24,7 +24,7 @@ import {
import { ApplicationSyncService } from 'src/engine/core-modules/application/application-manifest/application-sync.service';
import { CacheLockService } from 'src/engine/core-modules/cache-lock/cache-lock.service';
import { FileStorageService } from 'src/engine/core-modules/file-storage/file-storage.service';
import { SdkClientGenerationService } from 'src/engine/core-modules/sdk-client/sdk-client-generation.service';
@Injectable()
export class ApplicationInstallService {
private readonly logger = new Logger(ApplicationInstallService.name);
@@ -37,6 +37,7 @@ export class ApplicationInstallService {
private readonly applicationSyncService: ApplicationSyncService,
private readonly fileStorageService: FileStorageService,
private readonly cacheLockService: CacheLockService,
private readonly sdkClientGenerationService: SdkClientGenerationService,
) {}
async installApplication(params: {
@@ -108,7 +109,7 @@ export class ApplicationInstallService {
const universalIdentifier = appRegistration.universalIdentifier;
await this.ensureApplicationExists({
const { application, wasCreated } = await this.ensureApplicationExists({
universalIdentifier,
name: resolvedPackage.manifest.application.displayName,
workspaceId: params.workspaceId,
@@ -123,11 +124,20 @@ export class ApplicationInstallService {
params.workspaceId,
);
await this.applicationSyncService.synchronizeFromManifest({
workspaceId: params.workspaceId,
manifest: resolvedPackage.manifest,
applicationRegistrationId: appRegistration.id,
});
const { hasSchemaMetadataChanged } =
await this.applicationSyncService.synchronizeFromManifest({
workspaceId: params.workspaceId,
manifest: resolvedPackage.manifest,
applicationRegistrationId: appRegistration.id,
});
if (wasCreated || hasSchemaMetadataChanged) {
await this.sdkClientGenerationService.generateSdkClientForApplication({
workspaceId: params.workspaceId,
applicationId: application.id,
applicationUniversalIdentifier: universalIdentifier,
});
}
this.logger.log(
`Successfully installed app ${universalIdentifier} v${resolvedPackage.packageJson.version ?? 'unknown'}`,
@@ -232,17 +242,17 @@ export class ApplicationInstallService {
workspaceId: string;
applicationRegistrationId: string;
sourceType: ApplicationRegistrationSourceType;
}): Promise<ApplicationEntity> {
}): Promise<{ application: ApplicationEntity; wasCreated: boolean }> {
const existing = await this.applicationService.findByUniversalIdentifier({
universalIdentifier: params.universalIdentifier,
workspaceId: params.workspaceId,
});
if (isDefined(existing)) {
return existing;
return { application: existing, wasCreated: false };
}
return this.applicationService.create({
const application = await this.applicationService.create({
universalIdentifier: params.universalIdentifier,
name: params.name,
sourcePath: params.universalIdentifier,
@@ -250,5 +260,7 @@ export class ApplicationInstallService {
applicationRegistrationId: params.applicationRegistrationId,
workspaceId: params.workspaceId,
});
return { application, wasCreated: true };
}
}
@@ -47,7 +47,10 @@ export class ApplicationManifestMigrationService {
manifest: Manifest;
workspaceId: string;
ownerFlatApplication: FlatApplication;
}): Promise<WorkspaceMigration> {
}): Promise<{
workspaceMigration: WorkspaceMigration;
hasSchemaMetadataChanged: boolean;
}> {
const now = new Date().toISOString();
const { twentyStandardFlatApplication } =
@@ -122,7 +125,10 @@ export class ApplicationManifestMigrationService {
ownerFlatApplication,
});
return validateAndBuildResult.workspaceMigration;
return {
workspaceMigration: validateAndBuildResult.workspaceMigration,
hasSchemaMetadataChanged: validateAndBuildResult.hasSchemaMetadataChanged,
};
}
/**
@@ -47,7 +47,10 @@ export class ApplicationSyncService {
workspaceId: string;
manifest: Manifest;
applicationRegistrationId?: string;
}): Promise<WorkspaceMigration> {
}): Promise<{
workspaceMigration: WorkspaceMigration;
hasSchemaMetadataChanged: boolean;
}> {
const application = await this.syncApplication({
workspaceId,
manifest,
@@ -56,7 +59,7 @@ export class ApplicationSyncService {
const ownerFlatApplication: FlatApplication = application;
const workspaceMigration =
const syncResult =
await this.applicationManifestMigrationService.syncMetadataFromManifest({
manifest,
workspaceId,
@@ -65,7 +68,7 @@ export class ApplicationSyncService {
this.logger.log('Application sync from manifest completed');
return workspaceMigration;
return syncResult;
}
private async syncApplication({
@@ -21,6 +21,7 @@ export const fromFrontComponentManifestToUniversalFlatFrontComponent = ({
componentName: frontComponentManifest.componentName,
builtComponentChecksum: frontComponentManifest.builtComponentChecksum,
isHeadless: frontComponentManifest.isHeadless ?? false,
usesSdkClient: frontComponentManifest.usesSdkClient ?? false,
createdAt: now,
updatedAt: now,
};
@@ -97,6 +97,9 @@ export class ApplicationEntity extends WorkspaceRelatedEntity {
@Column({ nullable: false, type: 'boolean', default: true })
canBeUninstalled: boolean;
@Column({ nullable: false, type: 'boolean', default: false })
isSdkLayerStale: boolean;
@Column({ nullable: true, type: 'uuid' })
applicationRegistrationId: string | null;
@@ -1,7 +1,6 @@
import { Module } from '@nestjs/common';
import { TypeOrmModule } from '@nestjs/typeorm';
import { TypeORMModule } from 'src/database/typeorm/typeorm.module';
import { ApiKeyEntity } from 'src/engine/core-modules/api-key/api-key.entity';
import { ApiKeyModule } from 'src/engine/core-modules/api-key/api-key.module';
import { ApplicationRegistrationModule } from 'src/engine/core-modules/application/application-registration/application-registration.module';
@@ -58,18 +57,13 @@ import { UserEntity } from 'src/engine/core-modules/user/user.entity';
import { UserModule } from 'src/engine/core-modules/user/user.module';
import { WorkspaceInvitationModule } from 'src/engine/core-modules/workspace-invitation/workspace-invitation.module';
import { WorkspaceEntity } from 'src/engine/core-modules/workspace/workspace.entity';
import { WorkspaceModule } from 'src/engine/core-modules/workspace/workspace.module';
import { DataSourceModule } from 'src/engine/metadata-modules/data-source/data-source.module';
import { CalendarChannelDataAccessModule } from 'src/engine/metadata-modules/calendar-channel/data-access/calendar-channel-data-access.module';
import { ConnectedAccountEntity } from 'src/engine/metadata-modules/connected-account/entities/connected-account.entity';
import { ConnectedAccountDataAccessModule } from 'src/engine/metadata-modules/connected-account/data-access/connected-account-data-access.module';
import { MessageChannelDataAccessModule } from 'src/engine/metadata-modules/message-channel/data-access/message-channel-data-access.module';
import { ObjectMetadataEntity } from 'src/engine/metadata-modules/object-metadata/object-metadata.entity';
import { PermissionsModule } from 'src/engine/metadata-modules/permissions/permissions.module';
import { UserRoleModule } from 'src/engine/metadata-modules/user-role/user-role.module';
import { WorkspaceCacheModule } from 'src/engine/workspace-cache/workspace-cache.module';
import { WorkspaceDataSourceModule } from 'src/engine/workspace-datasource/workspace-datasource.module';
import { WorkspaceManagerModule } from 'src/engine/workspace-manager/workspace-manager.module';
import { CalendarChannelSyncStatusService } from 'src/modules/calendar/common/services/calendar-channel-sync-status.service';
import { ConnectedAccountModule } from 'src/modules/connected-account/connected-account.module';
import { MessagingCommonModule } from 'src/modules/messaging/common/messaging-common.module';
@@ -83,12 +77,9 @@ import { JwtAuthStrategy } from './strategies/jwt.auth.strategy';
@Module({
imports: [
JwtModule,
DataSourceModule,
WorkspaceDomainsModule,
TokenModule,
UserModule,
WorkspaceManagerModule,
TypeORMModule,
TypeOrmModule.forFeature([
WorkspaceEntity,
UserEntity,
@@ -104,9 +95,7 @@ import { JwtAuthStrategy } from './strategies/jwt.auth.strategy';
ConnectedAccountEntity,
]),
UserWorkspaceModule,
WorkspaceModule,
OnboardingModule,
WorkspaceDataSourceModule,
ConnectedAccountModule,
MessagingCommonModule,
MessagingFolderSyncManagerModule,
@@ -120,7 +109,6 @@ import { JwtAuthStrategy } from './strategies/jwt.auth.strategy';
GuardRedirectModule,
MetricsModule,
PermissionsModule,
UserRoleModule,
TwoFactorAuthenticationModule,
ApiKeyModule,
AuditModule,
@@ -7,19 +7,17 @@ import { ActorModule } from 'src/engine/core-modules/actor/actor.module';
import { AdminPanelModule } from 'src/engine/core-modules/admin-panel/admin-panel.module';
import { ApiKeyModule } from 'src/engine/core-modules/api-key/api-key.module';
import { AppTokenModule } from 'src/engine/core-modules/app-token/app-token.module';
import { ApplicationOAuthModule } from 'src/engine/core-modules/application/application-oauth/application-oauth.module';
import { ApplicationRegistrationModule } from 'src/engine/core-modules/application/application-registration/application-registration.module';
import { ApplicationDevelopmentModule } from 'src/engine/core-modules/application/application-development/application-development.module';
import { ApplicationInstallModule } from 'src/engine/core-modules/application/application-install/application-install.module';
import { MarketplaceModule } from 'src/engine/core-modules/application/application-marketplace/marketplace.module';
import { ApplicationOAuthModule } from 'src/engine/core-modules/application/application-oauth/application-oauth.module';
import { ApplicationRegistrationModule } from 'src/engine/core-modules/application/application-registration/application-registration.module';
import { ApplicationUpgradeModule } from 'src/engine/core-modules/application/application-upgrade/application-upgrade.module';
import { ApplicationModule } from 'src/engine/core-modules/application/application.module';
import { MarketplaceModule } from 'src/engine/core-modules/application/application-marketplace/marketplace.module';
import { EnvironmentModule } from 'src/engine/core-modules/environment/environment.module';
import { ApprovedAccessDomainModule } from 'src/engine/core-modules/approved-access-domain/approved-access-domain.module';
import { AuthModule } from 'src/engine/core-modules/auth/auth.module';
import { BillingWebhookModule } from 'src/engine/core-modules/billing-webhook/billing-webhook.module';
import { BillingModule } from 'src/engine/core-modules/billing/billing.module';
import { UsageModule } from 'src/engine/core-modules/usage/usage.module';
import { CacheStorageModule } from 'src/engine/core-modules/cache-storage/cache-storage.module';
import { TimelineCalendarEventModule } from 'src/engine/core-modules/calendar/timeline-calendar-event.module';
import { CaptchaModule } from 'src/engine/core-modules/captcha/captcha.module';
@@ -28,6 +26,7 @@ import { CodeInterpreterModule } from 'src/engine/core-modules/code-interpreter/
import { DnsManagerModule } from 'src/engine/core-modules/dns-manager/dns-manager.module';
import { EmailModule } from 'src/engine/core-modules/email/email.module';
import { EmailingDomainModule } from 'src/engine/core-modules/emailing-domain/emailing-domain.module';
import { EnvironmentModule } from 'src/engine/core-modules/environment/environment.module';
import { ExceptionHandlerModule } from 'src/engine/core-modules/exception-handler/exception-handler.module';
import { exceptionHandlerModuleFactory } from 'src/engine/core-modules/exception-handler/exception-handler.module-factory';
import { FeatureFlagModule } from 'src/engine/core-modules/feature-flag/feature-flag.module';
@@ -39,6 +38,7 @@ import { ImpersonationModule } from 'src/engine/core-modules/impersonation/imper
import { LabModule } from 'src/engine/core-modules/lab/lab.module';
import { LoggerModule } from 'src/engine/core-modules/logger/logger.module';
import { loggerModuleFactory } from 'src/engine/core-modules/logger/logger.module-factory';
import { LogicFunctionModule } from 'src/engine/core-modules/logic-function/logic-function.module';
import { MessageQueueModule } from 'src/engine/core-modules/message-queue/message-queue.module';
import { messageQueueModuleFactory } from 'src/engine/core-modules/message-queue/message-queue.module-factory';
import { TimelineMessagingModule } from 'src/engine/core-modules/messaging/timeline-messaging.module';
@@ -50,11 +50,11 @@ import { PublicDomainModule } from 'src/engine/core-modules/public-domain/public
import { RedisClientModule } from 'src/engine/core-modules/redis-client/redis-client.module';
import { RedisClientService } from 'src/engine/core-modules/redis-client/redis-client.service';
import { SearchModule } from 'src/engine/core-modules/search/search.module';
import { LogicFunctionModule } from 'src/engine/core-modules/logic-function/logic-function.module';
import { WorkspaceSSOModule } from 'src/engine/core-modules/sso/sso.module';
import { TelemetryModule } from 'src/engine/core-modules/telemetry/telemetry.module';
import { TwentyConfigModule } from 'src/engine/core-modules/twenty-config/twenty-config.module';
import { TwentyConfigService } from 'src/engine/core-modules/twenty-config/twenty-config.service';
import { UsageModule } from 'src/engine/core-modules/usage/usage.module';
import { UserModule } from 'src/engine/core-modules/user/user.module';
import { WorkflowApiModule } from 'src/engine/core-modules/workflow/workflow-api.module';
import { WorkspaceInvitationModule } from 'src/engine/core-modules/workspace-invitation/workspace-invitation.module';
@@ -57,6 +57,9 @@ export const fileFolderConfigs: Record<FileFolder, FileFolderConfig> = {
[FileFolder.AppTarball]: {
ignoreExpirationToken: false,
},
[FileFolder.GeneratedSdkClient]: {
ignoreExpirationToken: false,
},
};
export type AllowedFolders = KebabCase<keyof typeof FileFolder>;
@@ -6,6 +6,7 @@ import {
CreateFunctionCommand,
type CreateFunctionCommandInput,
DeleteFunctionCommand,
DeleteLayerVersionCommand,
GetFunctionCommand,
InvokeCommand,
type InvokeCommandInput,
@@ -24,9 +25,9 @@ import { AssumeRoleCommand, STSClient } from '@aws-sdk/client-sts';
import { isDefined } from 'twenty-shared/utils';
import {
type LogicFunctionDriver,
type LogicFunctionExecuteParams,
type LogicFunctionExecuteResult,
type LogicFunctionDriver,
type LogicFunctionTranspileParams,
type LogicFunctionTranspileResult,
} from 'src/engine/core-modules/logic-function/logic-function-drivers/interfaces/logic-function-driver.interface';
@@ -40,6 +41,9 @@ import { copyExecutor } from 'src/engine/core-modules/logic-function/logic-funct
import { copyYarnInstall } from 'src/engine/core-modules/logic-function/logic-function-drivers/utils/copy-yarn-install';
import { createZipFile } from 'src/engine/core-modules/logic-function/logic-function-drivers/utils/create-zip-file';
import { TemporaryDirManager } from 'src/engine/core-modules/logic-function/logic-function-drivers/utils/temporary-dir-manager';
import { type LogicFunctionResourceService } from 'src/engine/core-modules/logic-function/logic-function-resource/logic-function-resource.service';
import { type SdkClientArchiveService } from 'src/engine/core-modules/sdk-client/sdk-client-archive.service';
import { callWithTimeout } from 'src/engine/core-modules/logic-function/logic-function-drivers/utils/call-with-timeout';
import { LogicFunctionExecutionStatus } from 'src/engine/metadata-modules/logic-function/dtos/logic-function-execution-result.dto';
import { LogicFunctionRuntime } from 'src/engine/metadata-modules/logic-function/logic-function.entity';
import {
@@ -47,8 +51,6 @@ import {
LogicFunctionExceptionCode,
} from 'src/engine/metadata-modules/logic-function/logic-function.exception';
import { type FlatLogicFunction } from 'src/engine/metadata-modules/logic-function/types/flat-logic-function.type';
import { type LogicFunctionResourceService } from 'src/engine/core-modules/logic-function/logic-function-resource/logic-function-resource.service';
import { callWithTimeout } from 'src/engine/core-modules/logic-function/logic-function-drivers/utils/call-with-timeout';
const UPDATE_FUNCTION_DURATION_TIMEOUT_IN_SECONDS = 60;
const CREDENTIALS_DURATION_IN_SECONDS = 60 * 60; // 1h
@@ -105,6 +107,7 @@ export type BuilderLambdaResult = {
export interface LambdaDriverOptions extends LambdaClientConfig {
logicFunctionResourceService: LogicFunctionResourceService;
sdkClientArchiveService: SdkClientArchiveService;
region: string;
lambdaRole: string;
subhostingRole?: string;
@@ -120,11 +123,13 @@ export class LambdaDriver implements LogicFunctionDriver {
private credentialsExpiry: Date | null = null;
private readonly options: LambdaDriverOptions;
private readonly logicFunctionResourceService: LogicFunctionResourceService;
private readonly sdkClientArchiveService: SdkClientArchiveService;
constructor(options: LambdaDriverOptions) {
this.options = options;
this.lambdaClient = undefined;
this.logicFunctionResourceService = options.logicFunctionResourceService;
this.sdkClientArchiveService = options.sdkClientArchiveService;
}
private areAssumeRoleCredentialsExpired(): boolean {
@@ -223,8 +228,20 @@ export class LambdaDriver implements LogicFunctionDriver {
);
}
private getLayerName(flatApplication: FlatApplication) {
return flatApplication.yarnLockChecksum ?? 'default';
private getDepsLayerName(flatApplication: FlatApplication): string {
const checksum = flatApplication.yarnLockChecksum ?? 'default';
return `deps-${checksum}`;
}
private getSdkLayerName({
workspaceId,
applicationUniversalIdentifier,
}: {
workspaceId: string;
applicationUniversalIdentifier: string;
}): string {
return `sdk-${workspaceId}-${applicationUniversalIdentifier}`;
}
private yarnInstallFunctionName: string | undefined;
@@ -538,6 +555,33 @@ export class LambdaDriver implements LogicFunctionDriver {
return listLayerResult.LayerVersions?.[0]?.LayerVersionArn;
}
private async publishLayer({
layerName,
zipBuffer,
}: {
layerName: string;
zipBuffer: Buffer;
}): Promise<string> {
const result = await (
await this.getLambdaClient()
).send(
new PublishLayerVersionCommand({
LayerName: layerName,
Content: { ZipFile: zipBuffer },
CompatibleRuntimes: [
LogicFunctionRuntime.NODE18,
LogicFunctionRuntime.NODE22,
],
}),
);
if (!isDefined(result.LayerVersionArn)) {
throw new Error('New layer version ARN is undefined');
}
return result.LayerVersionArn;
}
private async getDependencyContents(
flatApplication: FlatApplication,
applicationUniversalIdentifier: string,
@@ -570,7 +614,7 @@ export class LambdaDriver implements LogicFunctionDriver {
flatApplication: FlatApplication;
applicationUniversalIdentifier: string;
}): Promise<void> {
const layerName = this.getLayerName(flatApplication);
const layerName = this.getDepsLayerName(flatApplication);
const existingArn = await this.getExistingLayerArn(layerName);
@@ -625,7 +669,7 @@ export class LambdaDriver implements LogicFunctionDriver {
flatApplication: FlatApplication;
applicationUniversalIdentifier: string;
}): Promise<string> {
const layerName = this.getLayerName(flatApplication);
const layerName = this.getDepsLayerName(flatApplication);
const existingArn = await this.getExistingLayerArn(layerName);
@@ -649,6 +693,89 @@ export class LambdaDriver implements LogicFunctionDriver {
return newArn;
}
private async ensureSdkLayer({
flatApplication,
applicationUniversalIdentifier,
}: {
flatApplication: FlatApplication;
applicationUniversalIdentifier: string;
}): Promise<string> {
const layerName = this.getSdkLayerName({
workspaceId: flatApplication.workspaceId,
applicationUniversalIdentifier,
});
if (!flatApplication.isSdkLayerStale) {
const existingArn = await this.getExistingLayerArn(layerName);
if (isDefined(existingArn)) {
return existingArn;
}
}
await this.deleteAllLayerVersions({
lambdaClient: await this.getLambdaClient(),
layerName,
});
const sdkArchiveBuffer =
await this.sdkClientArchiveService.downloadArchiveBuffer({
workspaceId: flatApplication.workspaceId,
applicationUniversalIdentifier,
});
const zipBuffer = await this.reprefixZipEntries({
sourceBuffer: sdkArchiveBuffer,
prefix: 'nodejs/node_modules/twenty-client-sdk',
});
const arn = await this.publishLayer({ layerName, zipBuffer });
await this.sdkClientArchiveService.markSdkLayerFresh({
applicationId: flatApplication.id,
workspaceId: flatApplication.workspaceId,
});
return arn;
}
// Re-wraps zip entries under a new prefix path without extracting to disk.
private async reprefixZipEntries({
sourceBuffer,
prefix,
}: {
sourceBuffer: Buffer;
prefix: string;
}): Promise<Buffer> {
const { default: unzipper } = await import('unzipper');
const archiver = (await import('archiver')).default;
const directory = await unzipper.Open.buffer(sourceBuffer);
const archive = archiver('zip', { zlib: { level: 9 } });
const chunks: Buffer[] = [];
archive.on('data', (chunk: Buffer) => chunks.push(chunk));
for (const entry of directory.files) {
if (entry.type === 'Directory') {
continue;
}
archive.append(entry.stream(), {
name: `${prefix}/${entry.path}`,
});
}
await new Promise<void>((resolve, reject) => {
archive.on('end', resolve);
archive.on('error', reject);
archive.finalize();
});
return Buffer.concat(chunks);
}
private async getLambdaExecutor(flatLogicFunction: FlatLogicFunction) {
try {
const getFunctionCommand: GetFunctionCommand = new GetFunctionCommand({
@@ -675,10 +802,50 @@ export class LambdaDriver implements LogicFunctionDriver {
}
}
private async isAlreadyBuilt(
flatLogicFunction: FlatLogicFunction,
flatApplication: FlatApplication,
) {
private async deleteAllLayerVersions({
lambdaClient,
layerName,
}: {
lambdaClient: Lambda;
layerName: string;
}): Promise<void> {
let marker: string | undefined;
do {
const listResult = await lambdaClient.send(
new ListLayerVersionsCommand({
LayerName: layerName,
MaxItems: 50,
Marker: marker,
}),
);
const versions = listResult.LayerVersions ?? [];
await Promise.all(
versions.map((version) =>
lambdaClient.send(
new DeleteLayerVersionCommand({
LayerName: layerName,
VersionNumber: version.Version,
}),
),
),
);
marker = listResult.NextMarker;
} while (isDefined(marker));
}
private async isAlreadyBuilt({
flatLogicFunction,
flatApplication,
applicationUniversalIdentifier,
}: {
flatLogicFunction: FlatLogicFunction;
flatApplication: FlatApplication;
applicationUniversalIdentifier: string;
}) {
const lambdaExecutor = await this.getLambdaExecutor(flatLogicFunction);
if (!isDefined(lambdaExecutor)) {
@@ -687,15 +854,23 @@ export class LambdaDriver implements LogicFunctionDriver {
const layers = lambdaExecutor.Configuration?.Layers;
if (!isDefined(layers) || layers.length !== 1) {
if (!isDefined(layers) || layers.length !== 2) {
await this.delete(flatLogicFunction);
return false;
}
const layerName = this.getLayerName(flatApplication);
const depsLayerName = this.getDepsLayerName(flatApplication);
const sdkLayerName = this.getSdkLayerName({
workspaceId: flatApplication.workspaceId,
applicationUniversalIdentifier,
});
if (layers[0].Arn?.includes(layerName)) {
const hasExpectedLayers =
layers.some((layer) => layer.Arn?.includes(depsLayerName)) &&
layers.some((layer) => layer.Arn?.includes(sdkLayerName));
if (hasExpectedLayers) {
return true;
}
@@ -713,11 +888,25 @@ export class LambdaDriver implements LogicFunctionDriver {
flatApplication: FlatApplication;
applicationUniversalIdentifier: string;
}) {
if (await this.isAlreadyBuilt(flatLogicFunction, flatApplication)) {
if (
!flatApplication.isSdkLayerStale &&
(await this.isAlreadyBuilt({
flatLogicFunction,
flatApplication,
applicationUniversalIdentifier,
}))
) {
return;
}
const layerArn = await this.getLayerArn({
await this.delete(flatLogicFunction);
const depsLayerArn = await this.getLayerArn({
flatApplication,
applicationUniversalIdentifier,
});
const sdkLayerArn = await this.ensureSdkLayer({
flatApplication,
applicationUniversalIdentifier,
});
@@ -727,27 +916,31 @@ export class LambdaDriver implements LogicFunctionDriver {
const { sourceTemporaryDir, lambdaZipPath } =
await temporaryDirManager.init();
await copyExecutor(sourceTemporaryDir);
try {
await copyExecutor(sourceTemporaryDir);
await createZipFile(sourceTemporaryDir, lambdaZipPath);
await createZipFile(sourceTemporaryDir, lambdaZipPath);
const params: CreateFunctionCommandInput = {
Code: {
ZipFile: await fs.readFile(lambdaZipPath),
},
FunctionName: flatLogicFunction.id,
Layers: [layerArn],
Handler: 'index.handler',
Role: this.options.lambdaRole,
Runtime: flatLogicFunction.runtime,
Timeout: 900, // timeout is handled by the logic function service
};
// SDK layer listed last so it overwrites the stub twenty-client-sdk
// from the deps layer (later layers take precedence in /opt merge).
const params: CreateFunctionCommandInput = {
Code: {
ZipFile: await fs.readFile(lambdaZipPath),
},
FunctionName: flatLogicFunction.id,
Layers: [depsLayerArn, sdkLayerArn],
Handler: 'index.handler',
Role: this.options.lambdaRole,
Runtime: flatLogicFunction.runtime,
Timeout: 900,
};
const command = new CreateFunctionCommand(params);
const command = new CreateFunctionCommand(params);
await (await this.getLambdaClient()).send(command);
await temporaryDirManager.clean();
await (await this.getLambdaClient()).send(command);
} finally {
await temporaryDirManager.clean();
}
}
private extractLogs(logString: string): string {
@@ -21,23 +21,41 @@ import { HANDLER_NAME_REGEX } from 'src/engine/metadata-modules/logic-function/c
import { LogicFunctionExecutionStatus } from 'src/engine/metadata-modules/logic-function/dtos/logic-function-execution-result.dto';
import { copyYarnEngineAndBuildDependencies } from 'src/engine/core-modules/application/application-package/utils/copy-yarn-engine-and-build-dependencies';
import type { LogicFunctionResourceService } from 'src/engine/core-modules/logic-function/logic-function-resource/logic-function-resource.service';
import type { SdkClientArchiveService } from 'src/engine/core-modules/sdk-client/sdk-client-archive.service';
export interface LocalDriverOptions {
logicFunctionResourceService: LogicFunctionResourceService;
sdkClientArchiveService: SdkClientArchiveService;
}
export class LocalDriver implements LogicFunctionDriver {
private readonly logicFunctionResourceService: LogicFunctionResourceService;
private readonly sdkClientArchiveService: SdkClientArchiveService;
constructor(options: LocalDriverOptions) {
this.logicFunctionResourceService = options.logicFunctionResourceService;
this.sdkClientArchiveService = options.sdkClientArchiveService;
}
private getInMemoryLayerFolderPath = (flatApplication: FlatApplication) => {
private getDepsLayerPath(flatApplication: FlatApplication): string {
const checksum = flatApplication.yarnLockChecksum ?? 'default';
return join(LOGIC_FUNCTION_EXECUTOR_TMPDIR_FOLDER, checksum);
};
return join(LOGIC_FUNCTION_EXECUTOR_TMPDIR_FOLDER, 'deps', checksum);
}
private getSdkLayerPath({
workspaceId,
applicationUniversalIdentifier,
}: {
workspaceId: string;
applicationUniversalIdentifier: string;
}): string {
return join(
LOGIC_FUNCTION_EXECUTOR_TMPDIR_FOLDER,
'sdk',
`${workspaceId}-${applicationUniversalIdentifier}`,
);
}
private async createLayerIfNotExist({
flatApplication,
@@ -45,20 +63,64 @@ export class LocalDriver implements LogicFunctionDriver {
}: {
flatApplication: FlatApplication;
applicationUniversalIdentifier: string;
}) {
const inMemoryLayerFolderPath =
this.getInMemoryLayerFolderPath(flatApplication);
}): Promise<void> {
const depsLayerPath = this.getDepsLayerPath(flatApplication);
try {
await fs.access(inMemoryLayerFolderPath);
await fs.access(depsLayerPath);
return;
} catch {
await this.logicFunctionResourceService.copyDependenciesInMemory({
applicationUniversalIdentifier,
workspaceId: flatApplication.workspaceId,
inMemoryFolderPath: inMemoryLayerFolderPath,
});
await copyYarnEngineAndBuildDependencies(inMemoryLayerFolderPath);
// Layer doesn't exist yet
}
await this.logicFunctionResourceService.copyDependenciesInMemory({
applicationUniversalIdentifier,
workspaceId: flatApplication.workspaceId,
inMemoryFolderPath: depsLayerPath,
});
await copyYarnEngineAndBuildDependencies(depsLayerPath);
}
private async ensureSdkLayer({
flatApplication,
applicationUniversalIdentifier,
}: {
flatApplication: FlatApplication;
applicationUniversalIdentifier: string;
}): Promise<void> {
const sdkLayerPath = this.getSdkLayerPath({
workspaceId: flatApplication.workspaceId,
applicationUniversalIdentifier,
});
const layerExists = await fs
.access(sdkLayerPath)
.then(() => true)
.catch(() => false);
if (layerExists && !flatApplication.isSdkLayerStale) {
return;
}
await fs.rm(sdkLayerPath, { recursive: true, force: true });
const sdkPackagePath = join(
sdkLayerPath,
'node_modules',
'twenty-client-sdk',
);
await this.sdkClientArchiveService.downloadAndExtractToPackage({
workspaceId: flatApplication.workspaceId,
applicationUniversalIdentifier,
targetPackagePath: sdkPackagePath,
});
await this.sdkClientArchiveService.markSdkLayerFresh({
applicationId: flatApplication.id,
workspaceId: flatApplication.workspaceId,
});
}
async transpile({
@@ -110,6 +172,59 @@ export class LocalDriver implements LogicFunctionDriver {
flatApplication,
applicationUniversalIdentifier,
});
await this.ensureSdkLayer({
flatApplication,
applicationUniversalIdentifier,
});
}
// Symlinks everything from the deps layer except twenty-client-sdk,
// which comes from the SDK layer (workspace-specific generated client).
private async assembleNodeModules({
sourceTemporaryDir,
flatApplication,
applicationUniversalIdentifier,
}: {
sourceTemporaryDir: string;
flatApplication: FlatApplication;
applicationUniversalIdentifier: string;
}): Promise<void> {
const depsNodeModules = join(
this.getDepsLayerPath(flatApplication),
'node_modules',
);
const sdkNodeModules = join(
this.getSdkLayerPath({
workspaceId: flatApplication.workspaceId,
applicationUniversalIdentifier,
}),
'node_modules',
);
const execNodeModules = join(sourceTemporaryDir, 'node_modules');
await fs.mkdir(execNodeModules, { recursive: true });
const entries = await fs.readdir(depsNodeModules, {
withFileTypes: true,
});
const symlinkPromises = entries
.filter((entry) => entry.name !== 'twenty-client-sdk')
.map((entry) =>
fs.symlink(
join(depsNodeModules, entry.name),
join(execNodeModules, entry.name),
entry.isDirectory() ? 'dir' : 'file',
),
);
await Promise.all(symlinkPromises);
await fs.symlink(
join(sdkNodeModules, 'twenty-client-sdk'),
join(execNodeModules, 'twenty-client-sdk'),
'dir',
);
}
async execute({
@@ -124,6 +239,10 @@ export class LocalDriver implements LogicFunctionDriver {
flatApplication,
applicationUniversalIdentifier,
});
await this.ensureSdkLayer({
flatApplication,
applicationUniversalIdentifier,
});
const startTime = Date.now();
@@ -140,20 +259,11 @@ export class LocalDriver implements LogicFunctionDriver {
inMemoryDestinationPath: sourceTemporaryDir,
});
try {
await fs.symlink(
join(
this.getInMemoryLayerFolderPath(flatApplication),
'node_modules',
),
join(sourceTemporaryDir, 'node_modules'),
'dir',
);
} catch (err) {
if (err.code !== 'EEXIST') {
throw err;
}
}
await this.assembleNodeModules({
sourceTemporaryDir,
flatApplication,
applicationUniversalIdentifier,
});
let logs = '';
@@ -169,7 +279,7 @@ export class LocalDriver implements LogicFunctionDriver {
(_key, value) => {
if (typeof value === 'object' && value !== null) {
if (seen.has(value)) {
return '[Circular]'; // Handle circular references
return '[Circular]';
}
seen.add(value);
}
@@ -374,7 +484,6 @@ export class LocalDriver implements LogicFunctionDriver {
if (settled) return;
settled = true;
if (code === 0) {
// Fallback path if no IPC (shouldn't happen with our stdio)
resolve({ ok: true, stdout, stderr });
} else {
resolve({
@@ -398,7 +507,6 @@ export class LocalDriver implements LogicFunctionDriver {
});
}, timeoutMs);
// Kick it off
child.send?.({ type: 'run', payload });
child.on('close', () => clearTimeout(t));
@@ -11,6 +11,7 @@ import { DisabledDriver } from 'src/engine/core-modules/logic-function/logic-fun
import { LambdaDriver } from 'src/engine/core-modules/logic-function/logic-function-drivers/drivers/lambda.driver';
import { LocalDriver } from 'src/engine/core-modules/logic-function/logic-function-drivers/drivers/local.driver';
import { LogicFunctionResourceService } from 'src/engine/core-modules/logic-function/logic-function-resource/logic-function-resource.service';
import { SdkClientArchiveService } from 'src/engine/core-modules/sdk-client/sdk-client-archive.service';
import { DriverFactoryBase } from 'src/engine/core-modules/twenty-config/dynamic-factory.base';
import { ConfigVariablesGroup } from 'src/engine/core-modules/twenty-config/enums/config-variables-group.enum';
import { TwentyConfigService } from 'src/engine/core-modules/twenty-config/twenty-config.service';
@@ -20,6 +21,7 @@ export class LogicFunctionDriverFactory extends DriverFactoryBase<LogicFunctionD
constructor(
twentyConfigService: TwentyConfigService,
private readonly logicFunctionResourceService: LogicFunctionResourceService,
private readonly sdkClientArchiveService: SdkClientArchiveService,
) {
super(twentyConfigService);
}
@@ -44,6 +46,7 @@ export class LogicFunctionDriverFactory extends DriverFactoryBase<LogicFunctionD
case LogicFunctionDriverType.LOCAL:
return new LocalDriver({
logicFunctionResourceService: this.logicFunctionResourceService,
sdkClientArchiveService: this.sdkClientArchiveService,
});
case LogicFunctionDriverType.LAMBDA: {
@@ -82,6 +85,7 @@ export class LogicFunctionDriverFactory extends DriverFactoryBase<LogicFunctionD
subhostingRole,
layerBucket,
layerBucketRegion,
sdkClientArchiveService: this.sdkClientArchiveService,
});
}
@@ -4,6 +4,7 @@ import { LogicFunctionDriverFactory } from 'src/engine/core-modules/logic-functi
import { LogicFunctionResourceModule } from 'src/engine/core-modules/logic-function/logic-function-resource/logic-function-resource.module';
import { LogicFunctionTriggerModule } from 'src/engine/core-modules/logic-function/logic-function-trigger/logic-function-trigger.module';
import { LogicFunctionExecutorModule } from 'src/engine/core-modules/logic-function/logic-function-executor/logic-function-executor.module';
import { SdkClientModule } from 'src/engine/core-modules/sdk-client/sdk-client.module';
import { TwentyConfigModule } from 'src/engine/core-modules/twenty-config/twenty-config.module';
@Global()
@@ -17,6 +18,7 @@ export class LogicFunctionModule {
LogicFunctionResourceModule,
LogicFunctionTriggerModule,
LogicFunctionExecutorModule,
SdkClientModule,
],
providers: [LogicFunctionDriverFactory],
exports: [
@@ -0,0 +1,3 @@
export const ALLOWED_SDK_MODULES = ['core', 'metadata'] as const;
export type SdkModuleName = (typeof ALLOWED_SDK_MODULES)[number];
@@ -0,0 +1,13 @@
import path from 'path';
import { ASSET_PATH } from 'src/constants/assets-path';
const IS_BUILT =
__dirname.includes('/dist/') && process.env.NODE_ENV !== 'development';
// In built mode the package is copied into dist/assets/ by the build step.
// In dev mode it lives in node_modules via the monorepo workspace — resolve
// from the twenty-client-sdk/core entry point and navigate up to the package root.
export const SDK_CLIENT_PACKAGE_DIRNAME = IS_BUILT
? path.join(ASSET_PATH, 'twenty-client-sdk')
: path.resolve(require.resolve('twenty-client-sdk/core'), '..', '..');
@@ -0,0 +1,69 @@
import {
Controller,
Get,
NotFoundException,
Param,
Res,
UseGuards,
} from '@nestjs/common';
import { Response } from 'express';
import { isDefined } from 'twenty-shared/utils';
import {
ALLOWED_SDK_MODULES,
type SdkModuleName,
} from 'src/engine/core-modules/sdk-client/constants/allowed-sdk-modules';
import { SdkClientArchiveService } from 'src/engine/core-modules/sdk-client/sdk-client-archive.service';
import { WorkspaceEntity } from 'src/engine/core-modules/workspace/workspace.entity';
import { AuthWorkspace } from 'src/engine/decorators/auth/auth-workspace.decorator';
import { NoPermissionGuard } from 'src/engine/guards/no-permission.guard';
import { WorkspaceAuthGuard } from 'src/engine/guards/workspace-auth.guard';
import { WorkspaceCacheService } from 'src/engine/workspace-cache/services/workspace-cache.service';
@Controller('rest/sdk-client')
@UseGuards(WorkspaceAuthGuard)
export class SdkClientController {
constructor(
private readonly workspaceCacheService: WorkspaceCacheService,
private readonly sdkClientArchiveService: SdkClientArchiveService,
) {}
@Get(':applicationId/:moduleName')
@UseGuards(NoPermissionGuard)
async getSdkModule(
@Res() res: Response,
@Param('applicationId') applicationId: string,
@Param('moduleName') moduleName: SdkModuleName,
@AuthWorkspace() workspace: WorkspaceEntity,
) {
if (!ALLOWED_SDK_MODULES.includes(moduleName)) {
throw new NotFoundException(
`SDK module "${moduleName}" not found. Allowed: ${ALLOWED_SDK_MODULES.join(', ')}`,
);
}
const { flatApplicationMaps } =
await this.workspaceCacheService.getOrRecompute(workspace.id, [
'flatApplicationMaps',
]);
const application = flatApplicationMaps.byId[applicationId];
if (!isDefined(application)) {
throw new NotFoundException(
`Application "${applicationId}" not found in workspace "${workspace.id}"`,
);
}
const fileBuffer =
await this.sdkClientArchiveService.getClientModuleFromArchive({
workspaceId: workspace.id,
applicationUniversalIdentifier: application.universalIdentifier,
moduleName,
});
res.setHeader('Content-Type', 'application/javascript');
res.send(fileBuffer);
}
}
@@ -0,0 +1,42 @@
import { type MessageDescriptor } from '@lingui/core';
import { msg } from '@lingui/core/macro';
import { assertUnreachable } from 'twenty-shared/utils';
import { CustomException } from 'src/utils/custom-exception';
export enum SdkClientExceptionCode {
ARCHIVE_NOT_FOUND = 'ARCHIVE_NOT_FOUND',
ARCHIVE_EXTRACTION_FAILED = 'ARCHIVE_EXTRACTION_FAILED',
FILE_NOT_FOUND_IN_ARCHIVE = 'FILE_NOT_FOUND_IN_ARCHIVE',
GENERATION_FAILED = 'GENERATION_FAILED',
}
const getSdkClientExceptionUserFriendlyMessage = (
code: SdkClientExceptionCode,
) => {
switch (code) {
case SdkClientExceptionCode.ARCHIVE_NOT_FOUND:
return msg`SDK client archive not found. The SDK client may not have been generated for this application.`;
case SdkClientExceptionCode.ARCHIVE_EXTRACTION_FAILED:
return msg`Failed to extract SDK client archive.`;
case SdkClientExceptionCode.FILE_NOT_FOUND_IN_ARCHIVE:
return msg`File not found in SDK client archive.`;
case SdkClientExceptionCode.GENERATION_FAILED:
return msg`Failed to generate SDK client.`;
default:
assertUnreachable(code);
}
};
export class SdkClientException extends CustomException<SdkClientExceptionCode> {
constructor(
message: string,
code: SdkClientExceptionCode,
{ userFriendlyMessage }: { userFriendlyMessage?: MessageDescriptor } = {},
) {
super(message, code, {
userFriendlyMessage:
userFriendlyMessage ?? getSdkClientExceptionUserFriendlyMessage(code),
});
}
}
@@ -0,0 +1,166 @@
import { Injectable } from '@nestjs/common';
import { InjectRepository } from '@nestjs/typeorm';
import { createWriteStream } from 'fs';
import * as fs from 'fs/promises';
import { join } from 'path';
import { type Readable } from 'stream';
import { pipeline } from 'stream/promises';
import { FileFolder } from 'twenty-shared/types';
import { Repository } from 'typeorm';
import { ApplicationEntity } from 'src/engine/core-modules/application/application.entity';
import { FileStorageService } from 'src/engine/core-modules/file-storage/file-storage.service';
import {
FileStorageException,
FileStorageExceptionCode,
} from 'src/engine/core-modules/file-storage/interfaces/file-storage-exception';
import { TemporaryDirManager } from 'src/engine/core-modules/logic-function/logic-function-drivers/utils/temporary-dir-manager';
import { type SdkModuleName } from 'src/engine/core-modules/sdk-client/constants/allowed-sdk-modules';
import {
SdkClientException,
SdkClientExceptionCode,
} from 'src/engine/core-modules/sdk-client/exceptions/sdk-client.exception';
import { WorkspaceCacheService } from 'src/engine/workspace-cache/services/workspace-cache.service';
import { streamToBuffer } from 'src/utils/stream-to-buffer';
const SDK_CLIENT_ARCHIVE_NAME = 'twenty-client-sdk.zip';
@Injectable()
export class SdkClientArchiveService {
constructor(
private readonly fileStorageService: FileStorageService,
@InjectRepository(ApplicationEntity)
private readonly applicationRepository: Repository<ApplicationEntity>,
private readonly workspaceCacheService: WorkspaceCacheService,
) {}
async downloadAndExtractToPackage({
workspaceId,
applicationUniversalIdentifier,
targetPackagePath,
}: {
workspaceId: string;
applicationUniversalIdentifier: string;
targetPackagePath: string;
}): Promise<void> {
const temporaryDirManager = new TemporaryDirManager();
try {
const { sourceTemporaryDir } = await temporaryDirManager.init();
const archivePath = join(sourceTemporaryDir, SDK_CLIENT_ARCHIVE_NAME);
const archiveStream = await this.readArchiveStream({
workspaceId,
applicationUniversalIdentifier,
});
await pipeline(archiveStream, createWriteStream(archivePath));
await fs.rm(targetPackagePath, { recursive: true, force: true });
await fs.mkdir(targetPackagePath, { recursive: true });
const { default: unzipper } = await import('unzipper');
const directory = await unzipper.Open.file(archivePath);
await directory.extract({ path: targetPackagePath });
} finally {
await temporaryDirManager.clean();
}
}
async downloadArchiveBuffer({
workspaceId,
applicationUniversalIdentifier,
}: {
workspaceId: string;
applicationUniversalIdentifier: string;
}): Promise<Buffer> {
const archiveStream = await this.readArchiveStream({
workspaceId,
applicationUniversalIdentifier,
});
return streamToBuffer(archiveStream);
}
async getClientModuleFromArchive({
workspaceId,
applicationUniversalIdentifier,
moduleName,
}: {
workspaceId: string;
applicationUniversalIdentifier: string;
moduleName: SdkModuleName;
}): Promise<Buffer> {
const filePath = `dist/${moduleName}.mjs`;
const archiveBuffer = await this.downloadArchiveBuffer({
workspaceId,
applicationUniversalIdentifier,
});
const { default: unzipper } = await import('unzipper');
const directory = await unzipper.Open.buffer(archiveBuffer);
const entry = directory.files.find(
(file) => file.path === filePath || file.path === `./${filePath}`,
);
if (!entry) {
throw new SdkClientException(
`Module "${moduleName}" not found in SDK client archive for application "${applicationUniversalIdentifier}" in workspace "${workspaceId}"`,
SdkClientExceptionCode.FILE_NOT_FOUND_IN_ARCHIVE,
);
}
return entry.buffer();
}
async markSdkLayerFresh({
applicationId,
workspaceId,
}: {
applicationId: string;
workspaceId: string;
}): Promise<void> {
await this.applicationRepository.update(
{ id: applicationId, workspaceId },
{ isSdkLayerStale: false },
);
await this.workspaceCacheService.invalidateAndRecompute(workspaceId, [
'flatApplicationMaps',
]);
}
private async readArchiveStream({
workspaceId,
applicationUniversalIdentifier,
}: {
workspaceId: string;
applicationUniversalIdentifier: string;
}): Promise<Readable> {
try {
return await this.fileStorageService.readFile({
workspaceId,
applicationUniversalIdentifier,
fileFolder: FileFolder.GeneratedSdkClient,
resourcePath: SDK_CLIENT_ARCHIVE_NAME,
});
} catch (error) {
if (
error instanceof FileStorageException &&
error.code === FileStorageExceptionCode.FILE_NOT_FOUND
) {
throw new SdkClientException(
`SDK client archive "${SDK_CLIENT_ARCHIVE_NAME}" not found for application "${applicationUniversalIdentifier}" in workspace "${workspaceId}".`,
SdkClientExceptionCode.ARCHIVE_NOT_FOUND,
);
}
throw error;
}
}
}
@@ -0,0 +1,131 @@
import { Injectable, Logger } from '@nestjs/common';
import { InjectRepository } from '@nestjs/typeorm';
import * as fs from 'fs/promises';
import { printSchema } from 'graphql';
import path, { join } from 'path';
import { replaceCoreClient } from 'twenty-client-sdk/generate';
import { FileFolder } from 'twenty-shared/types';
import { Repository } from 'typeorm';
import { WorkspaceSchemaFactory } from 'src/engine/api/graphql/workspace-schema.factory';
import { ApplicationEntity } from 'src/engine/core-modules/application/application.entity';
import { FileStorageService } from 'src/engine/core-modules/file-storage/file-storage.service';
import { createZipFile } from 'src/engine/core-modules/logic-function/logic-function-drivers/utils/create-zip-file';
import { TemporaryDirManager } from 'src/engine/core-modules/logic-function/logic-function-drivers/utils/temporary-dir-manager';
import { SDK_CLIENT_PACKAGE_DIRNAME } from 'src/engine/core-modules/sdk-client/constants/sdk-client-package-dirname';
import {
SdkClientException,
SdkClientExceptionCode,
} from 'src/engine/core-modules/sdk-client/exceptions/sdk-client.exception';
import { WorkspaceEntity } from 'src/engine/core-modules/workspace/workspace.entity';
import { WorkspaceCacheService } from 'src/engine/workspace-cache/services/workspace-cache.service';
const SDK_CLIENT_ARCHIVE_NAME = 'twenty-client-sdk.zip';
@Injectable()
export class SdkClientGenerationService {
private readonly logger = new Logger(SdkClientGenerationService.name);
constructor(
private readonly fileStorageService: FileStorageService,
@InjectRepository(ApplicationEntity)
private readonly applicationRepository: Repository<ApplicationEntity>,
private readonly workspaceCacheService: WorkspaceCacheService,
private readonly workspaceSchemaFactory: WorkspaceSchemaFactory,
) {}
async generateSdkClientForApplication({
workspaceId,
applicationId,
applicationUniversalIdentifier,
}: {
workspaceId: string;
applicationId: string;
applicationUniversalIdentifier: string;
}): Promise<void> {
const graphqlSchema = await this.workspaceSchemaFactory.createGraphQLSchema(
{ id: workspaceId } as WorkspaceEntity,
applicationId,
);
await this.generateAndStore({
workspaceId,
applicationId,
applicationUniversalIdentifier,
schema: printSchema(graphqlSchema),
});
this.logger.log(
`Generated SDK client for application ${applicationUniversalIdentifier}`,
);
}
private async generateAndStore({
workspaceId,
applicationId,
applicationUniversalIdentifier,
schema,
}: {
workspaceId: string;
applicationId: string;
applicationUniversalIdentifier: string;
schema: string;
}): Promise<void> {
const temporaryDirManager = new TemporaryDirManager();
try {
const { sourceTemporaryDir } = await temporaryDirManager.init();
const tempPackageRoot = join(sourceTemporaryDir, 'twenty-client-sdk');
await fs.cp(SDK_CLIENT_PACKAGE_DIRNAME, tempPackageRoot, {
recursive: true,
filter: (source) => {
const relativePath = path.relative(
SDK_CLIENT_PACKAGE_DIRNAME,
source,
);
return (
!relativePath.includes('node_modules') &&
!relativePath.startsWith('src')
);
},
});
await replaceCoreClient({ packageRoot: tempPackageRoot, schema });
const archivePath = join(sourceTemporaryDir, SDK_CLIENT_ARCHIVE_NAME);
await createZipFile(tempPackageRoot, archivePath);
await this.fileStorageService.writeFile({
workspaceId,
applicationUniversalIdentifier,
fileFolder: FileFolder.GeneratedSdkClient,
resourcePath: SDK_CLIENT_ARCHIVE_NAME,
sourceFile: await fs.readFile(archivePath),
mimeType: 'application/zip',
settings: { isTemporaryFile: false, toDelete: false },
});
await this.applicationRepository.update(
{ id: applicationId, workspaceId },
{ isSdkLayerStale: true },
);
await this.workspaceCacheService.invalidateAndRecompute(workspaceId, [
'flatApplicationMaps',
]);
} catch (error) {
throw new SdkClientException(
`Failed to generate SDK client for application "${applicationUniversalIdentifier}" in workspace "${workspaceId}": ${error instanceof Error ? error.message : String(error)}`,
SdkClientExceptionCode.GENERATION_FAILED,
);
} finally {
await temporaryDirManager.clean();
}
}
}
@@ -0,0 +1,21 @@
import { Module } from '@nestjs/common';
import { TypeOrmModule } from '@nestjs/typeorm';
import { CoreGraphQLApiModule } from 'src/engine/api/graphql/core-graphql-api.module';
import { ApplicationEntity } from 'src/engine/core-modules/application/application.entity';
import { SdkClientController } from 'src/engine/core-modules/sdk-client/controllers/sdk-client.controller';
import { SdkClientArchiveService } from 'src/engine/core-modules/sdk-client/sdk-client-archive.service';
import { SdkClientGenerationService } from 'src/engine/core-modules/sdk-client/sdk-client-generation.service';
import { WorkspaceCacheModule } from 'src/engine/workspace-cache/workspace-cache.module';
@Module({
imports: [
TypeOrmModule.forFeature([ApplicationEntity]),
WorkspaceCacheModule,
CoreGraphQLApiModule,
],
controllers: [SdkClientController],
providers: [SdkClientGenerationService, SdkClientArchiveService],
exports: [SdkClientGenerationService, SdkClientArchiveService],
})
export class SdkClientModule {}