Refactor twenty client sdk provisioning for logic function and front-component (#18544)

## 1. The `twenty-client-sdk` Package (Source of Truth) The monorepo package at `packages/twenty-client-sdk` ships with: - A **pre-built metadata client** (static, generated from a fixed schema) - A **stub core client** that throws at runtime (`CoreApiClient was not generated...`) - Both ESM (`.mjs`) and CJS (`.cjs`) bundles in `dist/` - A `package.json` with proper `exports` map for `twenty-client-sdk/core`, `twenty-client-sdk/metadata`, and `twenty-client-sdk/generate` ## 2. Generation & Upload (Server-Side, at Migration Time) **When**: `WorkspaceMigrationRunnerService.run()` executes after a metadata schema change. **What happens in `SdkClientGenerationService.generateAndStore()`**: 1. Copies the stub `twenty-client-sdk` package from the server's assets (resolved via `SDK_CLIENT_PACKAGE_DIRNAME` — from `dist/assets/twenty-client-sdk/` in production, or from `node_modules` in dev) 2. Filters out `node_modules/` and `src/` during copy — only `package.json` + `dist/` are kept (like an npm publish) 3. Calls `replaceCoreClient()` which uses `@genql/cli` to introspect the **application-scoped** GraphQL schema and generates a real `CoreApiClient`, then compiles it to ESM+CJS and overwrites `dist/core.mjs` and `dist/core.cjs` 4. Archives the **entire package** (with `package.json` + `dist/`) into `twenty-client-sdk.zip` 5. Uploads the single archive to S3 under `FileFolder.GeneratedSdkClient` 6. Sets `isSdkLayerStale = true` on the `ApplicationEntity` in the database ## 3. Invalidation Signal The `isSdkLayerStale` boolean column on `ApplicationEntity` is the invalidation mechanism: - **Set to `true`** by `generateAndStore()` after uploading a new client archive - **Checked** by both logic function drivers before execution — if `true`, they rebuild their local layer - **Set back to `false`** by `markSdkLayerFresh()` after the driver has successfully consumed the new archive Default is `false` so existing applications without a generated client aren't affected. ## 4a. Logic Functions — Local Driver **`ensureSdkLayer()`** is called before every execution: 1. Checks if the local SDK layer directory exists AND `isSdkLayerStale` is `false` → early return 2. Otherwise, cleans the local layer directory 3. Calls `downloadAndExtractToPackage()` which streams the zip from S3 directly to disk and extracts the full package into `<tmpdir>/sdk/<workspaceId>-<appId>/node_modules/twenty-client-sdk/` 4. Calls `markSdkLayerFresh()` to set `isSdkLayerStale = false` **At execution time**, `assembleNodeModules()` symlinks everything from the deps layer's `node_modules/` **except** `twenty-client-sdk`, which is symlinked from the SDK layer instead. This ensures the logic function's `import ... from 'twenty-client-sdk/core'` resolves to the generated client. ## 4b. Logic Functions — Lambda Driver **`ensureSdkLayer()`** is called during `build()`: 1. Checks if `isSdkLayerStale` is `false` and an existing Lambda layer ARN exists → early return 2. Otherwise, deletes all existing layer versions for this SDK layer name 3. Calls `downloadArchiveBuffer()` to get the raw zip from S3 (no disk extraction) 4. Calls `reprefixZipEntries()` which streams the zip entries into a **new zip** with the path prefix `nodejs/node_modules/twenty-client-sdk/` — this is the Lambda layer convention path. All done in memory, no disk round-trip 5. Publishes the re-prefixed zip as a new Lambda layer via `publishLayer()` 6. Calls `markSdkLayerFresh()` **At function creation**, the Lambda is created with **two layers**: `[depsLayerArn, sdkLayerArn]`. The SDK layer is listed last so it overwrites the stub `twenty-client-sdk` from the deps layer (later layers take precedence in Lambda's `/opt` merge). ## 5. Front Components Front components are built by `app:build` with `twenty-client-sdk/core` and `twenty-client-sdk/metadata` as **esbuild externals**. The stored `.mjs` in S3 has unresolved bare import specifiers like `import { CoreApiClient } from 'twenty-client-sdk/core'`. SDK import resolution is split between the **frontend host** (fetching & caching SDK modules) and the **Web Worker** (rewriting imports): **Server endpoints**: - `GET /rest/front-components/:id` — `FrontComponentService.getBuiltComponentStream()` returns the **raw `.mjs`** directly from file storage. No bundling, no SDK injection. - `GET /rest/sdk-client/:applicationId/:moduleName` — `SdkClientController` reads a single file (e.g. `dist/core.mjs`) from the generated SDK archive via `SdkClientGenerationService.readFileFromArchive()` and serves it as JavaScript. **Frontend host** (`FrontComponentRenderer` in `twenty-front`): 1. Queries `FindOneFrontComponent` which returns `applicationId`, `builtComponentChecksum`, `usesSdkClient`, and `applicationTokenPair` 2. If `usesSdkClient` is `true`, renders `FrontComponentRendererWithSdkClient` which calls the `useApplicationSdkClient` hook 3. `useApplicationSdkClient({ applicationId, accessToken })` checks the Jotai atom family cache for existing blob URLs. On cache miss, fetches both SDK modules from `GET /rest/sdk-client/:applicationId/core` and `/metadata`, creates **blob URLs** for each, and stores them in the atom family 4. Once the blob URLs are cached, passes them as `sdkClientUrls` (already blob URLs, not server URLs) to `SharedFrontComponentRenderer` → `FrontComponentWorkerEffect` → worker's `render()` call via `HostToWorkerRenderContext` **Worker** (`remote-worker.ts` in `twenty-sdk`): 1. Fetches the raw component `.mjs` source as text 2. If `sdkClientUrls` are provided and the source contains SDK import specifiers (`twenty-client-sdk/core`, `twenty-client-sdk/metadata`), **rewrites** the bare specifiers to the blob URLs received from the host (e.g. `'twenty-client-sdk/core'` → `'blob:...'`) 3. Creates a blob URL for the rewritten source and `import()`s it 4. Revokes only the component blob URL after the module is loaded — the SDK blob URLs are owned and managed by the host's Jotai cache This approach eliminates server-side esbuild bundling on every request, caches SDK modules per application in the frontend, and keeps the worker's job to a simple string rewrite. ## Summary Diagram ``` app:build (SDK) └─ twenty-client-sdk stub (metadata=real, core=stub) │ ▼ WorkspaceMigrationRunnerService.run() └─ SdkClientGenerationService.generateAndStore() ├─ Copy stub package (package.json + dist/) ├─ replaceCoreClient() → regenerate core.mjs/core.cjs ├─ Zip entire package → upload to S3 └─ Set isSdkLayerStale = true │ ┌────────┴────────────────────┐ ▼ ▼ Logic Functions Front Components │ │ ├─ Local Driver ├─ GET /rest/sdk-client/:appId/core │ └─ downloadAndExtract │ → core.mjs from archive │ → symlink into │ │ node_modules ├─ Host (useApplicationSdkClient) │ │ ├─ Fetch SDK modules └─ Lambda Driver │ ├─ Create blob URLs └─ downloadArchiveBuffer │ └─ Cache in Jotai atom family → reprefixZipEntries │ → publish as Lambda ├─ GET /rest/front-components/:id layer │ → raw .mjs (no bundling) │ └─ Worker (browser) ├─ Fetch component .mjs ├─ Rewrite imports → blob URLs └─ import() rewritten source ``` ## Next PR - Estimate perf improvement by implementing a redis caching for front component client storage ( we don't even cache front comp initially ) - Implem frontent blob invalidation sse event from server --------- Co-authored-by: Charles Bochet <charlesBochet@users.noreply.github.com>
2026-03-24 19:10:25 +01:00
parent 16451ee2ee
commit 4ea2e32366
189 changed files with 12487 additions and 9856 deletions
@@ -7,6 +7,7 @@ import { ApplicationDevelopmentResolver } from 'src/engine/core-modules/applicat
 import { TokenModule } from 'src/engine/core-modules/auth/token/token.module';
 import { FeatureFlagModule } from 'src/engine/core-modules/feature-flag/feature-flag.module';
 import { FileStorageModule } from 'src/engine/core-modules/file-storage/file-storage.module';
+import { SdkClientModule } from 'src/engine/core-modules/sdk-client/sdk-client.module';
 import { PermissionsModule } from 'src/engine/metadata-modules/permissions/permissions.module';
 import { WorkspaceMigrationGraphqlApiExceptionInterceptor } from 'src/engine/workspace-manager/workspace-migration/interceptors/workspace-migration-graphql-api-exception.interceptor';

@@ -16,6 +17,7 @@ import { WorkspaceMigrationGraphqlApiExceptionInterceptor } from 'src/engine/wor
    ApplicationManifestModule,
    ApplicationRegistrationModule,
    FeatureFlagModule,
+    SdkClientModule,
    TokenModule,
    FileStorageModule,
    PermissionsModule,
@@ -8,32 +8,34 @@ import { Args, Mutation } from '@nestjs/graphql';

 import GraphQLUpload from 'graphql-upload/GraphQLUpload.mjs';
 import { PermissionFlagType } from 'twenty-shared/constants';
-import { FileFolder, FeatureFlagKey } from 'twenty-shared/types';
+import { FeatureFlagKey, FileFolder } from 'twenty-shared/types';
+import { isDefined } from 'twenty-shared/utils';

 import type { FileUpload } from 'graphql-upload/processRequest.mjs';

-import { ApplicationRegistrationVariableService } from 'src/engine/core-modules/application/application-registration-variable/application-registration-variable.service';
-import { ApplicationRegistrationService } from 'src/engine/core-modules/application/application-registration/application-registration.service';
 import { MetadataResolver } from 'src/engine/api/graphql/graphql-config/decorators/metadata-resolver.decorator';
-import { ApplicationExceptionFilter } from 'src/engine/core-modules/application/application-exception-filter';
-import {
-  ApplicationException,
-  ApplicationExceptionCode,
-} from 'src/engine/core-modules/application/application.exception';
-import { ApplicationRegistrationSourceType } from 'src/engine/core-modules/application/application-registration/enums/application-registration-source-type.enum';
-import { ApplicationSyncService } from 'src/engine/core-modules/application/application-manifest/application-sync.service';
-import { ApplicationService } from 'src/engine/core-modules/application/application.service';
 import { ApplicationInput } from 'src/engine/core-modules/application/application-development/dtos/application.input';
 import { CreateDevelopmentApplicationInput } from 'src/engine/core-modules/application/application-development/dtos/create-development-application.input';
 import { DevelopmentApplicationDTO } from 'src/engine/core-modules/application/application-development/dtos/development-application.dto';
 import { GenerateApplicationTokenInput } from 'src/engine/core-modules/application/application-development/dtos/generate-application-token.input';
 import { UploadApplicationFileInput } from 'src/engine/core-modules/application/application-development/dtos/upload-application-file.input';
 import { WorkspaceMigrationDTO } from 'src/engine/core-modules/application/application-development/dtos/workspace-migration.dto';
+import { ApplicationExceptionFilter } from 'src/engine/core-modules/application/application-exception-filter';
+import { ApplicationSyncService } from 'src/engine/core-modules/application/application-manifest/application-sync.service';
 import { ApplicationTokenPairDTO } from 'src/engine/core-modules/application/application-oauth/dtos/application-token-pair.dto';
+import { ApplicationRegistrationVariableService } from 'src/engine/core-modules/application/application-registration-variable/application-registration-variable.service';
+import { ApplicationRegistrationService } from 'src/engine/core-modules/application/application-registration/application-registration.service';
+import { ApplicationRegistrationSourceType } from 'src/engine/core-modules/application/application-registration/enums/application-registration-source-type.enum';
+import {
+  ApplicationException,
+  ApplicationExceptionCode,
+} from 'src/engine/core-modules/application/application.exception';
+import { ApplicationService } from 'src/engine/core-modules/application/application.service';
 import { ApplicationTokenService } from 'src/engine/core-modules/auth/token/services/application-token.service';
 import { FileStorageService } from 'src/engine/core-modules/file-storage/file-storage.service';
 import { FileDTO } from 'src/engine/core-modules/file/dtos/file.dto';
 import { ResolverValidationPipe } from 'src/engine/core-modules/graphql/pipes/resolver-validation.pipe';
+import { SdkClientGenerationService } from 'src/engine/core-modules/sdk-client/sdk-client-generation.service';
 import { type WorkspaceEntity } from 'src/engine/core-modules/workspace/workspace.entity';
 import { AuthUser } from 'src/engine/decorators/auth/auth-user.decorator';
 import { AuthUserWorkspaceId } from 'src/engine/decorators/auth/auth-user-workspace-id.decorator';
@@ -66,6 +68,7 @@ export class ApplicationDevelopmentResolver {
    private readonly applicationRegistrationService: ApplicationRegistrationService,
    private readonly applicationRegistrationVariableService: ApplicationRegistrationVariableService,
    private readonly fileStorageService: FileStorageService,
+    private readonly sdkClientGenerationService: SdkClientGenerationService,
  ) {}

  @Mutation(() => DevelopmentApplicationDTO)
@@ -133,13 +136,38 @@ export class ApplicationDevelopmentResolver {
      workspaceId,
    );

-    const workspaceMigration =
+    const application = await this.applicationService.findByUniversalIdentifier(
+      {
+        universalIdentifier: manifest.application.universalIdentifier,
+        workspaceId,
+      },
+    );
+
+    if (!isDefined(application)) {
+      throw new ApplicationException(
+        `Application "${manifest.application.universalIdentifier}" not found in workspace "${workspaceId}". Run createDevelopmentApplication first.`,
+        ApplicationExceptionCode.APPLICATION_NOT_FOUND,
+      );
+    }
+
+    const isFirstSync = !isDefined(application.version);
+
+    const { workspaceMigration, hasSchemaMetadataChanged } =
      await this.applicationSyncService.synchronizeFromManifest({
        workspaceId,
        manifest,
        applicationRegistrationId,
      });

+    if (isFirstSync || hasSchemaMetadataChanged) {
+      await this.sdkClientGenerationService.generateSdkClientForApplication({
+        workspaceId,
+        applicationId: application.id,
+        applicationUniversalIdentifier:
+          manifest.application.universalIdentifier,
+      });
+    }
+
    await this.syncRegistrationMetadata(
      applicationRegistrationId,
      manifest,
@@ -10,6 +10,7 @@ import { ApplicationPackageModule } from 'src/engine/core-modules/application/ap
 import { ApplicationInstallResolver } from 'src/engine/core-modules/application/application-install/application-install.resolver';
 import { ApplicationInstallService } from 'src/engine/core-modules/application/application-install/application-install.service';
 import { FileStorageModule } from 'src/engine/core-modules/file-storage/file-storage.module';
+import { SdkClientModule } from 'src/engine/core-modules/sdk-client/sdk-client.module';
 import { PermissionsModule } from 'src/engine/metadata-modules/permissions/permissions.module';

@Module({
@@ -20,6 +21,7 @@ import { PermissionsModule } from 'src/engine/metadata-modules/permissions/permi
    ApplicationPackageModule,
    CacheLockModule,
    FeatureFlagModule,
+    SdkClientModule,
    PermissionsModule,
    FileStorageModule,
  ],
@@ -24,7 +24,7 @@ import {
 import { ApplicationSyncService } from 'src/engine/core-modules/application/application-manifest/application-sync.service';
 import { CacheLockService } from 'src/engine/core-modules/cache-lock/cache-lock.service';
 import { FileStorageService } from 'src/engine/core-modules/file-storage/file-storage.service';
-
+import { SdkClientGenerationService } from 'src/engine/core-modules/sdk-client/sdk-client-generation.service';
@Injectable()
 export class ApplicationInstallService {
  private readonly logger = new Logger(ApplicationInstallService.name);
@@ -37,6 +37,7 @@ export class ApplicationInstallService {
    private readonly applicationSyncService: ApplicationSyncService,
    private readonly fileStorageService: FileStorageService,
    private readonly cacheLockService: CacheLockService,
+    private readonly sdkClientGenerationService: SdkClientGenerationService,
  ) {}

  async installApplication(params: {
@@ -108,7 +109,7 @@ export class ApplicationInstallService {

      const universalIdentifier = appRegistration.universalIdentifier;

-      await this.ensureApplicationExists({
+      const { application, wasCreated } = await this.ensureApplicationExists({
        universalIdentifier,
        name: resolvedPackage.manifest.application.displayName,
        workspaceId: params.workspaceId,
@@ -123,11 +124,20 @@ export class ApplicationInstallService {
        params.workspaceId,
      );

-      await this.applicationSyncService.synchronizeFromManifest({
-        workspaceId: params.workspaceId,
-        manifest: resolvedPackage.manifest,
-        applicationRegistrationId: appRegistration.id,
-      });
+      const { hasSchemaMetadataChanged } =
+        await this.applicationSyncService.synchronizeFromManifest({
+          workspaceId: params.workspaceId,
+          manifest: resolvedPackage.manifest,
+          applicationRegistrationId: appRegistration.id,
+        });
+
+      if (wasCreated || hasSchemaMetadataChanged) {
+        await this.sdkClientGenerationService.generateSdkClientForApplication({
+          workspaceId: params.workspaceId,
+          applicationId: application.id,
+          applicationUniversalIdentifier: universalIdentifier,
+        });
+      }

      this.logger.log(
        `Successfully installed app ${universalIdentifier} v${resolvedPackage.packageJson.version ?? 'unknown'}`,
@@ -232,17 +242,17 @@ export class ApplicationInstallService {
    workspaceId: string;
    applicationRegistrationId: string;
    sourceType: ApplicationRegistrationSourceType;
-  }): Promise<ApplicationEntity> {
+  }): Promise<{ application: ApplicationEntity; wasCreated: boolean }> {
    const existing = await this.applicationService.findByUniversalIdentifier({
      universalIdentifier: params.universalIdentifier,
      workspaceId: params.workspaceId,
    });

    if (isDefined(existing)) {
-      return existing;
+      return { application: existing, wasCreated: false };
    }

-    return this.applicationService.create({
+    const application = await this.applicationService.create({
      universalIdentifier: params.universalIdentifier,
      name: params.name,
      sourcePath: params.universalIdentifier,
@@ -250,5 +260,7 @@ export class ApplicationInstallService {
      applicationRegistrationId: params.applicationRegistrationId,
      workspaceId: params.workspaceId,
    });
+
+    return { application, wasCreated: true };
  }
 }
@@ -47,7 +47,10 @@ export class ApplicationManifestMigrationService {
    manifest: Manifest;
    workspaceId: string;
    ownerFlatApplication: FlatApplication;
-  }): Promise<WorkspaceMigration> {
+  }): Promise<{
+    workspaceMigration: WorkspaceMigration;
+    hasSchemaMetadataChanged: boolean;
+  }> {
    const now = new Date().toISOString();

    const { twentyStandardFlatApplication } =
@@ -122,7 +125,10 @@ export class ApplicationManifestMigrationService {
      ownerFlatApplication,
    });

-    return validateAndBuildResult.workspaceMigration;
+    return {
+      workspaceMigration: validateAndBuildResult.workspaceMigration,
+      hasSchemaMetadataChanged: validateAndBuildResult.hasSchemaMetadataChanged,
+    };
  }

  /**
@@ -47,7 +47,10 @@ export class ApplicationSyncService {
    workspaceId: string;
    manifest: Manifest;
    applicationRegistrationId?: string;
-  }): Promise<WorkspaceMigration> {
+  }): Promise<{
+    workspaceMigration: WorkspaceMigration;
+    hasSchemaMetadataChanged: boolean;
+  }> {
    const application = await this.syncApplication({
      workspaceId,
      manifest,
@@ -56,7 +59,7 @@ export class ApplicationSyncService {

    const ownerFlatApplication: FlatApplication = application;

-    const workspaceMigration =
+    const syncResult =
      await this.applicationManifestMigrationService.syncMetadataFromManifest({
        manifest,
        workspaceId,
@@ -65,7 +68,7 @@ export class ApplicationSyncService {

    this.logger.log('Application sync from manifest completed');

-    return workspaceMigration;
+    return syncResult;
  }

  private async syncApplication({
@@ -21,6 +21,7 @@ export const fromFrontComponentManifestToUniversalFlatFrontComponent = ({
    componentName: frontComponentManifest.componentName,
    builtComponentChecksum: frontComponentManifest.builtComponentChecksum,
    isHeadless: frontComponentManifest.isHeadless ?? false,
+    usesSdkClient: frontComponentManifest.usesSdkClient ?? false,
    createdAt: now,
    updatedAt: now,
  };
@@ -97,6 +97,9 @@ export class ApplicationEntity extends WorkspaceRelatedEntity {
  @Column({ nullable: false, type: 'boolean', default: true })
  canBeUninstalled: boolean;

+  @Column({ nullable: false, type: 'boolean', default: false })
+  isSdkLayerStale: boolean;
+
  @Column({ nullable: true, type: 'uuid' })
  applicationRegistrationId: string | null;