80fb91c033dfd367f17ba58e2095526faa016e70
2 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Paul Rastoin
|
f04db9751f |
fix(client-sdk): bundle metadata client into a single self-contained file (#22085)
## Problem A front component that imports `MetadataApiClient` from `twenty-client-sdk/metadata` crashes at render time: ``` FrontComponent error: Failed to resolve module specifier "./chunk-Dqa2HsxW.mjs". Invalid relative url or base scheme isn't hierarchical. ``` (hash differs per build). The equivalent component using `CoreApiClient` from `twenty-client-sdk/core` works fine. ## Root cause The front-component renderer loads each SDK client as a **single in-memory blob-URL module** and only rewrites the two bare specifiers it knows (`twenty-client-sdk/core`, `twenty-client-sdk/metadata`). A blob-URL module cannot resolve a **relative** `import … from "./chunk-*.mjs"` (blob URLs aren't hierarchical), and that chunk isn't served anyway. Only two entrypoints are externalized by the front-component build (`FRONT_COMPONENT_EXTERNAL_MODULES`) and thus served as blob modules: `core` and `metadata`. Everything else (`rest`, `generate`) is bundled into the component and is unaffected. Of those two: | client | how `dist/*.mjs` is produced | self-contained? | |---|---|---| | **core** | esbuild single-file bundle (`compileGeneratedClient`), re-run per workspace at server runtime by `replaceCoreClient` | ✅ | | **metadata** | the shared multi-entry Vite build, which hoists shared code into a relative `chunk-*.mjs` | ❌ | The metadata client is built once at package-build time (it is not workspace-specific) and was shipped straight from the multi-entry Vite output, keeping the unresolvable relative chunk import. ## Regression trace This was **not** broken on arrival — it regressed via a transitive bundler swap: | Date | Commit | Event | |---|---|---| | 2026-05-20 | `a26fe3bb65` | Metadata-client-in-front-components shipped; `twenty-client-sdk` on **Vite 7 (Rollup)** | | 2026-06-08 | `d2e7dc0e74` (#21309, *"security: bump vulnerable direct dependencies"*) | Bumped **Vite 7 → 8**, introducing **Rolldown 1.0.3** (no rolldown entries in the lockfile before this commit) | Vite 7 is Rollup-based; Vite 8 uses Rolldown. The breaking artifact is literally a `\0rolldown/runtime.js` shared chunk — a Rolldown construct that could not have existed before the bump. So the metadata front-component path worked from 2026-05-20 until the 2026-06-08 security dependency bump silently changed the bundler and split out the shared runtime chunk. ## Fix Build the metadata client as its **own single-entry Vite library** (`vite.metadata.config.ts`) so its output is a single self-contained file with no shared chunk. `core` / `rest` / `generate` stay in the main multi-entry build (`vite.config.ts`); shared config (`isExternal`, `entryFileNames`) is factored into `vite.shared.ts`. The build pipeline runs `vite build && vite build -c vite.metadata.config.ts`. The server picks this up automatically: `SdkClientGenerationService` ships the pre-built package `dist/` and only regenerates the **core** client; it never regenerates metadata. No server-side change required. ## Regression guard (e2e) The postcard example's `card.front-component.tsx` previously used `CoreApiClient` only, so this metadata-only regression had no e2e coverage. It now loads and round-trips all three SDK clients (`Core`, `Metadata`, `Rest`) via an SDK health panel, and the e2e asserts the blob-served `core` + `metadata` probes reach `ok` — which only happens if those bundles resolve and function. A future chunk-import regression in either blob module would crash the component on load and fail the test. ## Verification - `npx nx build twenty-client-sdk` succeeds. - `dist/metadata.mjs` / `dist/metadata.cjs`: **0** `chunk-*` imports, **0** relative imports; both load and export `MetadataApiClient` + `MetadataSchema`. - `dist/metadata/index.d.ts` types still emitted. - `npx nx typecheck` + `npx nx lint twenty-client-sdk` pass; postcard app typecheck + lint pass. ## Notes - `dist/` is not committed (CI builds it); a running server must rebuild `twenty-client-sdk` for the fix to take effect. - The e2e was validated statically (typecheck + lint); running it end-to-end requires a live stack with a seeded postcard record. |
||
|
Paul Rastoin
|
4ea2e32366 |
Refactor twenty client sdk provisioning for logic function and front-component (#18544)
## 1. The `twenty-client-sdk` Package (Source of Truth)
The monorepo package at `packages/twenty-client-sdk` ships with:
- A **pre-built metadata client** (static, generated from a fixed
schema)
- A **stub core client** that throws at runtime (`CoreApiClient was not
generated...`)
- Both ESM (`.mjs`) and CJS (`.cjs`) bundles in `dist/`
- A `package.json` with proper `exports` map for
`twenty-client-sdk/core`, `twenty-client-sdk/metadata`, and
`twenty-client-sdk/generate`
## 2. Generation & Upload (Server-Side, at Migration Time)
**When**: `WorkspaceMigrationRunnerService.run()` executes after a
metadata schema change.
**What happens in `SdkClientGenerationService.generateAndStore()`**:
1. Copies the stub `twenty-client-sdk` package from the server's assets
(resolved via `SDK_CLIENT_PACKAGE_DIRNAME` — from
`dist/assets/twenty-client-sdk/` in production, or from `node_modules`
in dev)
2. Filters out `node_modules/` and `src/` during copy — only
`package.json` + `dist/` are kept (like an npm publish)
3. Calls `replaceCoreClient()` which uses `@genql/cli` to introspect the
**application-scoped** GraphQL schema and generates a real
`CoreApiClient`, then compiles it to ESM+CJS and overwrites
`dist/core.mjs` and `dist/core.cjs`
4. Archives the **entire package** (with `package.json` + `dist/`) into
`twenty-client-sdk.zip`
5. Uploads the single archive to S3 under
`FileFolder.GeneratedSdkClient`
6. Sets `isSdkLayerStale = true` on the `ApplicationEntity` in the
database
## 3. Invalidation Signal
The `isSdkLayerStale` boolean column on `ApplicationEntity` is the
invalidation mechanism:
- **Set to `true`** by `generateAndStore()` after uploading a new client
archive
- **Checked** by both logic function drivers before execution — if
`true`, they rebuild their local layer
- **Set back to `false`** by `markSdkLayerFresh()` after the driver has
successfully consumed the new archive
Default is `false` so existing applications without a generated client
aren't affected.
## 4a. Logic Functions — Local Driver
**`ensureSdkLayer()`** is called before every execution:
1. Checks if the local SDK layer directory exists AND `isSdkLayerStale`
is `false` → early return
2. Otherwise, cleans the local layer directory
3. Calls `downloadAndExtractToPackage()` which streams the zip from S3
directly to disk and extracts the full package into
`<tmpdir>/sdk/<workspaceId>-<appId>/node_modules/twenty-client-sdk/`
4. Calls `markSdkLayerFresh()` to set `isSdkLayerStale = false`
**At execution time**, `assembleNodeModules()` symlinks everything from
the deps layer's `node_modules/` **except** `twenty-client-sdk`, which
is symlinked from the SDK layer instead. This ensures the logic
function's `import ... from 'twenty-client-sdk/core'` resolves to the
generated client.
## 4b. Logic Functions — Lambda Driver
**`ensureSdkLayer()`** is called during `build()`:
1. Checks if `isSdkLayerStale` is `false` and an existing Lambda layer
ARN exists → early return
2. Otherwise, deletes all existing layer versions for this SDK layer
name
3. Calls `downloadArchiveBuffer()` to get the raw zip from S3 (no disk
extraction)
4. Calls `reprefixZipEntries()` which streams the zip entries into a
**new zip** with the path prefix
`nodejs/node_modules/twenty-client-sdk/` — this is the Lambda layer
convention path. All done in memory, no disk round-trip
5. Publishes the re-prefixed zip as a new Lambda layer via
`publishLayer()`
6. Calls `markSdkLayerFresh()`
**At function creation**, the Lambda is created with **two layers**:
`[depsLayerArn, sdkLayerArn]`. The SDK layer is listed last so it
overwrites the stub `twenty-client-sdk` from the deps layer (later
layers take precedence in Lambda's `/opt` merge).
## 5. Front Components
Front components are built by `app:build` with `twenty-client-sdk/core`
and `twenty-client-sdk/metadata` as **esbuild externals**. The stored
`.mjs` in S3 has unresolved bare import specifiers like `import {
CoreApiClient } from 'twenty-client-sdk/core'`.
SDK import resolution is split between the **frontend host** (fetching &
caching SDK modules) and the **Web Worker** (rewriting imports):
**Server endpoints**:
- `GET /rest/front-components/:id` —
`FrontComponentService.getBuiltComponentStream()` returns the **raw
`.mjs`** directly from file storage. No bundling, no SDK injection.
- `GET /rest/sdk-client/:applicationId/:moduleName` —
`SdkClientController` reads a single file (e.g. `dist/core.mjs`) from
the generated SDK archive via
`SdkClientGenerationService.readFileFromArchive()` and serves it as
JavaScript.
**Frontend host** (`FrontComponentRenderer` in `twenty-front`):
1. Queries `FindOneFrontComponent` which returns `applicationId`,
`builtComponentChecksum`, `usesSdkClient`, and `applicationTokenPair`
2. If `usesSdkClient` is `true`, renders
`FrontComponentRendererWithSdkClient` which calls the
`useApplicationSdkClient` hook
3. `useApplicationSdkClient({ applicationId, accessToken })` checks the
Jotai atom family cache for existing blob URLs. On cache miss, fetches
both SDK modules from `GET /rest/sdk-client/:applicationId/core` and
`/metadata`, creates **blob URLs** for each, and stores them in the atom
family
4. Once the blob URLs are cached, passes them as `sdkClientUrls`
(already blob URLs, not server URLs) to `SharedFrontComponentRenderer` →
`FrontComponentWorkerEffect` → worker's `render()` call via
`HostToWorkerRenderContext`
**Worker** (`remote-worker.ts` in `twenty-sdk`):
1. Fetches the raw component `.mjs` source as text
2. If `sdkClientUrls` are provided and the source contains SDK import
specifiers (`twenty-client-sdk/core`, `twenty-client-sdk/metadata`),
**rewrites** the bare specifiers to the blob URLs received from the host
(e.g. `'twenty-client-sdk/core'` → `'blob:...'`)
3. Creates a blob URL for the rewritten source and `import()`s it
4. Revokes only the component blob URL after the module is loaded — the
SDK blob URLs are owned and managed by the host's Jotai cache
This approach eliminates server-side esbuild bundling on every request,
caches SDK modules per application in the frontend, and keeps the
worker's job to a simple string rewrite.
## Summary Diagram
```
app:build (SDK)
└─ twenty-client-sdk stub (metadata=real, core=stub)
│
▼
WorkspaceMigrationRunnerService.run()
└─ SdkClientGenerationService.generateAndStore()
├─ Copy stub package (package.json + dist/)
├─ replaceCoreClient() → regenerate core.mjs/core.cjs
├─ Zip entire package → upload to S3
└─ Set isSdkLayerStale = true
│
┌────────┴────────────────────┐
▼ ▼
Logic Functions Front Components
│ │
├─ Local Driver ├─ GET /rest/sdk-client/:appId/core
│ └─ downloadAndExtract │ → core.mjs from archive
│ → symlink into │
│ node_modules ├─ Host (useApplicationSdkClient)
│ │ ├─ Fetch SDK modules
└─ Lambda Driver │ ├─ Create blob URLs
└─ downloadArchiveBuffer │ └─ Cache in Jotai atom family
→ reprefixZipEntries │
→ publish as Lambda ├─ GET /rest/front-components/:id
layer │ → raw .mjs (no bundling)
│
└─ Worker (browser)
├─ Fetch component .mjs
├─ Rewrite imports → blob URLs
└─ import() rewritten source
```
## Next PR
- Estimate perf improvement by implementing a redis caching for front
component client storage ( we don't even cache front comp initially )
- Implem frontent blob invalidation sse event from server
---------
Co-authored-by: Charles Bochet <charlesBochet@users.noreply.github.com>
|