houseassassin/twenty
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
13,750 Commits 1 Branch 0 Tags
71a1ff7ac81308c048419dc26e4ef277a141d9da
Commit Graph

2 Commits

Author SHA1 Message Date
Paul Rastoin 71a1ff7ac8 Cache twenty-client-sdk modules host-side via content-addressed URLs (#22981) 
## Context

Front component sources are fetched host-side and integrity-verified by
the SHA-256 checksum embedded in their URL, cached in Cache Storage — a
layer that exists specifically because their download URLs are presigned
and rotate. The `twenty-client-sdk` modules (`core` and `metadata`) were
re-fetched on every render and could not be cached safely: their URLs
carried no checksum and the server exposed no freshness signal.

This PR makes the SDK module URLs **content-addressed** and relies on
the **browser HTTP cache** for immutability, and it keys the checksums
on their real owners: the **application** for `core`, the **instance**
for `metadata`. The checksum does double duty: cache invalidation
(regeneration changes the checksum → the URL changes → guaranteed cache
miss) and a server-side cacheability guard (the server only grants
`immutable` when the checksum in the URL matches the authoritative
checksum it knows for that module — persisted at generation time for
`core`, hashed once at bootstrap for `metadata` — so no per-request
hashing of the served bytes). Note this is **not** an end-to-end
integrity guarantee: there is no client-side hash verification, and on a
fingerprint mismatch the server still serves the current bytes with
`no-store` (self-healing for stale URLs) rather than failing.

<img width="2412" height="926" alt="image"
src="https://github.com/user-attachments/assets/d97935d2-0fdb-4c44-89ac-596b7ca8ca64"
/>

Closes twentyhq/core-team-issues#2688.

## Routes

| Module | URL | Scope |
| --- | --- | --- |
| `core` | `/rest/sdk-client/{applicationId}/core[/{checksum}]` | Per
application (generated bundle) |
| `metadata` | `/rest/sdk-client/metadata[/{checksum}]` |
**Instance-wide**: no application segment, so every application
converges on one URL and the browser downloads the module once per
release instead of once per application |

The previous application-scoped metadata path
(`/rest/sdk-client/{applicationId}/metadata[/{checksum}]`) is **kept for
backward compatibility**, new clients just stop generating those URLs.
The instance-wide route is declared before the parameterized route so
`metadata/{checksum}` is not swallowed as `:applicationId/:moduleName`.

## Caching model

| Request | `Cache-Control` | Effect |
| --- | --- | --- |
| Fingerprinted URL, checksum matches the known module checksum |
`immutable` | Cached indefinitely by the browser HTTP cache; a new
checksum is a new URL |
| Fingerprinted URL, checksum does not match | `no-store` | Current
bytes served uncached (self-healing for stale URLs) |
| Bare URL (pre-generation fallback, `core` only in practice) |
`no-store` | Never cached |

- Both responses also set `X-Content-Type-Options: nosniff` and
`Content-Type: application/javascript`.
- SDK modules are intentionally **not** placed in Cache Storage. That
layer stays reserved for the presigned/rotating component-source URLs;
SDK modules are served directly and authenticated, so the browser HTTP
cache (keyed by the content-addressed URL) is their single cache layer.

## Checksum provenance

- **core** — per **application**, persisted on
`application.sdkClientCoreChecksum` at generation time and read back
from `flatApplicationMaps` (never re-hashed per request).
- **metadata** — **instance-wide**, hashed once from the installed
`twenty-client-sdk/dist/metadata.mjs` package (warmed at bootstrap,
memoized per process) and served straight from that package, so it is
fresh from the first request after a release with no archive dependency.

## Server (twenty-server)

- Hash `dist/core.mjs` at SDK generation and persist
`sdkClientCoreChecksum` via `applicationRepository.update`. Adds the
nullable text column to `application.entity.ts` (mirroring
`packageJsonChecksum`) plus a fast instance command with up/down;
`FlatApplication` picks it up automatically.
- New **application-scoped** query
`applicationSdkClientChecksums(applicationId: UUID!):
SdkClientChecksums` on `ApplicationResolver` (metadata schema,
`WorkspaceAuthGuard` + `NoPermissionGuard`). `SdkClientChecksums.core`
is **nullable** and stays `null` until the SDK has been generated at
least once; `metadata` is **always present** (bootstrap-warmed), so the
metadata module is cacheable from the very first render of any app. The
query itself returns `null` only for unknown applications.
- `SdkClientChecksumsDTO` now lives in the shared
`core-modules/sdk-client/dtos/`. `FrontComponentDTO` and the
`frontComponent` resolver no longer carry checksums (decoupled from the
front-component row).
- `sdk-client` controller: instance-wide `metadata[/:checksum]` route
(no workspace-cache or application lookup, serves the memoized installed
module) + application-scoped `:applicationId/:moduleName[/:checksum]`
route (serves `core` from the per-application archive, `metadata` kept
for back-compat). Cacheability compares the URL checksum against the
**known** checksum — persisted `sdkClientCoreChecksum` for `core`,
memoized package hash for `metadata` — instead of hashing the served
bytes on every request: `immutable` on match, `no-store` otherwise (bare
URL or stale fingerprint), plus `nosniff`. A persisted checksum out of
sync with the archive only downgrades to `no-store` until the next
regeneration.

## Front (twenty-front)

- New metadata query `GetApplicationSdkClientChecksums`, keyed by
`applicationId`; removed the `sdkClientChecksums` selection from
`FindOneFrontComponent`.
- `getSdkClientUrls` builds the two module URLs independently:
`/sdk-client/{applicationId}/core/{checksum}` and the **instance-wide**
`/sdk-client/metadata/{checksum}` (no application segment → one shared
browser cache entry per release across all applications). Each falls
back to its bare URL when its checksum is absent — since `core` is
nullable, a never-generated app still gets a content-addressed metadata
URL and only `core` falls back. The checksum type is sourced from the
codegen `SdkClientChecksums` type rather than a hand-maintained
duplicate.
- `FrontComponentRenderer` is split into a gating outer component (runs
`FindOneFrontComponent`, renders nothing while loading) and a content
component that receives a guaranteed-non-null `frontComponent`.
Following project conventions, the side effects live in dedicated effect
components: `FrontComponentLoadErrorSnackBarEffect` (query error →
snackbar) and `FrontComponentApplicationTokenPairEffect` (mirrors the
query-derived token pair into component state unconditionally, `null`
included, so revoked credentials can never be retained or refreshed).
The content component fetches checksums via the application-keyed query
and **gates the mount of SDK-using components on that query**, so the
very first module fetch is always the content-addressed (`immutable`)
URL instead of the bare `no-store` one. Non-SDK components skip the
query and are never blocked.
- **Live invalidation without reload:** SDK regeneration updates the
application row, and the server broadcasts an `application` metadata
event carrying the new core checksum.
`useOnApplicationSdkClientChecksumsUpdated` /
`useUpdateSdkClientChecksumsApolloCache` patch the application-keyed
checksum query cache (core only; the instance-wide metadata is
preserved), so every mounted component of that application picks up the
new URL at once. This replaces the previous frontComponent-derived field
and closes the earlier "known gap" (a mounted component staying on a
session-old checksum until a full reload). The cache-patching callback
is memoized (`useCallback`) so the window listener is registered once
per application, and the listener is **skipped entirely** for non-SDK
components (`useListenToMetadataOperationBrowserEvent` gained a `skip`
option) — they register no listener and never refetch a query they don't
consume.

## Renderer (twenty-front-component-renderer)

- SDK sources are fetched through a dedicated plain authenticated fetch,
`fetchJavaScriptModuleSourceText` (Bearer header, `credentials:
'omit'`), instead of the Cache Storage `fetchComponentSource` path;
`fetchSdkClientSources` uses it. Execution stays exclusively in the
opaque-origin worker via blob URLs; the host only fetches and forwards
source strings (no hashing host-side). Staleness self-resolves through
the checksum: new checksum → new URL → cache miss.

<!-- This is an auto-generated description by cubic. -->
<a
href="https://cubic.dev/pr/twentyhq/twenty/pull/22981?utm_source=github"
target="_blank" rel="noopener noreferrer"
data-no-image-dialog="true"><picture><source
media="(prefers-color-scheme: dark)"
srcset="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"><source
media="(prefers-color-scheme: light)"
srcset="https://www.cubic.dev/buttons/review-in-cubic-light.svg"><img
alt="Review in cubic"
src="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"></picture></a>
<!-- End of auto-generated description by cubic. -->
 2026-07-21 15:05:00 +00:00
Paul Rastoin 4ea2e32366 Refactor twenty client sdk provisioning for logic function and front-component (#18544) 
## 1. The `twenty-client-sdk` Package (Source of Truth)

The monorepo package at `packages/twenty-client-sdk` ships with:
- A **pre-built metadata client** (static, generated from a fixed
schema)
- A **stub core client** that throws at runtime (`CoreApiClient was not
generated...`)
- Both ESM (`.mjs`) and CJS (`.cjs`) bundles in `dist/`
- A `package.json` with proper `exports` map for
`twenty-client-sdk/core`, `twenty-client-sdk/metadata`, and
`twenty-client-sdk/generate`

## 2. Generation & Upload (Server-Side, at Migration Time)

**When**: `WorkspaceMigrationRunnerService.run()` executes after a
metadata schema change.

**What happens in `SdkClientGenerationService.generateAndStore()`**:
1. Copies the stub `twenty-client-sdk` package from the server's assets
(resolved via `SDK_CLIENT_PACKAGE_DIRNAME` — from
`dist/assets/twenty-client-sdk/` in production, or from `node_modules`
in dev)
2. Filters out `node_modules/` and `src/` during copy — only
`package.json` + `dist/` are kept (like an npm publish)
3. Calls `replaceCoreClient()` which uses `@genql/cli` to introspect the
**application-scoped** GraphQL schema and generates a real
`CoreApiClient`, then compiles it to ESM+CJS and overwrites
`dist/core.mjs` and `dist/core.cjs`
4. Archives the **entire package** (with `package.json` + `dist/`) into
`twenty-client-sdk.zip`
5. Uploads the single archive to S3 under
`FileFolder.GeneratedSdkClient`
6. Sets `isSdkLayerStale = true` on the `ApplicationEntity` in the
database

## 3. Invalidation Signal

The `isSdkLayerStale` boolean column on `ApplicationEntity` is the
invalidation mechanism:
- **Set to `true`** by `generateAndStore()` after uploading a new client
archive
- **Checked** by both logic function drivers before execution — if
`true`, they rebuild their local layer
- **Set back to `false`** by `markSdkLayerFresh()` after the driver has
successfully consumed the new archive

Default is `false` so existing applications without a generated client
aren't affected.

## 4a. Logic Functions — Local Driver

**`ensureSdkLayer()`** is called before every execution:
1. Checks if the local SDK layer directory exists AND `isSdkLayerStale`
is `false` → early return
2. Otherwise, cleans the local layer directory
3. Calls `downloadAndExtractToPackage()` which streams the zip from S3
directly to disk and extracts the full package into
`<tmpdir>/sdk/<workspaceId>-<appId>/node_modules/twenty-client-sdk/`
4. Calls `markSdkLayerFresh()` to set `isSdkLayerStale = false`

**At execution time**, `assembleNodeModules()` symlinks everything from
the deps layer's `node_modules/` **except** `twenty-client-sdk`, which
is symlinked from the SDK layer instead. This ensures the logic
function's `import ... from 'twenty-client-sdk/core'` resolves to the
generated client.

## 4b. Logic Functions — Lambda Driver

**`ensureSdkLayer()`** is called during `build()`:
1. Checks if `isSdkLayerStale` is `false` and an existing Lambda layer
ARN exists → early return
2. Otherwise, deletes all existing layer versions for this SDK layer
name
3. Calls `downloadArchiveBuffer()` to get the raw zip from S3 (no disk
extraction)
4. Calls `reprefixZipEntries()` which streams the zip entries into a
**new zip** with the path prefix
`nodejs/node_modules/twenty-client-sdk/` — this is the Lambda layer
convention path. All done in memory, no disk round-trip
5. Publishes the re-prefixed zip as a new Lambda layer via
`publishLayer()`
6. Calls `markSdkLayerFresh()`

**At function creation**, the Lambda is created with **two layers**:
`[depsLayerArn, sdkLayerArn]`. The SDK layer is listed last so it
overwrites the stub `twenty-client-sdk` from the deps layer (later
layers take precedence in Lambda's `/opt` merge).

## 5. Front Components

Front components are built by `app:build` with `twenty-client-sdk/core`
and `twenty-client-sdk/metadata` as **esbuild externals**. The stored
`.mjs` in S3 has unresolved bare import specifiers like `import {
CoreApiClient } from 'twenty-client-sdk/core'`.

SDK import resolution is split between the **frontend host** (fetching &
caching SDK modules) and the **Web Worker** (rewriting imports):

**Server endpoints**:
- `GET /rest/front-components/:id` —
`FrontComponentService.getBuiltComponentStream()` returns the **raw
`.mjs`** directly from file storage. No bundling, no SDK injection.
- `GET /rest/sdk-client/:applicationId/:moduleName` —
`SdkClientController` reads a single file (e.g. `dist/core.mjs`) from
the generated SDK archive via
`SdkClientGenerationService.readFileFromArchive()` and serves it as
JavaScript.

**Frontend host** (`FrontComponentRenderer` in `twenty-front`):
1. Queries `FindOneFrontComponent` which returns `applicationId`,
`builtComponentChecksum`, `usesSdkClient`, and `applicationTokenPair`
2. If `usesSdkClient` is `true`, renders
`FrontComponentRendererWithSdkClient` which calls the
`useApplicationSdkClient` hook
3. `useApplicationSdkClient({ applicationId, accessToken })` checks the
Jotai atom family cache for existing blob URLs. On cache miss, fetches
both SDK modules from `GET /rest/sdk-client/:applicationId/core` and
`/metadata`, creates **blob URLs** for each, and stores them in the atom
family
4. Once the blob URLs are cached, passes them as `sdkClientUrls`
(already blob URLs, not server URLs) to `SharedFrontComponentRenderer` →
`FrontComponentWorkerEffect` → worker's `render()` call via
`HostToWorkerRenderContext`

**Worker** (`remote-worker.ts` in `twenty-sdk`):
1. Fetches the raw component `.mjs` source as text
2. If `sdkClientUrls` are provided and the source contains SDK import
specifiers (`twenty-client-sdk/core`, `twenty-client-sdk/metadata`),
**rewrites** the bare specifiers to the blob URLs received from the host
(e.g. `'twenty-client-sdk/core'` → `'blob:...'`)
3. Creates a blob URL for the rewritten source and `import()`s it
4. Revokes only the component blob URL after the module is loaded — the
SDK blob URLs are owned and managed by the host's Jotai cache

This approach eliminates server-side esbuild bundling on every request,
caches SDK modules per application in the frontend, and keeps the
worker's job to a simple string rewrite.

## Summary Diagram

```
app:build (SDK)
  └─ twenty-client-sdk stub (metadata=real, core=stub)
       │
       ▼
WorkspaceMigrationRunnerService.run()
  └─ SdkClientGenerationService.generateAndStore()
       ├─ Copy stub package (package.json + dist/)
       ├─ replaceCoreClient() → regenerate core.mjs/core.cjs
       ├─ Zip entire package → upload to S3
       └─ Set isSdkLayerStale = true
              │
     ┌────────┴────────────────────┐
     ▼                             ▼
Logic Functions               Front Components
     │                             │
     ├─ Local Driver               ├─ GET /rest/sdk-client/:appId/core
     │   └─ downloadAndExtract     │    → core.mjs from archive
     │      → symlink into         │
     │        node_modules         ├─ Host (useApplicationSdkClient)
     │                             │    ├─ Fetch SDK modules
     └─ Lambda Driver              │    ├─ Create blob URLs
         └─ downloadArchiveBuffer  │    └─ Cache in Jotai atom family
            → reprefixZipEntries   │
            → publish as Lambda    ├─ GET /rest/front-components/:id
              layer                │    → raw .mjs (no bundling)
                                   │
                                   └─ Worker (browser)
                                        ├─ Fetch component .mjs
                                        ├─ Rewrite imports → blob URLs
                                        └─ import() rewritten source
```

## Next PR
- Estimate perf improvement by implementing a redis caching for front
component client storage ( we don't even cache front comp initially )
- Implem frontent blob invalidation sse event from server

---------

Co-authored-by: Charles Bochet <charlesBochet@users.noreply.github.com>
 2026-03-24 18:10:25 +00:00