4ea2e32366
## 1. The `twenty-client-sdk` Package (Source of Truth)
The monorepo package at `packages/twenty-client-sdk` ships with:
- A **pre-built metadata client** (static, generated from a fixed
schema)
- A **stub core client** that throws at runtime (`CoreApiClient was not
generated...`)
- Both ESM (`.mjs`) and CJS (`.cjs`) bundles in `dist/`
- A `package.json` with proper `exports` map for
`twenty-client-sdk/core`, `twenty-client-sdk/metadata`, and
`twenty-client-sdk/generate`
## 2. Generation & Upload (Server-Side, at Migration Time)
**When**: `WorkspaceMigrationRunnerService.run()` executes after a
metadata schema change.
**What happens in `SdkClientGenerationService.generateAndStore()`**:
1. Copies the stub `twenty-client-sdk` package from the server's assets
(resolved via `SDK_CLIENT_PACKAGE_DIRNAME` — from
`dist/assets/twenty-client-sdk/` in production, or from `node_modules`
in dev)
2. Filters out `node_modules/` and `src/` during copy — only
`package.json` + `dist/` are kept (like an npm publish)
3. Calls `replaceCoreClient()` which uses `@genql/cli` to introspect the
**application-scoped** GraphQL schema and generates a real
`CoreApiClient`, then compiles it to ESM+CJS and overwrites
`dist/core.mjs` and `dist/core.cjs`
4. Archives the **entire package** (with `package.json` + `dist/`) into
`twenty-client-sdk.zip`
5. Uploads the single archive to S3 under
`FileFolder.GeneratedSdkClient`
6. Sets `isSdkLayerStale = true` on the `ApplicationEntity` in the
database
## 3. Invalidation Signal
The `isSdkLayerStale` boolean column on `ApplicationEntity` is the
invalidation mechanism:
- **Set to `true`** by `generateAndStore()` after uploading a new client
archive
- **Checked** by both logic function drivers before execution — if
`true`, they rebuild their local layer
- **Set back to `false`** by `markSdkLayerFresh()` after the driver has
successfully consumed the new archive
Default is `false` so existing applications without a generated client
aren't affected.
## 4a. Logic Functions — Local Driver
**`ensureSdkLayer()`** is called before every execution:
1. Checks if the local SDK layer directory exists AND `isSdkLayerStale`
is `false` → early return
2. Otherwise, cleans the local layer directory
3. Calls `downloadAndExtractToPackage()` which streams the zip from S3
directly to disk and extracts the full package into
`<tmpdir>/sdk/<workspaceId>-<appId>/node_modules/twenty-client-sdk/`
4. Calls `markSdkLayerFresh()` to set `isSdkLayerStale = false`
**At execution time**, `assembleNodeModules()` symlinks everything from
the deps layer's `node_modules/` **except** `twenty-client-sdk`, which
is symlinked from the SDK layer instead. This ensures the logic
function's `import ... from 'twenty-client-sdk/core'` resolves to the
generated client.
## 4b. Logic Functions — Lambda Driver
**`ensureSdkLayer()`** is called during `build()`:
1. Checks if `isSdkLayerStale` is `false` and an existing Lambda layer
ARN exists → early return
2. Otherwise, deletes all existing layer versions for this SDK layer
name
3. Calls `downloadArchiveBuffer()` to get the raw zip from S3 (no disk
extraction)
4. Calls `reprefixZipEntries()` which streams the zip entries into a
**new zip** with the path prefix
`nodejs/node_modules/twenty-client-sdk/` — this is the Lambda layer
convention path. All done in memory, no disk round-trip
5. Publishes the re-prefixed zip as a new Lambda layer via
`publishLayer()`
6. Calls `markSdkLayerFresh()`
**At function creation**, the Lambda is created with **two layers**:
`[depsLayerArn, sdkLayerArn]`. The SDK layer is listed last so it
overwrites the stub `twenty-client-sdk` from the deps layer (later
layers take precedence in Lambda's `/opt` merge).
## 5. Front Components
Front components are built by `app:build` with `twenty-client-sdk/core`
and `twenty-client-sdk/metadata` as **esbuild externals**. The stored
`.mjs` in S3 has unresolved bare import specifiers like `import {
CoreApiClient } from 'twenty-client-sdk/core'`.
SDK import resolution is split between the **frontend host** (fetching &
caching SDK modules) and the **Web Worker** (rewriting imports):
**Server endpoints**:
- `GET /rest/front-components/:id` —
`FrontComponentService.getBuiltComponentStream()` returns the **raw
`.mjs`** directly from file storage. No bundling, no SDK injection.
- `GET /rest/sdk-client/:applicationId/:moduleName` —
`SdkClientController` reads a single file (e.g. `dist/core.mjs`) from
the generated SDK archive via
`SdkClientGenerationService.readFileFromArchive()` and serves it as
JavaScript.
**Frontend host** (`FrontComponentRenderer` in `twenty-front`):
1. Queries `FindOneFrontComponent` which returns `applicationId`,
`builtComponentChecksum`, `usesSdkClient`, and `applicationTokenPair`
2. If `usesSdkClient` is `true`, renders
`FrontComponentRendererWithSdkClient` which calls the
`useApplicationSdkClient` hook
3. `useApplicationSdkClient({ applicationId, accessToken })` checks the
Jotai atom family cache for existing blob URLs. On cache miss, fetches
both SDK modules from `GET /rest/sdk-client/:applicationId/core` and
`/metadata`, creates **blob URLs** for each, and stores them in the atom
family
4. Once the blob URLs are cached, passes them as `sdkClientUrls`
(already blob URLs, not server URLs) to `SharedFrontComponentRenderer` →
`FrontComponentWorkerEffect` → worker's `render()` call via
`HostToWorkerRenderContext`
**Worker** (`remote-worker.ts` in `twenty-sdk`):
1. Fetches the raw component `.mjs` source as text
2. If `sdkClientUrls` are provided and the source contains SDK import
specifiers (`twenty-client-sdk/core`, `twenty-client-sdk/metadata`),
**rewrites** the bare specifiers to the blob URLs received from the host
(e.g. `'twenty-client-sdk/core'` → `'blob:...'`)
3. Creates a blob URL for the rewritten source and `import()`s it
4. Revokes only the component blob URL after the module is loaded — the
SDK blob URLs are owned and managed by the host's Jotai cache
This approach eliminates server-side esbuild bundling on every request,
caches SDK modules per application in the frontend, and keeps the
worker's job to a simple string rewrite.
## Summary Diagram
```
app:build (SDK)
└─ twenty-client-sdk stub (metadata=real, core=stub)
│
▼
WorkspaceMigrationRunnerService.run()
└─ SdkClientGenerationService.generateAndStore()
├─ Copy stub package (package.json + dist/)
├─ replaceCoreClient() → regenerate core.mjs/core.cjs
├─ Zip entire package → upload to S3
└─ Set isSdkLayerStale = true
│
┌────────┴────────────────────┐
▼ ▼
Logic Functions Front Components
│ │
├─ Local Driver ├─ GET /rest/sdk-client/:appId/core
│ └─ downloadAndExtract │ → core.mjs from archive
│ → symlink into │
│ node_modules ├─ Host (useApplicationSdkClient)
│ │ ├─ Fetch SDK modules
└─ Lambda Driver │ ├─ Create blob URLs
└─ downloadArchiveBuffer │ └─ Cache in Jotai atom family
→ reprefixZipEntries │
→ publish as Lambda ├─ GET /rest/front-components/:id
layer │ → raw .mjs (no bundling)
│
└─ Worker (browser)
├─ Fetch component .mjs
├─ Rewrite imports → blob URLs
└─ import() rewritten source
```
## Next PR
- Estimate perf improvement by implementing a redis caching for front
component client storage ( we don't even cache front comp initially )
- Implem frontent blob invalidation sse event from server
---------
Co-authored-by: Charles Bochet <charlesBochet@users.noreply.github.com>
435 lines
11 KiB
TypeScript
435 lines
11 KiB
TypeScript
// Ambient type stubs for the genql-generated code this template gets
|
|
// injected into. They enable full typecheck/lint on this file.
|
|
// __STRIPPED_DURING_INJECTION_START__
|
|
type QueryGenqlSelection = Record<string, unknown>;
|
|
type MutationGenqlSelection = Record<string, unknown>;
|
|
type GraphqlOperation = Record<string, unknown>;
|
|
|
|
type ClientOptions = {
|
|
url?: string;
|
|
headers?: HeadersInit | (() => HeadersInit | Promise<HeadersInit>);
|
|
fetcher?: (
|
|
operation: GraphqlOperation | GraphqlOperation[],
|
|
) => Promise<unknown>;
|
|
fetch?: typeof globalThis.fetch;
|
|
batch?: unknown;
|
|
};
|
|
|
|
type Client = {
|
|
query: (
|
|
request: QueryGenqlSelection & { __name?: string },
|
|
) => Promise<unknown>;
|
|
mutation: (
|
|
request: MutationGenqlSelection & { __name?: string },
|
|
) => Promise<unknown>;
|
|
};
|
|
|
|
declare function createClient(options: ClientOptions): Client;
|
|
|
|
declare class GenqlError extends Error {
|
|
constructor(errors: unknown, data: unknown);
|
|
}
|
|
// __STRIPPED_DURING_INJECTION_END__
|
|
|
|
const APP_ACCESS_TOKEN_ENV_KEY = 'TWENTY_APP_ACCESS_TOKEN';
|
|
const API_KEY_ENV_KEY = 'TWENTY_API_KEY';
|
|
|
|
type TwentyGeneratedClientOptions = ClientOptions;
|
|
|
|
type ProcessEnvironment = Record<string, string | undefined>;
|
|
|
|
type GraphqlErrorPayloadEntry = {
|
|
message?: string;
|
|
extensions?: { code?: string };
|
|
};
|
|
|
|
type GraphqlResponsePayload = {
|
|
data?: Record<string, unknown>;
|
|
errors?: GraphqlErrorPayloadEntry[];
|
|
};
|
|
|
|
type GraphqlResponse = {
|
|
status: number;
|
|
statusText: string;
|
|
payload: GraphqlResponsePayload | null;
|
|
rawBody: string;
|
|
};
|
|
|
|
const getProcessEnvironment = (): ProcessEnvironment => {
|
|
const processObject = (
|
|
globalThis as { process?: { env?: ProcessEnvironment } }
|
|
).process;
|
|
|
|
return processObject?.env ?? {};
|
|
};
|
|
|
|
const getTokenFromAuthorizationHeader = (
|
|
authorizationHeader: string | undefined,
|
|
): string | null => {
|
|
if (typeof authorizationHeader !== 'string') {
|
|
return null;
|
|
}
|
|
|
|
const trimmedAuthorizationHeader = authorizationHeader.trim();
|
|
|
|
if (trimmedAuthorizationHeader.length === 0) {
|
|
return null;
|
|
}
|
|
|
|
if (trimmedAuthorizationHeader === 'Bearer') {
|
|
return null;
|
|
}
|
|
|
|
if (trimmedAuthorizationHeader.startsWith('Bearer ')) {
|
|
return trimmedAuthorizationHeader.slice('Bearer '.length).trim();
|
|
}
|
|
|
|
return trimmedAuthorizationHeader;
|
|
};
|
|
|
|
const getTokenFromHeaders = (
|
|
headers: HeadersInit | undefined,
|
|
): string | null => {
|
|
if (!headers) {
|
|
return null;
|
|
}
|
|
|
|
if (headers instanceof Headers) {
|
|
return getTokenFromAuthorizationHeader(
|
|
headers.get('Authorization') ?? undefined,
|
|
);
|
|
}
|
|
|
|
if (Array.isArray(headers)) {
|
|
const matchedAuthorizationHeader = headers.find(
|
|
([headerName]) => headerName.toLowerCase() === 'authorization',
|
|
);
|
|
|
|
return getTokenFromAuthorizationHeader(matchedAuthorizationHeader?.[1]);
|
|
}
|
|
|
|
const headersRecord = headers as Record<string, string | undefined>;
|
|
|
|
return getTokenFromAuthorizationHeader(
|
|
headersRecord.Authorization ?? headersRecord.authorization,
|
|
);
|
|
};
|
|
|
|
const hasAuthenticationErrorInGraphqlPayload = (
|
|
payload: GraphqlResponsePayload | null,
|
|
): boolean => {
|
|
if (!payload?.errors) {
|
|
return false;
|
|
}
|
|
|
|
return payload.errors.some((graphqlError) => {
|
|
return (
|
|
graphqlError.extensions?.code === 'UNAUTHENTICATED' ||
|
|
graphqlError.message?.toLowerCase() === 'unauthorized'
|
|
);
|
|
});
|
|
};
|
|
|
|
const defaultOptions: TwentyGeneratedClientOptions = {
|
|
url: '__TWENTY_DEFAULT_URL__',
|
|
headers: {
|
|
'Content-Type': 'application/json',
|
|
},
|
|
};
|
|
|
|
export class TwentyGeneratedClient {
|
|
private client: Client;
|
|
private url: string;
|
|
private requestOptions: RequestInit;
|
|
private headers: HeadersInit | (() => HeadersInit | Promise<HeadersInit>);
|
|
private fetchImplementation: typeof globalThis.fetch | null;
|
|
private authorizationToken: string | null;
|
|
private refreshAccessTokenPromise: Promise<string | null> | null = null;
|
|
|
|
constructor(options?: TwentyGeneratedClientOptions) {
|
|
const merged: TwentyGeneratedClientOptions = {
|
|
...defaultOptions,
|
|
...options,
|
|
};
|
|
|
|
const {
|
|
url,
|
|
headers,
|
|
fetch: customFetchImplementation,
|
|
fetcher: _fetcher,
|
|
batch: _batch,
|
|
...requestOptions
|
|
} = merged;
|
|
|
|
this.url = url ?? '';
|
|
this.requestOptions = requestOptions;
|
|
this.headers = headers ?? {};
|
|
this.fetchImplementation =
|
|
customFetchImplementation ?? globalThis.fetch ?? null;
|
|
|
|
const processEnvironment = getProcessEnvironment();
|
|
const tokenFromHeaders = getTokenFromHeaders(
|
|
typeof headers === 'function' ? undefined : headers,
|
|
);
|
|
|
|
// Priority: explicit header > app access token > api key (legacy).
|
|
this.authorizationToken =
|
|
tokenFromHeaders ??
|
|
processEnvironment[APP_ACCESS_TOKEN_ENV_KEY] ??
|
|
processEnvironment[API_KEY_ENV_KEY] ??
|
|
null;
|
|
|
|
this.client = createClient({
|
|
...merged,
|
|
headers: undefined,
|
|
fetcher: async (operation) =>
|
|
this.executeGraphqlRequestWithOptionalRefresh({
|
|
operation,
|
|
}),
|
|
});
|
|
}
|
|
|
|
query<R extends QueryGenqlSelection>(request: R & { __name?: string }) {
|
|
return this.client.query(request);
|
|
}
|
|
|
|
mutation<R extends MutationGenqlSelection>(request: R & { __name?: string }) {
|
|
return this.client.mutation(request);
|
|
}
|
|
|
|
// __UPLOAD_FILE_START__
|
|
async uploadFile(
|
|
fileBuffer: Buffer,
|
|
filename: string,
|
|
contentType: string = 'application/octet-stream',
|
|
fieldMetadataUniversalIdentifier: string,
|
|
): Promise<{
|
|
id: string;
|
|
path: string;
|
|
size: number;
|
|
createdAt: string;
|
|
url: string;
|
|
}> {
|
|
const form = new FormData();
|
|
|
|
form.append(
|
|
'operations',
|
|
JSON.stringify({
|
|
query: `mutation UploadFilesFieldFileByUniversalIdentifier($file: Upload!, $fieldMetadataUniversalIdentifier: String!) {
|
|
uploadFilesFieldFileByUniversalIdentifier(file: $file, fieldMetadataUniversalIdentifier: $fieldMetadataUniversalIdentifier) { id path size createdAt url }
|
|
}`,
|
|
variables: {
|
|
file: null,
|
|
fieldMetadataUniversalIdentifier,
|
|
},
|
|
}),
|
|
);
|
|
form.append('map', JSON.stringify({ '0': ['variables.file'] }));
|
|
form.append(
|
|
'0',
|
|
new Blob([fileBuffer as BlobPart], { type: contentType }),
|
|
filename,
|
|
);
|
|
|
|
const result = await this.executeGraphqlRequestWithOptionalRefresh({
|
|
operation: form,
|
|
headers: {},
|
|
requestInit: {
|
|
method: 'POST',
|
|
},
|
|
});
|
|
|
|
if (result.errors) {
|
|
throw new GenqlError(result.errors, result.data);
|
|
}
|
|
|
|
const data = result.data as Record<string, unknown>;
|
|
|
|
return data.uploadFilesFieldFileByUniversalIdentifier as {
|
|
id: string;
|
|
path: string;
|
|
size: number;
|
|
createdAt: string;
|
|
url: string;
|
|
};
|
|
}
|
|
// __UPLOAD_FILE_END__
|
|
|
|
private async executeGraphqlRequestWithOptionalRefresh({
|
|
operation,
|
|
headers,
|
|
requestInit,
|
|
}: {
|
|
operation: GraphqlOperation | GraphqlOperation[] | FormData;
|
|
headers?: HeadersInit;
|
|
requestInit?: RequestInit;
|
|
}) {
|
|
const firstResponse = await this.executeGraphqlRequest({
|
|
operation,
|
|
headers,
|
|
requestInit,
|
|
token: this.authorizationToken,
|
|
});
|
|
|
|
if (this.shouldRefreshToken(firstResponse)) {
|
|
const refreshedAccessToken = await this.requestRefreshedAccessToken();
|
|
|
|
if (refreshedAccessToken) {
|
|
const retryResponse = await this.executeGraphqlRequest({
|
|
operation,
|
|
headers,
|
|
requestInit,
|
|
token: refreshedAccessToken,
|
|
});
|
|
|
|
return this.assertResponseIsSuccessful(retryResponse);
|
|
}
|
|
}
|
|
|
|
return this.assertResponseIsSuccessful(firstResponse);
|
|
}
|
|
|
|
private async executeGraphqlRequest({
|
|
operation,
|
|
headers,
|
|
requestInit,
|
|
token,
|
|
}: {
|
|
operation: GraphqlOperation | GraphqlOperation[] | FormData;
|
|
headers?: HeadersInit;
|
|
requestInit?: RequestInit;
|
|
token: string | null;
|
|
}): Promise<GraphqlResponse> {
|
|
if (!this.fetchImplementation) {
|
|
throw new Error(
|
|
'Global `fetch` function is not available, ' +
|
|
'pass a fetch implementation to the Twenty client',
|
|
);
|
|
}
|
|
|
|
const resolvedHeaders = await this.resolveHeaders();
|
|
const requestHeaders = new Headers(resolvedHeaders);
|
|
|
|
if (headers) {
|
|
new Headers(headers).forEach((value, key) =>
|
|
requestHeaders.set(key, value),
|
|
);
|
|
}
|
|
|
|
if (operation instanceof FormData) {
|
|
requestHeaders.delete('Content-Type');
|
|
} else {
|
|
requestHeaders.set('Content-Type', 'application/json');
|
|
}
|
|
|
|
if (token) {
|
|
requestHeaders.set('Authorization', `Bearer ${token}`);
|
|
} else {
|
|
requestHeaders.delete('Authorization');
|
|
}
|
|
|
|
const response = await this.fetchImplementation.call(globalThis, this.url, {
|
|
...this.requestOptions,
|
|
...requestInit,
|
|
method: requestInit?.method ?? 'POST',
|
|
headers: requestHeaders,
|
|
body:
|
|
operation instanceof FormData ? operation : JSON.stringify(operation),
|
|
});
|
|
|
|
const rawBody = await response.text();
|
|
let payload: GraphqlResponsePayload | null = null;
|
|
|
|
if (rawBody.trim().length > 0) {
|
|
try {
|
|
payload = JSON.parse(rawBody) as GraphqlResponsePayload;
|
|
} catch {
|
|
payload = null;
|
|
}
|
|
}
|
|
|
|
return {
|
|
status: response.status,
|
|
statusText: response.statusText,
|
|
payload,
|
|
rawBody,
|
|
};
|
|
}
|
|
|
|
private async resolveHeaders(): Promise<HeadersInit> {
|
|
if (typeof this.headers === 'function') {
|
|
return (await this.headers()) ?? {};
|
|
}
|
|
|
|
return this.headers ?? {};
|
|
}
|
|
|
|
private shouldRefreshToken(response: GraphqlResponse): boolean {
|
|
if (response.status === 401) {
|
|
return true;
|
|
}
|
|
|
|
return hasAuthenticationErrorInGraphqlPayload(response.payload);
|
|
}
|
|
|
|
private assertResponseIsSuccessful(response: GraphqlResponse) {
|
|
if (response.status < 200 || response.status >= 300) {
|
|
throw new Error(`${response.statusText}: ${response.rawBody}`);
|
|
}
|
|
|
|
if (response.payload === null) {
|
|
throw new Error('Invalid JSON response');
|
|
}
|
|
|
|
return response.payload;
|
|
}
|
|
|
|
private async requestRefreshedAccessToken(): Promise<string | null> {
|
|
const refreshAccessTokenFunction = (
|
|
globalThis as {
|
|
frontComponentHostCommunicationApi?: {
|
|
requestAccessTokenRefresh?: () => Promise<string>;
|
|
};
|
|
}
|
|
).frontComponentHostCommunicationApi?.requestAccessTokenRefresh;
|
|
|
|
if (typeof refreshAccessTokenFunction !== 'function') {
|
|
return null;
|
|
}
|
|
|
|
if (!this.refreshAccessTokenPromise) {
|
|
this.refreshAccessTokenPromise = refreshAccessTokenFunction()
|
|
.then((refreshedAccessToken) => {
|
|
if (
|
|
typeof refreshedAccessToken !== 'string' ||
|
|
refreshedAccessToken.length === 0
|
|
) {
|
|
return null;
|
|
}
|
|
|
|
this.setAuthorizationToken(refreshedAccessToken);
|
|
|
|
return refreshedAccessToken;
|
|
})
|
|
.catch((refreshError: unknown) => {
|
|
console.error('Twenty client: token refresh failed', refreshError);
|
|
|
|
return null;
|
|
})
|
|
.finally(() => {
|
|
this.refreshAccessTokenPromise = null;
|
|
});
|
|
}
|
|
|
|
return this.refreshAccessTokenPromise;
|
|
}
|
|
|
|
private setAuthorizationToken(token: string) {
|
|
this.authorizationToken = token;
|
|
|
|
const processEnvironment = getProcessEnvironment();
|
|
|
|
processEnvironment[APP_ACCESS_TOKEN_ENV_KEY] = token;
|
|
}
|
|
}
|