4ea2e32366
## 1. The `twenty-client-sdk` Package (Source of Truth)
The monorepo package at `packages/twenty-client-sdk` ships with:
- A **pre-built metadata client** (static, generated from a fixed
schema)
- A **stub core client** that throws at runtime (`CoreApiClient was not
generated...`)
- Both ESM (`.mjs`) and CJS (`.cjs`) bundles in `dist/`
- A `package.json` with proper `exports` map for
`twenty-client-sdk/core`, `twenty-client-sdk/metadata`, and
`twenty-client-sdk/generate`
## 2. Generation & Upload (Server-Side, at Migration Time)
**When**: `WorkspaceMigrationRunnerService.run()` executes after a
metadata schema change.
**What happens in `SdkClientGenerationService.generateAndStore()`**:
1. Copies the stub `twenty-client-sdk` package from the server's assets
(resolved via `SDK_CLIENT_PACKAGE_DIRNAME` — from
`dist/assets/twenty-client-sdk/` in production, or from `node_modules`
in dev)
2. Filters out `node_modules/` and `src/` during copy — only
`package.json` + `dist/` are kept (like an npm publish)
3. Calls `replaceCoreClient()` which uses `@genql/cli` to introspect the
**application-scoped** GraphQL schema and generates a real
`CoreApiClient`, then compiles it to ESM+CJS and overwrites
`dist/core.mjs` and `dist/core.cjs`
4. Archives the **entire package** (with `package.json` + `dist/`) into
`twenty-client-sdk.zip`
5. Uploads the single archive to S3 under
`FileFolder.GeneratedSdkClient`
6. Sets `isSdkLayerStale = true` on the `ApplicationEntity` in the
database
## 3. Invalidation Signal
The `isSdkLayerStale` boolean column on `ApplicationEntity` is the
invalidation mechanism:
- **Set to `true`** by `generateAndStore()` after uploading a new client
archive
- **Checked** by both logic function drivers before execution — if
`true`, they rebuild their local layer
- **Set back to `false`** by `markSdkLayerFresh()` after the driver has
successfully consumed the new archive
Default is `false` so existing applications without a generated client
aren't affected.
## 4a. Logic Functions — Local Driver
**`ensureSdkLayer()`** is called before every execution:
1. Checks if the local SDK layer directory exists AND `isSdkLayerStale`
is `false` → early return
2. Otherwise, cleans the local layer directory
3. Calls `downloadAndExtractToPackage()` which streams the zip from S3
directly to disk and extracts the full package into
`<tmpdir>/sdk/<workspaceId>-<appId>/node_modules/twenty-client-sdk/`
4. Calls `markSdkLayerFresh()` to set `isSdkLayerStale = false`
**At execution time**, `assembleNodeModules()` symlinks everything from
the deps layer's `node_modules/` **except** `twenty-client-sdk`, which
is symlinked from the SDK layer instead. This ensures the logic
function's `import ... from 'twenty-client-sdk/core'` resolves to the
generated client.
## 4b. Logic Functions — Lambda Driver
**`ensureSdkLayer()`** is called during `build()`:
1. Checks if `isSdkLayerStale` is `false` and an existing Lambda layer
ARN exists → early return
2. Otherwise, deletes all existing layer versions for this SDK layer
name
3. Calls `downloadArchiveBuffer()` to get the raw zip from S3 (no disk
extraction)
4. Calls `reprefixZipEntries()` which streams the zip entries into a
**new zip** with the path prefix
`nodejs/node_modules/twenty-client-sdk/` — this is the Lambda layer
convention path. All done in memory, no disk round-trip
5. Publishes the re-prefixed zip as a new Lambda layer via
`publishLayer()`
6. Calls `markSdkLayerFresh()`
**At function creation**, the Lambda is created with **two layers**:
`[depsLayerArn, sdkLayerArn]`. The SDK layer is listed last so it
overwrites the stub `twenty-client-sdk` from the deps layer (later
layers take precedence in Lambda's `/opt` merge).
## 5. Front Components
Front components are built by `app:build` with `twenty-client-sdk/core`
and `twenty-client-sdk/metadata` as **esbuild externals**. The stored
`.mjs` in S3 has unresolved bare import specifiers like `import {
CoreApiClient } from 'twenty-client-sdk/core'`.
SDK import resolution is split between the **frontend host** (fetching &
caching SDK modules) and the **Web Worker** (rewriting imports):
**Server endpoints**:
- `GET /rest/front-components/:id` —
`FrontComponentService.getBuiltComponentStream()` returns the **raw
`.mjs`** directly from file storage. No bundling, no SDK injection.
- `GET /rest/sdk-client/:applicationId/:moduleName` —
`SdkClientController` reads a single file (e.g. `dist/core.mjs`) from
the generated SDK archive via
`SdkClientGenerationService.readFileFromArchive()` and serves it as
JavaScript.
**Frontend host** (`FrontComponentRenderer` in `twenty-front`):
1. Queries `FindOneFrontComponent` which returns `applicationId`,
`builtComponentChecksum`, `usesSdkClient`, and `applicationTokenPair`
2. If `usesSdkClient` is `true`, renders
`FrontComponentRendererWithSdkClient` which calls the
`useApplicationSdkClient` hook
3. `useApplicationSdkClient({ applicationId, accessToken })` checks the
Jotai atom family cache for existing blob URLs. On cache miss, fetches
both SDK modules from `GET /rest/sdk-client/:applicationId/core` and
`/metadata`, creates **blob URLs** for each, and stores them in the atom
family
4. Once the blob URLs are cached, passes them as `sdkClientUrls`
(already blob URLs, not server URLs) to `SharedFrontComponentRenderer` →
`FrontComponentWorkerEffect` → worker's `render()` call via
`HostToWorkerRenderContext`
**Worker** (`remote-worker.ts` in `twenty-sdk`):
1. Fetches the raw component `.mjs` source as text
2. If `sdkClientUrls` are provided and the source contains SDK import
specifiers (`twenty-client-sdk/core`, `twenty-client-sdk/metadata`),
**rewrites** the bare specifiers to the blob URLs received from the host
(e.g. `'twenty-client-sdk/core'` → `'blob:...'`)
3. Creates a blob URL for the rewritten source and `import()`s it
4. Revokes only the component blob URL after the module is loaded — the
SDK blob URLs are owned and managed by the host's Jotai cache
This approach eliminates server-side esbuild bundling on every request,
caches SDK modules per application in the frontend, and keeps the
worker's job to a simple string rewrite.
## Summary Diagram
```
app:build (SDK)
└─ twenty-client-sdk stub (metadata=real, core=stub)
│
▼
WorkspaceMigrationRunnerService.run()
└─ SdkClientGenerationService.generateAndStore()
├─ Copy stub package (package.json + dist/)
├─ replaceCoreClient() → regenerate core.mjs/core.cjs
├─ Zip entire package → upload to S3
└─ Set isSdkLayerStale = true
│
┌────────┴────────────────────┐
▼ ▼
Logic Functions Front Components
│ │
├─ Local Driver ├─ GET /rest/sdk-client/:appId/core
│ └─ downloadAndExtract │ → core.mjs from archive
│ → symlink into │
│ node_modules ├─ Host (useApplicationSdkClient)
│ │ ├─ Fetch SDK modules
└─ Lambda Driver │ ├─ Create blob URLs
└─ downloadArchiveBuffer │ └─ Cache in Jotai atom family
→ reprefixZipEntries │
→ publish as Lambda ├─ GET /rest/front-components/:id
layer │ → raw .mjs (no bundling)
│
└─ Worker (browser)
├─ Fetch component .mjs
├─ Rewrite imports → blob URLs
└─ import() rewritten source
```
## Next PR
- Estimate perf improvement by implementing a redis caching for front
component client storage ( we don't even cache front comp initially )
- Implem frontent blob invalidation sse event from server
---------
Co-authored-by: Charles Bochet <charlesBochet@users.noreply.github.com>
305 lines
8.1 KiB
TypeScript
305 lines
8.1 KiB
TypeScript
import crypto from 'crypto';
|
|
import { promises as fs } from 'fs';
|
|
import { join } from 'path';
|
|
import { tmpdir } from 'os';
|
|
|
|
import request from 'supertest';
|
|
import * as tar from 'tar';
|
|
import { type DataSource } from 'typeorm';
|
|
import { uploadAppTarball } from 'test/integration/metadata/suites/application/utils/upload-app-tarball.util';
|
|
|
|
const TEST_WORKSPACE_ID = '20202020-1c25-4d02-bf25-6aeccf7ea419';
|
|
|
|
const createTestTarball = async (
|
|
files: Record<string, string>,
|
|
): Promise<Buffer> => {
|
|
const tempId = crypto.randomUUID();
|
|
const sourceDir = join(tmpdir(), `test-tarball-src-${tempId}`);
|
|
const tarballPath = join(tmpdir(), `test-tarball-${tempId}.tar.gz`);
|
|
|
|
await fs.mkdir(sourceDir, { recursive: true });
|
|
|
|
for (const [name, content] of Object.entries(files)) {
|
|
const filePath = join(sourceDir, name);
|
|
const dir = filePath.substring(0, filePath.lastIndexOf('/'));
|
|
|
|
if (dir !== sourceDir) {
|
|
await fs.mkdir(dir, { recursive: true });
|
|
}
|
|
await fs.writeFile(filePath, content);
|
|
}
|
|
|
|
await tar.create(
|
|
{
|
|
file: tarballPath,
|
|
gzip: true,
|
|
cwd: sourceDir,
|
|
},
|
|
Object.keys(files),
|
|
);
|
|
|
|
const buffer = await fs.readFile(tarballPath);
|
|
|
|
await fs.rm(sourceDir, { recursive: true, force: true });
|
|
await fs.rm(tarballPath, { force: true });
|
|
|
|
return buffer;
|
|
};
|
|
|
|
const createValidManifest = (universalIdentifier: string) =>
|
|
JSON.stringify({
|
|
application: {
|
|
universalIdentifier,
|
|
displayName: 'Test Tarball App',
|
|
description: 'A test app',
|
|
icon: 'IconTestPipe',
|
|
defaultRoleUniversalIdentifier: crypto.randomUUID(),
|
|
applicationVariables: {},
|
|
packageJsonChecksum: null,
|
|
yarnLockChecksum: null,
|
|
},
|
|
roles: [],
|
|
skills: [],
|
|
objects: [],
|
|
fields: [],
|
|
logicFunctions: [],
|
|
frontComponents: [],
|
|
publicAssets: [],
|
|
views: [],
|
|
navigationMenuItems: [],
|
|
pageLayouts: [],
|
|
});
|
|
|
|
const insertRegistrationWithSource = async (
|
|
ds: DataSource,
|
|
params: {
|
|
universalIdentifier: string;
|
|
name: string;
|
|
sourceType: 'npm' | 'tarball' | 'local';
|
|
sourcePackage?: string;
|
|
},
|
|
): Promise<string> => {
|
|
const id = crypto.randomUUID();
|
|
const oAuthClientId = crypto.randomUUID();
|
|
|
|
await ds.query(
|
|
`INSERT INTO core."applicationRegistration"
|
|
(id, "universalIdentifier", name, "oAuthClientId",
|
|
"oAuthRedirectUris", "oAuthScopes", "workspaceId",
|
|
"sourceType", "sourcePackage")
|
|
VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9)`,
|
|
[
|
|
id,
|
|
params.universalIdentifier,
|
|
params.name,
|
|
oAuthClientId,
|
|
[],
|
|
[],
|
|
TEST_WORKSPACE_ID,
|
|
params.sourceType,
|
|
params.sourcePackage ?? null,
|
|
],
|
|
);
|
|
|
|
return id;
|
|
};
|
|
|
|
describe('App Distribution (integration)', () => {
|
|
let ds: DataSource;
|
|
const createdRegistrationIds: string[] = [];
|
|
|
|
beforeAll(() => {
|
|
jest.useRealTimers();
|
|
ds = global.testDataSource;
|
|
});
|
|
|
|
afterAll(async () => {
|
|
for (const id of createdRegistrationIds) {
|
|
await ds.query(
|
|
`DELETE FROM core."applicationRegistration" WHERE id = $1`,
|
|
[id],
|
|
);
|
|
}
|
|
jest.useFakeTimers();
|
|
});
|
|
|
|
describe('Upload tarball via GraphQL', () => {
|
|
it('should reject tarball without manifest.json', async () => {
|
|
const tarball = await createTestTarball({
|
|
'readme.txt': 'no manifest here',
|
|
});
|
|
|
|
const { errors } = await uploadAppTarball({
|
|
tarballBuffer: tarball,
|
|
expectToFail: true,
|
|
});
|
|
|
|
expect(errors).toBeDefined();
|
|
expect(
|
|
errors?.some((error: { message: string }) =>
|
|
error.message.includes('manifest.json'),
|
|
),
|
|
).toBe(true);
|
|
});
|
|
|
|
it('should successfully upload a valid tarball and create registration', async () => {
|
|
const uid = crypto.randomUUID();
|
|
const manifest = createValidManifest(uid);
|
|
|
|
const tarball = await createTestTarball({
|
|
'manifest.json': manifest,
|
|
'package.json': JSON.stringify({
|
|
name: 'test-app',
|
|
version: '1.0.0',
|
|
}),
|
|
});
|
|
|
|
const { data, errors } = await uploadAppTarball({
|
|
tarballBuffer: tarball,
|
|
universalIdentifier: uid,
|
|
});
|
|
|
|
expect(errors).toBeUndefined();
|
|
expect(data?.uploadAppTarball.id).toBeDefined();
|
|
expect(data?.uploadAppTarball.universalIdentifier).toBe(uid);
|
|
createdRegistrationIds.push(data!.uploadAppTarball.id);
|
|
|
|
const rows = await ds.query(
|
|
`SELECT "sourceType" FROM core."applicationRegistration"
|
|
WHERE id = $1`,
|
|
[data!.uploadAppTarball.id],
|
|
);
|
|
|
|
expect(rows[0].sourceType).toBe('tarball');
|
|
});
|
|
|
|
it('should update existing tarball registration on re-upload', async () => {
|
|
const uid = crypto.randomUUID();
|
|
const manifest = createValidManifest(uid);
|
|
|
|
const tarball = await createTestTarball({
|
|
'manifest.json': manifest,
|
|
'package.json': JSON.stringify({
|
|
name: 'test-app',
|
|
version: '1.0.0',
|
|
}),
|
|
});
|
|
|
|
const firstResult = await uploadAppTarball({
|
|
tarballBuffer: tarball,
|
|
universalIdentifier: uid,
|
|
});
|
|
|
|
createdRegistrationIds.push(firstResult.data!.uploadAppTarball.id);
|
|
|
|
const secondResult = await uploadAppTarball({
|
|
tarballBuffer: tarball,
|
|
universalIdentifier: uid,
|
|
});
|
|
|
|
expect(secondResult.data?.uploadAppTarball.id).toBe(
|
|
firstResult.data?.uploadAppTarball.id,
|
|
);
|
|
});
|
|
});
|
|
|
|
describe('Source channel enforcement', () => {
|
|
it('should reject tarball upload for npm-sourced registration', async () => {
|
|
const uid = crypto.randomUUID();
|
|
|
|
const regId = await insertRegistrationWithSource(ds, {
|
|
universalIdentifier: uid,
|
|
name: 'NPM App',
|
|
sourceType: 'npm',
|
|
sourcePackage: '@test/npm-app',
|
|
});
|
|
|
|
createdRegistrationIds.push(regId);
|
|
|
|
const manifest = createValidManifest(uid);
|
|
const tarball = await createTestTarball({
|
|
'manifest.json': manifest,
|
|
'package.json': JSON.stringify({
|
|
name: 'test-app',
|
|
version: '1.0.0',
|
|
}),
|
|
});
|
|
|
|
const { errors } = await uploadAppTarball({
|
|
tarballBuffer: tarball,
|
|
universalIdentifier: uid,
|
|
expectToFail: true,
|
|
});
|
|
|
|
expect(errors).toBeDefined();
|
|
expect(
|
|
errors?.some((error: { message: string }) =>
|
|
error.message.includes('registered as npm'),
|
|
),
|
|
).toBe(true);
|
|
});
|
|
|
|
it('should allow tarball upload for local-sourced registration', async () => {
|
|
const uid = crypto.randomUUID();
|
|
|
|
const regId = await insertRegistrationWithSource(ds, {
|
|
universalIdentifier: uid,
|
|
name: 'Local Source App',
|
|
sourceType: 'local',
|
|
});
|
|
|
|
createdRegistrationIds.push(regId);
|
|
|
|
const manifest = createValidManifest(uid);
|
|
const tarball = await createTestTarball({
|
|
'manifest.json': manifest,
|
|
'package.json': JSON.stringify({
|
|
name: 'test-app',
|
|
version: '1.0.0',
|
|
}),
|
|
});
|
|
|
|
const { data, errors } = await uploadAppTarball({
|
|
tarballBuffer: tarball,
|
|
universalIdentifier: uid,
|
|
});
|
|
|
|
expect(errors).toBeUndefined();
|
|
expect(data?.uploadAppTarball.id).toBe(regId);
|
|
|
|
const rows = await ds.query(
|
|
`SELECT "sourceType" FROM core."applicationRegistration"
|
|
WHERE id = $1`,
|
|
[regId],
|
|
);
|
|
|
|
expect(rows[0].sourceType).toBe('tarball');
|
|
});
|
|
});
|
|
|
|
describe('Authentication', () => {
|
|
it('should reject unauthenticated requests', async () => {
|
|
const baseUrl = `http://localhost:${APP_PORT}`;
|
|
|
|
const response = await request(baseUrl)
|
|
.post('/metadata')
|
|
.field(
|
|
'operations',
|
|
JSON.stringify({
|
|
query: `mutation UploadAppTarball($file: Upload!) {
|
|
uploadAppTarball(file: $file) { id }
|
|
}`,
|
|
variables: { file: null },
|
|
}),
|
|
)
|
|
.field('map', JSON.stringify({ '0': ['variables.file'] }))
|
|
.attach('0', Buffer.from('test'), {
|
|
filename: 'app.tar.gz',
|
|
contentType: 'application/gzip',
|
|
});
|
|
|
|
expect(response.body.errors).toBeDefined();
|
|
});
|
|
});
|
|
});
|