Cache twenty-client-sdk modules host-side via content-addressed URLs (#22981)

## Context

Front component sources are fetched host-side and integrity-verified by
the SHA-256 checksum embedded in their URL, cached in Cache Storage — a
layer that exists specifically because their download URLs are presigned
and rotate. The `twenty-client-sdk` modules (`core` and `metadata`) were
re-fetched on every render and could not be cached safely: their URLs
carried no checksum and the server exposed no freshness signal.

This PR makes the SDK module URLs **content-addressed** and relies on
the **browser HTTP cache** for immutability, and it keys the checksums
on their real owners: the **application** for `core`, the **instance**
for `metadata`. The checksum does double duty: cache invalidation
(regeneration changes the checksum → the URL changes → guaranteed cache
miss) and a server-side cacheability guard (the server only grants
`immutable` when the checksum in the URL matches the authoritative
checksum it knows for that module — persisted at generation time for
`core`, hashed once at bootstrap for `metadata` — so no per-request
hashing of the served bytes). Note this is **not** an end-to-end
integrity guarantee: there is no client-side hash verification, and on a
fingerprint mismatch the server still serves the current bytes with
`no-store` (self-healing for stale URLs) rather than failing.

<img width="2412" height="926" alt="image"
src="https://github.com/user-attachments/assets/d97935d2-0fdb-4c44-89ac-596b7ca8ca64"
/>

Closes twentyhq/core-team-issues#2688.

## Routes

| Module | URL | Scope |
| --- | --- | --- |
| `core` | `/rest/sdk-client/{applicationId}/core[/{checksum}]` | Per
application (generated bundle) |
| `metadata` | `/rest/sdk-client/metadata[/{checksum}]` |
**Instance-wide**: no application segment, so every application
converges on one URL and the browser downloads the module once per
release instead of once per application |

The previous application-scoped metadata path
(`/rest/sdk-client/{applicationId}/metadata[/{checksum}]`) is **kept for
backward compatibility**, new clients just stop generating those URLs.
The instance-wide route is declared before the parameterized route so
`metadata/{checksum}` is not swallowed as `:applicationId/:moduleName`.

## Caching model

| Request | `Cache-Control` | Effect |
| --- | --- | --- |
| Fingerprinted URL, checksum matches the known module checksum |
`immutable` | Cached indefinitely by the browser HTTP cache; a new
checksum is a new URL |
| Fingerprinted URL, checksum does not match | `no-store` | Current
bytes served uncached (self-healing for stale URLs) |
| Bare URL (pre-generation fallback, `core` only in practice) |
`no-store` | Never cached |

- Both responses also set `X-Content-Type-Options: nosniff` and
`Content-Type: application/javascript`.
- SDK modules are intentionally **not** placed in Cache Storage. That
layer stays reserved for the presigned/rotating component-source URLs;
SDK modules are served directly and authenticated, so the browser HTTP
cache (keyed by the content-addressed URL) is their single cache layer.

## Checksum provenance

- **core** — per **application**, persisted on
`application.sdkClientCoreChecksum` at generation time and read back
from `flatApplicationMaps` (never re-hashed per request).
- **metadata** — **instance-wide**, hashed once from the installed
`twenty-client-sdk/dist/metadata.mjs` package (warmed at bootstrap,
memoized per process) and served straight from that package, so it is
fresh from the first request after a release with no archive dependency.

## Server (twenty-server)

- Hash `dist/core.mjs` at SDK generation and persist
`sdkClientCoreChecksum` via `applicationRepository.update`. Adds the
nullable text column to `application.entity.ts` (mirroring
`packageJsonChecksum`) plus a fast instance command with up/down;
`FlatApplication` picks it up automatically.
- New **application-scoped** query
`applicationSdkClientChecksums(applicationId: UUID!):
SdkClientChecksums` on `ApplicationResolver` (metadata schema,
`WorkspaceAuthGuard` + `NoPermissionGuard`). `SdkClientChecksums.core`
is **nullable** and stays `null` until the SDK has been generated at
least once; `metadata` is **always present** (bootstrap-warmed), so the
metadata module is cacheable from the very first render of any app. The
query itself returns `null` only for unknown applications.
- `SdkClientChecksumsDTO` now lives in the shared
`core-modules/sdk-client/dtos/`. `FrontComponentDTO` and the
`frontComponent` resolver no longer carry checksums (decoupled from the
front-component row).
- `sdk-client` controller: instance-wide `metadata[/:checksum]` route
(no workspace-cache or application lookup, serves the memoized installed
module) + application-scoped `:applicationId/:moduleName[/:checksum]`
route (serves `core` from the per-application archive, `metadata` kept
for back-compat). Cacheability compares the URL checksum against the
**known** checksum — persisted `sdkClientCoreChecksum` for `core`,
memoized package hash for `metadata` — instead of hashing the served
bytes on every request: `immutable` on match, `no-store` otherwise (bare
URL or stale fingerprint), plus `nosniff`. A persisted checksum out of
sync with the archive only downgrades to `no-store` until the next
regeneration.

## Front (twenty-front)

- New metadata query `GetApplicationSdkClientChecksums`, keyed by
`applicationId`; removed the `sdkClientChecksums` selection from
`FindOneFrontComponent`.
- `getSdkClientUrls` builds the two module URLs independently:
`/sdk-client/{applicationId}/core/{checksum}` and the **instance-wide**
`/sdk-client/metadata/{checksum}` (no application segment → one shared
browser cache entry per release across all applications). Each falls
back to its bare URL when its checksum is absent — since `core` is
nullable, a never-generated app still gets a content-addressed metadata
URL and only `core` falls back. The checksum type is sourced from the
codegen `SdkClientChecksums` type rather than a hand-maintained
duplicate.
- `FrontComponentRenderer` is split into a gating outer component (runs
`FindOneFrontComponent`, renders nothing while loading) and a content
component that receives a guaranteed-non-null `frontComponent`.
Following project conventions, the side effects live in dedicated effect
components: `FrontComponentLoadErrorSnackBarEffect` (query error →
snackbar) and `FrontComponentApplicationTokenPairEffect` (mirrors the
query-derived token pair into component state unconditionally, `null`
included, so revoked credentials can never be retained or refreshed).
The content component fetches checksums via the application-keyed query
and **gates the mount of SDK-using components on that query**, so the
very first module fetch is always the content-addressed (`immutable`)
URL instead of the bare `no-store` one. Non-SDK components skip the
query and are never blocked.
- **Live invalidation without reload:** SDK regeneration updates the
application row, and the server broadcasts an `application` metadata
event carrying the new core checksum.
`useOnApplicationSdkClientChecksumsUpdated` /
`useUpdateSdkClientChecksumsApolloCache` patch the application-keyed
checksum query cache (core only; the instance-wide metadata is
preserved), so every mounted component of that application picks up the
new URL at once. This replaces the previous frontComponent-derived field
and closes the earlier "known gap" (a mounted component staying on a
session-old checksum until a full reload). The cache-patching callback
is memoized (`useCallback`) so the window listener is registered once
per application, and the listener is **skipped entirely** for non-SDK
components (`useListenToMetadataOperationBrowserEvent` gained a `skip`
option) — they register no listener and never refetch a query they don't
consume.

## Renderer (twenty-front-component-renderer)

- SDK sources are fetched through a dedicated plain authenticated fetch,
`fetchJavaScriptModuleSourceText` (Bearer header, `credentials:
'omit'`), instead of the Cache Storage `fetchComponentSource` path;
`fetchSdkClientSources` uses it. Execution stays exclusively in the
opaque-origin worker via blob URLs; the host only fetches and forwards
source strings (no hashing host-side). Staleness self-resolves through
the checksum: new checksum → new URL → cache miss.

<!-- This is an auto-generated description by cubic. -->
<a
href="https://cubic.dev/pr/twentyhq/twenty/pull/22981?utm_source=github"
target="_blank" rel="noopener noreferrer"
data-no-image-dialog="true"><picture><source
media="(prefers-color-scheme: dark)"
srcset="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"><source
media="(prefers-color-scheme: light)"
srcset="https://www.cubic.dev/buttons/review-in-cubic-light.svg"><img
alt="Review in cubic"
src="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"></picture></a>
<!-- End of auto-generated description by cubic. -->
This commit is contained in:
Paul Rastoin
2026-07-21 17:05:00 +02:00
committed by GitHub
parent 8f704e87e3
commit 71a1ff7ac8
58 changed files with 2137 additions and 1048 deletions
@@ -92,6 +92,7 @@ describe('ApplicationManifestApplyService', () => {
workspaceId: WORKSPACE_ID,
applicationId: APPLICATION_ID,
applicationUniversalIdentifier: 'my-app',
trigger: 'manifest-sync',
});
});
@@ -51,6 +51,7 @@ export class ApplicationManifestApplyService {
workspaceId,
applicationId: application.id,
applicationUniversalIdentifier: application.universalIdentifier,
trigger: 'manifest-sync',
});
}
@@ -163,6 +163,7 @@ export class ApplicationSyncService {
canBeUninstalled: true,
autoUpgrade: false,
isSdkLayerStale: false,
sdkClientCoreChecksum: null,
applicationRegistrationId: null,
primaryPublicDomainId: null,
createdAt: now,
@@ -134,6 +134,13 @@ export class ApplicationEntity extends WorkspaceRelatedEntity {
@Column({ nullable: false, type: 'boolean', default: false })
isSdkLayerStale: boolean;
@Column({ nullable: true, type: 'text' })
@WasIntroducedInUpgrade({
upgradeCommandName:
'2.23.0_AddSdkClientCoreChecksumToApplicationFastInstanceCommand_1784625638000',
})
sdkClientCoreChecksum: string | null;
@Column({ nullable: true, type: 'uuid' })
applicationRegistrationId: string | null;
@@ -1,20 +1,50 @@
import { UseGuards } from '@nestjs/common';
import { Parent, ResolveField } from '@nestjs/graphql';
import { Args, Parent, Query, ResolveField } from '@nestjs/graphql';
import { isAbsoluteUrl, isDefined } from 'twenty-shared/utils';
import { MetadataResolver } from 'src/engine/api/graphql/graphql-config/decorators/metadata-resolver.decorator';
import { UUIDScalarType } from 'src/engine/api/graphql/workspace-schema-builder/graphql-types/scalars';
import { ApplicationDTO } from 'src/engine/core-modules/application/dtos/application.dto';
import { SdkClientChecksumsDTO } from 'src/engine/core-modules/sdk-client/dtos/sdk-client-checksums.dto';
import { getInstalledSdkMetadataModule } from 'src/engine/core-modules/sdk-client/utils/get-installed-sdk-metadata-module.util';
import { TwentyConfigService } from 'src/engine/core-modules/twenty-config/twenty-config.service';
import { WorkspaceEntity } from 'src/engine/core-modules/workspace/workspace.entity';
import { AuthWorkspace } from 'src/engine/decorators/auth/auth-workspace.decorator';
import { NoPermissionGuard } from 'src/engine/guards/no-permission.guard';
import { WorkspaceAuthGuard } from 'src/engine/guards/workspace-auth.guard';
import { WorkspaceCacheService } from 'src/engine/workspace-cache/services/workspace-cache.service';
@UseGuards(WorkspaceAuthGuard, NoPermissionGuard)
@MetadataResolver(() => ApplicationDTO)
export class ApplicationResolver {
constructor(private readonly twentyConfigService: TwentyConfigService) {}
constructor(
private readonly twentyConfigService: TwentyConfigService,
private readonly workspaceCacheService: WorkspaceCacheService,
) {}
@Query(() => SdkClientChecksumsDTO, { nullable: true })
async applicationSdkClientChecksums(
@Args('applicationId', { type: () => UUIDScalarType })
applicationId: string,
@AuthWorkspace() workspace: WorkspaceEntity,
): Promise<SdkClientChecksumsDTO | null> {
const { flatApplicationMaps } =
await this.workspaceCacheService.getOrRecompute(workspace.id, [
'flatApplicationMaps',
]);
const application = flatApplicationMaps.byId[applicationId];
if (!isDefined(application)) {
return null;
}
return {
core: application.sdkClientCoreChecksum,
metadata: (await getInstalledSdkMetadataModule()).checksum,
};
}
// Resolves the display url of the logo bundled in the installed
// application's public assets, so clients never build file urls themselves.
@@ -65,6 +65,9 @@ export enum MetricsKeys {
AiChatTurnCompleted = 'ai-chat/turn-completed',
AiChatTurnFailed = 'ai-chat/turn-failed',
WorkspaceMetadataCacheLocalEviction = 'workspace-metadata-cache/local-eviction',
SdkClientGenerationSucceeded = 'sdk-client-generation/succeeded',
SdkClientGenerationFailed = 'sdk-client-generation/failed',
SdkClientGenerationDurationMs = 'sdk-client-generation/duration-ms',
ConnectedAccountSyncWebhookReceivedMessaging = 'connected-account-sync-webhook/received/messaging',
ConnectedAccountSyncWebhookReceivedCalendar = 'connected-account-sync-webhook/received/calendar',
}
@@ -10,9 +10,11 @@ import { FileStorageService } from 'src/engine/core-modules/file-storage/service
import { getQueueToken } from 'src/engine/core-modules/message-queue/utils/get-queue-token.util';
import { MessageQueue } from 'src/engine/core-modules/message-queue/message-queue.constants';
import { MessageQueueService } from 'src/engine/core-modules/message-queue/services/message-queue.service';
import { MetricsService } from 'src/engine/core-modules/metrics/metrics.service';
import { GENERATE_SDK_CLIENT_JOB_NAME } from 'src/engine/core-modules/sdk-client/jobs/generate-sdk-client.job-constants';
import { SdkClientGenerationService } from 'src/engine/core-modules/sdk-client/sdk-client-generation.service';
import { WorkspaceEntity } from 'src/engine/core-modules/workspace/workspace.entity';
import { WorkspaceEventBroadcaster } from 'src/engine/subscriptions/workspace-event-broadcaster/workspace-event-broadcaster.service';
import { WorkspaceCacheService } from 'src/engine/workspace-cache/services/workspace-cache.service';
describe('SdkClientGenerationService', () => {
@@ -24,6 +26,9 @@ describe('SdkClientGenerationService', () => {
>
>;
let messageQueueService: jest.Mocked<Pick<MessageQueueService, 'add'>>;
let workspaceCacheService: jest.Mocked<
Pick<WorkspaceCacheService, 'getOrRecompute'>
>;
beforeEach(async () => {
applicationService = {
@@ -32,6 +37,9 @@ describe('SdkClientGenerationService', () => {
messageQueueService = {
add: jest.fn().mockResolvedValue(undefined),
};
workspaceCacheService = {
getOrRecompute: jest.fn(),
};
const module: TestingModule = await Test.createTestingModule({
providers: [
@@ -45,13 +53,24 @@ describe('SdkClientGenerationService', () => {
provide: getRepositoryToken(WorkspaceEntity),
useValue: {} as Repository<WorkspaceEntity>,
},
{ provide: WorkspaceCacheService, useValue: {} },
{ provide: WorkspaceCacheService, useValue: workspaceCacheService },
{ provide: WorkspaceSchemaFactory, useValue: {} },
{ provide: ApplicationService, useValue: applicationService },
{
provide: getQueueToken(MessageQueue.workspaceQueue),
useValue: messageQueueService,
},
{
provide: WorkspaceEventBroadcaster,
useValue: { broadcast: jest.fn().mockResolvedValue(undefined) },
},
{
provide: MetricsService,
useValue: {
incrementCounterBy: jest.fn(),
recordHistogram: jest.fn(),
},
},
],
}).compile();
@@ -88,6 +107,7 @@ describe('SdkClientGenerationService', () => {
workspaceId,
applicationId: 'std-app-id',
applicationUniversalIdentifier: 'twenty-standard',
trigger: 'workspace-activation',
},
{
id: `sdk-client:${workspaceId}:std-app-id`,
@@ -101,6 +121,7 @@ describe('SdkClientGenerationService', () => {
workspaceId,
applicationId: 'custom-app-id',
applicationUniversalIdentifier: 'workspace-custom',
trigger: 'workspace-activation',
},
{
id: `sdk-client:${workspaceId}:custom-app-id`,
@@ -0,0 +1,257 @@
import { NotFoundException } from '@nestjs/common';
import { Test, type TestingModule } from '@nestjs/testing';
import { type Response } from 'express';
import {
SDK_CLIENT_MODULE_CACHE_CONTROL,
SDK_CLIENT_MODULE_NO_STORE_CACHE_CONTROL,
} from 'src/engine/core-modules/sdk-client/constants/sdk-client-module-cache-control';
import { SdkClientController } from 'src/engine/core-modules/sdk-client/controllers/sdk-client.controller';
import { SdkClientArchiveService } from 'src/engine/core-modules/sdk-client/sdk-client-archive.service';
import { getInstalledSdkMetadataModule } from 'src/engine/core-modules/sdk-client/utils/get-installed-sdk-metadata-module.util';
import { type WorkspaceEntity } from 'src/engine/core-modules/workspace/workspace.entity';
import { WorkspaceCacheService } from 'src/engine/workspace-cache/services/workspace-cache.service';
jest.mock(
'src/engine/core-modules/sdk-client/utils/get-installed-sdk-metadata-module.util',
() => ({
getInstalledSdkMetadataModule: jest.fn(),
}),
);
const mockGetInstalledSdkMetadataModule = jest.mocked(
getInstalledSdkMetadataModule,
);
const WORKSPACE_ID = 'workspace-1';
const APPLICATION_ID = 'app-1';
const CORE_MODULE_BUFFER = Buffer.from('core module from archive');
const PERSISTED_CORE_CHECKSUM = 'b'.repeat(64);
const workspace = { id: WORKSPACE_ID } as WorkspaceEntity;
describe('SdkClientController', () => {
let controller: SdkClientController;
let workspaceCacheService: jest.Mocked<
Pick<WorkspaceCacheService, 'getOrRecompute'>
>;
let sdkClientArchiveService: jest.Mocked<
Pick<SdkClientArchiveService, 'getClientModuleFromArchive'>
>;
let response: jest.Mocked<Pick<Response, 'setHeader' | 'send'>>;
beforeEach(async () => {
jest.clearAllMocks();
workspaceCacheService = {
getOrRecompute: jest.fn().mockResolvedValue({
flatApplicationMaps: {
byId: {
[APPLICATION_ID]: {
id: APPLICATION_ID,
universalIdentifier: 'my-app',
sdkClientCoreChecksum: PERSISTED_CORE_CHECKSUM,
},
},
},
}),
};
sdkClientArchiveService = {
getClientModuleFromArchive: jest
.fn()
.mockResolvedValue(CORE_MODULE_BUFFER),
};
response = {
setHeader: jest.fn(),
send: jest.fn(),
};
const module: TestingModule = await Test.createTestingModule({
controllers: [SdkClientController],
providers: [
{ provide: WorkspaceCacheService, useValue: workspaceCacheService },
{
provide: SdkClientArchiveService,
useValue: sdkClientArchiveService,
},
],
}).compile();
controller = module.get<SdkClientController>(SdkClientController);
});
describe('instance-wide metadata route', () => {
const INSTALLED_METADATA_BUFFER = Buffer.from('installed metadata module');
const INSTALLED_METADATA_CHECKSUM = 'a'.repeat(64);
beforeEach(() => {
mockGetInstalledSdkMetadataModule.mockResolvedValue({
moduleBuffer: INSTALLED_METADATA_BUFFER,
checksum: INSTALLED_METADATA_CHECKSUM,
});
});
it('serves the installed metadata module without touching workspace caches', async () => {
await controller.getInstanceSdkMetadataModule(
response as unknown as Response,
);
expect(workspaceCacheService.getOrRecompute).not.toHaveBeenCalled();
expect(
sdkClientArchiveService.getClientModuleFromArchive,
).not.toHaveBeenCalled();
expect(response.send).toHaveBeenCalledWith(INSTALLED_METADATA_BUFFER);
});
it('opts out of caching on the bare url', async () => {
await controller.getInstanceSdkMetadataModule(
response as unknown as Response,
);
expect(response.setHeader).toHaveBeenCalledWith(
'Cache-Control',
SDK_CLIENT_MODULE_NO_STORE_CACHE_CONTROL,
);
});
it('serves fingerprinted urls as immutable when the checksum matches the installed module', async () => {
await controller.getInstanceSdkMetadataModule(
response as unknown as Response,
INSTALLED_METADATA_CHECKSUM,
);
expect(response.setHeader).toHaveBeenCalledWith(
'Cache-Control',
SDK_CLIENT_MODULE_CACHE_CONTROL,
);
});
it('opts out of caching when the fingerprint does not match the installed module', async () => {
await controller.getInstanceSdkMetadataModule(
response as unknown as Response,
'c'.repeat(64),
);
expect(response.setHeader).toHaveBeenCalledWith(
'Cache-Control',
SDK_CLIENT_MODULE_NO_STORE_CACHE_CONTROL,
);
});
});
it('serves the metadata module from the installed package, not the archive', async () => {
const installedModuleBuffer = Buffer.from('installed metadata module');
mockGetInstalledSdkMetadataModule.mockResolvedValue({
moduleBuffer: installedModuleBuffer,
checksum: 'a'.repeat(64),
});
await controller.getSdkModule(
response as unknown as Response,
APPLICATION_ID,
'metadata',
workspace,
);
expect(
sdkClientArchiveService.getClientModuleFromArchive,
).not.toHaveBeenCalled();
expect(response.send).toHaveBeenCalledWith(installedModuleBuffer);
});
it('serves the core module from the application archive', async () => {
await controller.getSdkModule(
response as unknown as Response,
APPLICATION_ID,
'core',
workspace,
);
expect(
sdkClientArchiveService.getClientModuleFromArchive,
).toHaveBeenCalledWith({
workspaceId: WORKSPACE_ID,
applicationId: APPLICATION_ID,
applicationUniversalIdentifier: 'my-app',
moduleName: 'core',
});
expect(mockGetInstalledSdkMetadataModule).not.toHaveBeenCalled();
});
it('disables MIME sniffing and opts out of HTTP caching on the bare fallback url', async () => {
await controller.getSdkModule(
response as unknown as Response,
APPLICATION_ID,
'core',
workspace,
);
expect(response.setHeader).toHaveBeenCalledWith(
'Content-Type',
'application/javascript',
);
expect(response.setHeader).toHaveBeenCalledWith(
'X-Content-Type-Options',
'nosniff',
);
expect(response.setHeader).toHaveBeenCalledWith(
'Cache-Control',
SDK_CLIENT_MODULE_NO_STORE_CACHE_CONTROL,
);
});
it('serves fingerprinted urls as immutable when the checksum matches the persisted core checksum', async () => {
await controller.getSdkModule(
response as unknown as Response,
APPLICATION_ID,
'core',
workspace,
PERSISTED_CORE_CHECKSUM,
);
expect(response.setHeader).toHaveBeenCalledWith(
'Cache-Control',
SDK_CLIENT_MODULE_CACHE_CONTROL,
);
});
it('opts out of caching when the fingerprint does not match the persisted checksum', async () => {
await controller.getSdkModule(
response as unknown as Response,
APPLICATION_ID,
'core',
workspace,
'a'.repeat(64),
);
expect(response.setHeader).toHaveBeenCalledWith(
'Cache-Control',
SDK_CLIENT_MODULE_NO_STORE_CACHE_CONTROL,
);
});
it('rejects unknown module names', async () => {
await expect(
controller.getSdkModule(
response as unknown as Response,
APPLICATION_ID,
'evil' as never,
workspace,
),
).rejects.toBeInstanceOf(NotFoundException);
});
it('rejects unknown applications', async () => {
await expect(
controller.getSdkModule(
response as unknown as Response,
'unknown-app',
'metadata',
workspace,
),
).rejects.toBeInstanceOf(NotFoundException);
});
});
@@ -0,0 +1,4 @@
export const SDK_CLIENT_MODULE_CACHE_CONTROL =
'private, max-age=31536000, immutable';
export const SDK_CLIENT_MODULE_NO_STORE_CACHE_CONTROL = 'private, no-store';
@@ -14,7 +14,12 @@ import {
ALLOWED_SDK_MODULES,
type SdkModuleName,
} from 'src/engine/core-modules/sdk-client/constants/allowed-sdk-modules';
import {
SDK_CLIENT_MODULE_CACHE_CONTROL,
SDK_CLIENT_MODULE_NO_STORE_CACHE_CONTROL,
} from 'src/engine/core-modules/sdk-client/constants/sdk-client-module-cache-control';
import { SdkClientArchiveService } from 'src/engine/core-modules/sdk-client/sdk-client-archive.service';
import { getInstalledSdkMetadataModule } from 'src/engine/core-modules/sdk-client/utils/get-installed-sdk-metadata-module.util';
import { WorkspaceEntity } from 'src/engine/core-modules/workspace/workspace.entity';
import { AuthWorkspace } from 'src/engine/decorators/auth/auth-workspace.decorator';
import { NoPermissionGuard } from 'src/engine/guards/no-permission.guard';
@@ -29,13 +34,31 @@ export class SdkClientController {
private readonly sdkClientArchiveService: SdkClientArchiveService,
) {}
@Get(':applicationId/:moduleName')
@Get(['metadata', 'metadata/:checksum'])
@UseGuards(NoPermissionGuard)
async getInstanceSdkMetadataModule(
@Res() res: Response,
@Param('checksum') checksum?: string,
) {
const { moduleBuffer, checksum: servedModuleChecksum } =
await getInstalledSdkMetadataModule();
this.sendSdkModule({
res,
moduleBuffer,
servedModuleChecksum,
requestedChecksum: checksum,
});
}
@Get([':applicationId/:moduleName', ':applicationId/:moduleName/:checksum'])
@UseGuards(NoPermissionGuard)
async getSdkModule(
@Res() res: Response,
@Param('applicationId') applicationId: string,
@Param('moduleName') moduleName: SdkModuleName,
@AuthWorkspace() workspace: WorkspaceEntity,
@Param('checksum') checksum?: string,
) {
if (!ALLOWED_SDK_MODULES.includes(moduleName)) {
throw new NotFoundException(
@@ -56,15 +79,51 @@ export class SdkClientController {
);
}
const fileBuffer =
await this.sdkClientArchiveService.getClientModuleFromArchive({
workspaceId: workspace.id,
applicationId,
applicationUniversalIdentifier: application.universalIdentifier,
moduleName,
});
const { moduleBuffer, checksum: servedModuleChecksum } =
moduleName === 'metadata'
? await getInstalledSdkMetadataModule()
: {
moduleBuffer:
await this.sdkClientArchiveService.getClientModuleFromArchive({
workspaceId: workspace.id,
applicationId,
applicationUniversalIdentifier: application.universalIdentifier,
moduleName,
}),
checksum: application.sdkClientCoreChecksum,
};
this.sendSdkModule({
res,
moduleBuffer,
servedModuleChecksum,
requestedChecksum: checksum,
});
}
private sendSdkModule({
res,
moduleBuffer,
servedModuleChecksum,
requestedChecksum,
}: {
res: Response;
moduleBuffer: Buffer;
servedModuleChecksum: string | null;
requestedChecksum?: string;
}) {
const isChecksumMatch =
isDefined(requestedChecksum) &&
requestedChecksum === servedModuleChecksum;
res.setHeader('Content-Type', 'application/javascript');
res.send(fileBuffer);
res.setHeader('X-Content-Type-Options', 'nosniff');
res.setHeader(
'Cache-Control',
isChecksumMatch
? SDK_CLIENT_MODULE_CACHE_CONTROL
: SDK_CLIENT_MODULE_NO_STORE_CACHE_CONTROL,
);
res.send(moduleBuffer);
}
}
@@ -0,0 +1,16 @@
import { Field, ObjectType } from '@nestjs/graphql';
import { IsNotEmpty, IsOptional, IsString } from 'class-validator';
@ObjectType('SdkClientChecksums')
export class SdkClientChecksumsDTO {
@IsString()
@IsOptional()
@Field(() => String, { nullable: true })
core: string | null;
@IsString()
@IsNotEmpty()
@Field()
metadata: string;
}
@@ -1,7 +1,11 @@
import { type SdkClientGenerationTrigger } from 'src/engine/core-modules/sdk-client/types/sdk-client-generation-trigger.type';
export const GENERATE_SDK_CLIENT_JOB_NAME = 'GenerateSdkClientJob';
export type GenerateSdkClientJobData = {
workspaceId: string;
applicationId: string;
applicationUniversalIdentifier: string;
// Optional: jobs enqueued before this field existed carry no trigger
trigger?: SdkClientGenerationTrigger;
};
@@ -19,6 +19,7 @@ export class GenerateSdkClientJob {
workspaceId: data.workspaceId,
applicationId: data.applicationId,
applicationUniversalIdentifier: data.applicationUniversalIdentifier,
trigger: data.trigger,
});
}
}
@@ -165,6 +165,7 @@ export class SdkClientArchiveService {
workspaceId,
applicationId,
applicationUniversalIdentifier,
trigger: 'missing-archive',
});
}
}
@@ -1,6 +1,7 @@
import { Injectable, Logger } from '@nestjs/common';
import { InjectRepository } from '@nestjs/typeorm';
import { createHash } from 'crypto';
import * as fs from 'fs/promises';
import { printSchema } from 'graphql';
import path, { join } from 'path';
@@ -18,6 +19,9 @@ import { TemporaryDirManager } from 'src/engine/core-modules/logic-function/logi
import { InjectMessageQueue } from 'src/engine/core-modules/message-queue/decorators/message-queue.decorator';
import { MessageQueue } from 'src/engine/core-modules/message-queue/message-queue.constants';
import { MessageQueueService } from 'src/engine/core-modules/message-queue/services/message-queue.service';
import { MetricsService } from 'src/engine/core-modules/metrics/metrics.service';
import { MetricsKeys } from 'src/engine/core-modules/metrics/types/metrics-keys.type';
import { type SdkModuleName } from 'src/engine/core-modules/sdk-client/constants/allowed-sdk-modules';
import { SDK_CLIENT_PACKAGE_DIRNAME } from 'src/engine/core-modules/sdk-client/constants/sdk-client-package-dirname';
import {
SdkClientException,
@@ -27,8 +31,10 @@ import {
GENERATE_SDK_CLIENT_JOB_NAME,
type GenerateSdkClientJobData,
} from 'src/engine/core-modules/sdk-client/jobs/generate-sdk-client.job-constants';
import { type SdkClientGenerationTrigger } from 'src/engine/core-modules/sdk-client/types/sdk-client-generation-trigger.type';
import { fromWorkspaceEntityToFlat } from 'src/engine/core-modules/workspace/utils/from-workspace-entity-to-flat.util';
import { WorkspaceEntity } from 'src/engine/core-modules/workspace/workspace.entity';
import { WorkspaceEventBroadcaster } from 'src/engine/subscriptions/workspace-event-broadcaster/workspace-event-broadcaster.service';
import { WorkspaceCacheService } from 'src/engine/workspace-cache/services/workspace-cache.service';
const SDK_CLIENT_ARCHIVE_NAME = 'twenty-client-sdk.zip';
@@ -49,6 +55,8 @@ export class SdkClientGenerationService {
private readonly applicationService: ApplicationService,
@InjectMessageQueue(MessageQueue.workspaceQueue)
private readonly messageQueueService: MessageQueueService,
private readonly workspaceEventBroadcaster: WorkspaceEventBroadcaster,
private readonly metricsService: MetricsService,
) {}
async enqueueSdkClientGenerationForWorkspace(
@@ -62,52 +70,101 @@ export class SdkClientGenerationService {
await Promise.all(
[twentyStandardFlatApplication, workspaceCustomFlatApplication].map(
(application) =>
this.messageQueueService.add<GenerateSdkClientJobData>(
GENERATE_SDK_CLIENT_JOB_NAME,
{
workspaceId,
applicationId: application.id,
applicationUniversalIdentifier: application.universalIdentifier,
},
{
id: `sdk-client:${workspaceId}:${application.id}`,
retryLimit: SDK_CLIENT_GENERATION_RETRY_LIMIT,
},
),
this.enqueueSdkClientGenerationForApplication({
workspaceId,
applicationId: application.id,
applicationUniversalIdentifier: application.universalIdentifier,
trigger: 'workspace-activation',
}),
),
);
}
private async enqueueSdkClientGenerationForApplication({
workspaceId,
applicationId,
applicationUniversalIdentifier,
trigger,
}: {
workspaceId: string;
applicationId: string;
applicationUniversalIdentifier: string;
trigger: SdkClientGenerationTrigger;
}): Promise<void> {
await this.messageQueueService.add<GenerateSdkClientJobData>(
GENERATE_SDK_CLIENT_JOB_NAME,
{
workspaceId,
applicationId,
applicationUniversalIdentifier,
trigger,
},
{
id: `sdk-client:${workspaceId}:${applicationId}`,
retryLimit: SDK_CLIENT_GENERATION_RETRY_LIMIT,
},
);
}
async generateSdkClientForApplication({
workspaceId,
applicationId,
applicationUniversalIdentifier,
trigger = 'unknown',
}: {
workspaceId: string;
applicationId: string;
applicationUniversalIdentifier: string;
trigger?: SdkClientGenerationTrigger;
}): Promise<Buffer> {
const workspaceEntity = await this.workspaceRepository.findOneByOrFail({
id: workspaceId,
});
const generationStart = performance.now();
const graphqlSchema = await this.workspaceSchemaFactory.createGraphQLSchema(
fromWorkspaceEntityToFlat(workspaceEntity),
applicationId,
);
try {
const workspaceEntity = await this.workspaceRepository.findOneByOrFail({
id: workspaceId,
});
const archiveBuffer = await this.generateAndStore({
workspaceId,
applicationId,
applicationUniversalIdentifier,
schema: printSchema(graphqlSchema),
});
const graphqlSchema =
await this.workspaceSchemaFactory.createGraphQLSchema(
fromWorkspaceEntityToFlat(workspaceEntity),
applicationId,
);
this.logger.log(
`Generated SDK client for application ${applicationUniversalIdentifier}`,
);
const archiveBuffer = await this.generateAndStore({
workspaceId,
applicationId,
applicationUniversalIdentifier,
schema: printSchema(graphqlSchema),
});
return archiveBuffer;
const generationDurationMs = performance.now() - generationStart;
this.metricsService.incrementCounterBy({
key: MetricsKeys.SdkClientGenerationSucceeded,
amount: 1,
attributes: { trigger },
});
this.metricsService.recordHistogram({
key: MetricsKeys.SdkClientGenerationDurationMs,
value: generationDurationMs,
unit: 'ms',
attributes: { trigger },
});
this.logger.log(
`Generated SDK client for application ${applicationUniversalIdentifier} (trigger: ${trigger})`,
);
return archiveBuffer;
} catch (error) {
this.metricsService.incrementCounterBy({
key: MetricsKeys.SdkClientGenerationFailed,
amount: 1,
attributes: { trigger },
});
throw error;
}
}
private async generateAndStore({
@@ -145,6 +202,11 @@ export class SdkClientGenerationService {
await replaceCoreClient({ packageRoot: tempPackageRoot, schema });
const sdkClientCoreChecksum = await this.computeSdkModuleChecksum(
tempPackageRoot,
'core',
);
const archivePath = join(sourceTemporaryDir, SDK_CLIENT_ARCHIVE_NAME);
await createZipFile(tempPackageRoot, archivePath);
@@ -162,13 +224,22 @@ export class SdkClientGenerationService {
await this.applicationRepository.update(
{ id: applicationId, workspaceId },
{ isSdkLayerStale: true },
{
isSdkLayerStale: true,
sdkClientCoreChecksum,
},
);
await this.workspaceCacheService.invalidateAndRecompute(workspaceId, [
'flatApplicationMaps',
]);
await this.broadcastSdkClientCoreChecksumUpdate({
workspaceId,
applicationId,
sdkClientCoreChecksum,
});
return archiveBuffer;
} catch (error) {
throw new SdkClientException(
@@ -179,4 +250,52 @@ export class SdkClientGenerationService {
await temporaryDirManager.clean();
}
}
private async broadcastSdkClientCoreChecksumUpdate({
workspaceId,
applicationId,
sdkClientCoreChecksum,
}: {
workspaceId: string;
applicationId: string;
sdkClientCoreChecksum: string;
}): Promise<void> {
try {
await this.workspaceEventBroadcaster.broadcast({
workspaceId,
events: [
{
type: 'updated',
entityName: 'application',
recordId: applicationId,
properties: {
updatedFields: ['sdkClientCoreChecksum'],
after: {
id: applicationId,
sdkClientCoreChecksum,
},
},
},
],
});
} catch (error) {
this.logger.warn(
`Failed to broadcast SDK client core checksum update for application ${applicationId} in workspace ${workspaceId}`,
error,
);
}
}
// sha-256 (not md5) so the renderer can verify cached bundles against the URL
// checksum with WebCrypto, which has no md5 support
private async computeSdkModuleChecksum(
tempPackageRoot: string,
moduleName: SdkModuleName,
): Promise<string> {
const moduleBuffer = await fs.readFile(
join(tempPackageRoot, 'dist', `${moduleName}.mjs`),
);
return createHash('sha256').update(moduleBuffer).digest('hex');
}
}
@@ -1,12 +1,14 @@
import { Module } from '@nestjs/common';
import { Module, type OnApplicationBootstrap } from '@nestjs/common';
import { TypeOrmModule } from '@nestjs/typeorm';
import { CoreGraphQLApiModule } from 'src/engine/api/graphql/core-graphql-api.module';
import { ApplicationEntity } from 'src/engine/core-modules/application/application.entity';
import { ApplicationModule } from 'src/engine/core-modules/application/application.module';
import { MetricsModule } from 'src/engine/core-modules/metrics/metrics.module';
import { SdkClientController } from 'src/engine/core-modules/sdk-client/controllers/sdk-client.controller';
import { SdkClientArchiveService } from 'src/engine/core-modules/sdk-client/sdk-client-archive.service';
import { SdkClientGenerationService } from 'src/engine/core-modules/sdk-client/sdk-client-generation.service';
import { getInstalledSdkMetadataModule } from 'src/engine/core-modules/sdk-client/utils/get-installed-sdk-metadata-module.util';
import { WorkspaceEntity } from 'src/engine/core-modules/workspace/workspace.entity';
import { WorkspaceCacheModule } from 'src/engine/workspace-cache/workspace-cache.module';
@@ -16,9 +18,14 @@ import { WorkspaceCacheModule } from 'src/engine/workspace-cache/workspace-cache
WorkspaceCacheModule,
CoreGraphQLApiModule,
ApplicationModule,
MetricsModule,
],
controllers: [SdkClientController],
providers: [SdkClientGenerationService, SdkClientArchiveService],
exports: [SdkClientGenerationService, SdkClientArchiveService],
})
export class SdkClientModule {}
export class SdkClientModule implements OnApplicationBootstrap {
async onApplicationBootstrap(): Promise<void> {
await getInstalledSdkMetadataModule();
}
}
@@ -0,0 +1,9 @@
// What caused an SDK client generation. Emitted as a metric attribute so
// regeneration volume can be broken down by cause.
// 'unknown' covers jobs enqueued before the trigger field existed.
export type SdkClientGenerationTrigger =
| 'workspace-activation'
| 'manifest-sync'
| 'missing-archive'
| 'dev-seeder'
| 'unknown';
@@ -0,0 +1,32 @@
import { createHash } from 'crypto';
import * as fs from 'fs/promises';
import { join } from 'path';
import { SDK_CLIENT_PACKAGE_DIRNAME } from 'src/engine/core-modules/sdk-client/constants/sdk-client-package-dirname';
export type InstalledSdkMetadataModule = {
moduleBuffer: Buffer;
checksum: string;
};
// Warmed once at bootstrap (SdkClientModule.onApplicationBootstrap) and treated
// as a process-lifetime invariant: the module ships inside the server build and
// never changes at runtime. A read/hash failure blocks boot, so a running
// server always has a resolved value here.
let installedSdkMetadataModule: InstalledSdkMetadataModule | undefined;
export const getInstalledSdkMetadataModule =
async (): Promise<InstalledSdkMetadataModule> => {
if (!installedSdkMetadataModule) {
const moduleBuffer = await fs.readFile(
join(SDK_CLIENT_PACKAGE_DIRNAME, 'dist', 'metadata.mjs'),
);
installedSdkMetadataModule = {
moduleBuffer,
checksum: createHash('sha256').update(moduleBuffer).digest('hex'),
};
}
return installedSdkMetadataModule;
};
@@ -48,6 +48,7 @@ const MOCK_FLAT_APPLICATION: FlatApplication = {
updatedAt: new Date(),
deletedAt: null,
isSdkLayerStale: true,
sdkClientCoreChecksum: null,
};
const flatObjectMetadataMaps = [
@@ -43,6 +43,7 @@ const MOCK_FLAT_APPLICATION: FlatApplication = {
updatedAt: new Date(),
deletedAt: null,
isSdkLayerStale: true,
sdkClientCoreChecksum: null,
};
type GenerateMorphOrRelationFlatFieldMetadataPairTestInput = Parameters<
@@ -128,6 +128,7 @@ export class DevSeederService {
applicationId: twentyStandardFlatApplication.id,
applicationUniversalIdentifier:
twentyStandardFlatApplication.universalIdentifier,
trigger: 'dev-seeder',
});
await this.devSeederMetadataService.seed({
@@ -140,6 +141,7 @@ export class DevSeederService {
applicationId: workspaceCustomFlatApplication.id,
applicationUniversalIdentifier:
workspaceCustomFlatApplication.universalIdentifier,
trigger: 'dev-seeder',
});
await this.devSeederMetadataService.seedRelations({