houseassassin/twenty
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
14,240 Commits 1 Branch 0 Tags
4dbaafc65d8ee32ebb346df3cb426fab65467569
Commit Graph

7 Commits

Author SHA1 Message Date
Paul Rastoin 71a1ff7ac8 Cache twenty-client-sdk modules host-side via content-addressed URLs (#22981) 
## Context

Front component sources are fetched host-side and integrity-verified by
the SHA-256 checksum embedded in their URL, cached in Cache Storage — a
layer that exists specifically because their download URLs are presigned
and rotate. The `twenty-client-sdk` modules (`core` and `metadata`) were
re-fetched on every render and could not be cached safely: their URLs
carried no checksum and the server exposed no freshness signal.

This PR makes the SDK module URLs **content-addressed** and relies on
the **browser HTTP cache** for immutability, and it keys the checksums
on their real owners: the **application** for `core`, the **instance**
for `metadata`. The checksum does double duty: cache invalidation
(regeneration changes the checksum → the URL changes → guaranteed cache
miss) and a server-side cacheability guard (the server only grants
`immutable` when the checksum in the URL matches the authoritative
checksum it knows for that module — persisted at generation time for
`core`, hashed once at bootstrap for `metadata` — so no per-request
hashing of the served bytes). Note this is **not** an end-to-end
integrity guarantee: there is no client-side hash verification, and on a
fingerprint mismatch the server still serves the current bytes with
`no-store` (self-healing for stale URLs) rather than failing.

<img width="2412" height="926" alt="image"
src="https://github.com/user-attachments/assets/d97935d2-0fdb-4c44-89ac-596b7ca8ca64"
/>

Closes twentyhq/core-team-issues#2688.

## Routes

| Module | URL | Scope |
| --- | --- | --- |
| `core` | `/rest/sdk-client/{applicationId}/core[/{checksum}]` | Per
application (generated bundle) |
| `metadata` | `/rest/sdk-client/metadata[/{checksum}]` |
**Instance-wide**: no application segment, so every application
converges on one URL and the browser downloads the module once per
release instead of once per application |

The previous application-scoped metadata path
(`/rest/sdk-client/{applicationId}/metadata[/{checksum}]`) is **kept for
backward compatibility**, new clients just stop generating those URLs.
The instance-wide route is declared before the parameterized route so
`metadata/{checksum}` is not swallowed as `:applicationId/:moduleName`.

## Caching model

| Request | `Cache-Control` | Effect |
| --- | --- | --- |
| Fingerprinted URL, checksum matches the known module checksum |
`immutable` | Cached indefinitely by the browser HTTP cache; a new
checksum is a new URL |
| Fingerprinted URL, checksum does not match | `no-store` | Current
bytes served uncached (self-healing for stale URLs) |
| Bare URL (pre-generation fallback, `core` only in practice) |
`no-store` | Never cached |

- Both responses also set `X-Content-Type-Options: nosniff` and
`Content-Type: application/javascript`.
- SDK modules are intentionally **not** placed in Cache Storage. That
layer stays reserved for the presigned/rotating component-source URLs;
SDK modules are served directly and authenticated, so the browser HTTP
cache (keyed by the content-addressed URL) is their single cache layer.

## Checksum provenance

- **core** — per **application**, persisted on
`application.sdkClientCoreChecksum` at generation time and read back
from `flatApplicationMaps` (never re-hashed per request).
- **metadata** — **instance-wide**, hashed once from the installed
`twenty-client-sdk/dist/metadata.mjs` package (warmed at bootstrap,
memoized per process) and served straight from that package, so it is
fresh from the first request after a release with no archive dependency.

## Server (twenty-server)

- Hash `dist/core.mjs` at SDK generation and persist
`sdkClientCoreChecksum` via `applicationRepository.update`. Adds the
nullable text column to `application.entity.ts` (mirroring
`packageJsonChecksum`) plus a fast instance command with up/down;
`FlatApplication` picks it up automatically.
- New **application-scoped** query
`applicationSdkClientChecksums(applicationId: UUID!):
SdkClientChecksums` on `ApplicationResolver` (metadata schema,
`WorkspaceAuthGuard` + `NoPermissionGuard`). `SdkClientChecksums.core`
is **nullable** and stays `null` until the SDK has been generated at
least once; `metadata` is **always present** (bootstrap-warmed), so the
metadata module is cacheable from the very first render of any app. The
query itself returns `null` only for unknown applications.
- `SdkClientChecksumsDTO` now lives in the shared
`core-modules/sdk-client/dtos/`. `FrontComponentDTO` and the
`frontComponent` resolver no longer carry checksums (decoupled from the
front-component row).
- `sdk-client` controller: instance-wide `metadata[/:checksum]` route
(no workspace-cache or application lookup, serves the memoized installed
module) + application-scoped `:applicationId/:moduleName[/:checksum]`
route (serves `core` from the per-application archive, `metadata` kept
for back-compat). Cacheability compares the URL checksum against the
**known** checksum — persisted `sdkClientCoreChecksum` for `core`,
memoized package hash for `metadata` — instead of hashing the served
bytes on every request: `immutable` on match, `no-store` otherwise (bare
URL or stale fingerprint), plus `nosniff`. A persisted checksum out of
sync with the archive only downgrades to `no-store` until the next
regeneration.

## Front (twenty-front)

- New metadata query `GetApplicationSdkClientChecksums`, keyed by
`applicationId`; removed the `sdkClientChecksums` selection from
`FindOneFrontComponent`.
- `getSdkClientUrls` builds the two module URLs independently:
`/sdk-client/{applicationId}/core/{checksum}` and the **instance-wide**
`/sdk-client/metadata/{checksum}` (no application segment → one shared
browser cache entry per release across all applications). Each falls
back to its bare URL when its checksum is absent — since `core` is
nullable, a never-generated app still gets a content-addressed metadata
URL and only `core` falls back. The checksum type is sourced from the
codegen `SdkClientChecksums` type rather than a hand-maintained
duplicate.
- `FrontComponentRenderer` is split into a gating outer component (runs
`FindOneFrontComponent`, renders nothing while loading) and a content
component that receives a guaranteed-non-null `frontComponent`.
Following project conventions, the side effects live in dedicated effect
components: `FrontComponentLoadErrorSnackBarEffect` (query error →
snackbar) and `FrontComponentApplicationTokenPairEffect` (mirrors the
query-derived token pair into component state unconditionally, `null`
included, so revoked credentials can never be retained or refreshed).
The content component fetches checksums via the application-keyed query
and **gates the mount of SDK-using components on that query**, so the
very first module fetch is always the content-addressed (`immutable`)
URL instead of the bare `no-store` one. Non-SDK components skip the
query and are never blocked.
- **Live invalidation without reload:** SDK regeneration updates the
application row, and the server broadcasts an `application` metadata
event carrying the new core checksum.
`useOnApplicationSdkClientChecksumsUpdated` /
`useUpdateSdkClientChecksumsApolloCache` patch the application-keyed
checksum query cache (core only; the instance-wide metadata is
preserved), so every mounted component of that application picks up the
new URL at once. This replaces the previous frontComponent-derived field
and closes the earlier "known gap" (a mounted component staying on a
session-old checksum until a full reload). The cache-patching callback
is memoized (`useCallback`) so the window listener is registered once
per application, and the listener is **skipped entirely** for non-SDK
components (`useListenToMetadataOperationBrowserEvent` gained a `skip`
option) — they register no listener and never refetch a query they don't
consume.

## Renderer (twenty-front-component-renderer)

- SDK sources are fetched through a dedicated plain authenticated fetch,
`fetchJavaScriptModuleSourceText` (Bearer header, `credentials:
'omit'`), instead of the Cache Storage `fetchComponentSource` path;
`fetchSdkClientSources` uses it. Execution stays exclusively in the
opaque-origin worker via blob URLs; the host only fetches and forwards
source strings (no hashing host-side). Staleness self-resolves through
the checksum: new checksum → new URL → cache miss.

<!-- This is an auto-generated description by cubic. -->
<a
href="https://cubic.dev/pr/twentyhq/twenty/pull/22981?utm_source=github"
target="_blank" rel="noopener noreferrer"
data-no-image-dialog="true"><picture><source
media="(prefers-color-scheme: dark)"
srcset="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"><source
media="(prefers-color-scheme: light)"
srcset="https://www.cubic.dev/buttons/review-in-cubic-light.svg"><img
alt="Review in cubic"
src="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"></picture></a>
<!-- End of auto-generated description by cubic. -->
 2026-07-21 15:05:00 +00:00
Félix Malfait fd7387928c feat: queue messages + replace AI SDK with GraphQL SSE subscription (#19203) 
## Summary

- **Queue messages while streaming**: Messages sent during active AI
streaming are queued server-side and auto-flushed when the current
stream completes. Frontend renders queued messages optimistically in a
dedicated queue UI.
- **Drop `@ai-sdk/react` + `resumable-stream`**: Replace the dual HTTP
SSE + AI SDK client architecture with a single GraphQL SSE subscription
per thread. All events (token chunks, message persistence, queue
updates, errors) flow through Redis PubSub → GraphQL subscription.
- **Server-driven architecture**: The server decides whether to queue or
stream (via `POST /:threadId/message`). The frontend mirrors this
decision for optimistic rendering but defers to the server response.
- **Reuse AI SDK accumulation logic**: `readUIMessageStream` from the
`ai` package handles chunk-to-message accumulation on the frontend,
avoiding a custom 780-line accumulator.

## Key files

**Backend:**
- `agent-chat-event-publisher.service.ts` — publishes events to Redis
PubSub
- `agent-chat-subscription.resolver.ts` — GraphQL subscription resolver
- `stream-agent-chat.job.ts` — publishes chunks via PubSub instead of
resumable-stream
- `agent-chat.controller.ts` — unified `POST /:threadId/message`
endpoint

**Frontend:**
- `useAgentChatSubscription.ts` — subscribes to `onAgentChatEvent`,
bridges to `readUIMessageStream`
- `useAgentChat.ts` — send/stop/optimistic rendering (no more AI SDK)
- `AgentChatStreamSubscriptionEffect.tsx` — replaces
`AgentChatAiSdkStreamEffect.tsx`

## Test plan

- [ ] Send message on new thread → optimistic render, streaming response
appears
- [ ] Send message while streaming → queued instantly (no flash in main
thread)
- [ ] Queued message auto-flushes after current stream completes
- [ ] Remove queued message via queue UI
- [ ] Stop streaming mid-response
- [ ] Leave chat idle for several minutes → streaming still works after
(SSE client recycling)
- [ ] Token refresh during session → requests succeed (authenticated
fetch)
- [ ] Switch threads while streaming → clean subscription handoff


Made with [Cursor](https://cursor.com)

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
 2026-04-02 10:10:13 +02:00
Charles Bochet d5a7dec117 refactor: rename ObjectMetadataItem to EnrichedObjectMetadataItem and clean up metadata flows (#18830) 
## Summary

- Renames `ObjectMetadataItem` to `EnrichedObjectMetadataItem` across
the entire frontend (~440 files) to clarify that this type includes
derived fields (`readableFields`, `updatableFields`, nested `fields[]`,
`indexMetadatas[]`) computed at read time from the metadata store
- Creates `splitObjectMetadataGqlResponse` that goes directly from a
GraphQL `ObjectMetadataItemsQuery` response to flat store items
(combining the old
`mapPaginatedObjectMetadataItemsToObjectMetadataItems` +
`splitObjectMetadataItemWithRelated` two-step flow into one call)
- Removes `ObjectMetadataItemWithRelated` type and all "WithRelated"
naming
- Renames `generatedMockObjectMetadataItems` to
`generateTestEnrichedObjectMetadataItemsMock` to make it clear this is
test-only enriched data
- Deletes `useLoadMockedObjectMetadataItems` hook (consolidated into
`useLoadMockedMinimalMetadata`)
- Ensures nothing destined for the metadata store computes
`readableFields`/`updatableFields` (preventing the localStorage bloat
from #18809)

## Type hierarchy (before → after)

**Before:**
```
ObjectMetadataItemsQuery → mapPaginated → ObjectMetadataItemWithRelated → enrich → ObjectMetadataItem
                                        → split → FlatObjectMetadataItem (store)
```

**After:**
```
ObjectMetadataItemsQuery → splitObjectMetadataGqlResponse → FlatObjectMetadataItem (store)
                         → mapPaginated + enrich (tests only) → EnrichedObjectMetadataItem
```

## Test plan

- [x] `npx nx typecheck twenty-front` passes
- [x] `npx nx test twenty-front` passes (767 suites, 4505 tests)
- [x] `npx nx lint twenty-front` passes
- [ ] CI checks pass


Made with [Cursor](https://cursor.com)
 2026-03-22 15:53:47 +01:00
Charles Bochet 06efee1eef feat: hash-based metadata staleness detection (#18649) 
## Summary

Replace the single `metadataVersion` integer with per-entity-type
**collection hashes** for granular metadata staleness detection. The
backend already generates a UUID per flat entity map on each cache
recompute (`crypto.randomUUID()` in `WorkspaceCacheService`); we now
expose these via the minimal metadata endpoint and SSE events so the
frontend can compare and know exactly which entity types are stale.

### Key changes

**Backend:**
- `WorkspaceCacheService.getCacheHashes()` — new public method that
reads only `:hash` keys from Redis without fetching full data
- `MinimalMetadataDTO` — added `collectionHashes: Record<string,
string>` (JSON scalar mapping `AllMetadataName` → collection hash),
removed `metadataVersion`
- `MetadataEventDTO` — added optional `updatedCollectionHash` field to
SSE events
- `MetadataEventsToDbListener` — reads the collection hash for the
affected entity type after cache invalidation and attaches it to the SSE
event before publishing
- `MinimalMetadataService` — no longer queries the workspace table; uses
`getCacheHashes()` for all flat entity maps and maps cache keys to
`AllMetadataName` locally

**Frontend:**
- `metadataCollectionHashesState` — new Jotai atom with
`atomWithStorage` + `getOnInit: true` storing
`Partial<Record<MetadataEntityKey, string>>`
- `mapAllMetadataNameToEntityKey()` — explicit mapping from backend
`AllMetadataName` to frontend `MetadataEntityKey` (23 entries)
- `useLoadMinimalMetadata` — stores `collectionHashes` from server,
computes `staleEntityKeys` by comparing local vs server hashes
- `patchMetadataStoreFromSSEEvent()` — accepts optional
`updatedCollectionHash` and updates `metadataCollectionHashesState`
- All 11 SSE effect components — pass
`eventDetail.updatedCollectionHash` through to the patch function
- `useStaleMetadataEntities` — new hook returning entity keys missing
from collection hashes (not yet loaded/synced)
- `resetMetadataStore()` — also clears collection hashes
- Deleted `metadataVersionState` (superseded by collection hashes)

### Design decisions

- **No change to hash generation** — existing `crypto.randomUUID()` is
sufficient. Hashes are persisted in Redis, survive server restarts, and
change only on `invalidateAndRecompute`.
- **"Collection hash" naming** — used consistently to clarify the hash
represents an entire entity collection (e.g., all views), not a single
record.
- **Mapping localized** — backend `WorkspaceCacheKeyName` →
`AllMetadataName` mapping lives in the minimal metadata service.
Frontend `AllMetadataName` → `MetadataEntityKey` mapping lives in a
local utility. Nothing in `twenty-shared`.
- **Backward compatible** — `collectionHashes` is additive;
`updatedCollectionHash` is nullable.
 2026-03-14 23:38:37 +01:00
Lucas Bordeau cb3e32df86 Fix AI demo workspace skill (#18575) 
This PR fixes what allows to have a working demo workspace skill.

- Skill updated many times into something that works
- Fixed infinite loop in AI chat by memoizing ai-sdk output
- Finished navigateToView implementation
- Increased MAX_STEPS to 300 so the chat don't quit in the middle of a
long running skill
- Added CreateManyRelationFields
 2026-03-12 13:19:01 +01:00
Lucas Bordeau 2a82df7073 AI tools to create a demo workspace (#18236) 
This PR adds the necessary tool to create a demo workspace with :
relevant custom objects and fields, mock data and a real dashboard with
graph widgets.

It is still a bit under-optimized and slow but it works.

This PR also adds an AI tool that allows to see what happens in real
time, it navigates the app and waits when necessary.

---------

Co-authored-by: Félix Malfait <felix.malfait@gmail.com>
 2026-03-05 11:39:31 +01:00
Raphaël Bosi 2455c859b4 Add SSE for metadata and plug front components (#17998) 
Create the necessary tooling to listen to metadata events and plug it to
the front components. Now we have a hot reload like experience when we
edit a component in an app.

## Backend

- Split `EventWithQueryIds` into `ObjectRecordEventWithQueryIds` and
`MetadataEventWithQueryIds`
- Publish metadata event batches to active SSE streams in
`MetadataEventsToDbListener`

## Frontend

- Create a metadata event dispatching pipeline: SSE metadata events are
grouped by metadata name, transformed into
`MetadataOperationBrowserEventDetail` objects, and dispatched as browser
`CustomEvents`
- Add `useListenToMetadataOperationBrowserEvent` hook for consuming
metadata operation events filtered by metadata name and operation type
- Rename `useListenToObjectRecordEventsForQuery` to
`useListenToEventsForQuery`, now accepting both
`RecordGqlOperationSignature` and `MetadataGqlOperationSignature`
- Implement `useOnFrontComponentUpdated` which subscribes to front
component metadata events and updates the Apollo cache when the
component is modified
- Add `builtComponentChecksum` to the front component query and appends
it to the component URL for browser cache invalidation
 2026-02-18 11:26:20 +00:00