716e67a2762a8396a955ed3e1fffb39fc16da569
94 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Joshua Freedman
|
4a7324c0c8 |
fix(serverless): flush IPC message before exiting local function runner (#22920)
Closes #22925 ## Problem `LocalChildProcessRunnerService.writeBootstrapRunner` generates a child-process runner that returns the function result to the parent over the Node IPC channel: ```js const out = await handlerFn(msg.payload); process.send && process.send({ ok: true, result: out }); process.exit(0); ``` `process.send()` is **asynchronous**. When the serialized payload is larger than the OS pipe buffer (~64 KB on Linux), it can't be written in a single synchronous step, and the `process.exit(0)` on the next line tears the child down before the message is flushed. On the parent side (`runChildWithEnv`), the lost message means the `'message'` handler never fires — only `'exit'` with `code === 0` does, which resolves: ```js resolve({ ok: true, stdout, stderr }); // no `result` ``` `LocalDriver.execute` then returns `data: result ?? null` → **`null`**, so the function's return value is silently discarded while the step is reported as a *success* with an empty result. ## Symptom Larger serverless / workflow **Code** step results intermittently come back empty (`{}` / `null`). It is size- and load-dependent, so it presents as flakiness. A common downstream failure is a workflow **Iterator** fed the now-missing array: ``` Iterator input items must be an array ``` Results smaller than the pipe buffer always flush synchronously and never reproduce it — which is why only larger payloads are affected. ## Root cause `process.exit()` runs before the asynchronous `process.send()` (and the stdout fallback `process.stdout.write()`) has flushed — the classic Node footgun of exiting before pending async writes drain. ## Fix Wait for the `send()` / `write()` flush callback before exiting, on every exit path (success, error, stdout fallback, outer catch): ```js if (process.send) { process.send({ ok: true, result: out }, () => process.exit(0)); } else { process.exit(0); } ``` Behavior-preserving: it never delivers *less* than before; it only closes the window where a large result is dropped. No change to the small-payload happy path. ## Deterministic reproduction Reproduces the dropped IPC message under back-pressure (parent stalls before draining), comparing the current pattern vs the fix: ```js const { spawn } = require('node:child_process'); const fs = require('fs'); const SIZE = 3_000_000; // beyond any pipe buffer const child = (mode) => ` process.on('message', () => { const out = 'z'.repeat(${SIZE}); ${mode === 'fixed' ? 'process.send({ ok: true, result: out }, () => process.exit(0));' : 'process.send({ ok: true, result: out }); process.exit(0);'} });`; const busy = (ms) => { const e = Date.now() + ms; while (Date.now() < e) {} }; function trial(mode) { return new Promise((resolve) => { const f = `/tmp/child_${mode}.cjs`; fs.writeFileSync(f, child(mode)); const c = spawn(process.execPath, [f], { stdio: ['pipe', 'pipe', 'pipe', 'ipc'] }); let got = false; c.on('message', (m) => { got = m?.result?.length === SIZE; }); c.on('exit', () => resolve(got)); c.send({ type: 'run' }); busy(30); // stall parent so it doesn't drain the IPC pipe promptly }); } (async () => { for (const mode of ['current', 'fixed']) { let ok = 0; const N = 25; for (let i = 0; i < N; i += 5) { ok += (await Promise.all([...Array(5)].map(() => trial(mode)))).filter(Boolean).length; } console.log(`${mode}: result delivered ${ok}/${N}`); } })(); ``` Output: ``` current: result delivered 0/25 fixed: result delivered 25/25 ``` <!-- This is an auto-generated description by cubic. --> <a href="https://cubic.dev/pr/twentyhq/twenty/pull/22920?utm_source=github" target="_blank" rel="noopener noreferrer" data-no-image-dialog="true"><picture><source media="(prefers-color-scheme: dark)" srcset="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"><source media="(prefers-color-scheme: light)" srcset="https://www.cubic.dev/buttons/review-in-cubic-light.svg"><img alt="Review in cubic" src="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"></picture></a> <!-- End of auto-generated description by cubic. --> |
||
|
martmull
|
b733a79821 |
feat(server): support server-scoped files via nullable workspaceId on file table (#22587)
Part of the app settings architecture cleanup (twentyhq/core-team-issues#2456) — PR 1 of the server-level documents plan, reworked after the revert of #22560 (#22579). Same capability, different shape: **no new entity** — server-level documents live in the existing `file` table with a nullable `workspaceId`. ## Problem All file storage is workspace-scoped (`FileEntity.workspaceId NOT NULL`, `{workspaceId}/{app}/…` storage keys). Server-level data like application-registration manifests and tarballs for ownerless catalog registrations has no first-class home, forcing raw-driver bypasses (`DefaultAiCatalogService`, prototype #22556). ## Changes (core storage layer only — no HTTP serving, no GraphQL exposure) **`FileEntity` gains server scope** (mirrors `KeyValuePairEntity`, which already supports both instance-level and per-workspace rows): - `workspaceId` uuid becomes **nullable** — NULL means server-scoped; the entity no longer extends `WorkspaceRelatedEntity` and declares its columns directly - `applicationRegistrationId` nullable FK (`onDelete: CASCADE`) — registration-owned documents follow their registration - ownership checks: `workspaceId IS NOT NULL OR applicationRegistrationId IS NOT NULL` and `workspaceId IS NULL OR applicationRegistrationId IS NULL` — every row has exactly one owner - `IDX_FILE_APPLICATION_REGISTRATION_ID_PATH_UNIQUE` UNIQUE (`applicationRegistrationId`, `path`) — mirrors the workspace unique-constraint pattern; workspace rows are exempt via their NULL `applicationRegistrationId` **New `ServerFileStorageService`** (`file-storage/services/`, exported from the global `FileStorageModule`; `FileStorageService` moved alongside it): - storage keys `server/{fileFolder}/{applicationRegistrationId}/{resourcePath}` — the registration segment is injected by the service itself, so paths cannot collide across registrations; scope-validation util mirroring `validateStoragePathIsWithinWorkspaceOrThrow`; new `ServerFileFolder` enum in twenty-shared - `writeServerFile` (upsert on (`applicationRegistrationId`, `path`) + driver write; throws on failure), `readServerFile`/`readServerFileById` (missing row or bytes surfaces `FILE_NOT_FOUND`), `checkServerFileExists`, `deleteServerFile`/`deleteByServerFileId` (bytes best-effort, row authoritative), `deleteByApplicationRegistrationId` - rows are accessed through a plain repository pinned to `workspaceId: IsNull()` on every query; workspace-file code paths still go through `WorkspaceScopedRepository`, which never sees NULL rows **Null-safety ripples** (workspaceId is now `string | null`): - `WorkspaceScopedEntity` bound widened to `workspaceId: string | null` (the wrapper always filters with a concrete id) - `list-and-delete-orphaned-workspace-entities` now skips `workspaceId IS NULL` rows — previously `NOT EXISTS` would have flagged server rows as orphans and deleted them - `PendingFileCleanupService` sweeps only `workspaceId IS NOT NULL` rows; `application-package-fetcher` pins its tarball lookup to workspace rows (tarball migration to server scope is a follow-up PR) **Migration**: `allow-server-scoped-file` ships as a **2-20 fast instance command** (2.20.0 is current since #22639; re-slotted from 2-19 per review). Command runs are tracked by name, so instances that already executed the 2-20 `standardOverrides` drop command still pick this one up. Its realistic timestamp sorts before that drop command's fabricated `1825000000000`, which the `ci:allow-upgrade-command-timestamp-exception` label covers. ## Next PRs in the plan - PR 2: HTTP serving + token type for server files - PR 3: application-registration manifests stored as versioned server files (rework of draft #22556) - PR 4 (optional): registration tarballs migrate to server scope ## Verification - New spec `server-file-storage.service.spec.ts` (traversal table, upsert conflict semantics, row-before-bytes reads, best-effort byte deletion, registration cascade) + scope-validation util spec; affected suites all green - Typecheck (server + shared), `lint:diff-with-main`, full `oxfmt --check src/` on both packages clean - Fresh `database:reset` on the re-slotted branch: the 2-20 command executes, generator then reports **no schema drift**; both ownership checks and the composite unique verified live (dual-owner insert and duplicate registration+path both rejected) |
||
|
nitin
|
9657c59272 |
Inject functions URL into logic function env (#22583)
<!-- This is an auto-generated description by cubic. --> <a href="https://cubic.dev/pr/twentyhq/twenty/pull/22583?utm_source=github" target="_blank" rel="noopener noreferrer" data-no-image-dialog="true"><picture><source media="(prefers-color-scheme: dark)" srcset="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"><source media="(prefers-color-scheme: light)" srcset="https://www.cubic.dev/buttons/review-in-cubic-light.svg"><img alt="Review in cubic" src="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"></picture></a> <!-- End of auto-generated description by cubic. --> |
||
|
Thomas Trompette
|
fb0a54c73a |
fix(server): pace lambda control-plane calls to avoid 'Rate exceeded' on release (#22569)
## Problem Logic functions intermittently fail with: ``` Lambda invocation failed for function '<id>' during build: Rate exceeded ``` `Rate exceeded` is AWS Lambda's control-plane throttling (`TooManyRequestsException`), thrown during the **build** phase — before invoke — inside `buildExecutor`. ### Why it spikes on release A build is skipped (`canSkip = true`, zero control-plane calls) unless the executor is missing/inactive **or** `flatApplication.isSdkLayerStale` is true. `isSdkLayerStale` is flipped to `true` for the **whole application at once** whenever the SDK client regenerates (app install / development / schema change). So on release, every logic function in the app goes stale simultaneously → each enters `ensureExecutor` in its own per-function lock → a burst of `Create`/`Update`/`PublishLayer`/`GetFunction` calls across many functions at once → the low, account-region-wide control-plane quota is exceeded → `Rate exceeded`. Between releases everything is warm and no control-plane calls happen — hence "spikes on release, silent otherwise". The Lambda client was created with no retry override, so it used the SDK default (`standard` mode, `maxAttempts = 3`): a few retries with backoff, but no client-side pacing. ## Change Configure the shared Lambda client with: - `retryMode: 'adaptive'` — adds a client-side token-bucket rate limiter that slows outgoing requests when it sees throttling, instead of fire-then-backoff. - `maxAttempts: 8` — rides out the burst. Applied after the options spread so it always takes effect, and covers **every** control-plane call including the `waitUntilFunctionActive/UpdatedV2` pollers (same client). ## Scope / follow-up This is the cheap, high-leverage mitigation and dampens the burst per process. It does **not** add a cross-function/cross-pod concurrency cap, so a large enough release across multiple replicas could still exceed the account quota. A follow-up could add a limiter (in-process semaphore, or a distributed token bucket via the existing Redis cache-lock) around `ensureExecutor`. ## Testing - `tsc --noEmit` on twenty-server: clean. - Not runtime-tested — AWS control-plane throttling can't be reproduced locally. Worth confirming against a real release-time CloudWatch window after deploy. <!-- This is an auto-generated description by cubic. --> <a href="https://cubic.dev/pr/twentyhq/twenty/pull/22569?utm_source=github" target="_blank" rel="noopener noreferrer" data-no-image-dialog="true"><picture><source media="(prefers-color-scheme: dark)" srcset="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"><source media="(prefers-color-scheme: light)" srcset="https://www.cubic.dev/buttons/review-in-cubic-light.svg"><img alt="Review in cubic" src="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"></picture></a> <!-- End of auto-generated description by cubic. --> |
||
|
Charles Bochet
|
203da78b05 |
feat(logic-function): namespace shared Lambda resources per instance (#22509)
## Problem When several Twenty instances run in the **same AWS account + region** (our customer's setup), their logic-function Lambda resources collide. Four shared resources are named purely from a content hash: | Resource | Name | Derived from | |---|---|---| | Builder fn | `twenty-builder-<sha256(handler)>` | handler code (⇒ version) | | Yarn-install fn | `twenty-yarn-install-<sha256(handler)>` | handler code | | Common layer | `twenty-common-layer-<sha256(pkg+lock)>` | dependencies | | Deps layer | `deps-<yarnLockChecksum>` | app yarn.lock | Because the hash is identical across instances of the same version, every instance computes the **same name**. `ensureBuilderLambdaExists` / `ensureYarnInstallLambdaExists` do `GetFunction → if it exists, return` with **no role check**, so the first instance to create the function binds it to *its* `LOGIC_FUNCTION_LAMBDA_ROLE` and every other instance silently reuses it. When that role is later deleted or belongs to a different account, invokes fail with: > The role defined for the function cannot be assumed by Lambda. Nothing tears these shared resources down, so a poisoned function persists indefinitely. (Executors are UUID-named and SDK layers are workspace-scoped, so they don't collide.) ## Fix Namespace the four shared resources by a per-instance segment. - New optional config var **`LOGIC_FUNCTION_LAMBDA_RESOURCE_NAMESPACE`** (LOGIC_FUNCTION_CONFIG group). - When unset it defaults to `sha256(LOGIC_FUNCTION_LAMBDA_ROLE).slice(0, 10)`. Keying the namespace on the execution role makes the sharing boundary correct: - **same role → same names →** instances still dedupe (original intent preserved), - **different role → different names →** full isolation, invoke can never hit a role it can't assume, - **role change →** resources are recreated fresh under a new name (self-healing). Names become `twenty-builder-<namespace>-<checksum>`, `deps-<namespace>-<checksum>`, etc. — the owning instance stays legible in the AWS console. ### Defensive role-heal As a safety net (and to heal already-poisoned functions), after `GetFunction` succeeds we compare `Configuration.Role` to the configured role; on mismatch we delete and recreate the tool function. (`waitFunctionDeleted` polls `GetFunction` until `ResourceNotFoundException` — this SDK version has no `waitUntilFunctionNotExists` waiter.) ## Scope / compatibility - Executor functions (`<logicFunctionId>`) and SDK layers (`sdk-<workspaceId>-<appUUID>`) are unchanged — already unique. - On upgrade, shared-resource names change once (role-hash namespace), so each instance recreates its builder/yarn-install/common-layer/deps on first use; old ones are orphaned (harmless, unreferenced). - Operators who want explicit control can set `LOGIC_FUNCTION_LAMBDA_RESOURCE_NAMESPACE`. ## Ops note (immediate unblock, independent of this PR) Delete the poisoned `twenty-builder-<hash>` (and sibling `twenty-yarn-install-*`) in the affected region; it is recreated with the correct role on next use. ## Tests - `compute-hashed-lambda-resource-name.util.spec.ts` — namespace segment behavior - `get-lambda-deps-layer-name.util.spec.ts` — namespaced deps layer name - `get-lambda-resource-namespace.util.spec.ts` — stable, role-distinct namespace All pass; `lint:diff-with-main` clean; typecheck clean for changed files. <!-- This is an auto-generated description by cubic. --> <a href="https://cubic.dev/pr/twentyhq/twenty/pull/22509?utm_source=github" target="_blank" rel="noopener noreferrer" data-no-image-dialog="true"><picture><source media="(prefers-color-scheme: dark)" srcset="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"><source media="(prefers-color-scheme: light)" srcset="https://www.cubic.dev/buttons/review-in-cubic-light.svg"><img alt="Review in cubic" src="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"></picture></a> <!-- End of auto-generated description by cubic. --> |
||
|
martmull
|
25fe66565c |
feat(applications): add type and options to application variables (#22157)
## Before <img width="1452" height="709" alt="image" src="https://github.com/user-attachments/assets/cd384ffa-cbe6-49d5-a807-ca8d580f55a9" /> <img width="1074" height="452" alt="image" src="https://github.com/user-attachments/assets/720d38db-3495-4032-8831-17d24ec6a7e7" /> ## After <img width="1421" height="865" alt="image" src="https://github.com/user-attachments/assets/2275c996-c895-4800-8324-2aa2ddfddd43" /> <img width="1348" height="870" alt="image" src="https://github.com/user-attachments/assets/3e1a891d-6db0-4cbd-870a-2a5bbde4929d" /> ## Summary Adds typed application variables with optional select **options**. This is the other half of #22059, split out from the custom-settings-tab removal. ## Changes - **Shared types**: `ApplicationVariable` / `ServerVariables` gain an optional `type` (a `FieldMetadataType` subset — `TEXT`, `BOOLEAN`, `NUMBER`, `DATE`, `SELECT`, `MULTI_SELECT`, `RAW_JSON`, `RICH_TEXT`, `ARRAY`, …) and select `options`. New `serializeApplicationVariableValue` / `deserializeApplicationVariableValue` helpers convert typed values to/from the encrypted string storage. - **Server**: `type`/`options` columns on `applicationVariable` and `applicationRegistrationVariable` (entities + DTOs), a fast `2-17` instance command, manifest processing via the serialization helpers, and a `QueryDeepPartialEntity` cast where the manifest JSON column is persisted. - **Frontend**: a polymorphic `SettingsApplicationVariableInput` that renders the native `Form*` field component for each type (boolean, number, date/date-time, select, multi-select, array, raw JSON, rich text, text); fragment/query updates to fetch `type`/`options`. - **SDK**: `defineApplication` validates that `SELECT`/`MULTI_SELECT` variables declare non-empty `options` at build time (since `options` is kept structurally optional for TypeORM/SDK compatibility). Variables default to `TEXT` when no type is given, so existing manifests are unaffected. ## Notes The generated GraphQL artifacts (`type`/`options` on the variable types) are regenerated by codegen; that change accompanies this PR. https://claude.ai/code/session_013Z7UB35V2mvUozh55QHG23 --- _Generated by [Claude Code](https://claude.ai/code/session_013Z7UB35V2mvUozh55QHG23)_ <!-- This is an auto-generated description by cubic. --> <a href="https://cubic.dev/pr/twentyhq/twenty/pull/22157?utm_source=github" target="_blank" rel="noopener noreferrer" data-no-image-dialog="true"><picture><source media="(prefers-color-scheme: dark)" srcset="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"><source media="(prefers-color-scheme: light)" srcset="https://www.cubic.dev/buttons/review-in-cubic-light.svg"><img alt="Review in cubic" src="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"></picture></a> <!-- End of auto-generated description by cubic. --> |
||
|
Weiko
|
adf56cac56 |
fix(workflow): avoid 'file not found' when duplicating unbuilt code steps (#22179)
## Context Duplicating a workflow version (e.g. when editing an active workflow) clones each Code step's logic function via copyResources, which copied both the source and the built artifact unconditionally. Logic functions created from source are not built yet (no index.mjs, isBuildUpToDate=false), so copying the missing built file threw FILE_NOT_FOUND. Copy the built artifact only when it exists. This is safe because the duplicate inherits isBuildUpToDate=false and is rebuilt lazily on activation or run. In seed dev case for example, they are created with source but never built ## Test https://github.com/user-attachments/assets/5a7febba-413b-4041-985e-f84a99a5ebda |
||
|
Charles Bochet
|
ad61d6d8a3 |
fix(server): dispatch each cron trigger exactly once (#22113)
## Problem
App/logic-function crons occasionally fire **twice, ~1 minute apart**.
The most visible symptom is a notification cron sending the same Discord
DM (or channel post) at e.g. `17:00` and again at `17:01`.
## Root cause
`CronTriggerCronJob` runs every minute (`* * * * *`) and re-dispatches
any logic function whose pattern is "due" according to `shouldRunNow`:
```ts
const diff = Math.abs(prevTriggerDate.getTime() - now.getTime());
return diff < rootCronIntervalMs; // 60_000
```
The detection window (`60_000ms`) is **equal to** the 60s tick interval.
So when a root tick drifts across a minute boundary (runs slightly
early/late, or BullMQ fires a catch-up), two adjacent ticks can both see
the *same* trigger as "within the last 60s" and each enqueue a
`LogicFunctionTriggerJob`. The dispatch isn't idempotent, so the
function runs twice.
## Fix
Make dispatch idempotent, keyed on the trigger itself:
- New `getMatchingTriggerTimestamp(pattern, now)` returns the epoch-ms
of the matched trigger (stable regardless of *when* within the window
the root job runs), or `null`. `shouldRunNow` now delegates to it —
behaviour unchanged.
- Before enqueuing, `CronTriggerCronJob` claims a
`logic-function-cron:{workspace}:{function}:{triggerTs}` key in the
`EngineLock` cache. A second tick that resolves to the same trigger
finds the key and skips.
Distinct triggers always have distinct timestamps (hence distinct keys),
so a later legitimate run is never suppressed. The TTL (2 min) only
needs to outlive the detection window.
## Notes
- `WorkflowCronTriggerCronJob` uses the same `shouldRunNow` pattern and
has the same latent double-dispatch; left out of this PR to keep it
focused, but the new helper makes the same guard a small follow-up.
- The cache `get`-then-`set` isn't atomic; for the observed failure mode
(ticks ~1 min apart, sequential) it's reliable. A Redis `SET NX` would
also close the rare concurrent-multi-instance race.
## Test plan
- [x] `should-run-now.utils.spec.ts` extended: two ticks within one
window resolve to the same timestamp; out-of-window and invalid patterns
return `null`. All 8 pass.
- [x] `oxlint --type-aware` + `oxfmt` clean on changed files.
<!-- This is an auto-generated description by cubic. -->
<a
href="https://cubic.dev/pr/twentyhq/twenty/pull/22113?utm_source=github"
target="_blank" rel="noopener noreferrer"
data-no-image-dialog="true"><picture><source
media="(prefers-color-scheme: dark)"
srcset="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"><source
media="(prefers-color-scheme: light)"
srcset="https://www.cubic.dev/buttons/review-in-cubic-light.svg"><img
alt="Review in cubic"
src="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"></picture></a>
<!-- End of auto-generated description by cubic. -->
|
||
|
Félix Malfait
|
614bc7b7e6 |
feat: serve HTTP logic functions on isolated *.withtwenty.com domain (#22045)
## Summary Implements [core-team-issues#2473](https://github.com/twentyhq/core-team-issues/issues/2473): serve HTTP-triggered logic functions from a dedicated, **cookieless** public domain (`{workspaceSubdomain}.withtwenty.com`) instead of the same-site `/s/` route, so functions can safely return **arbitrary headers** — custom headers, `Permissions-Policy` (camera/mic/geolocation), `Cross-Origin-Opener-Policy: same-origin`, `Cross-Origin-Embedder-Policy: require-corp`, `Set-Cookie`, etc. The `/s/` route stays the strict, same-site path it is today. **Self-hosting is unchanged** — everything new is gated on `PUBLIC_DOMAIN_URL` being set. ### Why Today user-authored function responses are served same-site with the Twenty app, so the response-header allow-list is restricted to 5 safe headers and request headers are limited to a per-function allow-list. Serving from an origin that shares nothing with `*.twenty.com` removes that constraint safely — the same "user content domain" pattern as GitHub (`*.githubusercontent.com`) and CodeSandbox (`*.csb.app`). ## What's in here **Routing** - The **root-path → `/s` rewrite happens at the nginx ingress**, not in app code. The existing `api-ingress.yaml` already rewrites root paths onto `/s` (host-agnostically) when the edge sets `X-Twenty-Public-Domain: true`, so `*.withtwenty.com` and registered custom public domains are handled by the same mechanism. (An earlier in-app middleware was removed as a redundant, wrong-layer duplicate.) - `WorkspaceDomainsService.resolveWorkspaceAndPublicDomain` recognizes `*.` subdomains, resolves the workspace by subdomain, and returns `isIsolatedOrigin`. Explicitly registered public-domain rows still take precedence and keep their application scoping. The ingress preserves the `Host` header, so this resolution still fires. **Headers (server)** - Isolated origin → all response headers pass through and all request headers are forwarded. Same-site `/s/` keeps the strict allow-lists. (Global CORS already handles preflight/ACAO.) **`/s/` deprecation for new routes (cloud only)** - New `LOGIC_FUNCTION_LEGACY_ROUTE_CUTOFF` config var (ISO date, optional). When `PUBLIC_DOMAIN_URL` is set, functions created on/after the cutoff return **410 Gone** on `/s/` with the new URL. Existing routes and self-hosted instances are untouched. **Frontend education** - `publicFunctionDomain` added to `ClientConfig` (from `PUBLIC_DOMAIN_URL`). - The logic-function **Live URL** now resolves to `https://{workspaceSubdomain}.{publicFunctionDomain}{path}` on cloud, falling back to `/s/` for self-hosting. - Front components call their functions through the SDK (`RestApiClient`), which now targets the isolated domain via the injected `TWENTY_FUNCTIONS_URL`. - New **"Public URL"** section on the application **Settings** tab explaining the isolated domain (shown when the app exposes HTTP-triggered functions). **Docs**: note the `withtwenty.com` domain for external callers in the apps guide. ## Infra prerequisites (not code — needs dashboard work) - Wildcard DNS `*.withtwenty.com` (proxied) + wildcard TLS in the public-domain Cloudflare zone. - Edge (Cloudflare) sets `X-Twenty-Public-Domain: true` for `*.withtwenty.com` requests, so the existing nginx ingress rewrites them onto `/s` (same header the custom-domain flow already relies on). - Set `PUBLIC_DOMAIN_URL=https://withtwenty.com` on cloud. - Submit `withtwenty.com` to the **Public Suffix List** (required for cross-tenant cookie isolation before relying on `Set-Cookie`). ## Test plan - [x] `nx typecheck twenty-server`, `nx typecheck twenty-front` - [x] `lint:diff-with-main` + oxfmt clean (server + front) - [x] `npx jest route-trigger public-function-domain domain-server-config workspace-domains build-logic-function-event client-config` → server unit tests passing (resolution tiers, header passthrough vs allow-list, `/s/` cutoff 410) - [x] `npx jest getLogicFunctionHttpUrl` (front) and `nx test twenty-client-sdk` (RestApiClient routing) passing - [x] CI green (server, front, sdk, renderer, ui, zapier, example apps) - [ ] Manual: hit `{subdomain}.withtwenty.com/` end-to-end once infra is provisioned <a href="https://cubic.dev/pr/twentyhq/twenty/pull/22045?utm_source=github" rel="nofollow noreferrer noopener" target="_blank">``<img alt="Review in cubic" src="https://www.cubic.dev/buttons/review-in-cubic-dark.svg">``</a> |
||
|
Paul Rastoin
|
a5aac3c21e |
Logic function handler name hardened validation (#21956)
# Introduction Introduce centralized handlerName validation for the logic function handlerName inside the flat logic function validator Even if not safe by definition, avoid string interpolation inside the local driver executor when retrieving the handler name from the parent module <!-- This is an auto-generated description by cubic. --> <a href="https://cubic.dev/pr/twentyhq/twenty/pull/21956?utm_source=github" target="_blank" rel="noopener noreferrer" data-no-image-dialog="true"><picture><source media="(prefers-color-scheme: dark)" srcset="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"><source media="(prefers-color-scheme: light)" srcset="https://www.cubic.dev/buttons/review-in-cubic-light.svg"><img alt="Review in cubic" src="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"></picture></a> <!-- End of auto-generated description by cubic. --> |
||
|
martmull
|
6423c4cd3c |
Add recall io webhook endpoint (#21879)
## Context
Bot-recording integrations (e.g. the Recall.ai meeting bot) receive
webhooks from a third-party provider that delivers **every
tenant's events to a single URL**. Our existing `route-trigger` (`/s/…`)
resolves the workspace from the request host, which can't
work for one shared multi-tenant webhook URL. We need an instance-scoped
ingress that identifies the target workspace from the payload
instead.
## Strategy
Add a new **`ingress-trigger`** logic-function trigger, mirroring
`route-trigger`:
- A public endpoint keyed by the app's identifiers: `POST
/webhooks/ingress/:applicationRegistrationUniversalIdentifier/:logicFunctionUniversalIdentifier`.
- The logic function declares an `ingressTriggerSettings` block in its
manifest describing how to find the workspace in the payload
(`workspaceId: { source: 'body' | 'query' | 'header', path }`).
- Core only **resolves the workspace** (declarative, fail-closed,
prototype-safe path getter), verifies the app is installed in that
workspace, then runs the function **synchronously** so the provider sees
the response (status codes / retries).
- **Signature verification stays in the logic function** (it gets
`rawBody` + forwarded headers), keeping core provider-agnostic.
- Shared execution logic (`build event → execute → map response`)
extracted into `LogicFunctionTriggerService`, now reused by both
`route-trigger` and `ingress-trigger`.
## Major changes
- **twenty-shared**: new `ingressTriggerSettings` on
`LogicFunctionManifest` (`IngressTriggerSettings` type).
- **twenty-server**: new `ingress-trigger` module (controller, service,
exception + filter, workspace-id resolver util).
- **twenty-server**: extracted `LogicFunctionTriggerService` +
`route-trigger-response.util` (response builder + sender); refactored
`RouteTriggerService` and both controllers to reuse them.
- **twenty-docs**: documented the ingress trigger (endpoint, workspace
resolution, signature responsibility, provider HMAC examples).
- Unit tests for the resolver and the ingress service.
|
||
|
Paul Rastoin
|
98c35ea804 |
App uninstall lambda, layers cleanup (#21749)
# Introduction On a logic function deletion also remove the driver entry On a app uninstall also remove the sdk layer ( keeps the dep one as it can be shared across several lambda ) | Resource | Scope | Before this PR | After | |---|---|---|---| | DB metadata (functions, objects, fields…) | per-app | deleted | deleted | | Source folder (`FileFolder.Source`) | per-function | deleted | deleted | | Built handler file (`FileFolder.BuiltLogicFunction`) | per-function | deleted | deleted | | **Lambda function** | per-function | **leaked** | **deleted** (driver `delete`) | | **SDK layer** `sdk-<wsId>-<appId>` (all versions) | per-app | **leaked** | **deleted** (driver `deleteApplicationResources` → `deleteSdkLayer`) | | Deps layer `deps-<checksum>` | shared across apps/workspaces | not deleted | **intentionally not deleted** (content-addressed, GC'd) | ## What I don't like about all that Right now there's some non reversible side effect inside the workspace migration transaction - If the transaction fails we're facing data loss - It also slows down everything I'm about to create a new PR that allow population post transaction commit side effect / cleanup to be run later <!-- This is an auto-generated description by cubic. --> <a href="https://cubic.dev/pr/twentyhq/twenty/pull/21749?utm_source=github" target="_blank" rel="noopener noreferrer" data-no-image-dialog="true"><picture><source media="(prefers-color-scheme: dark)" srcset="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"><source media="(prefers-color-scheme: light)" srcset="https://www.cubic.dev/buttons/review-in-cubic-light.svg"><img alt="Review in cubic" src="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"></picture></a> <!-- End of auto-generated description by cubic. --> |
||
|
Paul Rastoin
|
26db3f5735 |
Deprecate legacy encryption (#21831)
# Introduction Still preserving the cross-upgrade flow close https://github.com/twentyhq/core-team-issues/issues/2465 <!-- This is an auto-generated description by cubic. --> <a href="https://cubic.dev/pr/twentyhq/twenty/pull/21831?utm_source=github" target="_blank" rel="noopener noreferrer" data-no-image-dialog="true"><picture><source media="(prefers-color-scheme: dark)" srcset="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"><source media="(prefers-color-scheme: light)" srcset="https://www.cubic.dev/buttons/review-in-cubic-light.svg"><img alt="Review in cubic" src="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"></picture></a> <!-- End of auto-generated description by cubic. --> |
||
|
Thomas Trompette
|
7a1cfc17cc |
fix(logic-function): treat invoke timeout as a user-level error, not a platform error (#21779)
Fixes https://twenty-v7.sentry.io/issues/7527156270?project=4507072499810304 ## Problem Sentry was flooded with high-severity alerts for logic functions that simply ran too long: > Lambda timed out for function '…' during invoke (functionState=Active, phase=invoke …) A function exceeding its configured `timeoutSeconds` is a **user-level outcome** (their code is too slow), not a platform failure — but it was being reported as one. ## Root cause The two timeout mechanisms were classified inconsistently: - **Lambda's own timeout** → returns `{ status: ERROR, … }` → handled as a user error (route returns 500 with `shouldBeCapturedBySentry: false`, queue job records it without failing). Not in Sentry. ✓ - **Client-side `AbortSignal` timeout** → **threw** `LOGIC_FUNCTION_EXECUTION_TIMEOUT`, which isn't mapped in `mapErrorToRouteTriggerCode`, so it fell through to `ROUTE_TRIGGER_PLATFORM_ERROR` (Sentry) and failed the BullMQ job (Sentry). ✗ Since the executor Lambda is fixed at 900s, the client abort is the *sole* timeout enforcement for every function with `timeoutSeconds < 900` — so essentially every slow function paged the team. The `local` driver already returns an ERROR result here; only the Lambda driver threw. ## Fix On an **invoke-phase** `TimeoutError`, return a structured ERROR result instead of throwing — mirroring the Lambda's own timeout and the local driver. The timeout now flows through the normal result path: surfaced to the caller as `status: ERROR`, recorded via `handleExecutionResult` (which the throw path skipped), and kept out of Sentry. **Build- and fetch-phase timeouts still throw** and stay in Sentry — those are platform-side (executor build / code fetch too slow, even for short user code), which is exactly what the phase instrumentation exists to catch. ## Tests - Unit test on the new `buildLogicFunctionTimeoutResult` util - `npx jest logic-function-drivers/drivers/lambda` green |
||
|
Charles Bochet
|
34362de7b7 |
fix(route-trigger): distinguish user vs platform logic function execution errors (#21715)
## What Splits route trigger logic-function failures into two cases instead of one catch-all: - **User error** — the function's own code threw an uncaught error. Returns `500` and is **not** sent to Sentry. - **Platform error** — an infrastructure/execution failure on our side. Returns `500` and **is** sent to Sentry. A disabled logic function now returns `403`. ## Why User-code failures were flooding Sentry: a single workspace's function hitting a transient upstream error generated tens of thousands of events. #21656 stopped the flood by muting the entire route-trigger execution error bucket — but muting everything also silenced genuine platform failures we *do* want to be alerted on. Splitting the bucket keeps the user-code noise out of Sentry (the original goal) while making sure real platform errors still surface. Users who want to return a specific status/body when their function fails can still catch the error and return a `Response` — that path is unchanged. <!-- This is an auto-generated description by cubic. --> <a href="https://cubic.dev/pr/twentyhq/twenty/pull/21715?utm_source=github" target="_blank" rel="noopener noreferrer" data-no-image-dialog="true"><picture><source media="(prefers-color-scheme: dark)" srcset="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"><source media="(prefers-color-scheme: light)" srcset="https://www.cubic.dev/buttons/review-in-cubic-light.svg"><img alt="Review in cubic" src="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"></picture></a> <!-- End of auto-generated description by cubic. --> |
||
|
Charles Bochet
|
8205c97b5b |
fix(route-trigger): return 422 instead of 500 for logic function execution errors (#21656)
## What Return HTTP 422 instead of 500 for `LOGIC_FUNCTION_EXECUTION_ERROR` in the route trigger exception filter. ## Why When a logic function's user code fails (e.g. an HTTP call inside the function returns a 502 from an upstream service), the exception was mapped to HTTP 500. This caused two problems: - **Sentry noise**: `shouldCaptureException` captures all 5xx responses, so every user-code failure was reported as a platform error. This generated ~56k Sentry events over 2 months for a single workspace's logic function hitting a transient upstream 502. - **Webhook retry loops**: Webhook senders like GitHub auto-retry on 5xx responses, amplifying the event count. `LOGIC_FUNCTION_EXECUTION_ERROR` is a user-code error, not a platform error. A 422 (Unprocessable Entity) correctly signals that the request could not be processed due to the logic function's own failure, without triggering Sentry capture or webhook retries. <!-- This is an auto-generated description by cubic. --> <a href="https://cubic.dev/pr/twentyhq/twenty/pull/21656?utm_source=github" target="_blank" rel="noopener noreferrer" data-no-image-dialog="true"><picture><source media="(prefers-color-scheme: dark)" srcset="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"><source media="(prefers-color-scheme: light)" srcset="https://www.cubic.dev/buttons/review-in-cubic-light.svg"><img alt="Review in cubic" src="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"></picture></a> <!-- End of auto-generated description by cubic. --> |
||
|
Paul Rastoin
|
fdab89ae02 |
Move twenty-client-sdk to dev dep (#21611)
# Introduction The `twenty-client-sdk` is always provided and injected at runtime by the twenty-server instance Which mean that even if in your app locally you're using twenty-client-sdk `1.0` installing this app on twenty instance `2.0` will result in injecting another `twenty-client-sdk` That's the expected behavior and tradeof The twenty-app devdep should only be used to guide local devxp following typesafety and so on A user can still locally generated its own twenty-client-sdk and publish it if necessary <!-- This is an auto-generated description by cubic. --> <a href="https://cubic.dev/pr/twentyhq/twenty/pull/21611?utm_source=github" target="_blank" rel="noopener noreferrer" data-no-image-dialog="true"><picture><source media="(prefers-color-scheme: dark)" srcset="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"><source media="(prefers-color-scheme: light)" srcset="https://www.cubic.dev/buttons/review-in-cubic-light.svg"><img alt="Review in cubic" src="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"></picture></a> <!-- End of auto-generated description by cubic. --> |
||
|
Charles Bochet
|
869680a5a1 |
fix(deps): esbuild ^0.28.1 floors + vite 7→8 (rolldown) upgrade (#21517)
## What this does Resolves the remaining esbuild security alerts on packages we own, and upgrades the repo to **Vite 8** (which drops esbuild entirely in favour of rolldown/oxc). ### 1. esbuild → `^0.28.1` (security) - Raised the declared `esbuild` floor in `twenty-sdk` and the logic-function common-layer (both were `^0.25.0`, which can only resolve to a vulnerable version). These are our packages, so this is just declaring the patched version — clears Dependabot **#1467** and **#1468**. ### 2. Vite 7 → 8 - Bumped `vite` to `^8` in the 5 packages that declare it, and `@vitejs/plugin-react-swc` to `^4.3.1` (the only plugin that needed a bump for Vite 8; everything else already supports it). - `twenty-front` keeps esbuild minification, so esbuild is now an explicit (patched) devDependency there — Vite 8 no longer ships it. ### Two Vite-8 fallout fixes (bundler internals changed) - **Storybook tests:** added React to `optimizeDeps.include` so Vite's dep optimizer doesn't re-bundle React mid-run and break in-flight imports in browser-mode tests. - **`hex-rgb`:** it's ESM-only and broke rolldown's CJS interop (a default import resolved to the wrong thing under jest). Replaced its one use with a tiny inline hex→rgb parse and dropped the dependency. ## Verified Vite resolves to a single `8.0.16` with no esbuild in its tree. Builds pass on Vite 8/rolldown: `twenty-front` production build, the SDKs, and Storybook; the previously-failing front and storybook test jobs now pass; `yarn install --immutable` is clean. ## Note This doesn't close root alert **#1469** — esbuild is still pulled by other third-party tools (storybook, tsx, lingui, zapier, etc.) that haven't shipped a patched release. The vulnerable code path (esbuild's dev server) isn't used here, so that one is best dismissed as not-affected. |
||
|
Charles Bochet
|
9bb98fa5b5 |
fix(billing): don't crash when workspace has no active subscription (#21510)
## Problem
Sentry (high severity, SLA-breaching): `Billing Subscription Not Found:
No active subscription found for workspace …`
The `billingSubscription` workspace-cache provider
(`WorkspaceBillingSubscriptionCacheService.computeForCache`) called
`getCurrentBillingSubscriptionOrThrow`. For a workspace whose
subscription is fully canceled, `getCurrentBillingSubscription` filters
out `Canceled` and returns `undefined`, so the provider **threw**
`BILLING_SUBSCRIPTION_NOT_FOUND`.
That cache key is read on every usage-recording path:
- workflow execution
(`WorkflowExecutorWorkspaceService.sendWorkflowNodeRunEvent`)
- AI usage (`AiBillingService`)
- logic-function execution (`LogicFunctionExecutorService`)
- app charges (`AppBillingService`)
- the gate `BillingUsageService.canFeatureBeUsed` /
`hasAvailableCredits` / `decrementAvailableCreditsInCache`
- the cancellation webhook
(`invalidateAndRecompute('billingSubscription')`)
So any of these throws an unhandled exception for a
no-active-subscription workspace. The intent was clearly to tolerate
this state — `canFeatureBeUsed` already guards with
`isDefined(billingSubscription)` and the workflow runner logs *"there is
no subscription for this workspace"* — but the throwing provider made
those guards unreachable.
## Fix
- `computeForCache` now returns `FlatBillingSubscription | null` via the
non-throwing `getCurrentBillingSubscription`, and the cache type allows
`null`.
- Every consumer guards the absent case (`isDefined` / optional
chaining) and no-ops: usage events still emit with an undefined
`periodStart`, credits aren't decremented, `hasAvailableCredits` returns
`false`.
- `getCurrentBillingSubscriptionOrThrow` is **left untouched** for the
many callers (resolver, subscription-update, etc.) that genuinely
require a subscription.
## Test
Adds `workspace-billing-subscription-cache.service.spec.ts`: the
provider returns `null` when there's no active subscription (regression)
and the flattened subscription when one exists.
All 142 tests across the billing / ai-billing / workflow-executor suites
pass; `oxlint --type-aware` and `oxfmt` are clean.
<!-- This is an auto-generated description by cubic. -->
<a
href="https://cubic.dev/pr/twentyhq/twenty/pull/21510?utm_source=github"
target="_blank" rel="noopener noreferrer"
data-no-image-dialog="true"><picture><source
media="(prefers-color-scheme: dark)"
srcset="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"><source
media="(prefers-color-scheme: light)"
srcset="https://www.cubic.dev/buttons/review-in-cubic-light.svg"><img
alt="Review in cubic"
src="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"></picture></a>
<!-- End of auto-generated description by cubic. -->
|
||
|
Thomas Trompette
|
0ac4f237c0 |
fix(server): stop redundant lambda rebuilds causing build-lock acquisition failures (#21442)
## Context `Lambda invocation failed for function '<id>' during build: Failed to acquire lock for key: lambda-build:<id>` fires ~1000 times/day in production. ## Root cause `LambdaExecutorManagerService.buildExecutor` re-checks `canSkip` inside the `lambda-build:<functionId>` lock, but the re-check reuses the `flatApplication` snapshot captured when the request started. `canSkip` depends on `!flatApplication.isSdkLayerStale`, so: 1. An app sync/install regenerates the SDK client and sets `isSdkLayerStale = true` 2. All in-flight executions of the function fail `canSkip` and queue on the lock 3. The first holder rebuilds and `markSdkLayerFresh` clears the flag in DB + workspace cache 4. Queued waiters can't see that fix — their in-memory snapshot still says stale — so **each waiter redoes the full rebuild serially** (download SDK archive, delete + republish layer, update function config, wait for update) 5. The lock is held back-to-back for minutes; everyone deeper in the queue exhausts the 120s retry budget and throws The local driver already handles this correctly (`LocalLayerManagerService` refreshes the flat application from the workspace cache inside its lock); the lambda driver missed it. ## Fix - Refresh `flatApplication` from the workspace cache inside the lock before re-checking `canSkip`, so waiters skip in ~100ms once the first holder finishes - Degrade gracefully on lock-acquisition timeout: re-check build status with fresh data and proceed with the invocation if the executor is already usable, instead of failing the run (introduces a typed `CacheLockAcquisitionError` so only that case is caught) ## Test plan - [x] `cache-lock.service.spec.ts` passes - [x] `lint:diff-with-main` + typecheck pass - [ ] Monitor `Failed to acquire lock for key: lambda-build:*` error rate in production after deploy |
||
|
nitin
|
20c83e1f86 |
fix(kanban): preserve scroll on board re-init + propagate same-column reorders via SSE (#20637)
closes https://discord.com/channels/1130383047699738754/1504130730840821860 https://github.com/user-attachments/assets/d5833031-01c6-4e46-b699-c29c42435a53 ## Summary Fixes two related issues with the kanban (board view) collaboration experience: 1. **Scroll-to-top on every data change** — `triggerRecordBoardInitialQuery` always scrolled the board to the top, even when re-initializing for a single-record data change (SSE echo of your own mutation, a collaborator's update). Scroll reset only makes sense when the dataset itself changes (filter / sort / group). 2. **Same-column reorders by other users did not propagate** — the server's diff function stripped `FieldMetadataType.POSITION`, so position-only updates produced empty `updatedFields` and short-circuited event emission entirely. SSE clients never received them. ## What's in here - **Frontend** — `useTriggerRecordBoardInitialQuery` now exposes a `triggerRecordBoardInitialQueryWithoutScrollReset` variant; data-driven re-inits in `RecordBoardDataChangedEffect` use it, while genuine filter / sort / group changes keep the scroll-resetting `triggerRecordBoardInitialQuery`. `getRecordBoardEffectsForUpdateInputs` classifies each update as `trigger-initial-query` / `reposition-records` / `none`. For position- or group-only changes we skip the re-query and reposition records in place in the store (`useRepositionRecordsOnBoard`), which avoids the flicker and preserves scroll. - **Server** — removes `POSITION` from `objectRecordChangedValues`' strip list, so position-only updates emit a non-empty diff and flow through SSE. Position is now treated as a field like any other across all event consumers (SSE, webhooks, workflows, logic functions); a trigger with an explicit field filter still excludes it. |
||
|
Charles Bochet
|
45ceb5d74c |
security: bump vulnerable deps in logic-function runtime lockfiles (#21317)
## What Two standalone dep-set lockfiles bundled for the logic-function / application-package sandbox carried vulnerable transitive versions. Within-range bumps only (these dep sets are handed to user logic functions, so no breaking majors): **`seed-dependencies`** - `nodemailer` `^8.0.4 → ^8.0.5` (resolves 8.0.10) — SMTP CRLF command injection (direct dep) - `brace-expansion`, `picomatch`, `lodash` → patched **`common-layer-dependencies`** - `lodash`, `brace-expansion` → patched ## Left for follow-up (can't be fixed within-range) - `qs` — pinned transitively by `body-parser`, stays at 6.14.2 (needs body-parser bump) - `ip-address` 9.x → 10.x and `uuid` 10.x → 11.x — major bumps; left out since these are exposed to user functions and would be breaking `axios` here is already on the patched `^1.16.1`. |
||
|
martmull
|
36567e96ab |
Remove twenty-sdk from package.json before build (#21305)
Follow up of https://github.com/twentyhq/twenty/pull/21179 Removes twenty-sdk from dependencies when building lambda layer |
||
|
Félix Malfait
|
91f2f08995 |
feat(server): unify workspace-event ingestion behind one EventSink pipeline (#21197)
## Why The five event-log streams (`workspaceEvent`, `pageview`, `objectEvent`, `usageEvent`, `applicationLog`) each wrote to ClickHouse through their own fire-and-forget writer (`AuditService`, `UsageEventWriterService`, and the `application-logs` driver), with the per-type knowledge (table names, normalization, access rules) spread across several modules. Three of them reimplemented the same ClickHouse insert, and the read side, the live stream, and the producers lived in different modules under two different names. This consolidates them into one `core-modules/event-logs/` subsystem (emit, write, live, read), with the per-type config in a single registry so adding an event type is roughly one file. The base Logs settings tab and free application logs shipped separately in #21180 (merged). This PR adds the unified backend, the registry, and the viewer's live mode and entitlement gating. ## Pipeline ```mermaid flowchart TB subgraph PROD["Producers"] A["auth, billing, impersonation,<br/>webhook, custom-domain"] U["usage listener"] F["logic-function executor (app logs)"] R["record CRUD (entity events)"] end EM["EventLogEmitterService<br/>createContext().insert* / dispatch()"] EQ(["entityEventsToDbQueue<br/>(existing, shared with timeline)"]) CIE["CreateEventLogFromInternalEvent"] SINK["WorkspaceEventSinkService.ingest()"] C1["ClickHouseEventSink"] C2["ConsoleEventSink"] LIVE["EventLogLiveService.publishWatched()<br/>(presence-gated)"] CH[("ClickHouse, 5 tables, async_insert")] CHAN(["WORKSPACE_EVENTS_CHANNEL"]) RS["EventLogsService (registry-driven read)"] LR["EventLogsLiveResolver"] UI["Settings > Logs"] A --> EM U --> EM F --> EM EM -->|direct| SINK R --> EQ --> CIE -->|ingest| SINK SINK --> C1 --> CH SINK --> C2 SINK --> LIVE -.->|if a viewer is watching| CHAN --> LR --> UI CH --> RS --> UI ``` ## What it does - Producers call `EventLogEmitterService.createContext().insert*()`, which builds a typed `WorkspaceEventEnvelope` and writes it through `WorkspaceEventSinkService` to the configured sinks (ClickHouse, Console) plus a presence-gated live fan-out. Record/CRUD events reach the same sink through the existing `entityEventsToDbQueue`. There is no dedicated queue; ClickHouse `async_insert` batches server-side. Writes are best-effort, as on main today. - `EVENT_LOG_TYPES[table]` is the per-type source of truth: the ClickHouse table, the required entitlement, the free-text filter column, and the row-to-GraphQL mapping. Read row shapes derive from the write rows. - Four modules along their dependency boundaries: `EventLogEmitterModule` (producer API), `EventLogIngestionModule` (sink layer), `EventLogLiveModule` (fan-out), and `EventLogsViewerModule` (the entitlement-gated GraphQL read, which is where billing/enterprise/permissions stay so producers stay light). - Logs viewer: per-table columns, filters (text, date, record), live mode, and an upgrade card that points to Billing on Cloud or the Admin Panel on self-hosted. Application logs are free on every plan; the other four require the `AUDIT_LOGS` entitlement (with a `NO_ENTITLEMENT` fallback to the upgrade card). - Renames `AuditService` to `EventLogEmitterService`, and the generic `Monitoring` event to a typed `Impersonation` event (`level` + `action`). - Removes `UsageEventWriterService`, the `application-logs` driver/module, and `AuditService`'s direct inserts. ## Durability Writes are best-effort, the same as main today (the old writers were fire-and-forget). A dedicated queue was tried mid-PR and removed: `async_insert` already batches server-side, so the queue only added durability, which isn't a requirement right now. The `EventSink` seam keeps a durable transport (e.g. a Redis-Streams buffer) easy to add later without touching producers. ## Out of scope S3 peer sink (seam only), Postgres or any second read path, `ReplicatedMergeTree`, ClickHouse table-schema changes, and the record-data `EVENT_STREAM_CHANNEL` (unchanged, separate concern). ## Testing Unit tests cover the registry definitions and row normalization, the entitlement gating, the envelope builders, and the producers. Integration tests cover the write paths (record create produces an `objectEvent`; the track mutation produces a `workspaceEvent`) and the read/query path across all five tables. Verified with typecheck, lint, a server boot, and GraphQL/SDK codegen. |
||
|
Félix Malfait
|
c3dd6b25a6 |
fix: use canonical oxlint rule id in lint-disable directives (#21253)
## What Many `oxlint-disable` / `eslint-disable` directives across the repo carry a corrupted rule id — `@typescripttypescript/<rule>` — most likely a find-and-replace accident that mangled the eslint-era `@typescript-eslint/` prefix. oxlint matches disable directives **loosely by rule name**, so these still suppress in practice (not a silent no-op), but the id is malformed and misleading. ## Change Replace them with the **canonical oxlint id** `typescript/<rule>` — matching the plugin name and rule keys declared in `.oxlintrc.json` — **127 files, 262 directives**: | rule | count | | --- | ----- | | `typescript/no-explicit-any` | 250 | | `typescript/ban-ts-comment` | 6 | | `typescript/no-misused-promises` | 4 | | `typescript/no-empty-object-type` | 2 | - `twenty-server`: 122 files - `twenty-front`: 5 files Comment-only — no code or runtime changes. ## Verification `oxlint --type-aware -c .oxlintrc.json` reports **0 warnings / 0 errors** for both `twenty-server` and `twenty-front`. Every changed line is exactly the id correction inside a disable directive (262 insertions / 262 deletions, no collateral edits). > Addresses the cubic review, which flagged that the canonical oxlint id is `typescript/...` (no `@`). Worth noting the original `@typescripttypescript/` was not actually a silent no-op — oxlint matches these directives loosely by rule name — but `typescript/` is the correct, config-aligned id. |
||
|
martmull
|
e0d42323af |
Add more control on http trigger (#21216)
add "new Response" utils to define response code or content type of http route triggered logic function responses follow up of https://github.com/twentyhq/twenty/pull/21214 --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Paul Rastoin
|
3d49642d12 |
[AUDIT] Run knip over twenty-server (#21159)
# Introduction Run [knip](https://knip.dev/) over twenty-server Used config: ```json { "$schema": "https://unpkg.com/knip@5/schema.json", "workspaces": { "packages/twenty-server": { "entry": [ "src/main.ts", "src/command/command.ts", "src/queue-worker/queue-worker.ts", "src/database/scripts/setup-db.ts", "src/database/scripts/truncate-db.ts", "src/database/clickHouse/migrations/run-migrations.ts", "src/database/clickHouse/seeds/run-seeds.ts", "src/instrument.ts", "lingui.config.ts", "test/integration/graphql/codegen/index.ts", "test/integration/utils/setup-test.ts", "test/integration/utils/teardown-test.ts", "scripts/**/*.ts", "**/*.spec.ts", "**/*.integration-spec.ts" ], "project": ["src/**/*.ts", "test/**/*.ts", "scripts/**/*.ts"], "ignore": [ "src/database/typeorm/**/migrations/**", "src/database/typeorm/**/*.entity.ts", "**/*.workspace-entity.ts", "**/logic-function-resource/constants/seed-project/**" ], "ignoreDependencies": ["@types/psl", "@types/aws-lambda"], "ignoreBinaries": ["nest", "lingui", "typeorm"] } } } ``` |
||
|
Charles Bochet
|
f6a17bc5f7 |
refactor(server): split LambdaDriver and LocalDriver into focused sub-services (#21153)
## Why `LambdaDriver` (1.4k lines) and `LocalDriver` (709 lines) each mixed many unrelated concerns in a single class. ## What changed **This PR only moves code** — every line is byte-for-byte the same as `main` (same logic, same comments, same constants, same lock keys/TTLs, same error mapping). The drivers are now thin orchestrators wiring co-located sub-services + pure utils (each with unit tests). | File | Before | After | |---|---|---| | `lambda.driver.ts` | 1425 | 252 | | `local.driver.ts` | 709 | 231 | | Largest sub-service | — | 371 (`lambda-executor-manager`) | | Unit tests on these paths | 0 | 15 | Lambda split: `lambda-aws-client` / `lambda-tool-functions` / `lambda-layer-manager` / `lambda-executor-manager` + `constants` + `types` + 5 pure utils. Local split: `local-layer-manager` / `local-child-process-runner` / `local-prebuilt-bundle` + `constants` + `types` + 3 pure utils. No behavior change, no signature change, no public-API change. |
||
|
Charles Bochet
|
58907b733c |
feat(logic-function): add LIVE / PREBUILT execution modes (#20873)
## Summary
### Why
1. Sending the code to the lambda (~1Mb usually) is heavy on network and
results to a constant traffic of ~30Mb/s on AWS which results into TB of
network data every month
2. eval(1MB of code) is not that fast, it's heavy on memory and CPU on
lambda side
### High level
Adds two execution modes for logic functions, gated behind the new
`IS_LOGIC_FUNCTION_PREBUILT_MODE_ENABLED` workspace feature flag (off
everywhere by default):
- **LIVE** (current behavior, preserved bit-for-bit): the compiled
bundle is read from object storage and shipped in every Lambda invoke
payload. Used for fast iteration in the workflow editor / Settings test
runs.
- **PREBUILT** (new): the bundle is installed onto the per-function
Lambda alongside the unified executor, and invocations carry only `{
params, env, handlerName }` — saving JSON payload egress and warm-start
`import()` cost on every call.
### Key design choices
- **Unified Lambda handler** (`constants/executor/index.mjs`) dispatches
at runtime: `event.code` present ? LIVE (write to `/tmp`, dynamic
import) : `import('./prebuilt-logic-function.mjs')`. Both code paths
always coexist on the deployment package, so the same Lambda can serve
either mode without redeploying.
- **Install runs inside the `validateBuildAndRun` migration pipeline**,
not at execute time. `Create/UpdateLogicFunctionActionHandlerService`
calls `driver.installPrebuiltBundle` when `executionMode` flips
LIVE?PREBUILT or `checksum` changes while PREBUILT, gated on
`isBuildUpToDate=true` and a fresh checksum.
- **Strict execute, no reconciliation**:
`LogicFunctionExecutorService.execute` resolves `effectiveExecutionMode`
(caller override > feature flag > entity column). For PREBUILT it asks
the driver `getInstalledBundleChecksum` (Lambda `twenty:bundle-checksum`
tag for AWS, sidecar file locally) and throws
`LOGIC_FUNCTION_PREBUILT_BUNDLE_NOT_INSTALLED` on mismatch.
- **Feature flag gates every side effect**: with the flag off the
executor forces LIVE, the action-handler install hooks bail before AWS,
and workflow activation does not flip the mode. Rollback is just turning
the flag off.
### Lifecycle
- New workflow CODE step ? `LIVE`, no install.
- Workflow activated ? build + activation flips `executionMode=PREBUILT`
? action-handler installs the bundle + sets the Lambda tag.
- Draft from active version ? duplicated logic function reset to `LIVE`.
- App install ? manifest converter sets `PREBUILT`, create-action
handler installs.
- Test runs (`executeOneFromSource`, workflow editor) pass
`executionMode=LIVE` explicitly.
### Observability
`[lambda-timing]` log lines now include `effectiveExecutionMode` and
`payloadBytes`; the action handler logs `install_duration_ms` for each
install.
## Test plan
- [x] `npx nx typecheck twenty-server` ? passes
- [x] `npx oxlint --type-aware` on all changed files ? 0 warnings, 0
errors
- [x] `npx nx test twenty-server` ? 588 suites / 5009 tests pass (no
regressions vs main)
- [x] New unit suite `flat-logic-function-validator.service.spec.ts` ?
9/9
- [x] Existing
`workflow-version-step-operations.workspace-service.spec.ts` ? 8/8
(verified the new token-based DI avoids a circular-import regression)
- [x] Snapshot for
`ALL_UNIVERSAL_FLAT_ENTITY_PROPERTIES_TO_COMPARE_AND_STRINGIFY` updated
to include `executionMode`
- [x] Integration suite `logic-function-execution.integration-spec.ts`
extended to assert `executionMode=LIVE` on newly-created functions and
continues to exercise the LIVE happy path
- [ ] Manual staging rollout: flip
`IS_LOGIC_FUNCTION_PREBUILT_MODE_ENABLED` per workspace, observe
`[lambda-timing]` `payloadBytes` drop + `install_duration_ms`, then ramp
in prod.
|
||
|
Paul Rastoin
|
ebfaca5b3d |
EncryptedString PlaintextString branded string types (#21001)
## Summary closes https://github.com/twentyhq/core-team-issues/issues/2464 Introduces compile-time branded types to distinguish encrypted ciphertext from plaintext strings, preventing mix-ups like the one fixed in #20819 — but at the type level rather in addition to the one existing at runtime. ### Branded string primitives - Created `EncryptedString` and `PlaintextString` as hard nominal brands using `z.string().brand(...)`, making them non-assignable to each other or to raw `string` - Created `isEncryptedString` type predicate to narrow `string` to `EncryptedString` based on the `enc:v2:` envelope prefix - Retyped `SecretEncryptionService`: `encryptVersioned` accepts `PlaintextString`, `decryptVersioned` returns `PlaintextString` ### Entity typing - Typed encrypted columns across entities: `SigningKeyEntity.privateKey`, `TwoFactorAuthenticationMethodEntity.secret`, `ApplicationRegistrationVariableEntity.encryptedValue`, `ApplicationVariableEntity.value` - Parameterized JSONB types for connected account connection parameters (`ImapSmtpCaldavParams<Pwd>`) with reusable aliases `EncryptedImapSmtpCaldavParams` / `DecryptedImapSmtpCaldavParams` - Typed DTOs (`CreateApplicationRegistrationVariableInput`, `UpdateApplicationRegistrationVariablePayload`, `UpdateApplicationVariableEntityInput`) with `PlaintextString` ### ApplicationVariable always-encrypt uniformization - Retyped `ApplicationVariableEntity.value` to `EncryptedString | ''` — all values are now encrypted regardless of `isSecret` - Updated `ApplicationVariableEntityService` to always encrypt on write and always decrypt on read - Simplified `UpdateApplicationVariableActionHandlerService` by removing conditional encrypt/decrypt-on-isSecret-toggle logic - Added slow instance command (`2.9.0`) to backfill-encrypt existing `isSecret=false` plaintext rows and tighten the `CHECK` constraint ### ConfigStorageService refactor - Split `convertAndSecureValue` (which used `any`) into two well-typed methods: `convertAndDecrypt` and `convertAndEncrypt` - Introduced `isSensitiveStringValue` type predicate to narrow values before encryption/decryption ### What's next - Typeorm entity derivation to strictly type sitemap configuration as code + handler logic for encryption rotation - https://github.com/twentyhq/core-team-issues/issues/2465 |
||
|
Thomas Trompette
|
6e2cf4cbc0 |
Fix lambda timeout diagnostics (#21002)
Trying to fix https://twenty-v7.sentry.io/issues/7420384466/?environment=prod&environment=prod-eu&project=4507072499810304&query=is%3Aunresolved%20%21issue.type%3A%5Bperformance_consecutive_db_queries%2Cperformance_consecutive_http%2Cperformance_file_io_main_thread%2Cperformance_db_main_thread%2Cperformance_n_plus_one_db_queries%2Cperformance_n_plus_one_api_calls%2Cperformance_p95_endpoint_regression%2Cperformance_slow_db_query%2Cperformance_render_blocking_asset_span%2Cperformance_uncompressed_assets%2Cperformance_http_overhead%2Cperformance_large_http_payload%5D%20timesSeen%3A%3E10&referrer=issue-stream&sort=date ## Summary - Prevent invoking Lambda functions stuck in `Pending` state by checking `Configuration.State === 'Active'` in `checkLambdaExecutorBuildStatus` — a Pending function now goes through `ensureLambdaExecutor` which waits for Active. - Track execution phase (`build`/`fetch-code`/`invoke`) via a `LambdaExecutionPhase` enum and include phase timing + function state in all error messages for faster debugging. Co-authored-by: Matt Van Horn <mvanhorn@users.noreply.github.com> |
||
|
Thomas Trompette
|
ff28547a36 |
tt-fix-logic-function-error-handling (#20928)
## Summary - **Fix `callWithTimeout` timer leak**: the `setTimeout` was never cleared when the callback resolved first, leaving orphaned timers (up to 15 minutes) in the Node.js event loop. Now uses `try/finally` with `clearTimeout`. - **Properly classify timeout errors**: introduced `ExecutionTimedOutError` so the Lambda driver can throw `LOGIC_FUNCTION_EXECUTION_TIMEOUT` instead of `LOGIC_FUNCTION_EXECUTION_FAILED` — users now see "Function execution timed out" instead of the generic "Function execution failed." ## Test plan - [x] Execute a logic function normally and verify it still works - [x] Trigger a logic function timeout and verify the error message says "Function execution timed out" (not "Function execution failed") - [x] Verify that a deleted logic function invocation logs a specific "was deleted" warning - [x] Verify no orphaned timers remain after logic function execution completes |
||
|
Paul Rastoin
|
b8de469f37 |
Refactor and centralize file mimeType integrity check and sanitization (#20889)
# Introduction closes https://github.com/twentyhq/private-issues/issues/484 This PR refactors the writeFile API to never expect to be passed a mimetype, its extract is done programmatically low level so any callers will pass through Same for the file sanitization ## IANA override Disclaimer for consistency we existing behavior we wanted to always have `application/typescript` - should we rather consider fallbacking to octect-steam instead ? - Any pulbic assets that has .ts will now also fallback to `application/typescript` instead of the official IANA ## Integration Added coverage |
||
|
Paul Rastoin
|
b8b115f4e3 |
FileStorageService Dedicated file and folder code flow + integrity check (#20831)
# Introduction
Next handling mimetype integrity check and checksum integrity check for
s3 storage type
Always expecting a trailing end slash when deleting a folder etc
## Application
Uninstalling an application now deletes all its related files
## File storage service
Making a distincton between folder path and file path
## Validation Pipeline
Every file operation in `FileStorageService.buildOnStoragePath` runs
through `validateResourcePath`, which chains three validators in order:
**1. `validateSafeRelativePath`** -- rejects path traversal attacks
| Input | Result | Error |
|---|---|---|
| `../../../etc/passwd` | Rejected | `Resource path must not contain
path traversal (..)` |
| `/etc/passwd` | Rejected | `Resource path must be relative, not
absolute` |
| `file\0.txt` | Rejected | `Resource path contains null bytes` |
| `..\\..\\etc\\passwd` | Rejected | `Resource path must not contain
backslashes` |
| _(empty)_ | Rejected | `Resource path must not be empty` |
**2. `validateFilenameIntegrity`** -- enforces safe characters, length
limits, extension required
| Input | Result | Error |
|---|---|---|
| `my folder/file.mjs` | Rejected | `A path segment contains invalid
characters...` |
| `Makefile` | Rejected | `Filename must have an extension` |
| `aaa...(256 chars).mjs` | Rejected | `A path segment exceeds the
maximum length of 255 characters` |
| `a/b/.../file.mjs` (1025+ chars) | Rejected | `Resource path exceeds
maximum length of 1024 characters` |
| `src/handlers/index.mjs` | Accepted | -- |
| `my-app/my_file.tsx` | Accepted | -- |
| `v1.0/module.config.mjs` | Accepted | -- |
Allowed characters per segment: `a-z`, `A-Z`, `0-9`, `.`, `-`, `_`
**3. `validateResourceExtension`** -- checks extension against the
`FileFolder` allowlist
| Input | FileFolder | Result | Error |
|---|---|---|---|
| `handler.js` | `BuiltLogicFunction` | Rejected | `Invalid file
extension. Allowed extensions: .mjs` |
| `card.tsx` | `BuiltFrontComponent` | Rejected | `Invalid file
extension. Allowed extensions: .mjs` |
| `script.js` | `PublicAsset` | Rejected | `Invalid file extension.
Allowed extensions: .png, .jpg, ...` |
| `index.mjs` | `BuiltLogicFunction` | Accepted | -- |
| `app.tsx` | `Source` | Accepted | -- |
| `photo.png` | `CorePicture` | Accepted | -- (unconfigured folder,
passes through) |
## Consumers
- **`FileStorageService`** -- calls `validateResourcePath`, throws
`FileStorageException` on failure (last-resort defense)
- **Resolver (`uploadApplicationFile`)** -- calls
`validateResourcePath`, throws `ApplicationException` on failure
(user-facing)
- **Flat validators** -- call `validateResourcePath`, push the error to
`validationResult.errors` (non-throwing, collects all errors)
All error messages are translated via Lingui `t` and returned in a
discriminated union `{ isValid: true } | { isValid: false, error: string
}`, letting each consumer decide how to handle failures.
|
||
|
Charles Bochet
|
7fa136f305 |
feat(twenty-server): migrate remaining at-rest encryption sites to versioned envelope (#20550)
## Summary Second PR in the encryption key rotation series. The previous PR (#20528) introduced `ENCRYPTION_KEY` + the versioned `enc:v2:<keyId>:<base64>` envelope inside `SecretEncryptionService` and migrated `ConnectedAccountTokenEncryptionService` as the first consumer. This PR routes every remaining at-rest encryption site through the versioned envelope so that `ENCRYPTION_KEY` (and the future `FALLBACK_ENCRYPTION_KEY`) actually covers them. The legacy unprefixed CTR ciphertext remains readable as a fallback during the rollout window — every migrated read site uses `decryptVersioned`, which transparently delegates to the legacy CTR decrypt when it sees an unprefixed payload. ### Service migrations - **`ApplicationVariableEntityService` (#8)** — workspace-scoped. HKDF info is bound to each row's `workspaceId`. A new `decryptAndMaskVersioned` helper lands on `SecretEncryptionService` for the resolver display path. - **`ApplicationRegistrationVariableService` (#7)** + consumers — **instance-scoped**. Registration variables are server-level config readable by every workspace that installs the application, so HKDF info is `instance`. Updated consumers: - `LogicFunctionExecutorService.buildServerVariableEnvMap` - `ConnectionProviderService.getClientCredentials` - **`LogicFunctionExecutorService.buildEnvVar` (#9)** — workspace-scoped. Each variable's `workspaceId` is threaded into `decryptVersioned`, so per-workspace HKDF contexts are honoured at execution time. - **`UpdateApplicationVariableActionHandlerService`** (workspace-migration runner) — threads `workspaceId` through the secret/non-secret toggle. - **`JwtKeyManagerService` (#3)** — instance-scoped. Signing keys are shared across the JWKS. - **`ConfigStorageService` (#6)** — instance-scoped sensitive STRING config variables. ### Slow instance commands (2.5.0) Each migrated site has a paired backfill that re-encrypts existing rows into the v2 envelope before the column is constrained: | timestamp | command | scope | CHECK constraint | |---|---|---|---| | `1798000005000` | encrypt-application-variable | workspaceId | `"isSecret" = false OR value = '' OR value LIKE 'enc:v2:%'` | | `1798000006000` | encrypt-application-registration-variable | instance | `"encryptedValue" = '' OR value LIKE 'enc:v2:%'` | | `1798000007000` | encrypt-signing-key-private-keys | instance | `"privateKey" IS NULL OR value LIKE 'enc:v2:%'` | | `1798000008000` | encrypt-sensitive-config-storage | instance | _none_ — heterogeneous jsonb column | All backfills are idempotent (the SELECT filter skips rows already in v2 form) and run before their respective `up()` adds the CHECK constraint. Every `down()` deliberately stops at dropping the CHECK constraint — they intentionally do not re-introduce plaintext on rollback. ### Tests - Unit specs for each new slow command cover the v2 upgrade path, the idempotency invariant, and the instance vs workspace HKDF scope. - New `JwtKeyManagerService` spec asserts `decryptVersioned`/`encryptVersioned` are called without `workspaceId` (instance scope). - Updated existing specs for `ApplicationVariableEntityService`, `ConfigStorageService`, and `buildEnvVar` to assert the versioned API and the workspace HKDF context plumbing. - New `SecretEncryptionService.decryptAndMaskVersioned` cases in the service spec. - Updated the `applicationRegistrationVariable` integration spec to assert the column now stores `enc:v2:<keyId>:<base64>` instead of raw legacy CTR. ### Out of scope (future PRs) - `PostgresCredentialsService` — bespoke `jwtWrapperService.generateAppSecret`–derived key + `encryptText`/`decryptText` from `auth.util.ts`; deserves its own migration. - `SimpleSecretEncryptionUtil` (TOTP) — entirely different `aes-256-cbc` `iv:enc` format; deserves its own migration. ## Test plan - [x] `npx nx typecheck twenty-server` - [x] `npx nx lint:diff-with-main twenty-server` (oxlint + prettier) - [x] Local jest run for `secret-encryption | connected-account-token | application-variable | application-registration-variable | build-env-var | jwt-key-manager | config-storage | encrypt-application-variable | encrypt-application-registration-variable | encrypt-signing-key | encrypt-sensitive-config-storage` — 17 suites, 106 tests pass. - [x] Local jest run for `upgrade | instance-command` — 12 suites, 86 tests pass. - [ ] CI green - [ ] Manual review of CHECK constraint shapes by a server reviewer (each one matches `enc:v2:%` rather than `enc:v_:%` since none of the migrated columns can legitimately hold `enc:v1:` ciphertext). |
||
|
Etienne
|
aecfe699f4 |
feat(ai-chat) - Stop ai thinking if credits exhausted (#20526)
Billing is now decremented per-step, not per-turn. The onStepFinish callback in chat-execution.service.ts calls a new decrementAndCheckAvailableCredits method on each model step, so Redis is debited incrementally as the agent runs rather than all at once at the end. Credit exhaustion stops the stream mid-run. When a step depletes the remaining credits, a hasNoMoreAvailableCredits flag is set and passed into the stopWhen predicate of streamText, causing the agent to halt before starting the next step. A new credits-exhausted event is introduced. After the stream drains and the response is persisted, if credits ran out the job publishes a dedicated credits-exhausted event to the frontend instead of the normal message-persisted event. The frontend handles this new event. useAgentChatSubscription has a new credits-exhausted case that sets a BILLING_CREDITS_EXHAUSTED-coded error on the atom, closes the writer, and stops the streaming state — triggering the existing AiChatCreditsExhaustedMessage UI. |
||
|
Paul Rastoin
|
a159a68e2c |
[twenty-server] no floating promises lint rule (#20499)
## Introduction
That's an audit + RFC
## Fire-and-forget (`void`) -- Intentional, correct
These are telemetry, metrics, and audit logging in hot paths or
non-critical contexts. `void` is the right choice.
| File | What was voided |
|---|---|
| `sign-in-up.service.ts` | `metricsService.incrementCounter` (sign-up
metric) + `auditService.insertWorkspaceEvent` (workspace created) |
| `use-graphql-error-handler.hook.ts` | 5x
`metricsService.incrementCounter` (GraphQL operation metrics) |
| `bullmq.driver.ts` | 2x `metricsService.incrementCounter` (job
completed/failed metrics) |
| `call-webhook.job.ts` | 2x `auditService.insertWorkspaceEvent` + 1x
`metricsService.incrementCounter` |
| `custom-domain-manager.service.ts` | `analytics.insertWorkspaceEvent`
(domain activation event) |
| `logic-function-executor.service.ts` |
`auditService.insertWorkspaceEvent` (function execution) |
| `workflow-runner.workspace-service.ts` |
`metricsService.incrementCounter` (throttle metric) |
| `cleaner.workspace-service.ts` | `metricsService.incrementCounter`
(deleted workspace metric) |
| `stream-agent-chat.job.ts` | Detached IIFE for streaming chunks
(intentional concurrent pipeline) |
| `workspace-auth-context.middleware.ts` |
`withWorkspaceAuthContext(...)` (AsyncLocalStorage, returns void anyway)
|
## Top-level script entry points (`void bootstrap()`)
These are module-level calls where the promise has no consumer. `void`
makes the lint rule happy and documents the intent.
| File | What changed |
|---|---|
| `main.ts` | `void bootstrap()` |
| `command.ts` | `void bootstrap()` |
| `queue-worker.ts` | `void bootstrap()` |
| `truncate-db.ts` | `void dropSchemasSequentially()` |
| `codegen/index.ts` | `void generateTests(forceArg)` |
## Now properly awaited -- Real bug fixes
These were floating promises that could silently fail, lose data, or
cause race conditions.
| File | What was fixed |
|---|---|
| `billing-sync-plans-data.command.ts` | `meters.map(async ...)` wrapped
in `Promise.all` -- was returning before upserts finished |
| `cache-storage.service.ts` | `setAdd` and `setPop` had `.then()`
chains that weren't returned/awaited |
| `create-audit-log-from-internal-event.ts` | 4x
`auditService.createObjectEvent` now awaited inside a job |
| `cleaner.workspace-service.ts` | 2x `emailService.send(...)` now
awaited -- emails could silently fail |
| `agent-async-executor.service.ts` | `calculateAndBillUsage` +
`billNativeWebSearchUsage` in `finally` block now awaited |
| `repair-tool-call.util.ts` | `calculateAndBillUsage` now awaited |
| `agent-title-generation.service.ts` | `calculateAndBillUsage` now
awaited |
| `chat-execution.service.ts` | `billNativeWebSearchUsage` now awaited |
| `ai-generate-text.controller.ts` | `calculateAndBillUsage` in
`finally` block now awaited |
| `agent-turn.resolver.ts` | `messageQueueService.add(...)` now awaited
|
| `command.ts` | `app.close()` now awaited (was exiting before graceful
shutdown) |
| `i18n.service.ts` | `loadTranslations()` in `onModuleInit` now awaited
|
| `workspace-query-hook.explorer.ts` | `explore()` in `onModuleInit` now
awaited |
| `message-queue.explorer.ts` | `handleProcessorGroupCollection` in
`onModuleInit` now awaited |
| `ai-billing.service.spec.ts` | Test now properly `await`s the async
call |
| `messaging-messages-import.service.spec.ts` | `expect(...)` now
properly `await`ed for async assertion |
| `archive.finalize()` (3 files) | Voided -- promise resolution already
handled by `pipeline()` / `on('end')` |
## Impersonation & security audit trail -- Upgraded from `void` to
`await`
These were previously fire-and-forget but are
security/compliance-critical events that must be reliably persisted.
| File | What was fixed |
|---|---|
| `impersonation.service.ts` | 4x `auditService.insertWorkspaceEvent`
now awaited (impersonation attempt, token generation
attempt/success/failure) |
| `auth.resolver.ts` | 5x `auditService.insertWorkspaceEvent` now
awaited (impersonation token exchange attempt/success/failure at server
and workspace levels) |
| `auth.service.ts` | 2x `analytics.insertWorkspaceEvent` now awaited
(impersonation attempted/issued) |
## Billing audit -- Upgraded from `void` to `await`
Payment events should be reliably persisted for financial/compliance
reporting.
| File | What was fixed |
|---|---|
| `billing-webhook-invoice.service.ts` |
`auditService.insertWorkspaceEvent(PAYMENT_RECEIVED_EVENT)` now awaited
inside Stripe webhook handler |
## Fire-and-forget with proper error handling -- Upgraded from bare
`void`
These remain non-blocking but now catch and log errors instead of
risking unhandled rejections.
| File | What was fixed |
|---|---|
| `logic-function-executor.service.ts` |
`applicationLogsService.writeLogs` now uses `.catch()` instead of bare
`void` -- user-facing logs should surface errors |
## Systemic infrastructure fixes
| File | What was fixed |
|---|---|
| `metrics.service.ts` | `incrementCounter`: Redis cache write
(`metricsCacheService.updateCounter`) now uses `.catch()` internally
instead of raw `await` -- prevents unhandled rejections across all `void
metricsService.incrementCounter(...)` call sites when Redis is unhealthy
|
| `audit.service.ts` | `preventIfDisabled`: made properly `async` with
`await` and consistent `Promise<{ success: boolean }>` return type.
Removed broken `catch` that returned an `AuditException` as a value
(wrong constructor args, unreachable dead code). Removed unused
`AuditException` import |
## Fixed in this session (beyond original PR)
| File | What changed |
|---|---|
| `telemetry.listener.ts` | Removed misleading `Promise.all` + `void`
combo; replaced with simple `for...of` + `void` |
| `message-queue.explorer.ts` | Changed from `void` to `await` so
startup crashes on registration failure |
|
||
|
martmull
|
7ade9e3aab |
Fix application variable issue (#20500)
Fixes wrong formatting converting secret value to ******** before saving Issued by https://discord.com/channels/1130383047699738754/1423290797079662602/1502674904770936903 |
||
|
Félix Malfait
|
34b927ff23 |
feat(public-domain): bind public domains to apps + reorganize settings (#20360)
## Summary - **Public domains can now be bound to a specific app.** When a request hits an app-bound public domain, route resolution restricts logic-function matching to that app's HTTP-routed functions only — isolating each app's routes to its own domain instead of letting routes from other apps in the workspace match nondeterministically. - **Settings sidebar reorganized.** Removed the standalone Domains page. Workspace Domain → General. Approved Domains + Invitations → Members "Access" tab. Emailing Domains + Public Domains → Apps "Developer" tab. Roles → Members "Roles" tab. ## Why The use case: someone building a partner portal app or a lead-collection app declares private objects (leads, partners…) plus a few public HTTP routes. Each app needs its own domain (`partners.acme.com`, `leads.acme.com`) without those domains exposing every other app's routes in the same workspace. Today's PublicDomainEntity is workspace-scoped only, so all HTTP-routed logic functions in a workspace compete for any public domain — first match wins nondeterministically. ## Backend - Added nullable `applicationId` FK to `PublicDomainEntity` (cascade-deleted with the app); indexed for the route-trigger lookup. - New fast instance command `2-4-instance-command-fast-1798000003000-add-application-id-to-public-domain` adds the column, index, and FK constraint. - `createPublicDomain(domain, applicationId)` accepts an optional app binding; new `updatePublicDomain(domain, applicationId)` mutation rebinds/unbinds an existing domain. Both validate the application belongs to the workspace. - `WorkspaceDomainsService.resolveWorkspaceAndPublicDomain(origin)` returns both the workspace and the matched public domain in one query — replacing the old back-to-back lookups in the route-trigger hot path. `getWorkspaceByOriginOrDefaultWorkspace` is preserved as a thin wrapper. - `RouteTriggerService` filters `logicFunction` by `applicationId` when the matched public domain is app-scoped; falls back to workspace-wide when unbound. - Three sequential validation queries in `createPublicDomain` now run in parallel via `Promise.all`. ## Frontend | Old location | New location | |---|---| | Settings sidebar → Domains (standalone page) | Removed | | Domains page → Workspace Domain | General page | | Domains page → Approved Domains | Members → Access tab | | Domains page → Emailing Domains | Apps → Developer tab | | Domains page → Public Domains | Apps → Developer tab | | Settings sidebar → Roles (standalone) | Members → Roles tab | | `pages/settings/roles/` | `pages/settings/members/roles/` | - The Public Domain detail page has an Application picker that uses `Select`'s native `emptyOption` + `null` value pattern (matches `SettingsDataModelObjectIdentifiersForm`). - Members page tabs use the existing `TabListFromUrlOptionalEffect` mechanism (rendered automatically by `TabList`) for hash-based tab activation. - `/settings/members/roles` redirects to `/settings/members#roles` so role sub-pages' `navigate(SettingsPath.Roles)` lands on the Members page with the Roles tab pre-selected. - All affected breadcrumbs updated to nest under their new parents. - `SettingsPath.Roles` and friends now nest under `members/`; `Subdomain` and `CustomDomain` under `general/`; `PublicDomain` and `EmailingDomain` under `applications/`. ## Test plan - [x] `nx typecheck twenty-front` passes - [x] `nx typecheck twenty-server` passes - [x] `oxlint --type-aware` clean on all touched files - [x] `prettier --check` clean on all touched files - [x] Migration applied locally; `publicDomain.applicationId` (uuid, nullable) confirmed in DB - [x] GraphQL schema exposes `PublicDomain.applicationId`, `createPublicDomain.applicationId`, `updatePublicDomain` mutation - [x] **End-to-end route resolution scenarios verified locally:** - Domain bound to App A, function in App A → route matches ✅ - Domain bound to App B, function in App A → route does NOT match (HTTP 404 `TRIGGER_NOT_FOUND`) ✅ - Domain unbound (`applicationId = NULL`) → route matches workspace-wide ✅ - Unknown path on bound domain → returns 404 cleanly ✅ - [x] UI sanity (browser-tested at `apple.localhost:3001`): - General page shows Workspace Domain card - Members page shows Team / Access / Roles tabs - Access tab combines Invite by link + by email + Approved Domains - Roles tab embeds the role list - `/settings/members/roles` direct URL → redirects + Roles tab pre-selected - Apps Developer tab shows Emailing Domains + Public Domains sections - Public Domain detail page has Application picker dropdown listing workspace apps - Sidebar nav: "Domains" and "Roles" no longer present (now folded into General/Members) ## Notes for reviewers - Creating a public domain via the UI still requires Cloudflare credentials in the dev `.env` (`CLOUDFLARE_API_KEY`, `CLOUDFLARE_PUBLIC_DOMAIN_ZONE_ID`, `PUBLIC_DOMAIN_URL`). The DNS step is unchanged from main. - The `applicationId` column is nullable, so existing public-domain rows continue to work workspace-wide — no data backfill required. - `SettingsRolesContainer` was deleted (no longer referenced after `SettingsRoles` index page was removed). 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
martmull
|
0d05788547 |
Protect sendEmail endpoint and thread user context through logic function executor (#20369)
- Thread userId and userWorkspaceId through LogicFunctionExecutorService.execute() so application access tokens carry user context when available. This allows logic functions triggered by authenticated HTTP routes to call sendEmail with proper user identity, making the existing verifyOwnership() check work naturally. - Gate the sendEmail resolver with SettingsPermissionGuard(PermissionFlagType.SEND_EMAIL_TOOL) instead of NoPermissionGuard, ensuring only callers with the SEND_EMAIL_TOOL permission can send emails. |
||
|
Etienne
|
9fc5be1c4c |
Billing - Migrate from Stripe metering (#20298)
**Overall strategy** **1. Introduce “Billing V2” behind a workspace flag** Gate the new model with FeatureFlagKey.IS_BILLING_V2_ENABLED so existing workspaces stay on the old behavior until they’re migrated or explicitly on V2. **2. Replace workflow metered SKUs with a resource-credit product** Conceptually, billable “workflow execution” usage is not the primary subscription line item anymore. Add a RESOURCE_CREDIT product (and keep WORKFLOW_NODE_EXECUTION as deprecated for the transition). Usage and limits are expressed through credit buckets (e.g. price metadata like credit_amount), so one product can represent pooled credits instead of a narrow workflow-only meter. **3. Migrate subscriptions in two layers** Schema/catalog: persist extra price metadata (instance upgrade) so the server knows credit amounts and can match Stripe prices to the new model. Per workspace: the registered workspace command upgrade:2-2:migrate-to-billing-v2 finds subscriptions that still have WORKFLOW_NODE_EXECUTION, swaps those items to the right RESOURCE_CREDIT prices (using existing Stripe schedule + BillingSubscriptionUpdateService stack), then treats the workspace as V2 (flag). Workspaces without that legacy item or without a subscription are skipped. **4. Unify subscription lifecycle + usage on the server** **5. Refresh the product surface in Settings** Test : - [x] Subscribe v1 + Update subscribe + Migrate - [x] Subscribe v2 + Update subscribe |
||
|
martmull
|
617f571400 |
20215 convert application variable to a syncable entity (#20269)
## Summary - Converts applicationVariable from a bespoke sync path to a proper SyncableEntity, unifying it with the workspace migration pipeline used by all other manifest-managed entities (agent, skill, frontComponent, webhook, etc.) - Removes the upsertManyApplicationVariableEntities method and its direct-DB-mutation approach in favor of the standard validate → build → run action handler pipeline - Adds universalIdentifier, deletedAt columns and makes applicationId NOT NULL via an instance command migration ## Motivation Before this change, applicationVariable was the only manifest-managed entity that bypassed ApplicationManifestMigrationService.syncMetadataFromManifest(). It used a bespoke service method called directly from syncApplication(), creating two mental models, two validation styles, and two cache invalidation patterns. Now there's one unified pipeline for all manifest entities. ## What changed ### Entity refactor: - ApplicationVariableEntity now extends SyncableEntity (gains universalIdentifier, non-nullable applicationId with CASCADE, soft-delete via deletedAt) ### New flat entity layer (flat-application-variable/): - Type, maps type, editable properties constant, entity-to-flat converter, cache service, module ### New migration pipeline wiring: - Manifest converter (fromApplicationVariableManifestToUniversalFlatApplicationVariable) - Validator service (FlatApplicationVariableValidatorService) - Builder service (WorkspaceMigrationApplicationVariableActionsBuilderService) - Create/Update/Delete action handlers with secret encryption hooks - Registered in orchestrator, builder module, runner module, and all type registries ### Removed bespoke path: - Deleted upsertManyApplicationVariableEntities from ApplicationVariableEntityService - Removed its call from ApplicationSyncService.syncApplication() - Kept update() (operator-set value at runtime) and getDisplayValue() (runtime display) ### Database migration: - Instance command to add columns, backfill universalIdentifier, enforce NOT NULL constraints, and update indexes ## Test plan - npx nx typecheck twenty-server passes (0 errors) - Unit tests pass (application-variable.service.spec.ts, build-env-var.spec.ts) - Install an app with applicationVariables in its manifest → variables appear with correct universalIdentifier - Update app manifest (add/remove/modify a variable) → migration pipeline handles diff correctly - Operator-set value via update endpoint persists correctly with encryption - Uninstall app → variables cascade-deleted - app dev --once on example app syncs without errors |
||
|
Félix Malfait
|
9e94045fa5 |
feat(apps): generic OAuth provider support for app SDK (#20181)
## Summary
App developers can now declare third-party OAuth integrations (GitHub,
Linear, Slack, etc.) in their manifest and the platform handles the full
authorize → callback → token-exchange → refresh → injection lifecycle.
The dev writes ~10 lines of config and reads tokens via
`useOAuth('linear')` inside any logic function.
```ts
// app/src/oauth-providers/linear.ts
export default defineOAuthProvider({
universalIdentifier: '...',
name: 'linear',
displayName: 'Linear',
authorizationEndpoint: 'https://linear.app/oauth/authorize',
tokenEndpoint: 'https://api.linear.app/oauth/token',
scopes: ['read', 'write'],
connectionMode: 'per-user',
clientIdVariable: 'LINEAR_CLIENT_ID',
clientSecretVariable: 'LINEAR_CLIENT_SECRET',
tokenRequestContentType: 'form-urlencoded',
});
// app/src/logic-functions/handlers/...
const { accessToken } = useOAuth('linear'); // throws OAuthNotConnectedError if missing
```
## Architecture
- **Storage**: extends the existing `connectedAccount` table — new
nullable `applicationOAuthProviderId` FK + new `app` value on the
`ConnectedAccountProvider` enum. Existing Google/Microsoft flows are
untouched.
- **OAuth flow**: a single `/apps/oauth/authorize` +
`/apps/oauth/callback` controller pair handles every app provider. State
travels in a JWT signed via the existing `JwtWrapperService` (new
`APP_OAUTH_STATE` token type).
- **Token exchange**: goes through
`SecureHttpClientService.createSsrfSafeFetch()` (so an installed app
can't point `tokenEndpoint` at internal hosts).
- **Refresh**: piggybacks on the existing
`ConnectedAccountRefreshTokensService` dispatch — Google/Microsoft
drivers untouched, new app driver lives engine-side under
`application-oauth-provider/refresh/`.
- **Injection**: the executor injects refreshed tokens as env vars
(`OAUTH_<NAME>_ACCESS_TOKEN`, `_HANDLE`, `_SCOPES`, `_CONNECTED`); the
SDK helpers `useOAuth` / `useOptionalOAuth` read them.
- **Frontend**: auto-rendered "OAuth Connections" section under each
app's settings tab (no custom front component needed). App-managed
connections are filtered out of `/settings/accounts` so the
email/calendar page stays focused.
- **Disconnect**: best-effort revoke against the manifest's
`revokeEndpoint` before deleting the row.
## Reference app
`packages/twenty-apps/internal/twenty-linear/` exercises the full
pipeline:
- `defineOAuthProvider` for Linear
- `POST /linear/create-issue` and `GET /linear/teams` HTTP-route logic
functions
- Vitest tests for the handlers
## Tests
- 14 server-side Jest tests: token-exchange util (form-urlencoded vs
JSON, PKCE, error paths), flow service (authorize URL shape, state
binding, ConnectedAccount upsert on first/reconnect, per-workspace mode,
invalid state)
- 8 app-level Vitest tests: handler error paths, GraphQL request shape,
Linear error propagation
- All 4 packages clean: `npx nx lint:diff-with-main` and `npx tsc
--noEmit`
## Test plan
- [ ] Apply migration on a dev DB: `npx nx run
twenty-server:database:migrate:prod`
- [ ] Regenerate frontend types: `npx nx run
twenty-front:graphql:generate --configuration=metadata`
- [ ] Create a Linear OAuth app at
https://linear.app/settings/api/applications/new with redirect URI
`<SERVER_URL>/apps/oauth/callback`
- [ ] Deploy + install `twenty-linear` on a workspace, paste the Linear
client id/secret into the app's variables
- [ ] Click "Connect Linear" in the app's settings tab → complete OAuth
→ verify `connectedAccount` row created with `provider = 'app'`
- [ ] Trigger `POST /linear/create-issue` with a valid teamId → verify
issue lands in Linear
- [ ] Disconnect → verify the row is deleted and (if Linear's revoke
endpoint is configured in the manifest) the revoke call fires
- [ ] Verify `/settings/accounts` does NOT show the Linear connection —
it appears only under the Linear app's settings tab
## Out of scope (deliberately)
- **Cron + per-user providers**: a cron-triggered function with a
per-user OAuth provider currently returns `CONNECTED=false` (no user
context). The follow-up design is `useOAuthForUser(name,
userWorkspaceId)` paired with a `POST /apps/oauth/connection-token`
endpoint, deferred to keep this PR focused.
- **Token encryption at rest**: tokens stored as plain `varchar`
matching the existing Google/Microsoft pattern. Worth a separate
cross-cutting PR.
- **Manifest endpoint pinning**: a malicious app upgrade could change
`tokenEndpoint` silently. Same trust model as logic-function source code
(which already runs arbitrary server-side); worth tightening across the
whole upgrade pipeline rather than just OAuth.
- **CLI helpers** (`twenty oauth show-callback-url`, `twenty oauth
connect`): manual setup for v1.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
|
||
|
Etienne
|
fbaea0639a |
Billing - optimize usageEvent CH table (#20019)
- Update usageEvent clickhouse table, partitioning, indexing and projection (auto materialized view) to optimize credit usage queries - Add caching for available credits and billing subscription To do in next PR: deprecate enforceCapUsage cron. Bonus : real-time on billingSubscription |
||
|
Charles Bochet
|
3db1af9a17 |
fix(logic-function): forward raw request body for HMAC signature verification (#20061)
## Summary - Add optional `rawBody?: string` to `LogicFunctionEvent` and forward it from the route trigger so HMAC-based webhook signatures (GitHub's `X-Hub-Signature-256`, Stripe, …) can be verified by user logic functions. - Update `github-connector`'s `getRawBodyForSignature` to prefer `event.rawBody` (with the existing string/base64/null fallbacks kept for older runtimes). ## Why GitHub computes `X-Hub-Signature-256` over the **raw bytes** of the request body. The receiver must verify against those exact bytes — key order, whitespace and unicode escaping all matter, so the parsed JSON body cannot be re-serialized to them. Today the route trigger calls `extractBody(request)` which returns the parsed object only. NestJS already preserves the raw body on `request.rawBody` (the app is bootstrapped with `rawBody: true` in `main.ts`), but it was never propagated into `LogicFunctionEvent`. As a result the github-connector's webhook handler always took the "raw body unavailable" branch and rejected every delivery (after #19961 / 962c2b3c14). With this change, signature verification can succeed end-to-end. --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Charles Bochet
|
499067ae14 |
fix(logic-function): serialize LocalDriver layer builds with cache lock (#20054)
## Summary
Fixes a race condition in `LocalDriver` where concurrent `execute()`
calls for the same application can trash each other's layer build,
causing logic function executions to fail with either:
- `Error: ENOENT: process.cwd failed with error no such file or
directory, the current working directory was likely removed without
changing the working directory, uv_cwd` (the yarn-install child's `cwd`
got wiped mid-run), or
- `ENOENT: no such file or directory, open '…/<pkg>/<file>.js.map'`
raised from yarn's berry link step (files under the deps layer vanish
while yarn is extracting).
Both are thrown from `…/deps/<checksum>/.yarn/releases/yarn-4.9.2.cjs` —
i.e. the yarn process `copyYarnEngineAndBuildDependencies` spawns with
`cwd: buildDirectory`.
## Root cause
`LocalDriver.createLayerIfNotExist` (and `ensureSdkLayer`) both follow a
check-then-act pattern with no mutual exclusion:
```ts
if (await pathExists(depsNodeModulesPath)) return;
await fs.rm(depsLayerPath, { recursive: true, force: true });
await copyDependenciesInMemory(...);
await copyYarnEngineAndBuildDependencies(depsLayerPath); // spawns yarn with cwd=depsLayerPath
```
The deps layer path is shared across all `execute()` calls that match a
given `yarnLockChecksum`. Two concurrent callers (e.g. a webhook
invocation + a cron-triggered logic function firing while the first
run's layer is still being built) both see no `node_modules`, both
`fs.rm` the directory, and one's yarn child ends up with its `cwd` or
its extraction target gone.
## Fix
Wrap the critical sections with `CacheLockService.withLock` — the same
cache-backed lock the Lambda driver already uses for its layer/executor
builds:
- `createLayerIfNotExist` → lock key
`local-driver-deps-layer:${yarnLockChecksum ?? 'default'}`
- `ensureSdkLayer` → lock key
`local-driver-sdk-layer:${workspaceId}:${applicationUniversalIdentifier}`
A fast-path `pathExists` check is kept outside the lock (so warm
executions still skip lock acquisition entirely), and the existence +
staleness check is **repeated inside the lock** so followers no-op after
the leader finishes.
Lock parameters mirror the Lambda driver's layer-build lock (`ttl=120s`,
`retry=500ms`, `maxRetries=240`).
`cacheLockService` is now passed to `LocalDriver` from
`LogicFunctionDriverFactory` (it was already injected there for the
Lambda driver).
|
||
|
Félix Malfait
|
4f938aa097 |
feat(app): infrastructure for pre-installed apps (#19973)
**PR 1 of 2.** Follow-up PR ships the Exa app, sets it as a default
pre-installed app, and removes the current `WebSearchTool` /
`WebSearchService` / `ExaDriver`. This PR adds the plumbing; no
user-visible change yet.
## Summary
- Server admins can declare a list of npm app packages to auto-install
on every new workspace and backfill onto existing workspaces via CLI.
- Server-level secrets (like Exa's API key) live on the
`ApplicationRegistration` (one row per server, encrypted) and are
injected into logic function execution env at runtime. No more
per-workspace storage of global secrets.
- A generic `POST /app/billing/charge` endpoint lets app logic functions
emit workspace usage events for metered features. Exa uses it in PR 2;
future apps (call recorder, etc.) reuse it.
- `LogicFunctionToolProvider` tool name prefix changes `logic_function_`
→ `app_`. Shorter, accurate (they come from installed apps).
## What's in this PR
**Logic function executor — server-level variables**
- `LogicFunctionExecutorService.getExecutionEnvVariables` now resolves
env vars in the order: hardcoded defaults →
`ApplicationRegistrationVariable[]` (server-level) →
`ApplicationVariable[]` (workspace-level override). The manifest
`serverVariables` schema has existed; this closes the loop.
**Config**
- `PRE_INSTALLED_APPS` — comma-separated list of npm packages. Default:
empty.
**\`PreInstalledAppsService\`** (new module)
- \`onApplicationBootstrap()\` — fetches each package's manifest from
the app registry CDN, upserts an \`ApplicationRegistration\`, and seeds
declared \`serverVariables\` from matching env vars (e.g.
\`EXA_API_KEY\` env → encrypted registration variable).
- \`installOnWorkspace(workspaceId)\` — installs all pre-installed apps
on a single workspace. Tolerates per-app failures.
**Auto-install on new workspace activation**
- \`WorkspaceService.prefillCreatedWorkspaceRecords\` invokes
\`installOnWorkspace\` after prefilling standard records. Non-blocking
on failure.
**Backfill CLI command**
- \`install-pre-installed-apps\` — iterates active and suspended
workspaces, installs pre-installed apps that aren't yet installed.
Idempotent. Run after changing \`PRE_INSTALLED_APPS\`.
**App billing endpoint**
- \`POST /app/billing/charge\`. Authenticated via \`APPLICATION_ACCESS\`
token (already injected into logic function execution env as
\`DEFAULT_APP_ACCESS_TOKEN\`). Body: \`{ creditsUsedMicro, quantity,
unit, operationType, resourceContext? }\`. Emits \`USAGE_RECORDED\` with
\`applicationId\` as \`resourceId\`. Generic — reusable by any app.
**Tool name prefix**
- \`LogicFunctionToolProvider.buildLogicFunctionToolName\` now produces
\`app_<name>\` instead of \`logic_function_<name>\`. Only affects tools
sourced from logic functions; other tool providers unchanged.
## Stats
- 16 files, +501 / −2
- 7 new files (1 command, 1 service × 2, 1 controller, 1 DTO, 2 modules)
- Typecheck: 7 pre-existing errors, zero new
- Prettier clean
## Behavior deltas
- **\`PRE_INSTALLED_APPS\` default = empty**: existing servers see no
change on merge.
- **\`ApplicationRegistrationVariable\` is now read by the executor**:
apps that were using manifest \`serverVariables\` but expecting them to
be ignored by the executor will now see them injected. No apps ship with
\`isTool: true\` logic functions today, so this is latent — first
consumer is Exa in PR 2.
- **Tool prefix**: currently no logic-function tools are named
\`logic_function_*\` in any production flow. The prefix change affects
only future tools emitted by \`LogicFunctionToolProvider\`.
## Risks
- **CDN unavailability at startup**: if the app registry CDN is down,
\`ensureRegistrationsExist\` logs warnings but doesn't block server
start. Installation on new workspaces during this window will find no
registrations and log a non-blocking error. Backfill command can retry
after CDN recovers.
- **Cold-start overhead**: \`ensureRegistrationsExist\` is called once
per process on bootstrap. Current configurable default is empty, so zero
overhead. When an admin sets \`PRE_INSTALLED_APPS\`, they accept one
HTTP call per package at boot.
- **Server-level variables flow**:
\`ApplicationRegistrationVariable.encryptedValue\` is shared by all
workspaces of a server. Appropriate for a single-tenant Exa key. Not
appropriate for per-tenant keys — those go in workspace-level
\`ApplicationVariable\` and override.
## Test plan
- [ ] \`npx nx typecheck twenty-server\` passes (verified: 7
pre-existing unrelated errors, zero new)
- [ ] Set \`PRE_INSTALLED_APPS=@twenty-apps/hello-world\` (or any real
npm-published app), \`HELLO_WORLD_API_KEY=xxx\`, restart server:
\`ApplicationRegistration\` row is upserted,
\`ApplicationRegistrationVariable\` for HELLO_WORLD_API_KEY is populated
(encrypted).
- [ ] Create a new workspace: the app is auto-installed,
\`ApplicationEntity\` row created, \`LogicFunctionEntity\` rows created.
- [ ] Existing workspace: run \`yarn nx run twenty-server:command
install-pre-installed-apps\`: apps install across all workspaces,
idempotent on re-run.
- [ ] Trigger a logic function that reads
\`process.env.HELLO_WORLD_API_KEY\`: value resolves from the
server-level \`ApplicationRegistrationVariable\`.
- [ ] Log a charge from the handler: \`POST /app/billing/charge\` with
\`Authorization: Bearer \$DEFAULT_APP_ACCESS_TOKEN\` body
\`{creditsUsedMicro: 1000, quantity: 1, unit: "INVOCATION",
operationType: "WEB_SEARCH"}\` → returns \`{success: true}\`,
\`USAGE_RECORDED\` event emitted with correct
\`resourceId=applicationId\`.
- [ ] Tool name generated by \`LogicFunctionToolProvider\` starts with
\`app_\`.
## What's NOT in this PR (PR 2 scope)
- The Exa app itself (\`packages/twenty-apps/...\` directory)
- Removing \`WebSearchTool\`, \`WebSearchService\`, \`ExaDriver\`,
\`web-search\` module
- Removing \`WEB_SEARCH_DRIVER\` config var
- Removing the current \`exa_web_search\` entry in
\`ActionToolProvider\`
- Chat preload list updated to \`app_exa_web_search\`
- Frontend \`getToolDisplayMessage\` branch for \`app_exa_web_search\`
- Setting \`PRE_INSTALLED_APPS\` default to include \`@twenty-apps/exa\`
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
Etienne
|
117909e10a | Billing - Adapt to new unit (#19886) | ||
|
Etienne
|
e68842c268 |
Billing - fixes (#19867)
- Uniformize credit formating : In UI, 1$=1credit. In BE 1 UI credit = 1_000_000 BE "crédits" - Add crédit rollover information + Link to documentation + Documentation update <img width="291" height="317" alt="Screenshot 2026-04-17 at 18 22 59" src="https://github.com/user-attachments/assets/2519fb9f-159d-4c85-95f4-a6e005a8a1a3" /> <img width="848" height="763" alt="Screenshot 2026-04-17 at 14 12 20" src="https://github.com/user-attachments/assets/a3cc0874-f275-49ea-819f-305ec314bdfe" /> <img width="797" height="757" alt="Screenshot 2026-04-17 at 14 12 13" src="https://github.com/user-attachments/assets/9048409b-d5a2-435a-b735-70370705e668" /> - Enable direct top-up (or subscription if in trial) from AI chat <img width="333" height="215" alt="Screenshot 2026-04-17 at 22 52 00" src="https://github.com/user-attachments/assets/7a20c627-2806-4bcf-a037-b45752232be9" /> <img width="457" height="769" alt="Screenshot 2026-04-17 at 22 51 41" src="https://github.com/user-attachments/assets/d2a90c1b-271f-4fe9-8891-baeb2fabb86d" /> - Inform users if credit limit is reached - Banner <img width="1130" height="127" alt="Screenshot 2026-04-17 at 19 15 11" src="https://github.com/user-attachments/assets/30723e5e-c07e-462f-8eb8-e08f52bbab1c" /> |
||
|
Charles Bochet
|
70a73534c0 |
perf(server): reuse ESM module cache across warm Lambda invocations of logic functions (#19830)
## Summary Lambda warm-invocations of logic functions were spending **~440 ms** re-parsing and re-evaluating the user bundle on every call. The executor wrote the user code to a **randomly-named** temp file and `import()`-ed it, so each warm call resolved to a new URL and Node's ESM cache could never reuse the previous module record. This PR makes the executor write to a **content-hash filename**, skip the write when the file already exists, and stop deleting it. Identical code now reuses the same module record across warm calls in the same container, dropping warm-invocation overhead by **~30–40%**. ## What changed - `executor/index.mjs`: temp filename derived from `sha256(code)`, write skipped when file exists, no `fs.rm` on cleanup. - `lambda.driver.ts`: single structured `[lambda-timing]` log per invocation with `totalMs / buildExecutorMs / getBuiltCodeMs / payloadBytes / invokeSendMs / reportDurationMs / billedMs / initDurationMs / coldStart`. Goes through the standard NestJS `Logger`. No behavioural change for callers: same input → same output, same error semantics. ### Caveat: module-scope state now persists across warm calls With a stable filename, the user bundle is evaluated **once per warm container**. Any module-scoped state or top-level side-effects in user code are now shared across invocations of the same container, instead of re-running on every call. This is documented in the executor and is the intended trade-off — module scope should be treated as a per-container cache, not as per-call isolation. ## Findings — measured impact Same logic function (`fetch-prs`, ~12k PRs to page through), same workspace, same Lambda config (eu-west-3, 512 MB), token cache primed. ### Warm invocations | Phase | Before fix | After fix | Δ | | -------------------------------------- | ------------- | -------------- | ------------ | | Executor `import(userBundle)` | ~440 ms | **~0 ms** | **-440 ms** | | Lambda billed duration | ~1.5–1.7 s | **~1.0–1.1 s** | **~30–40%** | | Server-perceived round-trip | ~1.7–2.0 s | **~1.0–1.2 s** | **~30–40%** | ### Cold starts Unchanged — the cache helps subsequent warm calls in the same container, not the first one. Init Duration stays ~130–170 ms; total cold call ~2.5–3.0 s. ### Stress Could not reproduce the previously-reported \"every ~10th call times out\" behaviour after the fix: - 30 sequential calls: max 1.7 s, median ~1.1 s, 0 timeouts - 50 concurrent calls: max 9.4 s (clear cold-start cluster), median ~1.5 s, 0 timeouts Hypothesis: the warm-import overhead was eating into the headroom against the function timeout under bursty load; removing it pushed everything well below the limit. ## Observability One structured log line per invocation, sent through the standard NestJS logger: \`\`\` [lambda-timing] fnId=abc123 totalMs=1187 buildExecutorMs=2 getBuiltCodeMs=3 payloadBytes=1466321 invokeSendMs=1180 reportDurationMs=992 billedMs=1000 initDurationMs=n/a coldStart=false \`\`\` \`coldStart=true\` whenever Lambda spun up a fresh container; on warm calls \`buildExecutorMs\` and \`getBuiltCodeMs\` collapse to single-digit ms, confirming the cache fix is working. ## Test plan - [ ] CI green. - [ ] Deploy to a Lambda-backed env, trigger a logic function several times in a row. - [ ] Confirm \`[lambda-timing]\` warm invocations show \`totalMs\` ~30–40% lower than before, and \`coldStart=false\` after the first call in a container. - [ ] Push a new version of an app; confirm the next call shows higher \`buildExecutorMs\` (new hash, new file written) followed by warm calls again. - [ ] Smoke test: errors thrown by the user handler are still surfaced correctly. Made with [Cursor](https://cursor.com) |
