0ac4f237c03e0987d078650999ac08c9fa7f032f
75 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Thomas Trompette
|
0ac4f237c0 |
fix(server): stop redundant lambda rebuilds causing build-lock acquisition failures (#21442)
## Context `Lambda invocation failed for function '<id>' during build: Failed to acquire lock for key: lambda-build:<id>` fires ~1000 times/day in production. ## Root cause `LambdaExecutorManagerService.buildExecutor` re-checks `canSkip` inside the `lambda-build:<functionId>` lock, but the re-check reuses the `flatApplication` snapshot captured when the request started. `canSkip` depends on `!flatApplication.isSdkLayerStale`, so: 1. An app sync/install regenerates the SDK client and sets `isSdkLayerStale = true` 2. All in-flight executions of the function fail `canSkip` and queue on the lock 3. The first holder rebuilds and `markSdkLayerFresh` clears the flag in DB + workspace cache 4. Queued waiters can't see that fix — their in-memory snapshot still says stale — so **each waiter redoes the full rebuild serially** (download SDK archive, delete + republish layer, update function config, wait for update) 5. The lock is held back-to-back for minutes; everyone deeper in the queue exhausts the 120s retry budget and throws The local driver already handles this correctly (`LocalLayerManagerService` refreshes the flat application from the workspace cache inside its lock); the lambda driver missed it. ## Fix - Refresh `flatApplication` from the workspace cache inside the lock before re-checking `canSkip`, so waiters skip in ~100ms once the first holder finishes - Degrade gracefully on lock-acquisition timeout: re-check build status with fresh data and proceed with the invocation if the executor is already usable, instead of failing the run (introduces a typed `CacheLockAcquisitionError` so only that case is caught) ## Test plan - [x] `cache-lock.service.spec.ts` passes - [x] `lint:diff-with-main` + typecheck pass - [ ] Monitor `Failed to acquire lock for key: lambda-build:*` error rate in production after deploy |
||
|
nitin
|
20c83e1f86 |
fix(kanban): preserve scroll on board re-init + propagate same-column reorders via SSE (#20637)
closes https://discord.com/channels/1130383047699738754/1504130730840821860 https://github.com/user-attachments/assets/d5833031-01c6-4e46-b699-c29c42435a53 ## Summary Fixes two related issues with the kanban (board view) collaboration experience: 1. **Scroll-to-top on every data change** — `triggerRecordBoardInitialQuery` always scrolled the board to the top, even when re-initializing for a single-record data change (SSE echo of your own mutation, a collaborator's update). Scroll reset only makes sense when the dataset itself changes (filter / sort / group). 2. **Same-column reorders by other users did not propagate** — the server's diff function stripped `FieldMetadataType.POSITION`, so position-only updates produced empty `updatedFields` and short-circuited event emission entirely. SSE clients never received them. ## What's in here - **Frontend** — `useTriggerRecordBoardInitialQuery` now exposes a `triggerRecordBoardInitialQueryWithoutScrollReset` variant; data-driven re-inits in `RecordBoardDataChangedEffect` use it, while genuine filter / sort / group changes keep the scroll-resetting `triggerRecordBoardInitialQuery`. `getRecordBoardEffectsForUpdateInputs` classifies each update as `trigger-initial-query` / `reposition-records` / `none`. For position- or group-only changes we skip the re-query and reposition records in place in the store (`useRepositionRecordsOnBoard`), which avoids the flicker and preserves scroll. - **Server** — removes `POSITION` from `objectRecordChangedValues`' strip list, so position-only updates emit a non-empty diff and flow through SSE. Position is now treated as a field like any other across all event consumers (SSE, webhooks, workflows, logic functions); a trigger with an explicit field filter still excludes it. |
||
|
Charles Bochet
|
45ceb5d74c |
security: bump vulnerable deps in logic-function runtime lockfiles (#21317)
## What Two standalone dep-set lockfiles bundled for the logic-function / application-package sandbox carried vulnerable transitive versions. Within-range bumps only (these dep sets are handed to user logic functions, so no breaking majors): **`seed-dependencies`** - `nodemailer` `^8.0.4 → ^8.0.5` (resolves 8.0.10) — SMTP CRLF command injection (direct dep) - `brace-expansion`, `picomatch`, `lodash` → patched **`common-layer-dependencies`** - `lodash`, `brace-expansion` → patched ## Left for follow-up (can't be fixed within-range) - `qs` — pinned transitively by `body-parser`, stays at 6.14.2 (needs body-parser bump) - `ip-address` 9.x → 10.x and `uuid` 10.x → 11.x — major bumps; left out since these are exposed to user functions and would be breaking `axios` here is already on the patched `^1.16.1`. |
||
|
martmull
|
36567e96ab |
Remove twenty-sdk from package.json before build (#21305)
Follow up of https://github.com/twentyhq/twenty/pull/21179 Removes twenty-sdk from dependencies when building lambda layer |
||
|
Félix Malfait
|
91f2f08995 |
feat(server): unify workspace-event ingestion behind one EventSink pipeline (#21197)
## Why The five event-log streams (`workspaceEvent`, `pageview`, `objectEvent`, `usageEvent`, `applicationLog`) each wrote to ClickHouse through their own fire-and-forget writer (`AuditService`, `UsageEventWriterService`, and the `application-logs` driver), with the per-type knowledge (table names, normalization, access rules) spread across several modules. Three of them reimplemented the same ClickHouse insert, and the read side, the live stream, and the producers lived in different modules under two different names. This consolidates them into one `core-modules/event-logs/` subsystem (emit, write, live, read), with the per-type config in a single registry so adding an event type is roughly one file. The base Logs settings tab and free application logs shipped separately in #21180 (merged). This PR adds the unified backend, the registry, and the viewer's live mode and entitlement gating. ## Pipeline ```mermaid flowchart TB subgraph PROD["Producers"] A["auth, billing, impersonation,<br/>webhook, custom-domain"] U["usage listener"] F["logic-function executor (app logs)"] R["record CRUD (entity events)"] end EM["EventLogEmitterService<br/>createContext().insert* / dispatch()"] EQ(["entityEventsToDbQueue<br/>(existing, shared with timeline)"]) CIE["CreateEventLogFromInternalEvent"] SINK["WorkspaceEventSinkService.ingest()"] C1["ClickHouseEventSink"] C2["ConsoleEventSink"] LIVE["EventLogLiveService.publishWatched()<br/>(presence-gated)"] CH[("ClickHouse, 5 tables, async_insert")] CHAN(["WORKSPACE_EVENTS_CHANNEL"]) RS["EventLogsService (registry-driven read)"] LR["EventLogsLiveResolver"] UI["Settings > Logs"] A --> EM U --> EM F --> EM EM -->|direct| SINK R --> EQ --> CIE -->|ingest| SINK SINK --> C1 --> CH SINK --> C2 SINK --> LIVE -.->|if a viewer is watching| CHAN --> LR --> UI CH --> RS --> UI ``` ## What it does - Producers call `EventLogEmitterService.createContext().insert*()`, which builds a typed `WorkspaceEventEnvelope` and writes it through `WorkspaceEventSinkService` to the configured sinks (ClickHouse, Console) plus a presence-gated live fan-out. Record/CRUD events reach the same sink through the existing `entityEventsToDbQueue`. There is no dedicated queue; ClickHouse `async_insert` batches server-side. Writes are best-effort, as on main today. - `EVENT_LOG_TYPES[table]` is the per-type source of truth: the ClickHouse table, the required entitlement, the free-text filter column, and the row-to-GraphQL mapping. Read row shapes derive from the write rows. - Four modules along their dependency boundaries: `EventLogEmitterModule` (producer API), `EventLogIngestionModule` (sink layer), `EventLogLiveModule` (fan-out), and `EventLogsViewerModule` (the entitlement-gated GraphQL read, which is where billing/enterprise/permissions stay so producers stay light). - Logs viewer: per-table columns, filters (text, date, record), live mode, and an upgrade card that points to Billing on Cloud or the Admin Panel on self-hosted. Application logs are free on every plan; the other four require the `AUDIT_LOGS` entitlement (with a `NO_ENTITLEMENT` fallback to the upgrade card). - Renames `AuditService` to `EventLogEmitterService`, and the generic `Monitoring` event to a typed `Impersonation` event (`level` + `action`). - Removes `UsageEventWriterService`, the `application-logs` driver/module, and `AuditService`'s direct inserts. ## Durability Writes are best-effort, the same as main today (the old writers were fire-and-forget). A dedicated queue was tried mid-PR and removed: `async_insert` already batches server-side, so the queue only added durability, which isn't a requirement right now. The `EventSink` seam keeps a durable transport (e.g. a Redis-Streams buffer) easy to add later without touching producers. ## Out of scope S3 peer sink (seam only), Postgres or any second read path, `ReplicatedMergeTree`, ClickHouse table-schema changes, and the record-data `EVENT_STREAM_CHANNEL` (unchanged, separate concern). ## Testing Unit tests cover the registry definitions and row normalization, the entitlement gating, the envelope builders, and the producers. Integration tests cover the write paths (record create produces an `objectEvent`; the track mutation produces a `workspaceEvent`) and the read/query path across all five tables. Verified with typecheck, lint, a server boot, and GraphQL/SDK codegen. |
||
|
Félix Malfait
|
c3dd6b25a6 |
fix: use canonical oxlint rule id in lint-disable directives (#21253)
## What Many `oxlint-disable` / `eslint-disable` directives across the repo carry a corrupted rule id — `@typescripttypescript/<rule>` — most likely a find-and-replace accident that mangled the eslint-era `@typescript-eslint/` prefix. oxlint matches disable directives **loosely by rule name**, so these still suppress in practice (not a silent no-op), but the id is malformed and misleading. ## Change Replace them with the **canonical oxlint id** `typescript/<rule>` — matching the plugin name and rule keys declared in `.oxlintrc.json` — **127 files, 262 directives**: | rule | count | | --- | ----- | | `typescript/no-explicit-any` | 250 | | `typescript/ban-ts-comment` | 6 | | `typescript/no-misused-promises` | 4 | | `typescript/no-empty-object-type` | 2 | - `twenty-server`: 122 files - `twenty-front`: 5 files Comment-only — no code or runtime changes. ## Verification `oxlint --type-aware -c .oxlintrc.json` reports **0 warnings / 0 errors** for both `twenty-server` and `twenty-front`. Every changed line is exactly the id correction inside a disable directive (262 insertions / 262 deletions, no collateral edits). > Addresses the cubic review, which flagged that the canonical oxlint id is `typescript/...` (no `@`). Worth noting the original `@typescripttypescript/` was not actually a silent no-op — oxlint matches these directives loosely by rule name — but `typescript/` is the correct, config-aligned id. |
||
|
martmull
|
e0d42323af |
Add more control on http trigger (#21216)
add "new Response" utils to define response code or content type of http route triggered logic function responses follow up of https://github.com/twentyhq/twenty/pull/21214 --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Paul Rastoin
|
3d49642d12 |
[AUDIT] Run knip over twenty-server (#21159)
# Introduction Run [knip](https://knip.dev/) over twenty-server Used config: ```json { "$schema": "https://unpkg.com/knip@5/schema.json", "workspaces": { "packages/twenty-server": { "entry": [ "src/main.ts", "src/command/command.ts", "src/queue-worker/queue-worker.ts", "src/database/scripts/setup-db.ts", "src/database/scripts/truncate-db.ts", "src/database/clickHouse/migrations/run-migrations.ts", "src/database/clickHouse/seeds/run-seeds.ts", "src/instrument.ts", "lingui.config.ts", "test/integration/graphql/codegen/index.ts", "test/integration/utils/setup-test.ts", "test/integration/utils/teardown-test.ts", "scripts/**/*.ts", "**/*.spec.ts", "**/*.integration-spec.ts" ], "project": ["src/**/*.ts", "test/**/*.ts", "scripts/**/*.ts"], "ignore": [ "src/database/typeorm/**/migrations/**", "src/database/typeorm/**/*.entity.ts", "**/*.workspace-entity.ts", "**/logic-function-resource/constants/seed-project/**" ], "ignoreDependencies": ["@types/psl", "@types/aws-lambda"], "ignoreBinaries": ["nest", "lingui", "typeorm"] } } } ``` |
||
|
Charles Bochet
|
f6a17bc5f7 |
refactor(server): split LambdaDriver and LocalDriver into focused sub-services (#21153)
## Why `LambdaDriver` (1.4k lines) and `LocalDriver` (709 lines) each mixed many unrelated concerns in a single class. ## What changed **This PR only moves code** — every line is byte-for-byte the same as `main` (same logic, same comments, same constants, same lock keys/TTLs, same error mapping). The drivers are now thin orchestrators wiring co-located sub-services + pure utils (each with unit tests). | File | Before | After | |---|---|---| | `lambda.driver.ts` | 1425 | 252 | | `local.driver.ts` | 709 | 231 | | Largest sub-service | — | 371 (`lambda-executor-manager`) | | Unit tests on these paths | 0 | 15 | Lambda split: `lambda-aws-client` / `lambda-tool-functions` / `lambda-layer-manager` / `lambda-executor-manager` + `constants` + `types` + 5 pure utils. Local split: `local-layer-manager` / `local-child-process-runner` / `local-prebuilt-bundle` + `constants` + `types` + 3 pure utils. No behavior change, no signature change, no public-API change. |
||
|
Charles Bochet
|
58907b733c |
feat(logic-function): add LIVE / PREBUILT execution modes (#20873)
## Summary
### Why
1. Sending the code to the lambda (~1Mb usually) is heavy on network and
results to a constant traffic of ~30Mb/s on AWS which results into TB of
network data every month
2. eval(1MB of code) is not that fast, it's heavy on memory and CPU on
lambda side
### High level
Adds two execution modes for logic functions, gated behind the new
`IS_LOGIC_FUNCTION_PREBUILT_MODE_ENABLED` workspace feature flag (off
everywhere by default):
- **LIVE** (current behavior, preserved bit-for-bit): the compiled
bundle is read from object storage and shipped in every Lambda invoke
payload. Used for fast iteration in the workflow editor / Settings test
runs.
- **PREBUILT** (new): the bundle is installed onto the per-function
Lambda alongside the unified executor, and invocations carry only `{
params, env, handlerName }` — saving JSON payload egress and warm-start
`import()` cost on every call.
### Key design choices
- **Unified Lambda handler** (`constants/executor/index.mjs`) dispatches
at runtime: `event.code` present ? LIVE (write to `/tmp`, dynamic
import) : `import('./prebuilt-logic-function.mjs')`. Both code paths
always coexist on the deployment package, so the same Lambda can serve
either mode without redeploying.
- **Install runs inside the `validateBuildAndRun` migration pipeline**,
not at execute time. `Create/UpdateLogicFunctionActionHandlerService`
calls `driver.installPrebuiltBundle` when `executionMode` flips
LIVE?PREBUILT or `checksum` changes while PREBUILT, gated on
`isBuildUpToDate=true` and a fresh checksum.
- **Strict execute, no reconciliation**:
`LogicFunctionExecutorService.execute` resolves `effectiveExecutionMode`
(caller override > feature flag > entity column). For PREBUILT it asks
the driver `getInstalledBundleChecksum` (Lambda `twenty:bundle-checksum`
tag for AWS, sidecar file locally) and throws
`LOGIC_FUNCTION_PREBUILT_BUNDLE_NOT_INSTALLED` on mismatch.
- **Feature flag gates every side effect**: with the flag off the
executor forces LIVE, the action-handler install hooks bail before AWS,
and workflow activation does not flip the mode. Rollback is just turning
the flag off.
### Lifecycle
- New workflow CODE step ? `LIVE`, no install.
- Workflow activated ? build + activation flips `executionMode=PREBUILT`
? action-handler installs the bundle + sets the Lambda tag.
- Draft from active version ? duplicated logic function reset to `LIVE`.
- App install ? manifest converter sets `PREBUILT`, create-action
handler installs.
- Test runs (`executeOneFromSource`, workflow editor) pass
`executionMode=LIVE` explicitly.
### Observability
`[lambda-timing]` log lines now include `effectiveExecutionMode` and
`payloadBytes`; the action handler logs `install_duration_ms` for each
install.
## Test plan
- [x] `npx nx typecheck twenty-server` ? passes
- [x] `npx oxlint --type-aware` on all changed files ? 0 warnings, 0
errors
- [x] `npx nx test twenty-server` ? 588 suites / 5009 tests pass (no
regressions vs main)
- [x] New unit suite `flat-logic-function-validator.service.spec.ts` ?
9/9
- [x] Existing
`workflow-version-step-operations.workspace-service.spec.ts` ? 8/8
(verified the new token-based DI avoids a circular-import regression)
- [x] Snapshot for
`ALL_UNIVERSAL_FLAT_ENTITY_PROPERTIES_TO_COMPARE_AND_STRINGIFY` updated
to include `executionMode`
- [x] Integration suite `logic-function-execution.integration-spec.ts`
extended to assert `executionMode=LIVE` on newly-created functions and
continues to exercise the LIVE happy path
- [ ] Manual staging rollout: flip
`IS_LOGIC_FUNCTION_PREBUILT_MODE_ENABLED` per workspace, observe
`[lambda-timing]` `payloadBytes` drop + `install_duration_ms`, then ramp
in prod.
|
||
|
Paul Rastoin
|
ebfaca5b3d |
EncryptedString PlaintextString branded string types (#21001)
## Summary closes https://github.com/twentyhq/core-team-issues/issues/2464 Introduces compile-time branded types to distinguish encrypted ciphertext from plaintext strings, preventing mix-ups like the one fixed in #20819 — but at the type level rather in addition to the one existing at runtime. ### Branded string primitives - Created `EncryptedString` and `PlaintextString` as hard nominal brands using `z.string().brand(...)`, making them non-assignable to each other or to raw `string` - Created `isEncryptedString` type predicate to narrow `string` to `EncryptedString` based on the `enc:v2:` envelope prefix - Retyped `SecretEncryptionService`: `encryptVersioned` accepts `PlaintextString`, `decryptVersioned` returns `PlaintextString` ### Entity typing - Typed encrypted columns across entities: `SigningKeyEntity.privateKey`, `TwoFactorAuthenticationMethodEntity.secret`, `ApplicationRegistrationVariableEntity.encryptedValue`, `ApplicationVariableEntity.value` - Parameterized JSONB types for connected account connection parameters (`ImapSmtpCaldavParams<Pwd>`) with reusable aliases `EncryptedImapSmtpCaldavParams` / `DecryptedImapSmtpCaldavParams` - Typed DTOs (`CreateApplicationRegistrationVariableInput`, `UpdateApplicationRegistrationVariablePayload`, `UpdateApplicationVariableEntityInput`) with `PlaintextString` ### ApplicationVariable always-encrypt uniformization - Retyped `ApplicationVariableEntity.value` to `EncryptedString | ''` — all values are now encrypted regardless of `isSecret` - Updated `ApplicationVariableEntityService` to always encrypt on write and always decrypt on read - Simplified `UpdateApplicationVariableActionHandlerService` by removing conditional encrypt/decrypt-on-isSecret-toggle logic - Added slow instance command (`2.9.0`) to backfill-encrypt existing `isSecret=false` plaintext rows and tighten the `CHECK` constraint ### ConfigStorageService refactor - Split `convertAndSecureValue` (which used `any`) into two well-typed methods: `convertAndDecrypt` and `convertAndEncrypt` - Introduced `isSensitiveStringValue` type predicate to narrow values before encryption/decryption ### What's next - Typeorm entity derivation to strictly type sitemap configuration as code + handler logic for encryption rotation - https://github.com/twentyhq/core-team-issues/issues/2465 |
||
|
Thomas Trompette
|
6e2cf4cbc0 |
Fix lambda timeout diagnostics (#21002)
Trying to fix https://twenty-v7.sentry.io/issues/7420384466/?environment=prod&environment=prod-eu&project=4507072499810304&query=is%3Aunresolved%20%21issue.type%3A%5Bperformance_consecutive_db_queries%2Cperformance_consecutive_http%2Cperformance_file_io_main_thread%2Cperformance_db_main_thread%2Cperformance_n_plus_one_db_queries%2Cperformance_n_plus_one_api_calls%2Cperformance_p95_endpoint_regression%2Cperformance_slow_db_query%2Cperformance_render_blocking_asset_span%2Cperformance_uncompressed_assets%2Cperformance_http_overhead%2Cperformance_large_http_payload%5D%20timesSeen%3A%3E10&referrer=issue-stream&sort=date ## Summary - Prevent invoking Lambda functions stuck in `Pending` state by checking `Configuration.State === 'Active'` in `checkLambdaExecutorBuildStatus` — a Pending function now goes through `ensureLambdaExecutor` which waits for Active. - Track execution phase (`build`/`fetch-code`/`invoke`) via a `LambdaExecutionPhase` enum and include phase timing + function state in all error messages for faster debugging. Co-authored-by: Matt Van Horn <mvanhorn@users.noreply.github.com> |
||
|
Thomas Trompette
|
ff28547a36 |
tt-fix-logic-function-error-handling (#20928)
## Summary - **Fix `callWithTimeout` timer leak**: the `setTimeout` was never cleared when the callback resolved first, leaving orphaned timers (up to 15 minutes) in the Node.js event loop. Now uses `try/finally` with `clearTimeout`. - **Properly classify timeout errors**: introduced `ExecutionTimedOutError` so the Lambda driver can throw `LOGIC_FUNCTION_EXECUTION_TIMEOUT` instead of `LOGIC_FUNCTION_EXECUTION_FAILED` — users now see "Function execution timed out" instead of the generic "Function execution failed." ## Test plan - [x] Execute a logic function normally and verify it still works - [x] Trigger a logic function timeout and verify the error message says "Function execution timed out" (not "Function execution failed") - [x] Verify that a deleted logic function invocation logs a specific "was deleted" warning - [x] Verify no orphaned timers remain after logic function execution completes |
||
|
Paul Rastoin
|
b8de469f37 |
Refactor and centralize file mimeType integrity check and sanitization (#20889)
# Introduction closes https://github.com/twentyhq/private-issues/issues/484 This PR refactors the writeFile API to never expect to be passed a mimetype, its extract is done programmatically low level so any callers will pass through Same for the file sanitization ## IANA override Disclaimer for consistency we existing behavior we wanted to always have `application/typescript` - should we rather consider fallbacking to octect-steam instead ? - Any pulbic assets that has .ts will now also fallback to `application/typescript` instead of the official IANA ## Integration Added coverage |
||
|
Paul Rastoin
|
b8b115f4e3 |
FileStorageService Dedicated file and folder code flow + integrity check (#20831)
# Introduction
Next handling mimetype integrity check and checksum integrity check for
s3 storage type
Always expecting a trailing end slash when deleting a folder etc
## Application
Uninstalling an application now deletes all its related files
## File storage service
Making a distincton between folder path and file path
## Validation Pipeline
Every file operation in `FileStorageService.buildOnStoragePath` runs
through `validateResourcePath`, which chains three validators in order:
**1. `validateSafeRelativePath`** -- rejects path traversal attacks
| Input | Result | Error |
|---|---|---|
| `../../../etc/passwd` | Rejected | `Resource path must not contain
path traversal (..)` |
| `/etc/passwd` | Rejected | `Resource path must be relative, not
absolute` |
| `file\0.txt` | Rejected | `Resource path contains null bytes` |
| `..\\..\\etc\\passwd` | Rejected | `Resource path must not contain
backslashes` |
| _(empty)_ | Rejected | `Resource path must not be empty` |
**2. `validateFilenameIntegrity`** -- enforces safe characters, length
limits, extension required
| Input | Result | Error |
|---|---|---|
| `my folder/file.mjs` | Rejected | `A path segment contains invalid
characters...` |
| `Makefile` | Rejected | `Filename must have an extension` |
| `aaa...(256 chars).mjs` | Rejected | `A path segment exceeds the
maximum length of 255 characters` |
| `a/b/.../file.mjs` (1025+ chars) | Rejected | `Resource path exceeds
maximum length of 1024 characters` |
| `src/handlers/index.mjs` | Accepted | -- |
| `my-app/my_file.tsx` | Accepted | -- |
| `v1.0/module.config.mjs` | Accepted | -- |
Allowed characters per segment: `a-z`, `A-Z`, `0-9`, `.`, `-`, `_`
**3. `validateResourceExtension`** -- checks extension against the
`FileFolder` allowlist
| Input | FileFolder | Result | Error |
|---|---|---|---|
| `handler.js` | `BuiltLogicFunction` | Rejected | `Invalid file
extension. Allowed extensions: .mjs` |
| `card.tsx` | `BuiltFrontComponent` | Rejected | `Invalid file
extension. Allowed extensions: .mjs` |
| `script.js` | `PublicAsset` | Rejected | `Invalid file extension.
Allowed extensions: .png, .jpg, ...` |
| `index.mjs` | `BuiltLogicFunction` | Accepted | -- |
| `app.tsx` | `Source` | Accepted | -- |
| `photo.png` | `CorePicture` | Accepted | -- (unconfigured folder,
passes through) |
## Consumers
- **`FileStorageService`** -- calls `validateResourcePath`, throws
`FileStorageException` on failure (last-resort defense)
- **Resolver (`uploadApplicationFile`)** -- calls
`validateResourcePath`, throws `ApplicationException` on failure
(user-facing)
- **Flat validators** -- call `validateResourcePath`, push the error to
`validationResult.errors` (non-throwing, collects all errors)
All error messages are translated via Lingui `t` and returned in a
discriminated union `{ isValid: true } | { isValid: false, error: string
}`, letting each consumer decide how to handle failures.
|
||
|
Charles Bochet
|
7fa136f305 |
feat(twenty-server): migrate remaining at-rest encryption sites to versioned envelope (#20550)
## Summary Second PR in the encryption key rotation series. The previous PR (#20528) introduced `ENCRYPTION_KEY` + the versioned `enc:v2:<keyId>:<base64>` envelope inside `SecretEncryptionService` and migrated `ConnectedAccountTokenEncryptionService` as the first consumer. This PR routes every remaining at-rest encryption site through the versioned envelope so that `ENCRYPTION_KEY` (and the future `FALLBACK_ENCRYPTION_KEY`) actually covers them. The legacy unprefixed CTR ciphertext remains readable as a fallback during the rollout window — every migrated read site uses `decryptVersioned`, which transparently delegates to the legacy CTR decrypt when it sees an unprefixed payload. ### Service migrations - **`ApplicationVariableEntityService` (#8)** — workspace-scoped. HKDF info is bound to each row's `workspaceId`. A new `decryptAndMaskVersioned` helper lands on `SecretEncryptionService` for the resolver display path. - **`ApplicationRegistrationVariableService` (#7)** + consumers — **instance-scoped**. Registration variables are server-level config readable by every workspace that installs the application, so HKDF info is `instance`. Updated consumers: - `LogicFunctionExecutorService.buildServerVariableEnvMap` - `ConnectionProviderService.getClientCredentials` - **`LogicFunctionExecutorService.buildEnvVar` (#9)** — workspace-scoped. Each variable's `workspaceId` is threaded into `decryptVersioned`, so per-workspace HKDF contexts are honoured at execution time. - **`UpdateApplicationVariableActionHandlerService`** (workspace-migration runner) — threads `workspaceId` through the secret/non-secret toggle. - **`JwtKeyManagerService` (#3)** — instance-scoped. Signing keys are shared across the JWKS. - **`ConfigStorageService` (#6)** — instance-scoped sensitive STRING config variables. ### Slow instance commands (2.5.0) Each migrated site has a paired backfill that re-encrypts existing rows into the v2 envelope before the column is constrained: | timestamp | command | scope | CHECK constraint | |---|---|---|---| | `1798000005000` | encrypt-application-variable | workspaceId | `"isSecret" = false OR value = '' OR value LIKE 'enc:v2:%'` | | `1798000006000` | encrypt-application-registration-variable | instance | `"encryptedValue" = '' OR value LIKE 'enc:v2:%'` | | `1798000007000` | encrypt-signing-key-private-keys | instance | `"privateKey" IS NULL OR value LIKE 'enc:v2:%'` | | `1798000008000` | encrypt-sensitive-config-storage | instance | _none_ — heterogeneous jsonb column | All backfills are idempotent (the SELECT filter skips rows already in v2 form) and run before their respective `up()` adds the CHECK constraint. Every `down()` deliberately stops at dropping the CHECK constraint — they intentionally do not re-introduce plaintext on rollback. ### Tests - Unit specs for each new slow command cover the v2 upgrade path, the idempotency invariant, and the instance vs workspace HKDF scope. - New `JwtKeyManagerService` spec asserts `decryptVersioned`/`encryptVersioned` are called without `workspaceId` (instance scope). - Updated existing specs for `ApplicationVariableEntityService`, `ConfigStorageService`, and `buildEnvVar` to assert the versioned API and the workspace HKDF context plumbing. - New `SecretEncryptionService.decryptAndMaskVersioned` cases in the service spec. - Updated the `applicationRegistrationVariable` integration spec to assert the column now stores `enc:v2:<keyId>:<base64>` instead of raw legacy CTR. ### Out of scope (future PRs) - `PostgresCredentialsService` — bespoke `jwtWrapperService.generateAppSecret`–derived key + `encryptText`/`decryptText` from `auth.util.ts`; deserves its own migration. - `SimpleSecretEncryptionUtil` (TOTP) — entirely different `aes-256-cbc` `iv:enc` format; deserves its own migration. ## Test plan - [x] `npx nx typecheck twenty-server` - [x] `npx nx lint:diff-with-main twenty-server` (oxlint + prettier) - [x] Local jest run for `secret-encryption | connected-account-token | application-variable | application-registration-variable | build-env-var | jwt-key-manager | config-storage | encrypt-application-variable | encrypt-application-registration-variable | encrypt-signing-key | encrypt-sensitive-config-storage` — 17 suites, 106 tests pass. - [x] Local jest run for `upgrade | instance-command` — 12 suites, 86 tests pass. - [ ] CI green - [ ] Manual review of CHECK constraint shapes by a server reviewer (each one matches `enc:v2:%` rather than `enc:v_:%` since none of the migrated columns can legitimately hold `enc:v1:` ciphertext). |
||
|
Etienne
|
aecfe699f4 |
feat(ai-chat) - Stop ai thinking if credits exhausted (#20526)
Billing is now decremented per-step, not per-turn. The onStepFinish callback in chat-execution.service.ts calls a new decrementAndCheckAvailableCredits method on each model step, so Redis is debited incrementally as the agent runs rather than all at once at the end. Credit exhaustion stops the stream mid-run. When a step depletes the remaining credits, a hasNoMoreAvailableCredits flag is set and passed into the stopWhen predicate of streamText, causing the agent to halt before starting the next step. A new credits-exhausted event is introduced. After the stream drains and the response is persisted, if credits ran out the job publishes a dedicated credits-exhausted event to the frontend instead of the normal message-persisted event. The frontend handles this new event. useAgentChatSubscription has a new credits-exhausted case that sets a BILLING_CREDITS_EXHAUSTED-coded error on the atom, closes the writer, and stops the streaming state — triggering the existing AiChatCreditsExhaustedMessage UI. |
||
|
Paul Rastoin
|
a159a68e2c |
[twenty-server] no floating promises lint rule (#20499)
## Introduction
That's an audit + RFC
## Fire-and-forget (`void`) -- Intentional, correct
These are telemetry, metrics, and audit logging in hot paths or
non-critical contexts. `void` is the right choice.
| File | What was voided |
|---|---|
| `sign-in-up.service.ts` | `metricsService.incrementCounter` (sign-up
metric) + `auditService.insertWorkspaceEvent` (workspace created) |
| `use-graphql-error-handler.hook.ts` | 5x
`metricsService.incrementCounter` (GraphQL operation metrics) |
| `bullmq.driver.ts` | 2x `metricsService.incrementCounter` (job
completed/failed metrics) |
| `call-webhook.job.ts` | 2x `auditService.insertWorkspaceEvent` + 1x
`metricsService.incrementCounter` |
| `custom-domain-manager.service.ts` | `analytics.insertWorkspaceEvent`
(domain activation event) |
| `logic-function-executor.service.ts` |
`auditService.insertWorkspaceEvent` (function execution) |
| `workflow-runner.workspace-service.ts` |
`metricsService.incrementCounter` (throttle metric) |
| `cleaner.workspace-service.ts` | `metricsService.incrementCounter`
(deleted workspace metric) |
| `stream-agent-chat.job.ts` | Detached IIFE for streaming chunks
(intentional concurrent pipeline) |
| `workspace-auth-context.middleware.ts` |
`withWorkspaceAuthContext(...)` (AsyncLocalStorage, returns void anyway)
|
## Top-level script entry points (`void bootstrap()`)
These are module-level calls where the promise has no consumer. `void`
makes the lint rule happy and documents the intent.
| File | What changed |
|---|---|
| `main.ts` | `void bootstrap()` |
| `command.ts` | `void bootstrap()` |
| `queue-worker.ts` | `void bootstrap()` |
| `truncate-db.ts` | `void dropSchemasSequentially()` |
| `codegen/index.ts` | `void generateTests(forceArg)` |
## Now properly awaited -- Real bug fixes
These were floating promises that could silently fail, lose data, or
cause race conditions.
| File | What was fixed |
|---|---|
| `billing-sync-plans-data.command.ts` | `meters.map(async ...)` wrapped
in `Promise.all` -- was returning before upserts finished |
| `cache-storage.service.ts` | `setAdd` and `setPop` had `.then()`
chains that weren't returned/awaited |
| `create-audit-log-from-internal-event.ts` | 4x
`auditService.createObjectEvent` now awaited inside a job |
| `cleaner.workspace-service.ts` | 2x `emailService.send(...)` now
awaited -- emails could silently fail |
| `agent-async-executor.service.ts` | `calculateAndBillUsage` +
`billNativeWebSearchUsage` in `finally` block now awaited |
| `repair-tool-call.util.ts` | `calculateAndBillUsage` now awaited |
| `agent-title-generation.service.ts` | `calculateAndBillUsage` now
awaited |
| `chat-execution.service.ts` | `billNativeWebSearchUsage` now awaited |
| `ai-generate-text.controller.ts` | `calculateAndBillUsage` in
`finally` block now awaited |
| `agent-turn.resolver.ts` | `messageQueueService.add(...)` now awaited
|
| `command.ts` | `app.close()` now awaited (was exiting before graceful
shutdown) |
| `i18n.service.ts` | `loadTranslations()` in `onModuleInit` now awaited
|
| `workspace-query-hook.explorer.ts` | `explore()` in `onModuleInit` now
awaited |
| `message-queue.explorer.ts` | `handleProcessorGroupCollection` in
`onModuleInit` now awaited |
| `ai-billing.service.spec.ts` | Test now properly `await`s the async
call |
| `messaging-messages-import.service.spec.ts` | `expect(...)` now
properly `await`ed for async assertion |
| `archive.finalize()` (3 files) | Voided -- promise resolution already
handled by `pipeline()` / `on('end')` |
## Impersonation & security audit trail -- Upgraded from `void` to
`await`
These were previously fire-and-forget but are
security/compliance-critical events that must be reliably persisted.
| File | What was fixed |
|---|---|
| `impersonation.service.ts` | 4x `auditService.insertWorkspaceEvent`
now awaited (impersonation attempt, token generation
attempt/success/failure) |
| `auth.resolver.ts` | 5x `auditService.insertWorkspaceEvent` now
awaited (impersonation token exchange attempt/success/failure at server
and workspace levels) |
| `auth.service.ts` | 2x `analytics.insertWorkspaceEvent` now awaited
(impersonation attempted/issued) |
## Billing audit -- Upgraded from `void` to `await`
Payment events should be reliably persisted for financial/compliance
reporting.
| File | What was fixed |
|---|---|
| `billing-webhook-invoice.service.ts` |
`auditService.insertWorkspaceEvent(PAYMENT_RECEIVED_EVENT)` now awaited
inside Stripe webhook handler |
## Fire-and-forget with proper error handling -- Upgraded from bare
`void`
These remain non-blocking but now catch and log errors instead of
risking unhandled rejections.
| File | What was fixed |
|---|---|
| `logic-function-executor.service.ts` |
`applicationLogsService.writeLogs` now uses `.catch()` instead of bare
`void` -- user-facing logs should surface errors |
## Systemic infrastructure fixes
| File | What was fixed |
|---|---|
| `metrics.service.ts` | `incrementCounter`: Redis cache write
(`metricsCacheService.updateCounter`) now uses `.catch()` internally
instead of raw `await` -- prevents unhandled rejections across all `void
metricsService.incrementCounter(...)` call sites when Redis is unhealthy
|
| `audit.service.ts` | `preventIfDisabled`: made properly `async` with
`await` and consistent `Promise<{ success: boolean }>` return type.
Removed broken `catch` that returned an `AuditException` as a value
(wrong constructor args, unreachable dead code). Removed unused
`AuditException` import |
## Fixed in this session (beyond original PR)
| File | What changed |
|---|---|
| `telemetry.listener.ts` | Removed misleading `Promise.all` + `void`
combo; replaced with simple `for...of` + `void` |
| `message-queue.explorer.ts` | Changed from `void` to `await` so
startup crashes on registration failure |
|
||
|
martmull
|
7ade9e3aab |
Fix application variable issue (#20500)
Fixes wrong formatting converting secret value to ******** before saving Issued by https://discord.com/channels/1130383047699738754/1423290797079662602/1502674904770936903 |
||
|
Félix Malfait
|
34b927ff23 |
feat(public-domain): bind public domains to apps + reorganize settings (#20360)
## Summary - **Public domains can now be bound to a specific app.** When a request hits an app-bound public domain, route resolution restricts logic-function matching to that app's HTTP-routed functions only — isolating each app's routes to its own domain instead of letting routes from other apps in the workspace match nondeterministically. - **Settings sidebar reorganized.** Removed the standalone Domains page. Workspace Domain → General. Approved Domains + Invitations → Members "Access" tab. Emailing Domains + Public Domains → Apps "Developer" tab. Roles → Members "Roles" tab. ## Why The use case: someone building a partner portal app or a lead-collection app declares private objects (leads, partners…) plus a few public HTTP routes. Each app needs its own domain (`partners.acme.com`, `leads.acme.com`) without those domains exposing every other app's routes in the same workspace. Today's PublicDomainEntity is workspace-scoped only, so all HTTP-routed logic functions in a workspace compete for any public domain — first match wins nondeterministically. ## Backend - Added nullable `applicationId` FK to `PublicDomainEntity` (cascade-deleted with the app); indexed for the route-trigger lookup. - New fast instance command `2-4-instance-command-fast-1798000003000-add-application-id-to-public-domain` adds the column, index, and FK constraint. - `createPublicDomain(domain, applicationId)` accepts an optional app binding; new `updatePublicDomain(domain, applicationId)` mutation rebinds/unbinds an existing domain. Both validate the application belongs to the workspace. - `WorkspaceDomainsService.resolveWorkspaceAndPublicDomain(origin)` returns both the workspace and the matched public domain in one query — replacing the old back-to-back lookups in the route-trigger hot path. `getWorkspaceByOriginOrDefaultWorkspace` is preserved as a thin wrapper. - `RouteTriggerService` filters `logicFunction` by `applicationId` when the matched public domain is app-scoped; falls back to workspace-wide when unbound. - Three sequential validation queries in `createPublicDomain` now run in parallel via `Promise.all`. ## Frontend | Old location | New location | |---|---| | Settings sidebar → Domains (standalone page) | Removed | | Domains page → Workspace Domain | General page | | Domains page → Approved Domains | Members → Access tab | | Domains page → Emailing Domains | Apps → Developer tab | | Domains page → Public Domains | Apps → Developer tab | | Settings sidebar → Roles (standalone) | Members → Roles tab | | `pages/settings/roles/` | `pages/settings/members/roles/` | - The Public Domain detail page has an Application picker that uses `Select`'s native `emptyOption` + `null` value pattern (matches `SettingsDataModelObjectIdentifiersForm`). - Members page tabs use the existing `TabListFromUrlOptionalEffect` mechanism (rendered automatically by `TabList`) for hash-based tab activation. - `/settings/members/roles` redirects to `/settings/members#roles` so role sub-pages' `navigate(SettingsPath.Roles)` lands on the Members page with the Roles tab pre-selected. - All affected breadcrumbs updated to nest under their new parents. - `SettingsPath.Roles` and friends now nest under `members/`; `Subdomain` and `CustomDomain` under `general/`; `PublicDomain` and `EmailingDomain` under `applications/`. ## Test plan - [x] `nx typecheck twenty-front` passes - [x] `nx typecheck twenty-server` passes - [x] `oxlint --type-aware` clean on all touched files - [x] `prettier --check` clean on all touched files - [x] Migration applied locally; `publicDomain.applicationId` (uuid, nullable) confirmed in DB - [x] GraphQL schema exposes `PublicDomain.applicationId`, `createPublicDomain.applicationId`, `updatePublicDomain` mutation - [x] **End-to-end route resolution scenarios verified locally:** - Domain bound to App A, function in App A → route matches ✅ - Domain bound to App B, function in App A → route does NOT match (HTTP 404 `TRIGGER_NOT_FOUND`) ✅ - Domain unbound (`applicationId = NULL`) → route matches workspace-wide ✅ - Unknown path on bound domain → returns 404 cleanly ✅ - [x] UI sanity (browser-tested at `apple.localhost:3001`): - General page shows Workspace Domain card - Members page shows Team / Access / Roles tabs - Access tab combines Invite by link + by email + Approved Domains - Roles tab embeds the role list - `/settings/members/roles` direct URL → redirects + Roles tab pre-selected - Apps Developer tab shows Emailing Domains + Public Domains sections - Public Domain detail page has Application picker dropdown listing workspace apps - Sidebar nav: "Domains" and "Roles" no longer present (now folded into General/Members) ## Notes for reviewers - Creating a public domain via the UI still requires Cloudflare credentials in the dev `.env` (`CLOUDFLARE_API_KEY`, `CLOUDFLARE_PUBLIC_DOMAIN_ZONE_ID`, `PUBLIC_DOMAIN_URL`). The DNS step is unchanged from main. - The `applicationId` column is nullable, so existing public-domain rows continue to work workspace-wide — no data backfill required. - `SettingsRolesContainer` was deleted (no longer referenced after `SettingsRoles` index page was removed). 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
martmull
|
0d05788547 |
Protect sendEmail endpoint and thread user context through logic function executor (#20369)
- Thread userId and userWorkspaceId through LogicFunctionExecutorService.execute() so application access tokens carry user context when available. This allows logic functions triggered by authenticated HTTP routes to call sendEmail with proper user identity, making the existing verifyOwnership() check work naturally. - Gate the sendEmail resolver with SettingsPermissionGuard(PermissionFlagType.SEND_EMAIL_TOOL) instead of NoPermissionGuard, ensuring only callers with the SEND_EMAIL_TOOL permission can send emails. |
||
|
Etienne
|
9fc5be1c4c |
Billing - Migrate from Stripe metering (#20298)
**Overall strategy** **1. Introduce “Billing V2” behind a workspace flag** Gate the new model with FeatureFlagKey.IS_BILLING_V2_ENABLED so existing workspaces stay on the old behavior until they’re migrated or explicitly on V2. **2. Replace workflow metered SKUs with a resource-credit product** Conceptually, billable “workflow execution” usage is not the primary subscription line item anymore. Add a RESOURCE_CREDIT product (and keep WORKFLOW_NODE_EXECUTION as deprecated for the transition). Usage and limits are expressed through credit buckets (e.g. price metadata like credit_amount), so one product can represent pooled credits instead of a narrow workflow-only meter. **3. Migrate subscriptions in two layers** Schema/catalog: persist extra price metadata (instance upgrade) so the server knows credit amounts and can match Stripe prices to the new model. Per workspace: the registered workspace command upgrade:2-2:migrate-to-billing-v2 finds subscriptions that still have WORKFLOW_NODE_EXECUTION, swaps those items to the right RESOURCE_CREDIT prices (using existing Stripe schedule + BillingSubscriptionUpdateService stack), then treats the workspace as V2 (flag). Workspaces without that legacy item or without a subscription are skipped. **4. Unify subscription lifecycle + usage on the server** **5. Refresh the product surface in Settings** Test : - [x] Subscribe v1 + Update subscribe + Migrate - [x] Subscribe v2 + Update subscribe |
||
|
martmull
|
617f571400 |
20215 convert application variable to a syncable entity (#20269)
## Summary - Converts applicationVariable from a bespoke sync path to a proper SyncableEntity, unifying it with the workspace migration pipeline used by all other manifest-managed entities (agent, skill, frontComponent, webhook, etc.) - Removes the upsertManyApplicationVariableEntities method and its direct-DB-mutation approach in favor of the standard validate → build → run action handler pipeline - Adds universalIdentifier, deletedAt columns and makes applicationId NOT NULL via an instance command migration ## Motivation Before this change, applicationVariable was the only manifest-managed entity that bypassed ApplicationManifestMigrationService.syncMetadataFromManifest(). It used a bespoke service method called directly from syncApplication(), creating two mental models, two validation styles, and two cache invalidation patterns. Now there's one unified pipeline for all manifest entities. ## What changed ### Entity refactor: - ApplicationVariableEntity now extends SyncableEntity (gains universalIdentifier, non-nullable applicationId with CASCADE, soft-delete via deletedAt) ### New flat entity layer (flat-application-variable/): - Type, maps type, editable properties constant, entity-to-flat converter, cache service, module ### New migration pipeline wiring: - Manifest converter (fromApplicationVariableManifestToUniversalFlatApplicationVariable) - Validator service (FlatApplicationVariableValidatorService) - Builder service (WorkspaceMigrationApplicationVariableActionsBuilderService) - Create/Update/Delete action handlers with secret encryption hooks - Registered in orchestrator, builder module, runner module, and all type registries ### Removed bespoke path: - Deleted upsertManyApplicationVariableEntities from ApplicationVariableEntityService - Removed its call from ApplicationSyncService.syncApplication() - Kept update() (operator-set value at runtime) and getDisplayValue() (runtime display) ### Database migration: - Instance command to add columns, backfill universalIdentifier, enforce NOT NULL constraints, and update indexes ## Test plan - npx nx typecheck twenty-server passes (0 errors) - Unit tests pass (application-variable.service.spec.ts, build-env-var.spec.ts) - Install an app with applicationVariables in its manifest → variables appear with correct universalIdentifier - Update app manifest (add/remove/modify a variable) → migration pipeline handles diff correctly - Operator-set value via update endpoint persists correctly with encryption - Uninstall app → variables cascade-deleted - app dev --once on example app syncs without errors |
||
|
Félix Malfait
|
9e94045fa5 |
feat(apps): generic OAuth provider support for app SDK (#20181)
## Summary
App developers can now declare third-party OAuth integrations (GitHub,
Linear, Slack, etc.) in their manifest and the platform handles the full
authorize → callback → token-exchange → refresh → injection lifecycle.
The dev writes ~10 lines of config and reads tokens via
`useOAuth('linear')` inside any logic function.
```ts
// app/src/oauth-providers/linear.ts
export default defineOAuthProvider({
universalIdentifier: '...',
name: 'linear',
displayName: 'Linear',
authorizationEndpoint: 'https://linear.app/oauth/authorize',
tokenEndpoint: 'https://api.linear.app/oauth/token',
scopes: ['read', 'write'],
connectionMode: 'per-user',
clientIdVariable: 'LINEAR_CLIENT_ID',
clientSecretVariable: 'LINEAR_CLIENT_SECRET',
tokenRequestContentType: 'form-urlencoded',
});
// app/src/logic-functions/handlers/...
const { accessToken } = useOAuth('linear'); // throws OAuthNotConnectedError if missing
```
## Architecture
- **Storage**: extends the existing `connectedAccount` table — new
nullable `applicationOAuthProviderId` FK + new `app` value on the
`ConnectedAccountProvider` enum. Existing Google/Microsoft flows are
untouched.
- **OAuth flow**: a single `/apps/oauth/authorize` +
`/apps/oauth/callback` controller pair handles every app provider. State
travels in a JWT signed via the existing `JwtWrapperService` (new
`APP_OAUTH_STATE` token type).
- **Token exchange**: goes through
`SecureHttpClientService.createSsrfSafeFetch()` (so an installed app
can't point `tokenEndpoint` at internal hosts).
- **Refresh**: piggybacks on the existing
`ConnectedAccountRefreshTokensService` dispatch — Google/Microsoft
drivers untouched, new app driver lives engine-side under
`application-oauth-provider/refresh/`.
- **Injection**: the executor injects refreshed tokens as env vars
(`OAUTH_<NAME>_ACCESS_TOKEN`, `_HANDLE`, `_SCOPES`, `_CONNECTED`); the
SDK helpers `useOAuth` / `useOptionalOAuth` read them.
- **Frontend**: auto-rendered "OAuth Connections" section under each
app's settings tab (no custom front component needed). App-managed
connections are filtered out of `/settings/accounts` so the
email/calendar page stays focused.
- **Disconnect**: best-effort revoke against the manifest's
`revokeEndpoint` before deleting the row.
## Reference app
`packages/twenty-apps/internal/twenty-linear/` exercises the full
pipeline:
- `defineOAuthProvider` for Linear
- `POST /linear/create-issue` and `GET /linear/teams` HTTP-route logic
functions
- Vitest tests for the handlers
## Tests
- 14 server-side Jest tests: token-exchange util (form-urlencoded vs
JSON, PKCE, error paths), flow service (authorize URL shape, state
binding, ConnectedAccount upsert on first/reconnect, per-workspace mode,
invalid state)
- 8 app-level Vitest tests: handler error paths, GraphQL request shape,
Linear error propagation
- All 4 packages clean: `npx nx lint:diff-with-main` and `npx tsc
--noEmit`
## Test plan
- [ ] Apply migration on a dev DB: `npx nx run
twenty-server:database:migrate:prod`
- [ ] Regenerate frontend types: `npx nx run
twenty-front:graphql:generate --configuration=metadata`
- [ ] Create a Linear OAuth app at
https://linear.app/settings/api/applications/new with redirect URI
`<SERVER_URL>/apps/oauth/callback`
- [ ] Deploy + install `twenty-linear` on a workspace, paste the Linear
client id/secret into the app's variables
- [ ] Click "Connect Linear" in the app's settings tab → complete OAuth
→ verify `connectedAccount` row created with `provider = 'app'`
- [ ] Trigger `POST /linear/create-issue` with a valid teamId → verify
issue lands in Linear
- [ ] Disconnect → verify the row is deleted and (if Linear's revoke
endpoint is configured in the manifest) the revoke call fires
- [ ] Verify `/settings/accounts` does NOT show the Linear connection —
it appears only under the Linear app's settings tab
## Out of scope (deliberately)
- **Cron + per-user providers**: a cron-triggered function with a
per-user OAuth provider currently returns `CONNECTED=false` (no user
context). The follow-up design is `useOAuthForUser(name,
userWorkspaceId)` paired with a `POST /apps/oauth/connection-token`
endpoint, deferred to keep this PR focused.
- **Token encryption at rest**: tokens stored as plain `varchar`
matching the existing Google/Microsoft pattern. Worth a separate
cross-cutting PR.
- **Manifest endpoint pinning**: a malicious app upgrade could change
`tokenEndpoint` silently. Same trust model as logic-function source code
(which already runs arbitrary server-side); worth tightening across the
whole upgrade pipeline rather than just OAuth.
- **CLI helpers** (`twenty oauth show-callback-url`, `twenty oauth
connect`): manual setup for v1.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
|
||
|
Etienne
|
fbaea0639a |
Billing - optimize usageEvent CH table (#20019)
- Update usageEvent clickhouse table, partitioning, indexing and projection (auto materialized view) to optimize credit usage queries - Add caching for available credits and billing subscription To do in next PR: deprecate enforceCapUsage cron. Bonus : real-time on billingSubscription |
||
|
Charles Bochet
|
3db1af9a17 |
fix(logic-function): forward raw request body for HMAC signature verification (#20061)
## Summary - Add optional `rawBody?: string` to `LogicFunctionEvent` and forward it from the route trigger so HMAC-based webhook signatures (GitHub's `X-Hub-Signature-256`, Stripe, …) can be verified by user logic functions. - Update `github-connector`'s `getRawBodyForSignature` to prefer `event.rawBody` (with the existing string/base64/null fallbacks kept for older runtimes). ## Why GitHub computes `X-Hub-Signature-256` over the **raw bytes** of the request body. The receiver must verify against those exact bytes — key order, whitespace and unicode escaping all matter, so the parsed JSON body cannot be re-serialized to them. Today the route trigger calls `extractBody(request)` which returns the parsed object only. NestJS already preserves the raw body on `request.rawBody` (the app is bootstrapped with `rawBody: true` in `main.ts`), but it was never propagated into `LogicFunctionEvent`. As a result the github-connector's webhook handler always took the "raw body unavailable" branch and rejected every delivery (after #19961 / 962c2b3c14). With this change, signature verification can succeed end-to-end. --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Charles Bochet
|
499067ae14 |
fix(logic-function): serialize LocalDriver layer builds with cache lock (#20054)
## Summary
Fixes a race condition in `LocalDriver` where concurrent `execute()`
calls for the same application can trash each other's layer build,
causing logic function executions to fail with either:
- `Error: ENOENT: process.cwd failed with error no such file or
directory, the current working directory was likely removed without
changing the working directory, uv_cwd` (the yarn-install child's `cwd`
got wiped mid-run), or
- `ENOENT: no such file or directory, open '…/<pkg>/<file>.js.map'`
raised from yarn's berry link step (files under the deps layer vanish
while yarn is extracting).
Both are thrown from `…/deps/<checksum>/.yarn/releases/yarn-4.9.2.cjs` —
i.e. the yarn process `copyYarnEngineAndBuildDependencies` spawns with
`cwd: buildDirectory`.
## Root cause
`LocalDriver.createLayerIfNotExist` (and `ensureSdkLayer`) both follow a
check-then-act pattern with no mutual exclusion:
```ts
if (await pathExists(depsNodeModulesPath)) return;
await fs.rm(depsLayerPath, { recursive: true, force: true });
await copyDependenciesInMemory(...);
await copyYarnEngineAndBuildDependencies(depsLayerPath); // spawns yarn with cwd=depsLayerPath
```
The deps layer path is shared across all `execute()` calls that match a
given `yarnLockChecksum`. Two concurrent callers (e.g. a webhook
invocation + a cron-triggered logic function firing while the first
run's layer is still being built) both see no `node_modules`, both
`fs.rm` the directory, and one's yarn child ends up with its `cwd` or
its extraction target gone.
## Fix
Wrap the critical sections with `CacheLockService.withLock` — the same
cache-backed lock the Lambda driver already uses for its layer/executor
builds:
- `createLayerIfNotExist` → lock key
`local-driver-deps-layer:${yarnLockChecksum ?? 'default'}`
- `ensureSdkLayer` → lock key
`local-driver-sdk-layer:${workspaceId}:${applicationUniversalIdentifier}`
A fast-path `pathExists` check is kept outside the lock (so warm
executions still skip lock acquisition entirely), and the existence +
staleness check is **repeated inside the lock** so followers no-op after
the leader finishes.
Lock parameters mirror the Lambda driver's layer-build lock (`ttl=120s`,
`retry=500ms`, `maxRetries=240`).
`cacheLockService` is now passed to `LocalDriver` from
`LogicFunctionDriverFactory` (it was already injected there for the
Lambda driver).
|
||
|
Félix Malfait
|
4f938aa097 |
feat(app): infrastructure for pre-installed apps (#19973)
**PR 1 of 2.** Follow-up PR ships the Exa app, sets it as a default
pre-installed app, and removes the current `WebSearchTool` /
`WebSearchService` / `ExaDriver`. This PR adds the plumbing; no
user-visible change yet.
## Summary
- Server admins can declare a list of npm app packages to auto-install
on every new workspace and backfill onto existing workspaces via CLI.
- Server-level secrets (like Exa's API key) live on the
`ApplicationRegistration` (one row per server, encrypted) and are
injected into logic function execution env at runtime. No more
per-workspace storage of global secrets.
- A generic `POST /app/billing/charge` endpoint lets app logic functions
emit workspace usage events for metered features. Exa uses it in PR 2;
future apps (call recorder, etc.) reuse it.
- `LogicFunctionToolProvider` tool name prefix changes `logic_function_`
→ `app_`. Shorter, accurate (they come from installed apps).
## What's in this PR
**Logic function executor — server-level variables**
- `LogicFunctionExecutorService.getExecutionEnvVariables` now resolves
env vars in the order: hardcoded defaults →
`ApplicationRegistrationVariable[]` (server-level) →
`ApplicationVariable[]` (workspace-level override). The manifest
`serverVariables` schema has existed; this closes the loop.
**Config**
- `PRE_INSTALLED_APPS` — comma-separated list of npm packages. Default:
empty.
**\`PreInstalledAppsService\`** (new module)
- \`onApplicationBootstrap()\` — fetches each package's manifest from
the app registry CDN, upserts an \`ApplicationRegistration\`, and seeds
declared \`serverVariables\` from matching env vars (e.g.
\`EXA_API_KEY\` env → encrypted registration variable).
- \`installOnWorkspace(workspaceId)\` — installs all pre-installed apps
on a single workspace. Tolerates per-app failures.
**Auto-install on new workspace activation**
- \`WorkspaceService.prefillCreatedWorkspaceRecords\` invokes
\`installOnWorkspace\` after prefilling standard records. Non-blocking
on failure.
**Backfill CLI command**
- \`install-pre-installed-apps\` — iterates active and suspended
workspaces, installs pre-installed apps that aren't yet installed.
Idempotent. Run after changing \`PRE_INSTALLED_APPS\`.
**App billing endpoint**
- \`POST /app/billing/charge\`. Authenticated via \`APPLICATION_ACCESS\`
token (already injected into logic function execution env as
\`DEFAULT_APP_ACCESS_TOKEN\`). Body: \`{ creditsUsedMicro, quantity,
unit, operationType, resourceContext? }\`. Emits \`USAGE_RECORDED\` with
\`applicationId\` as \`resourceId\`. Generic — reusable by any app.
**Tool name prefix**
- \`LogicFunctionToolProvider.buildLogicFunctionToolName\` now produces
\`app_<name>\` instead of \`logic_function_<name>\`. Only affects tools
sourced from logic functions; other tool providers unchanged.
## Stats
- 16 files, +501 / −2
- 7 new files (1 command, 1 service × 2, 1 controller, 1 DTO, 2 modules)
- Typecheck: 7 pre-existing errors, zero new
- Prettier clean
## Behavior deltas
- **\`PRE_INSTALLED_APPS\` default = empty**: existing servers see no
change on merge.
- **\`ApplicationRegistrationVariable\` is now read by the executor**:
apps that were using manifest \`serverVariables\` but expecting them to
be ignored by the executor will now see them injected. No apps ship with
\`isTool: true\` logic functions today, so this is latent — first
consumer is Exa in PR 2.
- **Tool prefix**: currently no logic-function tools are named
\`logic_function_*\` in any production flow. The prefix change affects
only future tools emitted by \`LogicFunctionToolProvider\`.
## Risks
- **CDN unavailability at startup**: if the app registry CDN is down,
\`ensureRegistrationsExist\` logs warnings but doesn't block server
start. Installation on new workspaces during this window will find no
registrations and log a non-blocking error. Backfill command can retry
after CDN recovers.
- **Cold-start overhead**: \`ensureRegistrationsExist\` is called once
per process on bootstrap. Current configurable default is empty, so zero
overhead. When an admin sets \`PRE_INSTALLED_APPS\`, they accept one
HTTP call per package at boot.
- **Server-level variables flow**:
\`ApplicationRegistrationVariable.encryptedValue\` is shared by all
workspaces of a server. Appropriate for a single-tenant Exa key. Not
appropriate for per-tenant keys — those go in workspace-level
\`ApplicationVariable\` and override.
## Test plan
- [ ] \`npx nx typecheck twenty-server\` passes (verified: 7
pre-existing unrelated errors, zero new)
- [ ] Set \`PRE_INSTALLED_APPS=@twenty-apps/hello-world\` (or any real
npm-published app), \`HELLO_WORLD_API_KEY=xxx\`, restart server:
\`ApplicationRegistration\` row is upserted,
\`ApplicationRegistrationVariable\` for HELLO_WORLD_API_KEY is populated
(encrypted).
- [ ] Create a new workspace: the app is auto-installed,
\`ApplicationEntity\` row created, \`LogicFunctionEntity\` rows created.
- [ ] Existing workspace: run \`yarn nx run twenty-server:command
install-pre-installed-apps\`: apps install across all workspaces,
idempotent on re-run.
- [ ] Trigger a logic function that reads
\`process.env.HELLO_WORLD_API_KEY\`: value resolves from the
server-level \`ApplicationRegistrationVariable\`.
- [ ] Log a charge from the handler: \`POST /app/billing/charge\` with
\`Authorization: Bearer \$DEFAULT_APP_ACCESS_TOKEN\` body
\`{creditsUsedMicro: 1000, quantity: 1, unit: "INVOCATION",
operationType: "WEB_SEARCH"}\` → returns \`{success: true}\`,
\`USAGE_RECORDED\` event emitted with correct
\`resourceId=applicationId\`.
- [ ] Tool name generated by \`LogicFunctionToolProvider\` starts with
\`app_\`.
## What's NOT in this PR (PR 2 scope)
- The Exa app itself (\`packages/twenty-apps/...\` directory)
- Removing \`WebSearchTool\`, \`WebSearchService\`, \`ExaDriver\`,
\`web-search\` module
- Removing \`WEB_SEARCH_DRIVER\` config var
- Removing the current \`exa_web_search\` entry in
\`ActionToolProvider\`
- Chat preload list updated to \`app_exa_web_search\`
- Frontend \`getToolDisplayMessage\` branch for \`app_exa_web_search\`
- Setting \`PRE_INSTALLED_APPS\` default to include \`@twenty-apps/exa\`
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
Etienne
|
117909e10a | Billing - Adapt to new unit (#19886) | ||
|
Etienne
|
e68842c268 |
Billing - fixes (#19867)
- Uniformize credit formating : In UI, 1$=1credit. In BE 1 UI credit = 1_000_000 BE "crédits" - Add crédit rollover information + Link to documentation + Documentation update <img width="291" height="317" alt="Screenshot 2026-04-17 at 18 22 59" src="https://github.com/user-attachments/assets/2519fb9f-159d-4c85-95f4-a6e005a8a1a3" /> <img width="848" height="763" alt="Screenshot 2026-04-17 at 14 12 20" src="https://github.com/user-attachments/assets/a3cc0874-f275-49ea-819f-305ec314bdfe" /> <img width="797" height="757" alt="Screenshot 2026-04-17 at 14 12 13" src="https://github.com/user-attachments/assets/9048409b-d5a2-435a-b735-70370705e668" /> - Enable direct top-up (or subscription if in trial) from AI chat <img width="333" height="215" alt="Screenshot 2026-04-17 at 22 52 00" src="https://github.com/user-attachments/assets/7a20c627-2806-4bcf-a037-b45752232be9" /> <img width="457" height="769" alt="Screenshot 2026-04-17 at 22 51 41" src="https://github.com/user-attachments/assets/d2a90c1b-271f-4fe9-8891-baeb2fabb86d" /> - Inform users if credit limit is reached - Banner <img width="1130" height="127" alt="Screenshot 2026-04-17 at 19 15 11" src="https://github.com/user-attachments/assets/30723e5e-c07e-462f-8eb8-e08f52bbab1c" /> |
||
|
Charles Bochet
|
70a73534c0 |
perf(server): reuse ESM module cache across warm Lambda invocations of logic functions (#19830)
## Summary Lambda warm-invocations of logic functions were spending **~440 ms** re-parsing and re-evaluating the user bundle on every call. The executor wrote the user code to a **randomly-named** temp file and `import()`-ed it, so each warm call resolved to a new URL and Node's ESM cache could never reuse the previous module record. This PR makes the executor write to a **content-hash filename**, skip the write when the file already exists, and stop deleting it. Identical code now reuses the same module record across warm calls in the same container, dropping warm-invocation overhead by **~30–40%**. ## What changed - `executor/index.mjs`: temp filename derived from `sha256(code)`, write skipped when file exists, no `fs.rm` on cleanup. - `lambda.driver.ts`: single structured `[lambda-timing]` log per invocation with `totalMs / buildExecutorMs / getBuiltCodeMs / payloadBytes / invokeSendMs / reportDurationMs / billedMs / initDurationMs / coldStart`. Goes through the standard NestJS `Logger`. No behavioural change for callers: same input → same output, same error semantics. ### Caveat: module-scope state now persists across warm calls With a stable filename, the user bundle is evaluated **once per warm container**. Any module-scoped state or top-level side-effects in user code are now shared across invocations of the same container, instead of re-running on every call. This is documented in the executor and is the intended trade-off — module scope should be treated as a per-container cache, not as per-call isolation. ## Findings — measured impact Same logic function (`fetch-prs`, ~12k PRs to page through), same workspace, same Lambda config (eu-west-3, 512 MB), token cache primed. ### Warm invocations | Phase | Before fix | After fix | Δ | | -------------------------------------- | ------------- | -------------- | ------------ | | Executor `import(userBundle)` | ~440 ms | **~0 ms** | **-440 ms** | | Lambda billed duration | ~1.5–1.7 s | **~1.0–1.1 s** | **~30–40%** | | Server-perceived round-trip | ~1.7–2.0 s | **~1.0–1.2 s** | **~30–40%** | ### Cold starts Unchanged — the cache helps subsequent warm calls in the same container, not the first one. Init Duration stays ~130–170 ms; total cold call ~2.5–3.0 s. ### Stress Could not reproduce the previously-reported \"every ~10th call times out\" behaviour after the fix: - 30 sequential calls: max 1.7 s, median ~1.1 s, 0 timeouts - 50 concurrent calls: max 9.4 s (clear cold-start cluster), median ~1.5 s, 0 timeouts Hypothesis: the warm-import overhead was eating into the headroom against the function timeout under bursty load; removing it pushed everything well below the limit. ## Observability One structured log line per invocation, sent through the standard NestJS logger: \`\`\` [lambda-timing] fnId=abc123 totalMs=1187 buildExecutorMs=2 getBuiltCodeMs=3 payloadBytes=1466321 invokeSendMs=1180 reportDurationMs=992 billedMs=1000 initDurationMs=n/a coldStart=false \`\`\` \`coldStart=true\` whenever Lambda spun up a fresh container; on warm calls \`buildExecutorMs\` and \`getBuiltCodeMs\` collapse to single-digit ms, confirming the cache fix is working. ## Test plan - [ ] CI green. - [ ] Deploy to a Lambda-backed env, trigger a logic function several times in a row. - [ ] Confirm \`[lambda-timing]\` warm invocations show \`totalMs\` ~30–40% lower than before, and \`coldStart=false\` after the first call in a container. - [ ] Push a new version of an app; confirm the next call shows higher \`buildExecutorMs\` (new hash, new file written) followed by warm calls again. - [ ] Smoke test: errors thrown by the user handler are still surfaced correctly. Made with [Cursor](https://cursor.com) |
||
|
Charles Bochet
|
e878d646ed |
chore(server): bump logic-function executor lambda memory to 512MB (#19826)
## Summary The executor lambda for user logic functions is created in `LambdaDriver` without a `MemorySize` parameter, so AWS Lambda falls back to its 128 MB default. That cap is too tight for non-trivial logic functions — large upstream GraphQL responses, JSON parsing of paginated batches, and chained Twenty Core API mutations push the process over the limit and trigger an OOM SIGKILL surfaced to the user as: ``` Runtime exited with error: signal: killed ``` This bumps the executor lambda memory to **512 MB** (matching the existing `BUILDER_LAMBDA_MEMORY_MB`). The change is applied on both: - The `CreateFunctionCommand` path used when a logic function is first deployed. - The `UpdateFunctionConfigurationCommand` path used when the deps/SDK layer wiring is refreshed — so existing functions get reconfigured on next deploy without any additional manual action. ## Why 512 MB Lambda compute is allocated proportionally to memory. 512 MB: - Matches the builder lambda already in this file (`BUILDER_LAMBDA_MEMORY_MB`). - Is comfortably above the 128 MB default that the existing OOMs are hitting. - Stays well below the higher tiers, keeping the per-invocation cost increase modest. Made with [Cursor](https://cursor.com) |
||
|
martmull
|
381f3ba7d9 |
Fix app design 1/2 (#19735)
comply with https://www.figma.com/design/xt8O9mFeLl46C5InWwoMrN/Twenty?node-id=96977-349627&m=dev ## After <img width="1512" height="909" alt="Capture d’écran 2026-04-16 à 09 40 37" src="https://github.com/user-attachments/assets/6d80191a-79a9-4f0f-aa4f-0e447fff4f6d" /> <img width="1512" height="909" alt="Capture d’écran 2026-04-16 à 09 40 22" src="https://github.com/user-attachments/assets/4f763272-027e-4246-b455-7d46babf7d8c" /> <img width="1512" height="909" alt="Capture d’écran 2026-04-16 à 09 39 11" src="https://github.com/user-attachments/assets/b9b35e18-8068-447e-821d-5ec28bb5bd16" /> <img width="1512" height="909" alt="Capture d’écran 2026-04-16 à 09 39 05" src="https://github.com/user-attachments/assets/57d9318a-902f-4fd7-a2a3-5795ebe0b9dc" /> <img width="1512" height="909" alt="Capture d’écran 2026-04-16 à 09 39 02" src="https://github.com/user-attachments/assets/78a33fa8-6bdd-484e-a82d-bd0f7592a623" /> <img width="1512" height="909" alt="Capture d’écran 2026-04-16 à 09 38 58" src="https://github.com/user-attachments/assets/f7987aed-c6e1-4032-a611-86817655137d" /> <img width="1512" height="909" alt="Capture d’écran 2026-04-16 à 09 38 55" src="https://github.com/user-attachments/assets/d1c451ab-1d2d-41e4-a059-cf4303ecabe7" /> <img width="1512" height="909" alt="Capture d’écran 2026-04-16 à 09 38 48" src="https://github.com/user-attachments/assets/593cae36-2320-443f-a955-93b211a6ee3f" /> <img width="1512" height="909" alt="Capture d’écran 2026-04-16 à 09 37 40" src="https://github.com/user-attachments/assets/c9f602b1-8de3-4e82-a3a6-344594a0c153" /> <img width="1512" height="909" alt="Capture d’écran 2026-04-16 à 09 37 34" src="https://github.com/user-attachments/assets/b54ddddf-5dda-46c8-ace3-cffe6015825a" /> ## before <img width="1512" height="909" alt="Capture d’écran 2026-04-16 à 09 42 18" src="https://github.com/user-attachments/assets/c0976a0a-0124-48ec-8e7c-78627cea7063" /> <img width="1512" height="909" alt="Capture d’écran 2026-04-16 à 09 42 16" src="https://github.com/user-attachments/assets/d2db926c-4040-411d-9091-8b60e7c519e6" /> <img width="1512" height="909" alt="Capture d’écran 2026-04-16 à 09 42 13" src="https://github.com/user-attachments/assets/2d69f2ff-f26e-4249-91a3-2cf3d261e840" /> <img width="1512" height="909" alt="Capture d’écran 2026-04-16 à 09 42 07" src="https://github.com/user-attachments/assets/1028aabc-77ac-4c51-a8c3-9a194faba87f" /> <img width="1512" height="909" alt="Capture d’écran 2026-04-16 à 09 42 01" src="https://github.com/user-attachments/assets/1caa9f5e-3eaa-433c-9d3b-e0f094f16e8e" /> <img width="1512" height="909" alt="Capture d’écran 2026-04-16 à 09 41 56" src="https://github.com/user-attachments/assets/f42b6976-3a8f-4591-9283-bda79bdb424b" /> <img width="1512" height="909" alt="Capture d’écran 2026-04-16 à 09 41 53" src="https://github.com/user-attachments/assets/93d00df8-0091-4dfa-9ac0-f6f376be5962" /> <img width="1512" height="909" alt="Capture d’écran 2026-04-16 à 09 41 43" src="https://github.com/user-attachments/assets/9deae7e5-39c1-4518-a463-6d79bc5bf132" /> <img width="1512" height="909" alt="Capture d’écran 2026-04-16 à 09 41 37" src="https://github.com/user-attachments/assets/3e21b521-c47d-482c-ad41-66abfe973772" /> |
||
|
Charles Bochet
|
446c39d1c0 |
Fix silent failures in logic function route trigger execution (#19698)
## Summary Route-triggered logic functions were returning empty 500 responses with zero server-side logging when the Lambda build chain failed. This PR makes those failures observable and returns meaningful HTTP responses to API clients. - **Observability** — Log errors (with stack traces) at each layer of the execution chain: `LambdaDriver` (deps-layer fetch, SDK-layer fetch, invocation), `LogicFunctionExecutorService`, and `RouteTriggerService`. - **Typed exceptions** — Replace raw `throw error` sites with `LogicFunctionException` carrying an appropriate code and `userFriendlyMessage` (new codes: `LOGIC_FUNCTION_EXECUTION_FAILED`, `LOGIC_FUNCTION_LAYER_BUILD_FAILED`). - **Correct HTTP semantics** — `RouteTriggerService` maps inner exception codes to the right `RouteTriggerExceptionCode` so `LOGIC_FUNCTION_NOT_FOUND` returns 404 and `RATE_LIMIT_EXCEEDED` returns 429 (new code + filter case) instead of a generic 500. - **User-facing messages** — Forward the inner `CustomException.userFriendlyMessage` when wrapping into `RouteTriggerException`, without leaking raw internal error text into the public exception message. - **Infra** — Bump Lambda ephemeral storage from 2048 to 4096 MB to prevent `ENOSPC` errors during yarn install layer builds (root cause of the original silent failures). |
||
|
neo773
|
88c0b24d9e |
Add per-workspace error handling to CronTriggerCronJob (#19640)
fix sonarly 19618 |
||
|
Paul Rastoin
|
a82d078906 |
Lambda build update instead of delete existing logic function while building (#19116)
# Introduction Avoid having a time window where the logic function is unavailable while being built, by implementing an update flow instead of delete and create everytime fixes https://twenty-v7.sentry.io/issues/7371110591/ **Note: executor code propagation (pre-existing limitation)** The Lambda executor shim (`logic-function-drivers/constants/executor/index.mjs`) is only deployed when a Lambda is first created. If this file is edited, the change won't propagate to existing Lambdas — neither before nor after this PR. A follow-up could compare the deployed `CodeSha256` against a local checksum to detect drift and trigger a code update via `UpdateFunctionCodeCommand`. Should implem as code versioning or checksum diffing |
||
|
neo773
|
8a10071253 |
add workspaceId to indirect entities (#19522)
Required for `workspace:export` command --------- Co-authored-by: Charles Bochet <charles@twenty.com> |
||
|
Thomas Trompette
|
00208dbf1f |
Build lambda error - catch user code compilation errors (#19516)
Fixes https://twenty-v7.sentry.io/issues/7335177730/events/latest/?environment=prod&project=4507072499810304&query=is%3Aunresolved&referrer=latest-event - Provide a more explicit error for user - Remove from sentry while keeping others due to infra issues |
||
|
Charles Bochet
|
36fbfca069 |
Add application-logs module with driver pattern for logic function log persistence (#19486)
## Summary
- Introduces a new `application-logs` core module with a driver pattern
(disabled/console/clickhouse) to capture and persist logic function
execution logs
- Adds a ClickHouse `applicationLog` table with per-line log storage,
30-day TTL, and `ORDER BY (workspaceId, timestamp, applicationId,
logicFunctionId)`
- Surfaces application logs in the existing frontend audit logs table as
a new "Application Logs" source with dedicated columns (Function,
Timestamp, Level, Message, Execution ID)
## Details
**Write path**: `LogicFunctionExecutorService.handleExecutionResult()`
parses the multi-line log string from driver output into individual `{
timestamp, level, message }` entries, generates an execution UUID, and
passes them to `ApplicationLogsService.writeLogs()` which delegates to
the configured driver.
**Driver pattern**: Follows the exception-handler module style (Symbol
injection token + `forRootAsync` dynamic module). Three drivers:
- `DISABLED` (default) — no-op, prevents information leaking
- `CONSOLE` — structured stdout logging with level-based `console.*`
calls
- `CLICKHOUSE` — inserts rows into the `applicationLog` ClickHouse table
**Read path**: Extends the existing event-logs module by adding
`APPLICATION_LOG` to the `EventLogTable` enum, table name mapping, and
normalization logic.
**Config**: New `APPLICATION_LOG_DRIVER_TYPE` environment variable
(default: `DISABLED`).
|
||
|
Félix Malfait
|
6073bb6706 |
Fix AI model registry staleness on self-hosted instances (#19427)
## Summary Fixes #19422. Self-hosted users hitting "No AI models are available" after configuring API keys via the admin panel were victims of a stale `AiModelRegistryService` cache. Only the four `addAiProvider` / `removeAiProvider` / `addModelToProvider` / `removeModelFromProvider` mutations called `refreshRegistry()` — setting an API key through `set/update/deleteDatabaseConfigVariable` left the registry pointing at the pre-mutation provider state. Rather than patch each mutation site (and re-introduce the same class of bug on the next one), the registry now invalidates lazily based on the LLM config-group hash, mirroring the pattern `WebSearchDriverFactory` already uses via `DriverFactoryBase`. Any mutation to an LLM-tagged config variable is picked up automatically on the next read — callers never have to remember to refresh. - Extracted `getConfigGroupHash` into a shared util reused by both `DriverFactoryBase` and `AiModelRegistryService` (and switched the hash to `JSON.stringify` so object-typed config vars like `AI_PROVIDERS` actually contribute meaningfully). - `AiModelRegistryService` gates all internal `Map` access behind private getters that call `ensureFresh()`, so future read paths can't accidentally observe stale state. Build path uses underscored backing fields directly to avoid recursing. - Dropped the now-redundant `refreshRegistry()` public method and its four call sites in `admin-panel.resolver.ts`. ## Test plan - [x] `nx typecheck twenty-server` passes - [x] Existing admin-panel + ai-models specs pass (79 tests) - [ ] Manual: on a self-hosted instance, set `OPENAI_API_KEY` (or another provider key) via the admin panel `setDatabaseConfigVariable` mutation and confirm AI features work without a server restart - [ ] Manual: existing flows (`addAiProvider`, `addModelToProvider`, etc.) still pick up new providers on the next read 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> |
||
|
Thomas Trompette
|
9a58f3d459 |
Add a lock on function creation (#19428)
Fixes https://twenty-v7.sentry.io/issues/7368127776/?environment=prod&project=4507072499810304&query=is%3Aunresolved&referrer=issue-stream - Add a distributed lock (via CacheLockService) around the Lambda executor build in LambdaDriver to prevent concurrent workflow runs from racing on CreateFunctionCommand for the same logic function - Use double-checked locking: isAlreadyBuilt is checked lock-free first (fast path), then re-checked inside the lock to avoid redundant rebuilds |
||
|
martmull
|
c91d642f29 |
App feedbacks front (#19417)
## Before <img width="777" height="284" alt="image" src="https://github.com/user-attachments/assets/22a8c318-ed7c-4ec5-b7a8-a688098ec103" /> ## After <img width="897" height="329" alt="image" src="https://github.com/user-attachments/assets/16ad78bf-a449-4f25-99f6-13c35e448f1b" /> |
||
|
Paul Rastoin
|
137e068ced |
Fix yarn-install-lambda and lambda-build target (#19407)
# Size issue
```
Yarn install Lambda failed: {"errorType":"Error","errorMessage":"yarn install failed: ➤ Y/tmp/3bac8baafe6354db414389434726b02c/nodejs/node_modules/tar ENOSPC: no space left on device, write","➤ YN0000: └ Completed in 3s 57ms","➤ YN0000: · Failed with errors in 11s 684ms",""," at runYarnInstall (file:///var/task/index.mjs:48:11)"," at process.processTicksAndRejections (node:internal/process/task_queues:105:5)"," at async Runtime.handler (file:///var/task/index.mjs:119:5)"]}
```
# target esnext
setting the same target for each logic function build funnels
- sdk
- local driver
- lambda driver
|
||
|
Paul Rastoin
|
a2533baa66 |
Local driver layer resolution wipe partial local layers relicas (#19267)
# Introduction Wipe partial layer local driver relicas that do not implem a node_modules folder |
||
|
Charles Bochet
|
c407341912 |
feat: optimize hot database queries with multi-layer caching (#19068)
## Summary Introduces multi-layer caching for the 5 most frequent database queries identified in production (Sentry data), targeting the JWT authentication hot path and cron job logic. ### Problem Our database is under heavy load from uncached queries on the auth hot path: - `WorkspaceEntity` lookups: **638 queries/min** - `ApiKeyEntity` lookups: **491 queries/min** - `UserEntity` lookups: **147 queries/min** - `UserWorkspaceEntity` lookups: **143 queries/min** - `LogicFunctionEntity` lookups: **1800 queries/min** (cron job) ### Solution **1. New `CoreEntityCacheService`** for non-workspace-scoped entities (Workspace, User, UserWorkspace): - Mirrors `WorkspaceCacheService` architecture (in-process Map + Redis with hash validation) - Provider pattern with `@CoreEntityCache` decorator - Keyed by entity primary key (not workspaceId) - 100ms local TTL, Redis-backed hash validation for cross-instance consistency - Three providers: `WorkspaceEntityCacheProviderService`, `UserEntityCacheProviderService`, `UserWorkspaceEntityCacheProviderService` **2. New `apiKeyMap` WorkspaceCache** for workspace-scoped API key lookups: - `WorkspaceApiKeyMapCacheService` loads all API keys for a workspace into a map by ID - Leverages existing `WorkspaceCacheService` infrastructure - Cache invalidation on API key create/update/revoke **3. `CronTriggerCronJob` refactored** to use existing `flatLogicFunctionMaps` workspace cache: - Eliminates per-workspace `LogicFunctionEntity` repository queries (~1800/min) - Filters cached data in-memory instead **4. `JwtAuthStrategy` refactored** to use caches for all entity lookups: - Workspace, User, UserWorkspace → `CoreEntityCacheService` - ApiKey → `WorkspaceCacheService` (`apiKeyMap`) - Impersonation queries kept as direct DB queries (rare path, requires relations) **5. Cache invalidation** wired into mutation paths: - `WorkspaceService` → invalidates `workspaceEntity` on save/update/delete - `ApiKeyService` → invalidates `apiKeyMap` on create/update/revoke ### Architecture ``` Request → JwtAuthStrategy ├── Workspace lookup → CoreEntityCacheService (in-process → Redis → DB) ├── User lookup → CoreEntityCacheService (in-process → Redis → DB) ├── UserWorkspace lookup → CoreEntityCacheService (in-process → Redis → DB) └── ApiKey lookup → WorkspaceCacheService (in-process → Redis → DB) CronTriggerCronJob └── LogicFunction lookup → WorkspaceCacheService (flatLogicFunctionMaps) ``` ### Expected Impact | Query | Before | After | |-------|--------|-------| | WorkspaceEntity | 638/min | ~0 (cached) | | ApiKeyEntity | 491/min | ~0 (cached) | | UserEntity | 147/min | ~0 (cached) | | UserWorkspaceEntity | 143/min | ~0 (cached) | | LogicFunctionEntity | 1800/min | ~0 (cached) | ### Not included (ongoing separately) - DataSourceEntity query optimization (IS_DATASOURCE_MIGRATED migration) - ObjectMetadataEntity query optimization (already partially cached) |
||
|
Thomas des Francs
|
50ea560e57 |
Seed company workflow for email upserts (#18909)
@thomtrp --------- Co-authored-by: Charles Bochet <charles@twenty.com> |
||
|
Paul Rastoin
|
db9da3194f | Refactor client auto heal (#19032) | ||
|
Paul Rastoin
|
4ea2e32366 |
Refactor twenty client sdk provisioning for logic function and front-component (#18544)
## 1. The `twenty-client-sdk` Package (Source of Truth)
The monorepo package at `packages/twenty-client-sdk` ships with:
- A **pre-built metadata client** (static, generated from a fixed
schema)
- A **stub core client** that throws at runtime (`CoreApiClient was not
generated...`)
- Both ESM (`.mjs`) and CJS (`.cjs`) bundles in `dist/`
- A `package.json` with proper `exports` map for
`twenty-client-sdk/core`, `twenty-client-sdk/metadata`, and
`twenty-client-sdk/generate`
## 2. Generation & Upload (Server-Side, at Migration Time)
**When**: `WorkspaceMigrationRunnerService.run()` executes after a
metadata schema change.
**What happens in `SdkClientGenerationService.generateAndStore()`**:
1. Copies the stub `twenty-client-sdk` package from the server's assets
(resolved via `SDK_CLIENT_PACKAGE_DIRNAME` — from
`dist/assets/twenty-client-sdk/` in production, or from `node_modules`
in dev)
2. Filters out `node_modules/` and `src/` during copy — only
`package.json` + `dist/` are kept (like an npm publish)
3. Calls `replaceCoreClient()` which uses `@genql/cli` to introspect the
**application-scoped** GraphQL schema and generates a real
`CoreApiClient`, then compiles it to ESM+CJS and overwrites
`dist/core.mjs` and `dist/core.cjs`
4. Archives the **entire package** (with `package.json` + `dist/`) into
`twenty-client-sdk.zip`
5. Uploads the single archive to S3 under
`FileFolder.GeneratedSdkClient`
6. Sets `isSdkLayerStale = true` on the `ApplicationEntity` in the
database
## 3. Invalidation Signal
The `isSdkLayerStale` boolean column on `ApplicationEntity` is the
invalidation mechanism:
- **Set to `true`** by `generateAndStore()` after uploading a new client
archive
- **Checked** by both logic function drivers before execution — if
`true`, they rebuild their local layer
- **Set back to `false`** by `markSdkLayerFresh()` after the driver has
successfully consumed the new archive
Default is `false` so existing applications without a generated client
aren't affected.
## 4a. Logic Functions — Local Driver
**`ensureSdkLayer()`** is called before every execution:
1. Checks if the local SDK layer directory exists AND `isSdkLayerStale`
is `false` → early return
2. Otherwise, cleans the local layer directory
3. Calls `downloadAndExtractToPackage()` which streams the zip from S3
directly to disk and extracts the full package into
`<tmpdir>/sdk/<workspaceId>-<appId>/node_modules/twenty-client-sdk/`
4. Calls `markSdkLayerFresh()` to set `isSdkLayerStale = false`
**At execution time**, `assembleNodeModules()` symlinks everything from
the deps layer's `node_modules/` **except** `twenty-client-sdk`, which
is symlinked from the SDK layer instead. This ensures the logic
function's `import ... from 'twenty-client-sdk/core'` resolves to the
generated client.
## 4b. Logic Functions — Lambda Driver
**`ensureSdkLayer()`** is called during `build()`:
1. Checks if `isSdkLayerStale` is `false` and an existing Lambda layer
ARN exists → early return
2. Otherwise, deletes all existing layer versions for this SDK layer
name
3. Calls `downloadArchiveBuffer()` to get the raw zip from S3 (no disk
extraction)
4. Calls `reprefixZipEntries()` which streams the zip entries into a
**new zip** with the path prefix
`nodejs/node_modules/twenty-client-sdk/` — this is the Lambda layer
convention path. All done in memory, no disk round-trip
5. Publishes the re-prefixed zip as a new Lambda layer via
`publishLayer()`
6. Calls `markSdkLayerFresh()`
**At function creation**, the Lambda is created with **two layers**:
`[depsLayerArn, sdkLayerArn]`. The SDK layer is listed last so it
overwrites the stub `twenty-client-sdk` from the deps layer (later
layers take precedence in Lambda's `/opt` merge).
## 5. Front Components
Front components are built by `app:build` with `twenty-client-sdk/core`
and `twenty-client-sdk/metadata` as **esbuild externals**. The stored
`.mjs` in S3 has unresolved bare import specifiers like `import {
CoreApiClient } from 'twenty-client-sdk/core'`.
SDK import resolution is split between the **frontend host** (fetching &
caching SDK modules) and the **Web Worker** (rewriting imports):
**Server endpoints**:
- `GET /rest/front-components/:id` —
`FrontComponentService.getBuiltComponentStream()` returns the **raw
`.mjs`** directly from file storage. No bundling, no SDK injection.
- `GET /rest/sdk-client/:applicationId/:moduleName` —
`SdkClientController` reads a single file (e.g. `dist/core.mjs`) from
the generated SDK archive via
`SdkClientGenerationService.readFileFromArchive()` and serves it as
JavaScript.
**Frontend host** (`FrontComponentRenderer` in `twenty-front`):
1. Queries `FindOneFrontComponent` which returns `applicationId`,
`builtComponentChecksum`, `usesSdkClient`, and `applicationTokenPair`
2. If `usesSdkClient` is `true`, renders
`FrontComponentRendererWithSdkClient` which calls the
`useApplicationSdkClient` hook
3. `useApplicationSdkClient({ applicationId, accessToken })` checks the
Jotai atom family cache for existing blob URLs. On cache miss, fetches
both SDK modules from `GET /rest/sdk-client/:applicationId/core` and
`/metadata`, creates **blob URLs** for each, and stores them in the atom
family
4. Once the blob URLs are cached, passes them as `sdkClientUrls`
(already blob URLs, not server URLs) to `SharedFrontComponentRenderer` →
`FrontComponentWorkerEffect` → worker's `render()` call via
`HostToWorkerRenderContext`
**Worker** (`remote-worker.ts` in `twenty-sdk`):
1. Fetches the raw component `.mjs` source as text
2. If `sdkClientUrls` are provided and the source contains SDK import
specifiers (`twenty-client-sdk/core`, `twenty-client-sdk/metadata`),
**rewrites** the bare specifiers to the blob URLs received from the host
(e.g. `'twenty-client-sdk/core'` → `'blob:...'`)
3. Creates a blob URL for the rewritten source and `import()`s it
4. Revokes only the component blob URL after the module is loaded — the
SDK blob URLs are owned and managed by the host's Jotai cache
This approach eliminates server-side esbuild bundling on every request,
caches SDK modules per application in the frontend, and keeps the
worker's job to a simple string rewrite.
## Summary Diagram
```
app:build (SDK)
└─ twenty-client-sdk stub (metadata=real, core=stub)
│
▼
WorkspaceMigrationRunnerService.run()
└─ SdkClientGenerationService.generateAndStore()
├─ Copy stub package (package.json + dist/)
├─ replaceCoreClient() → regenerate core.mjs/core.cjs
├─ Zip entire package → upload to S3
└─ Set isSdkLayerStale = true
│
┌────────┴────────────────────┐
▼ ▼
Logic Functions Front Components
│ │
├─ Local Driver ├─ GET /rest/sdk-client/:appId/core
│ └─ downloadAndExtract │ → core.mjs from archive
│ → symlink into │
│ node_modules ├─ Host (useApplicationSdkClient)
│ │ ├─ Fetch SDK modules
└─ Lambda Driver │ ├─ Create blob URLs
└─ downloadArchiveBuffer │ └─ Cache in Jotai atom family
→ reprefixZipEntries │
→ publish as Lambda ├─ GET /rest/front-components/:id
layer │ → raw .mjs (no bundling)
│
└─ Worker (browser)
├─ Fetch component .mjs
├─ Rewrite imports → blob URLs
└─ import() rewritten source
```
## Next PR
- Estimate perf improvement by implementing a redis caching for front
component client storage ( we don't even cache front comp initially )
- Implem frontent blob invalidation sse event from server
---------
Co-authored-by: Charles Bochet <charlesBochet@users.noreply.github.com>
|
||
|
martmull
|
d3c7b0131d |
Fix lambda driver (#18907)
as title |
||
|
Félix Malfait
|
c6f11d8adb |
fix: migrate driver modules to DriverFactoryBase lazy-loading pattern (#18731)
## Summary - Migrates `LogicFunctionModule`, `CodeInterpreterModule`, and `CaptchaModule` from the `forRootAsync` + injection token pattern to the `DriverFactoryBase` lazy-loading pattern (matching `EmailModule` and `FileStorageModule`) - Fixes #18724 where `LOGIC_FUNCTION_TYPE` was not respected in worker processes because the driver was created at module boot time before the DB config cache was loaded - Removes `isEnvOnly` from `LOGIC_FUNCTION_TYPE`, `CODE_INTERPRETER_TYPE`, `CAPTCHA_DRIVER`, `IS_MULTIWORKSPACE_ENABLED`, and `FRONTEND_URL` — these can now be safely configured via the database at runtime ## How it works Each migrated module now uses a `DriverFactory` (extending `DriverFactoryBase`) instead of a module-level async factory + Symbol injection token: 1. **Lazy creation**: `getCurrentDriver()` creates the driver on first call, after `DatabaseConfigDriver.onModuleInit()` has loaded the DB cache 2. **Auto-recreation**: If config changes in the DB, the next `getCurrentDriver()` call detects the key mismatch and creates a new driver instance 3. **Unified config**: Both server and worker read from the same database — driver config only needs to be set once ### Files deleted (old pattern) - `logic-function-module.factory.ts`, `logic-function-drivers.module.ts`, `logic-function-driver.constants.ts` - `code-interpreter-module.factory.ts` - `captcha.module-factory.ts`, `captcha-driver.constants.ts` ### Files created (new pattern) - `logic-function-driver.factory.ts` - `code-interpreter-driver.factory.ts` - `captcha-driver.factory.ts` Net: **-150 lines** ## Test plan - [x] `npx nx typecheck twenty-server` passes - [x] `npx nx lint:diff-with-main twenty-server` passes - [ ] Integration tests pass (`npx nx run twenty-server:test:integration:with-db-reset`) - [ ] Verify logic functions execute in workflow runs (the original bug) - [ ] Verify code interpreter works in workflow code steps - [ ] Verify captcha validation works on sign-up (when captcha is configured) Made with [Cursor](https://cursor.com) |