f8e3fd110d35778bef7341ef563c2607d065bd2e
8 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Weiko
|
fc6a95a37f |
Throttle local cache expiration sweeps (#23579)
## Context The workspace cache and core entity cache keep bounded in-process maps. Entries that have not been read for 30 minutes are removed by an expiration sweep. Before this PR, every cache read synchronously walked the entire local cache, including every stored version, before performing the actual lookup. The cost therefore grew with the number of cached entries even when there was nothing to expire. Both caches are used on common server request paths, so these repeated full-map scans add unnecessary CPU work and short-lived allocations, which can contribute to event-loop and garbage-collection pressure under load. ## What changed - Run each cache's expiration sweep at most once per minute. - Keep the existing expiration logic unchanged and use one captured timestamp for the complete sweep. - Cover both cache services with tests proving that repeated reads within the interval trigger one sweep and that sweeping resumes after the interval. Normal cache reads now pay only for a timestamp check and branch. The full `O(cache size)` scan runs at most once per minute per process. ## Safety This does not change cache freshness: - The 100 ms local freshness window and Redis hash validation still run as before. - Explicit cache invalidation is unchanged. - The 30-minute inactivity threshold is unchanged. - LRU eviction still runs when entries are inserted. - Existing local-cache size limits remain unchanged. An unused entry can remain in memory for at most one additional minute before the next sweep. This may marginally increase average retained memory, but it cannot cause unbounded growth or allow stale data to bypass the existing hash validation. ## Expected impact This removes a cache-size-dependent operation from a high-frequency path. The expected benefit is lower CPU and allocation overhead, less garbage-collection pressure, and improved tail latency when local caches are populated. This is intentionally a narrow optimization. It does not claim to address every source of API tail latency. ## Validation - `yarn nx typecheck twenty-server` - Type-aware Oxlint on the changed files - Oxfmt on the changed files - Targeted workspace-cache and core-entity-cache Jest suites, 23 tests passing |
||
|
martmull
|
9423af7f67 |
feat(server): add public marketplace resolver for vetted app catalog (#22647)
## What Adds a public GraphQL resolver so unauthenticated clients (the public website) can read the listed/vetted marketplace catalog without a workspace token. - `MarketplacePublicResolver` (metadata schema) exposes two public queries guarded by `PublicEndpointGuard` + `NoPermissionGuard`: - `publicMarketplaceApps` - `publicMarketplaceAppDetail(universalIdentifier)` Both delegate to the existing `MarketplaceQueryService` (no new logic, no new REST routing). The existing workspace-guarded `findManyMarketplaceApps` / `findMarketplaceAppDetail` queries are untouched. - Adds a shared `ApplicationCategory` type in `twenty-shared` (known values plus `string` for backward compatibility) used to type `ApplicationManifest.category`. A warning is logged server-side when an app declares a category outside the known set. ## Why This is the backend half of the public apps marketplace on the website. Splitting it out so the server-side catalog exposure can be reviewed independently from the website UI. ## Follow-up The website PR (the `/apps` marketplace UI) consumes `publicMarketplaceApps` and should merge after this one. --- _Generated by [Claude Code](https://claude.ai/code/session_01GBfegArtJcoiTLSsnWPH8R)_ <!-- This is an auto-generated description by cubic. --> <a href="https://cubic.dev/pr/twentyhq/twenty/pull/22647?utm_source=github" target="_blank" rel="noopener noreferrer" data-no-image-dialog="true"><picture><source media="(prefers-color-scheme: dark)" srcset="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"><source media="(prefers-color-scheme: light)" srcset="https://www.cubic.dev/buttons/review-in-cubic-light.svg"><img alt="Review in cubic" src="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"></picture></a> <!-- End of auto-generated description by cubic. --> --------- Co-authored-by: martmull <martin@twenty.com> |
||
|
Paul Rastoin
|
3d49642d12 |
[AUDIT] Run knip over twenty-server (#21159)
# Introduction Run [knip](https://knip.dev/) over twenty-server Used config: ```json { "$schema": "https://unpkg.com/knip@5/schema.json", "workspaces": { "packages/twenty-server": { "entry": [ "src/main.ts", "src/command/command.ts", "src/queue-worker/queue-worker.ts", "src/database/scripts/setup-db.ts", "src/database/scripts/truncate-db.ts", "src/database/clickHouse/migrations/run-migrations.ts", "src/database/clickHouse/seeds/run-seeds.ts", "src/instrument.ts", "lingui.config.ts", "test/integration/graphql/codegen/index.ts", "test/integration/utils/setup-test.ts", "test/integration/utils/teardown-test.ts", "scripts/**/*.ts", "**/*.spec.ts", "**/*.integration-spec.ts" ], "project": ["src/**/*.ts", "test/**/*.ts", "scripts/**/*.ts"], "ignore": [ "src/database/typeorm/**/migrations/**", "src/database/typeorm/**/*.entity.ts", "**/*.workspace-entity.ts", "**/logic-function-resource/constants/seed-project/**" ], "ignoreDependencies": ["@types/psl", "@types/aws-lambda"], "ignoreBinaries": ["nest", "lingui", "typeorm"] } } } ``` |
||
|
Charles Bochet
|
9e515afb13 |
feat(server): asymmetric JWT signing with kid + key rotation table (#20467)
## Context Today every JWT issued by Twenty (access, refresh, login, file, etc.) is HMAC-signed with a per-token-type secret derived from the global `APP_SECRET`. Rotating that secret invalidates **every** active token at once and there is no way to scope a leak to a subset of tokens. This PR is the first slice of a broader effort to **decouple stateful encryption (`APP_SECRET`-derived secrets) from stateless encryption (JWTs)**. It introduces an asymmetric (private/public key) signing path for `ACCESS` and `REFRESH` tokens and a signing-key registry to enable **safe rotation**: leaked keys can be revoked by flipping `revokedAt`/`isCurrent` on the matching row without invalidating tokens issued by other keys. > Out of scope (intentionally): swapping stateful encryption for `APP_SECRET`, asymmetric signing for token types other than `ACCESS`/`REFRESH`, an admin-panel rotation UI, and an enterprise re-encryption command. Those will land in follow-up PRs. ## What changes - **New `core.signingKey` table** (instance command `2.5.0` / `1778550000000`) storing both the public key (PEM, in clear) and the private key (PEM, encrypted with `APP_SECRET` via `SecretEncryptionService`). One row is marked `isCurrent = true` (enforced by a partial unique index). The row's UUID `id` is used directly as the JWT `kid`. - When a key is rotated out, its `privateKey` is nulled (we never keep historical private keys) but the `publicKey` row stays so previously issued tokens can still be verified. - **`JwtKeyManagerService`** lazily loads-or-generates the current signing key on first use: - If a row with `isCurrent = true` exists → decrypts and uses it. - Otherwise → generates a fresh EC P-256 keypair, encrypts the private key, inserts the row (UUID id = kid). Handles concurrent insert races via the unique constraint. - **`JwtWrapperService.signAsync()`** signs `ACCESS`/`REFRESH` payloads with `ES256` and a `kid` header. Falls back to `HS256` if no signing key is available (boot-time DB error, transient failure). - **Dual-path verification** in both `JwtWrapperService.verifyJwtToken` and the Passport `JwtAuthStrategy.secretOrKeyProvider`: - JWT with a `kid` header → resolve the public key PEM by id and verify with `ES256`, - otherwise → fall back to the existing `APP_SECRET`-derived `HS256` path (unchanged). - **`AccessTokenService` / `RefreshTokenService`** now sign through `signAsync` (single public surface; the routing detail stays internal to the wrapper). - **Public key cache**: a new `SigningKeyEntityCacheProviderService` plugs into `CoreEntityCacheService` (`signingKeyPublicKey` namespace) and serves PEMs by id, with the standard local-memo + Redis layering. - **PEM strings end-to-end**: `jsonwebtoken` accepts PEM strings directly for both sign and verify, so the manager never converts to a Node `KeyObject` and the cache hands the PEM straight to `jwt.verify`. ## Why ES256 (and not EdDSA / RS256) - `@nestjs/jwt` is backed by `jsonwebtoken`, which does **not** support EdDSA today. - ES256 keys are tiny (~120 bytes vs 1.6 kB for RS256), signatures are short (~64 bytes), and signing/verification is fast — important since JWT verification runs on every authenticated request. - ES256 is widely supported and standardized (RFC 7518), with mature ecosystem support. ## Why store the private key in DB (not env) - No new secret to provision: existing instances already have `APP_SECRET`, which we reuse to encrypt the private key at rest. - Self-healing: a fresh instance auto-generates its first signing key on first boot. Nothing to copy/paste. - Rotation is a SQL operation against `core.signingKey`, not a redeploy + env mutation. ## Backward compatibility - All previously-issued tokens (no `kid`) keep verifying through the legacy HS256 path with their existing `APP_SECRET`-derived secret. No forced re-login. - Token types not in scope (`WORKSPACE_AGNOSTIC`, `API_KEY`, `FILE`, `LOGIN`, `EMAIL_VERIFICATION`, etc.) keep their current HS256 behavior unchanged — they still go through the synchronous `JwtWrapperService.sign(payload, options)` with a caller-supplied secret. - `signWithAppSecret` is kept intentionally as the HS256 fallback path; it will be deprecated in a follow-up PR. - If the DB lookup/generation fails for any reason, the wrapper logs the error and falls back to HS256 — no startup crash, no silent regression. ## Rotation story 1. Bootstrap: first signing call lazily inserts a row in `core.signingKey` with `isCurrent = true`, `privateKey = encrypt(pem_A)`. New tokens carry `kid_A`. 2. Rotate: `UPDATE core."signingKey" SET "isCurrent" = false, "privateKey" = NULL WHERE id = '<kid_A>';` then insert a new row with `isCurrent = true`. New tokens carry `kid_B`. Tokens still in flight with `kid_A` keep verifying because the public-key row for `kid_A` is still there. 3. Revoke: `UPDATE core."signingKey" SET "revokedAt" = now() WHERE id = '<kid_A>';`. All tokens with `kid_A` now fail verification cleanly with `UNAUTHENTICATED` (no 500). 4. Tokens with no `kid` (legacy) are unaffected throughout. ## Test plan - [x] Unit: `JwtWrapperService` dual-path verification (HS256 no-kid vs ES256 with-kid), unknown-kid → `UNAUTHENTICATED`, `signAsync` happy path + `null` when no key, `signAsync` rejection for non-rotatable types. - [x] Unit: `JwtAuthStrategy` `secretOrKeyProvider` dual-path resolution and algorithm validation. - [x] All existing JWT/auth/application unit tests adjusted to the renamed public method. - [x] Integration (`jwt-key-rotation.integration-spec.ts`): - **Happy path**: signed-up user's `ACCESS` token has `alg=ES256` + correct UUID `kid`, the `isCurrent=true` row exists in `core.signingKey`, `getCurrentUser` resolves. - **Legacy fallback**: hand-crafted no-kid HS256 token verifies via the legacy `APP_SECRET`-derived path. - **Previous-key rotation**: token signed by a hardcoded *previous* key whose row is pre-inserted with `privateKey = NULL` (rotated-out) still verifies — proves the leaked-key revocation flow works in both directions. - **Unknown kid**: token signed with an orphan UUID `kid` is cleanly rejected (no 500). - [x] `npx nx typecheck twenty-server` - [x] `npx nx test twenty-server` - [x] `npx nx run twenty-server:lint` |
||
|
Marie
|
4852ac401a |
Add server upgrade status on admin panel (#20107)
## Summary Adds an admin upgrade-status panel that surfaces per-instance and per-workspace migration health, backed by a Redis-cached aggregate to keep the page snappy on large fleets. <img width="827" height="880" alt="Screenshot 2026-04-28 at 10 21 03" src="https://github.com/user-attachments/assets/8f88baa9-7268-4eff-bf6a-906a7f06ca91" /> <img width="804" height="892" alt="Screenshot 2026-04-28 at 10 21 11" src="https://github.com/user-attachments/assets/1e6decf8-766a-4d0e-96b1-03a9962bba3c" /> ## Computed metrics **Instance** (`InstanceUpgradeStatus`) - `inferredVersion` — version derived from the latest non-initial instance command name - `health` — `upToDate` | `behind` | `failed`, derived from the latest attempt vs. the last expected instance step in the upgrade sequence - `latestCommand` — `{ name, status, executedByVersion, errorMessage, createdAt }` from the most recent attempt **Per-workspace** (`WorkspaceUpgradeStatus`) - `workspaceId`, `displayName` - `inferredVersion`, `health`, `latestCommand` (same shape as instance), computed against the latest expected step in the sequence **Aggregate** (`AllWorkspacesUpgradeStatus`, only across `ACTIVE` / `SUSPENDED` workspaces) - `instanceUpgradeStatus` - `totalCount`, `upToDateCount`, `behindCount`, `failedCount` - `workspacesBehindIds[]`, `workspacesFailedIds[]` - `computedAt` ## Fetching strategy All reads go through `UpgradeStatusCacheService` (cache namespace: `EngineHealth`). - **Aggregate read** (`getAllWorkspacesStatus` → `getAllWorkspacesUpgradeStatus` query): reads summary + behind-ids + failed-ids in parallel; if any of the three keys is missing, full recompute (`recomputeAllWorkspaces`) is triggered, which also primes per-workspace entries. - **Per-workspace read** (`getWorkspacesStatus(ids)` → `getUpgradeStatus(ids)` query): `mget` on workspace keys; misses are recomputed individually (`recomputeWorkspace`), and aggregates are reconciled in place (count + id list deltas) without a full recompute. - **Recompute on demand**: `refreshUpgradeStatus` mutation calls `recomputeAllWorkspaces` to bypass cache and rewrite all keys. - **Auto-invalidation**: `InstanceCommandRunnerService` (fast + slow paths) and `WorkspaceCommandRunnerService` invalidate after every run via `safeInvalidateUpgradeStatusCache()` (`flushByPattern('upgrade-status:*')`). Failures in cache invalidation are swallowed and logged so they never break the migration runner. - **TTL**: `60 * 60 * 1000` ms (1 hour) on every key — protects against stale data even if a runner crashes before invalidating. ## Introduced cache keys All under the `EngineHealth` cache-storage namespace: | Key | Type | Purpose | | --- | --- | --- | | `upgrade-status:all-workspaces:summary` | `CachedAllWorkspacesStatusSummary` | Counts + instance status + `computedAt` | | `upgrade-status:all-workspaces:behind-ids` | `string[]` | Workspace ids in `behind` state | | `upgrade-status:all-workspaces:failed-ids` | `string[]` | Workspace ids in `failed` state | | `upgrade-status:workspace:<workspaceId>` | `CachedWorkspaceUpgradeStatus` | Per-workspace status (one key per workspace) | Full invalidation uses the pattern `upgrade-status:*`. ## Index added on `upgradeMigration` (already added on prod) Migration `2-2-instance-command-fast-1777308014234-addUpgradeMigrationWorkspaceIdIndex.ts`: ```sql CREATE INDEX "IDX_upgradeMigration_workspaceId_name_attempt" ON "core"."upgradeMigration" ("workspaceId", "name", "attempt") WHERE "workspaceId" IS NOT NULL; |
||
|
Etienne
|
3306d66f5b | cleaning - remove logs (#19445) | ||
|
Etienne
|
579714b62f |
Investigate memory leak (#19213)
In search for cache leak + Remove old instrumentation |
||
|
Charles Bochet
|
c407341912 |
feat: optimize hot database queries with multi-layer caching (#19068)
## Summary Introduces multi-layer caching for the 5 most frequent database queries identified in production (Sentry data), targeting the JWT authentication hot path and cron job logic. ### Problem Our database is under heavy load from uncached queries on the auth hot path: - `WorkspaceEntity` lookups: **638 queries/min** - `ApiKeyEntity` lookups: **491 queries/min** - `UserEntity` lookups: **147 queries/min** - `UserWorkspaceEntity` lookups: **143 queries/min** - `LogicFunctionEntity` lookups: **1800 queries/min** (cron job) ### Solution **1. New `CoreEntityCacheService`** for non-workspace-scoped entities (Workspace, User, UserWorkspace): - Mirrors `WorkspaceCacheService` architecture (in-process Map + Redis with hash validation) - Provider pattern with `@CoreEntityCache` decorator - Keyed by entity primary key (not workspaceId) - 100ms local TTL, Redis-backed hash validation for cross-instance consistency - Three providers: `WorkspaceEntityCacheProviderService`, `UserEntityCacheProviderService`, `UserWorkspaceEntityCacheProviderService` **2. New `apiKeyMap` WorkspaceCache** for workspace-scoped API key lookups: - `WorkspaceApiKeyMapCacheService` loads all API keys for a workspace into a map by ID - Leverages existing `WorkspaceCacheService` infrastructure - Cache invalidation on API key create/update/revoke **3. `CronTriggerCronJob` refactored** to use existing `flatLogicFunctionMaps` workspace cache: - Eliminates per-workspace `LogicFunctionEntity` repository queries (~1800/min) - Filters cached data in-memory instead **4. `JwtAuthStrategy` refactored** to use caches for all entity lookups: - Workspace, User, UserWorkspace → `CoreEntityCacheService` - ApiKey → `WorkspaceCacheService` (`apiKeyMap`) - Impersonation queries kept as direct DB queries (rare path, requires relations) **5. Cache invalidation** wired into mutation paths: - `WorkspaceService` → invalidates `workspaceEntity` on save/update/delete - `ApiKeyService` → invalidates `apiKeyMap` on create/update/revoke ### Architecture ``` Request → JwtAuthStrategy ├── Workspace lookup → CoreEntityCacheService (in-process → Redis → DB) ├── User lookup → CoreEntityCacheService (in-process → Redis → DB) ├── UserWorkspace lookup → CoreEntityCacheService (in-process → Redis → DB) └── ApiKey lookup → WorkspaceCacheService (in-process → Redis → DB) CronTriggerCronJob └── LogicFunction lookup → WorkspaceCacheService (flatLogicFunctionMaps) ``` ### Expected Impact | Query | Before | After | |-------|--------|-------| | WorkspaceEntity | 638/min | ~0 (cached) | | ApiKeyEntity | 491/min | ~0 (cached) | | UserEntity | 147/min | ~0 (cached) | | UserWorkspaceEntity | 143/min | ~0 (cached) | | LogicFunctionEntity | 1800/min | ~0 (cached) | ### Not included (ongoing separately) - DataSourceEntity query optimization (IS_DATASOURCE_MIGRATED migration) - ObjectMetadataEntity query optimization (already partially cached) |
