47689e676bbac1f4e7aced1e3a0bb81da12ac565
11 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
ce2d77be2a |
feat(server): in-app server-level admin management (#19785) (#21321)
## Closes #19785 In-app management of **server-level admin rights** (`canAccessFullAdminPanel`, `canImpersonate`) so self-hosters no longer need raw SQL + a Redis flush + restart to grant access. > **Draft** — feature complete; `/code-review` + `/security-review` run and addressed. ### Background `AdminPanelGuard` / `ServerLevelImpersonateGuard` read `request.user.{canAccessFullAdminPanel,canImpersonate}`, hydrated each request from `CoreEntityCacheService.get('user', …)` (local 30-min + Redis no-TTL). The cache was only invalidated on soft-delete, so a raw `UPDATE core."user"` never took effect. The **first** signup auto-gets both flags; every subsequent admin previously needed raw SQL. ### UX - **Admin Panel → General → Administrators**: a read-only overview of every user with server-level access; each row links to that user's admin page. - **Find anyone** via the user search (Recent Users) — available to full admins and impersonators — then open their **admin user page**. - On the user page, an **"Administrator access"** card (gated on `canAccessFullAdminPanel`) has two toggles — *Full admin panel access* and *Impersonation* — that work for **any** user (a user with no access shows both off). Mirrors how **Impersonate** already works (find user → user page → act). Each change opens a confirm dialog with a **2FA code** field; the last full admin's toggle is disabled. ### Backend / security - **Cache fix** — invalidate the user entity cache on committed user updates (not just soft-delete) so privilege changes propagate (~100 ms, cluster-wide) with no restart. - `getServerAdmins` query + `updateServerAdminAccess` mutation (any `targetUserId`), gated on `canAccessFullAdminPanel`. - `NoImpersonationGuard` on both — an impersonated full-admin session can't be used to escalate an impersonator. - Fresh **2FA TOTP step-up** (enrolled+verified method **and** a fresh code; genuine 2FA errors surface; dev-skip on trusted `NODE_ENV`). - **Last-admin lockout** in a transaction with a pessimistic row lock (no TOCTOU). - **Email-to-all-admins + affected user** (rendered once per locale), structured log, audit event-log emit. - **Authorization**: the read-only `userLookupAdminPanel` + `adminPanelRecentUsers` lookups now accept `canAccessFullAdminPanel OR canImpersonate` (new `AdminPanelOrImpersonateGuard`), so a full admin without impersonate can still find users to manage. Workspace/impersonation queries stay impersonate-gated. ### Reviews - `/code-review` (max effort): 3 security findings (impersonation-escalation sink, lockout TOCTOU, step-up accepting PENDING 2FA) — **all fixed**. `/simplify`: applied. `/security-review`: **no high/medium vulnerabilities**. ### Follow-ups (not in this PR) - Unit tests for `AdminPanelServerAdminService` + a frontend test. - Point the self-host troubleshooting docs at the new UI. - OTP retry UX: `ConfirmationModal` closes on confirm, so a wrong code needs a reopen (kept to reuse the existing modal; no new pattern). ### Notes for reviewers - `generated-admin/graphql.ts` entries were hand-added to match codegen output (admin codegen needs a running server); re-run `nx graphql:generate twenty-front --configuration=admin` to confirm parity. - First-admin bootstrap (first signup) is unchanged. --------- Co-authored-by: claude[bot] <41898282+claude[bot]@users.noreply.github.com> |
||
|
|
91f2f08995 |
feat(server): unify workspace-event ingestion behind one EventSink pipeline (#21197)
## Why The five event-log streams (`workspaceEvent`, `pageview`, `objectEvent`, `usageEvent`, `applicationLog`) each wrote to ClickHouse through their own fire-and-forget writer (`AuditService`, `UsageEventWriterService`, and the `application-logs` driver), with the per-type knowledge (table names, normalization, access rules) spread across several modules. Three of them reimplemented the same ClickHouse insert, and the read side, the live stream, and the producers lived in different modules under two different names. This consolidates them into one `core-modules/event-logs/` subsystem (emit, write, live, read), with the per-type config in a single registry so adding an event type is roughly one file. The base Logs settings tab and free application logs shipped separately in #21180 (merged). This PR adds the unified backend, the registry, and the viewer's live mode and entitlement gating. ## Pipeline ```mermaid flowchart TB subgraph PROD["Producers"] A["auth, billing, impersonation,<br/>webhook, custom-domain"] U["usage listener"] F["logic-function executor (app logs)"] R["record CRUD (entity events)"] end EM["EventLogEmitterService<br/>createContext().insert* / dispatch()"] EQ(["entityEventsToDbQueue<br/>(existing, shared with timeline)"]) CIE["CreateEventLogFromInternalEvent"] SINK["WorkspaceEventSinkService.ingest()"] C1["ClickHouseEventSink"] C2["ConsoleEventSink"] LIVE["EventLogLiveService.publishWatched()<br/>(presence-gated)"] CH[("ClickHouse, 5 tables, async_insert")] CHAN(["WORKSPACE_EVENTS_CHANNEL"]) RS["EventLogsService (registry-driven read)"] LR["EventLogsLiveResolver"] UI["Settings > Logs"] A --> EM U --> EM F --> EM EM -->|direct| SINK R --> EQ --> CIE -->|ingest| SINK SINK --> C1 --> CH SINK --> C2 SINK --> LIVE -.->|if a viewer is watching| CHAN --> LR --> UI CH --> RS --> UI ``` ## What it does - Producers call `EventLogEmitterService.createContext().insert*()`, which builds a typed `WorkspaceEventEnvelope` and writes it through `WorkspaceEventSinkService` to the configured sinks (ClickHouse, Console) plus a presence-gated live fan-out. Record/CRUD events reach the same sink through the existing `entityEventsToDbQueue`. There is no dedicated queue; ClickHouse `async_insert` batches server-side. Writes are best-effort, as on main today. - `EVENT_LOG_TYPES[table]` is the per-type source of truth: the ClickHouse table, the required entitlement, the free-text filter column, and the row-to-GraphQL mapping. Read row shapes derive from the write rows. - Four modules along their dependency boundaries: `EventLogEmitterModule` (producer API), `EventLogIngestionModule` (sink layer), `EventLogLiveModule` (fan-out), and `EventLogsViewerModule` (the entitlement-gated GraphQL read, which is where billing/enterprise/permissions stay so producers stay light). - Logs viewer: per-table columns, filters (text, date, record), live mode, and an upgrade card that points to Billing on Cloud or the Admin Panel on self-hosted. Application logs are free on every plan; the other four require the `AUDIT_LOGS` entitlement (with a `NO_ENTITLEMENT` fallback to the upgrade card). - Renames `AuditService` to `EventLogEmitterService`, and the generic `Monitoring` event to a typed `Impersonation` event (`level` + `action`). - Removes `UsageEventWriterService`, the `application-logs` driver/module, and `AuditService`'s direct inserts. ## Durability Writes are best-effort, the same as main today (the old writers were fire-and-forget). A dedicated queue was tried mid-PR and removed: `async_insert` already batches server-side, so the queue only added durability, which isn't a requirement right now. The `EventSink` seam keeps a durable transport (e.g. a Redis-Streams buffer) easy to add later without touching producers. ## Out of scope S3 peer sink (seam only), Postgres or any second read path, `ReplicatedMergeTree`, ClickHouse table-schema changes, and the record-data `EVENT_STREAM_CHANNEL` (unchanged, separate concern). ## Testing Unit tests cover the registry definitions and row normalization, the entitlement gating, the envelope builders, and the producers. Integration tests cover the write paths (record create produces an `objectEvent`; the track mutation produces a `workspaceEvent`) and the read/query path across all five tables. Verified with typecheck, lint, a server boot, and GraphQL/SDK codegen. |
||
|
|
2ac515894b |
feat(settings): add Logs as a dedicated tab in General settings (#21180)
## What & why The audit-log viewer lived as a full-screen page reachable only via a "View Logs" button buried in the **Security** tab. This surfaces it as the **third tab in General settings** (`General | Security | Logs`), consistent with the other tabs. ## Changes - **Relocated** the event-logs module `pages/settings/security/event-logs/` → `modules/settings/event-logs/` and render it as tab content instead of a `FullScreenContainer` page. Dropped `SettingsPath.EventLogs`, its route, and the fullscreen handling in favor of the `general#logs` hash tab. - **Security tab:** removed the "View Logs" entry; kept the log-retention setting there. - **In-tab gating** (shown to users with the Security permission): Enterprise upgrade card when not entitled, a clear "ClickHouse not configured" placeholder otherwise (derived from client config), and the query is skipped when disabled. Replaces a bespoke error component that string-matched error messages with the shared `SettingsEmptyPlaceholder` / `SettingsEnterpriseFeatureGateCard`. - **Layout:** boxed content column with the table selector + filters grouped in a `Card` and the results table below, matching settings conventions. Kept the existing fixed filters (page/event name, member, period) rather than recreating the record-view filter chips (those are tightly coupled to record/view context). Frontend + `twenty-shared` only — no changes to the log query or data. ## Test plan - [x] `npx nx typecheck twenty-front` and `npx nx lint twenty-front` pass - [x] Settings → General shows three tabs; Logs is the third; breadcrumb stays "Workspace / General" - [x] With Enterprise + ClickHouse: table selector, filters, refresh, and the paginated table work - [x] Non-Enterprise: Enterprise upgrade card shown; no failing query fires - [ ] Enterprise without ClickHouse: shows the "ClickHouse not configured" placeholder - [ ] Security tab still shows the log-retention setting and the "View Logs" button is gone - [ ] A user without the Security permission sees neither the Security nor Logs tab |
||
|
|
fbaea0639a |
Billing - optimize usageEvent CH table (#20019)
- Update usageEvent clickhouse table, partitioning, indexing and projection (auto materialized view) to optimize credit usage queries - Add caching for available credits and billing subscription To do in next PR: deprecate enforceCapUsage cron. Bonus : real-time on billingSubscription |
||
|
|
36fbfca069 |
Add application-logs module with driver pattern for logic function log persistence (#19486)
## Summary
- Introduces a new `application-logs` core module with a driver pattern
(disabled/console/clickhouse) to capture and persist logic function
execution logs
- Adds a ClickHouse `applicationLog` table with per-line log storage,
30-day TTL, and `ORDER BY (workspaceId, timestamp, applicationId,
logicFunctionId)`
- Surfaces application logs in the existing frontend audit logs table as
a new "Application Logs" source with dedicated columns (Function,
Timestamp, Level, Message, Execution ID)
## Details
**Write path**: `LogicFunctionExecutorService.handleExecutionResult()`
parses the multi-line log string from driver output into individual `{
timestamp, level, message }` entries, generates an execution UUID, and
passes them to `ApplicationLogsService.writeLogs()` which delegates to
the configured driver.
**Driver pattern**: Follows the exception-handler module style (Symbol
injection token + `forRootAsync` dynamic module). Three drivers:
- `DISABLED` (default) — no-op, prevents information leaking
- `CONSOLE` — structured stdout logging with level-based `console.*`
calls
- `CLICKHOUSE` — inserts rows into the `applicationLog` ClickHouse table
**Read path**: Extends the existing event-logs module by adding
`APPLICATION_LOG` to the `EventLogTable` enum, table name mapping, and
normalization logic.
**Config**: New `APPLICATION_LOG_DRIVER_TYPE` environment variable
(default: `DISABLED`).
|
||
|
|
77d4bd9158 |
Add billing usage analytics dashboard with ClickHouse integration (#18592)
## Summary This PR adds a comprehensive billing usage analytics feature that provides detailed breakdowns of credit consumption across execution types, users, resources, and time periods. The implementation includes a new ClickHouse-backed analytics service, GraphQL API endpoint, and a frontend dashboard component. ## Key Changes ### Backend - **New BillingAnalyticsService**: Queries ClickHouse for usage breakdowns by user, resource, execution type, and time series data - **BillingEventWriterService**: Writes billing events to ClickHouse for analytics while maintaining best-effort semantics (never blocks Stripe billing) - **ClickHouse Schema**: Added `billingEvent` table with 3-year TTL for storing detailed billing event data - **GraphQL Resolver**: New `getBillingAnalytics` query that aggregates usage data for the current billing period, protected by feature flag and billing permissions - **Enhanced BillingUsageEvent**: Added `userWorkspaceId` field to track per-user credit consumption - **AI Billing Integration**: Updated AI billing service to pass `userWorkspaceId` when recording usage events ### Frontend - **SettingsBillingAnalyticsSection**: New component displaying: - Usage breakdown by execution type with progress bars - Daily usage time series chart (28-day view) - Per-user credit consumption breakdown - Per-resource (agent/workflow) credit consumption breakdown - **SettingsUsage Page**: Dedicated page for viewing usage analytics - **GraphQL Query**: `GetBillingAnalytics` query with generated hooks - **Navigation**: Added Usage menu item in settings (feature-flagged) - **Mock Data**: Included screenshot mock data for preview/testing ### Feature Flag - Added `IS_USAGE_ANALYTICS_ENABLED` feature flag to control visibility and access to analytics features ## Implementation Details - Analytics data is queried in parallel for performance - ClickHouse writes are non-blocking to ensure billing operations never fail - Progress bars use dynamic coloring from a predefined palette - Time series visualization normalizes bar heights relative to max value - Empty state handling when no analytics data is available - Responsive UI with proper text truncation for long names https://claude.ai/code/session_01Y1EqrX6PFq3EJxJq89h7DF --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
|
c1da7be6d7 |
Billing for self-hosts (#18075)
## Summary Implements enterprise licensing and per-seat billing for self-hosted environments, with Stripe as the single source of truth for subscription data. ### Components - **twenty-website** hosts the private key to sign `ENTERPRISE_KEY` and `ENTERPRISE_VALIDITY_TOKEN`. It communicates with Stripe to emit the daily `ENTERPRISE_VALIDITY_TOKEN` if the subscription is active, based on the user's Stripe subscription ID stored in `ENTERPRISE_KEY`. - **Stripe** is the single source of truth for subscription data (status, seats, billing). - **The client** (twenty-server + DB + workers) saves `ENTERPRISE_KEY` in the `keyValuePair` table (or `.env` if `IS_CONFIG_VARIABLES_IN_DB_ENABLED` is false) and the daily-renewed `ENTERPRISE_VALIDITY_TOKEN` in the `appToken` table. `ENTERPRISE_VALIDITY_TOKEN` is verified client-side using a public key to grant access to enterprise features (RLS, SSO, audit logs, etc.). ### Flow 1. When requesting an upgrade to an enterprise plan (from **Enterprise** in settings), the user is shown a modal to choose monthly/yearly billing, then redirected to Stripe to enter payment details. After checkout, they land on twenty-website where they are exposed to their `ENTERPRISE_KEY`, which they paste in the UI. It is saved in the `keyValuePair` table. On activation, a first `ENTERPRISE_VALIDITY_TOKEN` with 30-day validity is stored in the `appToken` table. 2. **Every day**, a cron job runs and does two things: - **Refreshes the validity token**: communicates with twenty-website to get a new `ENTERPRISE_VALIDITY_TOKEN` with 30-day validity if the Stripe subscription is still active. If the subscription is in cancellation, the emitted token has a validity equal to the cancellation date. If it's no longer valid, the token is not replaced. The cron only needs to run every 30 days in practice, but runs daily so it's resilient to occasional failures. - **Reports seat count**: counts active (non-soft-deleted) `UserWorkspace` entries and sends the count to twenty-website, which updates the Stripe subscription quantity with proration. Seats are also reported on first activation. If the subscription is canceled or scheduled for cancellation, the seat update is skipped. 3. `ENTERPRISE_VALIDITY_TOKEN` is verified server-side via a public key to grant access to enterprise features. ### Key concepts Three distinct checks are exposed as GraphQL fields on `Workspace`: | Field | Meaning | |---|---| | `hasValidEnterpriseKey` | Has any valid enterprise key (signed JWT **or** legacy plain string) | | `hasValidSignedEnterpriseKey` | `ENTERPRISE_KEY` is a properly signed JWT (billing portal makes sense) | | `hasValidEnterpriseValidityToken` | `ENTERPRISE_VALIDITY_TOKEN` is present and not expired (expiration depends on signed token payload, not on "expiresAt" on appToken table which is only indicative) | Feature access is gated by `isValid()` = `hasValidEnterpriseValidityToken || hasValidEnterpriseKey` (to support both new and legacy keys during transition). After transition isValid() = hasValidEnterpriseValidityToken ### Frontend states The Enterprise settings page handles multiple states: - **No key**: show "Get Enterprise" with checkout modal - **Orphaned validity token** (token valid but no signed key): prompt user to set a valid enterprise key - **Active/trialing but no validity token**: show subscription status with a "Reload validity token" action - **Active/trialing**: show full subscription info, billing portal access, cancel option - **Cancellation scheduled**: show cancellation date, billing portal - **Canceled**: show billing history link and option to start a new subscription - **Past due / Incomplete**: prompt to update payment or restart ### Temporary retro-compatibility: legacy plain-text keys Previously, enterprise features were gated by a simple check: any non-empty string in `ENTERPRISE_KEY` granted access. With this PR, we transition to a controlled system relying on signed JWTs. To avoid breaking existing self-hosted users: - **Legacy plain-text keys still grant access** to enterprise features. `hasValidEnterpriseKey` returns `true` for both signed JWTs and plain strings, and `isValid()` checks `hasValidEnterpriseKey` as a fallback when no validity token is present. - **A deprecation banner** is shown at the top of the app when `hasValidEnterpriseKey` is `true` but `hasValidSignedEnterpriseKey` is `false`, informing the user that their key format is deprecated and they should activate a new signed key. - **No billing portal or subscription management** is available for legacy keys since there is no Stripe subscription to manage. This retro-compatibility will be removed in a future version. At that point, `isValid()` will only check `hasValidEnterpriseValidityToken`. ### Edge cases - **Air-gapped / production environments**: for self-hosted clients that block external traffic (or for our own production), provide a long-lived `ENTERPRISE_VALIDITY_TOKEN` (e.g. 99 years) directly in the `appToken` table, with no `ENTERPRISE_KEY`. The daily cron will skip the refresh (no enterprise key to authenticate with), but the pre-seeded validity token will be used to grant feature access. No billing or seat reporting occurs in this mode. - **`IS_CONFIG_VARIABLES_IN_DB_ENABLED` is false**: if the user tries to activate an enterprise key but DB config writes are disabled, the backend returns a clear error asking them to add `ENTERPRISE_KEY` to their `.env` file manually. - **Canceled subscriptions**: the `/seats` endpoint skips Stripe updates for canceled or cancellation-scheduled subscriptions to avoid Stripe API errors. ### How to test - launch twenty-website on a different url (eg localhost:1002) - add ENTERPRISE_API_URL=http://localhost:3002/api/enterprise (or else) in your server .env - ask me for twenty-website's .env file content (STRIPE_SECRET_KEY; STRIPE_ENTERPRISE_MONTHLY_PRICE_ID;STRIPE_ENTERPRISE_YEARLY_PRICE_ID; ENTERPRISE_JWT_PRIVATE_KEY; ENTERPRISE_JWT_PUBLIC_KEY; NEXT_PUBLIC_WEBSITE_URL) - visit Admin panel / enterprise |
||
|
|
012d819557 |
OAuth Client — Unified ApplicationRegistration, OAuth server, and frontend (#18267)
## Summary Consolidates three separate PRs (#18260, #18261, #18262) into a single unified branch with all review feedback addressed: ### New features - **ApplicationRegistration entity** — server-level registration for OAuth apps with encrypted server variables - **OAuth 2.0 server** — authorization code, client credentials, refresh token grants with PKCE support - **OAuth discovery endpoint** — `.well-known/oauth-authorization-server` metadata - **Frontend UI** — app registration details page with credential management, redirect URI editing, and server variable configuration - **CLI integration** — `twenty dev` auto-registers apps and stores OAuth credentials locally - **Authorize consent screen** — OAuth consent page at `/authorize` showing requested scopes ### Review feedback addressed **Renames (PR #18260):** - `appRegistration` → `applicationRegistration` (entity, tables, files, imports, GraphQL types) - `appRegistrationVariable` → `applicationRegistrationVariable` - `clientId` → `oAuthClientId`, `clientSecretHash` → `oAuthClientSecretHash`, `redirectUris` → `oAuthRedirectUris`, `scopes` → `oAuthScopes` **Security fixes (PR #18261):** - Fixed redirect URI validation bypass when `oAuthRedirectUris` is an empty array - Fixed workspace isolation in `clientCredentialsGrant` — now uses `find()` with explicit handling for multiple installations - Added error logging in refresh token `catch` block instead of silently swallowing **Code quality (PR #18262):** - Split `VersionDistributionEntry` into its own file (one export per file) - Split GraphQL queries and mutations into individual files with a shared fragment - Removed unused `OAuth` entry from `AuthProviderEnum` - Added loading state to `handleRotateSecret` - Removed 27 narration-style comments from test files - Added proper guards (`PublicEndpointGuard`, `NoPermissionGuard`) to controllers and resolvers ## Test plan - [ ] Verify `twenty dev` registers an app and stores OAuth credentials - [ ] Test OAuth authorization code flow end-to-end (authorize → token → API call) - [ ] Test client credentials grant - [ ] Verify redirect URI validation rejects requests when no URIs are registered - [ ] Verify app registration detail page renders correctly - [ ] Test secret rotation with loading state - [ ] Verify server variable editing and saving - [ ] Run `npx nx database:reset twenty-server` to validate migration Closes #18260, #18261, #18262 Made with [Cursor](https://cursor.com) --------- Co-authored-by: claude[bot] <41898282+claude[bot]@users.noreply.github.com> |
||
|
|
cfad24da48 |
Fix empty user id clickhouse (#18238)
- Fixes: - Make Workspace User select work; previously, it didn't work as we were not fetching the workspace users correctly - Send Object Events with valid record id and object id ## Audit logs demo https://github.com/user-attachments/assets/92437037-d253-4810-a138-7c709550755d |
||
|
|
9e21e55db4 |
Prevent leak between /metadata and /graphql GQL schemas (#17845)
## Fix resolver schema leaking between `/metadata` and `/graphql` endpoints ### Summary - Patch `@nestjs/graphql` to support a `resolverSchemaScope` option that filters resolvers at both schema generation and runtime, preventing cross-endpoint leaking - Introduce `@CoreResolver()` and `@MetadataResolver()` decorators to explicitly scope each resolver to its endpoint - Move most resolvers (auth, billing, workspace, user, etc.) to the metadata schema where the frontend expects them; only workflow and timeline calendar/messaging resolvers remain on `/graphql` - Fix frontend `SSEQuerySubscribeEffect` to use the default (metadata) Apollo client instead of the core client ### Problem NestJS GraphQL's module-based resolver discovery traverses transitive imports, causing resolvers from `/metadata` modules to leak into the `/graphql` schema and vice versa. This made the schemas unpredictable and tightly coupled to module import order. ### Approach - Added `resolverSchemaScope` to `GqlModuleOptions` via a patch on `@nestjs/graphql`, filtering in both `filterResolvers()` (runtime binding) and `getAllCtors()` (schema generation) - Each resolver is explicitly decorated with `@CoreResolver()` or `@MetadataResolver()` - Organized decorator, constant, and type files under `graphql-config/` following project conventions Core GQL Schema: (see: no more fields!) <img width="827" height="894" alt="image" src="https://github.com/user-attachments/assets/668f3f0f-485e-43f0-92be-4345aeccacb6" /> Metadata GQL Schema (see no more getTimelineCalendarEventsFromCompany) <img width="827" height="894" alt="image" src="https://github.com/user-attachments/assets/443913db-e5fe-4161-b0e7-4a971cc80a71" /> |
||
|
|
177cae8c53 | feat: audit Logs (#17660) |