## Summary
**All OTel metrics in twenty-server have been silently dropped since
April 30.**
### Root cause
PR #20149 (`bump @sentry/profiling-node 10.27→10.51`) pulled in
`@sentry/node@10.51.0`, which declares `@opentelemetry/api: ^1.9.1` as a
**dependency** (not peer). Yarn installed it as a **nested** copy at
`1.9.1`, while the hoisted copy stayed at `1.9.0`.
At startup in `instrument.ts`:
1. `Sentry.init()` uses the **nested `1.9.1`** to register `trace`,
`propagation`, `context` on the OTel global → global version becomes
**`1.9.1`**
2. `setGlobalMeterProvider()` uses the **hoisted `1.9.0`** →
`registerGlobal` sees version mismatch (`1.9.1` ≠ `1.9.0`) → **silently
returns `false`**
3. Global stays `NoopMeterProvider` → every counter, gauge, and
histogram in the server is a no-op
### What this PR does
1. **Reverts three troubleshooting PRs** that are no longer needed now
that the root cause is identified:
- #20230 — heartbeat gauge
- #20228 — OTLP export lifecycle logs
- #20221 — Sentry revert to 10.27 (which never actually downgraded in
`yarn.lock` since `^10.27.0` resolved to `10.51.0`)
2. **Fixes the root cause**:
- Root Yarn resolution pinning `@opentelemetry/api` to `1.9.1` → single
copy in the entire tree, Sentry and Twenty share the same instance
- Named import in `instrument.ts` (`import { metrics as otelMetrics }`
instead of default import) as defense-in-depth against CJS interop
issues
### Verified on dev cluster
Exec'd into the running pod and confirmed:
- `@sentry/node` nests `@opentelemetry/api@1.9.1`, hoisted is `1.9.0`
- `Sentry.init()` → global version `1.9.1` → `setGlobalMeterProvider`
with VERSION `1.9.0` → returns `false` → `NoopMeterProvider`
- Same-version registration returns `true` → `MeterProvider` ✓
## Test plan
- [ ] CI passes (lint, typecheck, build)
- [ ] Deploy to dev cluster and verify metrics flow to collector
- [ ] Confirm `node_modules/@opentelemetry/api/package.json` shows
`1.9.1` with no nested copy under `@sentry/`
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
## Summary
- Adds a trivial always-on observable gauge (`twenty.heartbeat = 1`) so
the OTLP exporter fires on every 10s collection tick, even on idle pods.
Without this, `PeriodicExportingMetricReader` skips the export when
`scopeMetrics` is empty, so the process-log wrapper never runs and we
can't prove OTLP connectivity.
- Adds a one-time "first periodic export attempt" log line inside the
wrapped exporter, completing the startup log sequence: `OTLP reader
enabled` → `first periodic export attempt` → `first export ok` / `export
failed`.
Co-authored-by: Cursor <cursoragent@cursor.com>
## Summary
Adds **grep-friendly** `console` logging around the OpenTelemetry
metrics OTLP exporter in
[`packages/twenty-server/src/instrument.ts`](packages/twenty-server/src/instrument.ts)
so production / staging can confirm whether the app is exporting metrics
and why exports fail.
## Log format
- Prefix: **`[Twenty OTEL metrics]`** (easy to filter in Loki / `kubectl
logs | grep`).
- **Startup:** whether the OTLP reader is enabled, `exportIntervalMs`,
and endpoint as `protocol//host/path` only (no credentials).
- **First successful export:** one `console.log` per process (`first
export ok`) with metric data point count — avoids spamming every 10s.
- **Each failed export:** `console.warn` with result code, point count,
and serialized error (including nested `AggregateError` causes when
present).
Co-authored-by: Cursor <cursoragent@cursor.com>
## Summary
Reverts **#20064** (`feat(sentry): propagate workspace context to all
spans`) and downgrades **@sentry** packages from **10.51** back to
**10.27** (reversing **#20149**), to validate in production whether
recent Sentry/instrumentation changes correlate with OTLP/metrics
issues.
## Changes
1. **Revert #20064** — removes `beforeSendSpan` from `instrument.ts`,
restores `WorkspaceAuthContextMiddleware` / `BullMQDriver` behavior, and
deletes the three `apply-workspace-sentry-*` utils added in that PR.
2. **Sentry versions** — `packages/twenty-server` (`@sentry/nestjs`,
`@sentry/node`, `@sentry/profiling-node`) and `packages/twenty-front`
(`@sentry/react`) set to `^10.27.0`; `yarn.lock` regenerated via `yarn
install`.
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
## Summary
After the 2026-04-26 token-usage incident, identifying the responsible
workspace from a Sentry trace required a Postgres scavenger hunt —
Vercel AI SDK auto-instrumentation captures token counts and model name
but no twenty-specific identifiers, and that same gap exists for every
other auto-instrumented span (HTTP outbound, Postgres queries, GraphQL
resolvers, Redis, etc.).
This PR plugs that gap globally, not just for AI:
- A small utility
(`packages/twenty-server/src/engine/core-modules/sentry/utils/sentry-workspace-context.util.ts`)
that writes workspace identifiers onto Sentry's active isolation scope
as a `twenty` context block plus filterable tags and a `Sentry.setUser`
call.
- Two hook points covering all server traffic:
- **`WorkspaceAuthContextMiddleware`** — already runs after token
hydration on the GraphQL, metadata, admin-panel, and REST routes. It now
calls the utility once per authenticated request, before delegating to
`withWorkspaceAuthContext`.
- **`BullMQDriver.work` and `SyncDriver.processJob`** — every queue job
now runs inside `Sentry.withIsolationScope` and applies workspace
context from `job.data.workspaceId` (skipping silently for system jobs
that don't carry one).
- A `beforeSendSpan` hook in `instrument.ts` that reads the scope's
`twenty` context block back and projects it onto every span as
`twenty.workspace.id` and (when available) `twenty.user_workspace.id` —
dotted-namespace naming consistent with OTel/Sentry conventions like
`user.id` and `http.response.status_code`. Spans without a workspace
context (unauthenticated traffic) pass through untouched.
## Why this shape
Sentry's docs position `beforeSendSpan` as a per-span hook. The previous
iteration set context only at AI-specific call sites, which left non-AI
spans (DB queries, outbound HTTP, regular GraphQL queries, workflow
steps not touching AI) entirely unenriched. Hooking the two existing
global boundaries — auth middleware for HTTP/GraphQL/REST, and the queue
driver `work()` callback for background jobs — covers every
authenticated span across the app with no per-handler instrumentation.
## What's not in this PR
AI-specific identifiers (`twenty.agent.id`, `twenty.thread.id`,
`twenty.turn.id`, `twenty.workflow_run.id`) are out of scope here.
They're useful additions but require either propagating the IDs through
the call stack or a more fine-grained scope (per-step, per-turn) than
the request/job boundary, which is best handled in follow-up PRs that
target those specific call sites.
## Test plan
- [ ] Make any authenticated GraphQL request locally and confirm the
resulting span(s) in Sentry carry `twenty.workspace.id` and (for
user-authenticated routes) `twenty.user_workspace.id`.
- [ ] Make any authenticated REST request and confirm the same.
- [ ] Trigger a queue job (chat stream, agent turn evaluation, workflow
run, etc.) and confirm spans produced inside the worker carry
`twenty.workspace.id`.
- [ ] Confirm that DB and outbound HTTP spans produced under the
request/job also carry the workspace tag — these previously had no
twenty-specific identifiers.
- [ ] In the Sentry UI, filter events by the `twenty.workspace.id` tag
and confirm matching events appear.
## Notes for review
- Sentry init lives in `instrument.ts`, loaded before Nest bootstraps,
so `beforeSendSpan` runs outside Nest DI and reads context off the
isolation scope rather than holding a service reference.
- The middleware change is three lines; the BullMQ wrap is a single
`Sentry.withIsolationScope` around the existing job handler body; the
SyncDriver wrap mirrors it for the dev/test path. No new modules or DI
providers.
- The previous iteration's `AiCallContextService` and per-handler
`setContext` / `withContext` calls have been removed in favor of these
two hooks.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This PR implements Sentry's AI agent monitoring by:
- Configuring vercelAIIntegration with recordInputs and recordOutputs
options
- Adding sendDefaultPii to Sentry.init() for better debugging
- Creating a shared AI_TELEMETRY_CONFIG constant to DRY up the telemetry
configuration
- Adding experimental_telemetry to all AI SDK calls (generateText,
generateObject, streamText)
All AI operations are now fully monitored in Sentry with complete
input/output recording for debugging and performance analysis.
Note: Currently on Sentry v9.26.0, which is compatible with this
implementation. No breaking changes from the v9-to-v10 migration guide
were found in the codebase.
Following https://github.com/twentyhq/twenty/pull/14869
Dynamically clearing the cache, as one entry was forgotten
It seems like a permission test deletes a role that's expected to be
existing in following tests, as now cache is getting invalidated tests
are failing
Seems to be related to
https://discord.com/channels/1130383047699738754/1423768505911869460
## Singleton local cache key collision
When set for the first time caches local keys looks like
`workspaceId:undefined", singleton is shared between several cache
instances
If the given key is undefined it will read on other entity cache entry.
Let's deprecate Sentry Release and use APP_VERSION instead.
It'll make it more clear in the interface to use named version for bug
analysis, than commit sha
Done :
- move metrics and health cache services from health module to metrics
module
- refactor metrics counter from specific method to set up from enum keys
- add OpenTelemetry (Otel) instrumentation for metrics
- set up Otel SDK to send metrics to Otel collector
To do later :
- implement Otel instrumentation for traces + plug Sentry on top
- Improve snackbar to enable displaying multi-line message (so far we
only displayed the first few words which was very frustrating)
- Followup on previous issue to enable tim@apple.dev on the demo
workspace (prefilled automatically)
- Fix sentry tracing which had been broken when migrating from v7 to v8
Charles Bochet