Files
twenty/packages/twenty-server/src/instrument.ts
T
Félix Malfait 875795cc30 feat(sentry): propagate workspace context to all spans (#20064)
## Summary

After the 2026-04-26 token-usage incident, identifying the responsible
workspace from a Sentry trace required a Postgres scavenger hunt —
Vercel AI SDK auto-instrumentation captures token counts and model name
but no twenty-specific identifiers, and that same gap exists for every
other auto-instrumented span (HTTP outbound, Postgres queries, GraphQL
resolvers, Redis, etc.).

This PR plugs that gap globally, not just for AI:

- A small utility
(`packages/twenty-server/src/engine/core-modules/sentry/utils/sentry-workspace-context.util.ts`)
that writes workspace identifiers onto Sentry's active isolation scope
as a `twenty` context block plus filterable tags and a `Sentry.setUser`
call.
- Two hook points covering all server traffic:
- **`WorkspaceAuthContextMiddleware`** — already runs after token
hydration on the GraphQL, metadata, admin-panel, and REST routes. It now
calls the utility once per authenticated request, before delegating to
`withWorkspaceAuthContext`.
- **`BullMQDriver.work` and `SyncDriver.processJob`** — every queue job
now runs inside `Sentry.withIsolationScope` and applies workspace
context from `job.data.workspaceId` (skipping silently for system jobs
that don't carry one).
- A `beforeSendSpan` hook in `instrument.ts` that reads the scope's
`twenty` context block back and projects it onto every span as
`twenty.workspace.id` and (when available) `twenty.user_workspace.id` —
dotted-namespace naming consistent with OTel/Sentry conventions like
`user.id` and `http.response.status_code`. Spans without a workspace
context (unauthenticated traffic) pass through untouched.

## Why this shape

Sentry's docs position `beforeSendSpan` as a per-span hook. The previous
iteration set context only at AI-specific call sites, which left non-AI
spans (DB queries, outbound HTTP, regular GraphQL queries, workflow
steps not touching AI) entirely unenriched. Hooking the two existing
global boundaries — auth middleware for HTTP/GraphQL/REST, and the queue
driver `work()` callback for background jobs — covers every
authenticated span across the app with no per-handler instrumentation.

## What's not in this PR

AI-specific identifiers (`twenty.agent.id`, `twenty.thread.id`,
`twenty.turn.id`, `twenty.workflow_run.id`) are out of scope here.
They're useful additions but require either propagating the IDs through
the call stack or a more fine-grained scope (per-step, per-turn) than
the request/job boundary, which is best handled in follow-up PRs that
target those specific call sites.

## Test plan

- [ ] Make any authenticated GraphQL request locally and confirm the
resulting span(s) in Sentry carry `twenty.workspace.id` and (for
user-authenticated routes) `twenty.user_workspace.id`.
- [ ] Make any authenticated REST request and confirm the same.
- [ ] Trigger a queue job (chat stream, agent turn evaluation, workflow
run, etc.) and confirm spans produced inside the worker carry
`twenty.workspace.id`.
- [ ] Confirm that DB and outbound HTTP spans produced under the
request/job also carry the workspace tag — these previously had no
twenty-specific identifiers.
- [ ] In the Sentry UI, filter events by the `twenty.workspace.id` tag
and confirm matching events appear.

## Notes for review

- Sentry init lives in `instrument.ts`, loaded before Nest bootstraps,
so `beforeSendSpan` runs outside Nest DI and reads context off the
isolation scope rather than holding a service reference.
- The middleware change is three lines; the BullMQ wrap is a single
`Sentry.withIsolationScope` around the existing job handler body; the
SyncDriver wrap mirrors it for the dev/test path. No new modules or DI
providers.
- The previous iteration's `AiCallContextService` and per-handler
`setContext` / `withContext` calls have been removed in favor of these
two hooks.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-27 20:53:50 +02:00

106 lines
3.1 KiB
TypeScript

import process from 'process';
import opentelemetry from '@opentelemetry/api';
import { OTLPMetricExporter } from '@opentelemetry/exporter-metrics-otlp-http';
import { PrometheusExporter } from '@opentelemetry/exporter-prometheus';
import {
AggregationTemporality,
ConsoleMetricExporter,
MeterProvider,
PeriodicExportingMetricReader,
} from '@opentelemetry/sdk-metrics';
import * as Sentry from '@sentry/node';
import { nodeProfilingIntegration } from '@sentry/profiling-node';
import { NodeEnvironment } from 'src/engine/core-modules/twenty-config/interfaces/node-environment.interface';
import { ExceptionHandlerDriver } from 'src/engine/core-modules/exception-handler/interfaces';
import { MeterDriver } from 'src/engine/core-modules/metrics/types/meter-driver.type';
import { parseArrayEnvVar } from 'src/utils/parse-array-env-var';
const meterDrivers = parseArrayEnvVar(
process.env.METER_DRIVER,
Object.values(MeterDriver),
[],
);
if (process.env.EXCEPTION_HANDLER_DRIVER === ExceptionHandlerDriver.SENTRY) {
Sentry.init({
environment: process.env.SENTRY_ENVIRONMENT,
release: process.env.APP_VERSION,
dsn: process.env.SENTRY_DSN,
integrations: [
Sentry.redisIntegration(),
Sentry.httpIntegration(),
Sentry.expressIntegration(),
Sentry.graphqlIntegration(),
Sentry.postgresIntegration(),
Sentry.vercelAIIntegration({
recordInputs: true,
recordOutputs: true,
}),
nodeProfilingIntegration(),
],
tracesSampleRate: 0.1,
profilesSampleRate: 0.3,
sendDefaultPii: true,
debug: process.env.NODE_ENV === NodeEnvironment.DEVELOPMENT,
beforeSendSpan: (span) => {
const twentyContext = Sentry.getIsolationScope().getScopeData().contexts
?.twenty as
| {
workspace_id?: string;
user_workspace_id?: string;
}
| undefined;
if (!twentyContext?.workspace_id) {
return span;
}
span.data = {
...span.data,
'twenty.workspace.id': twentyContext.workspace_id,
...(twentyContext.user_workspace_id && {
'twenty.user_workspace.id': twentyContext.user_workspace_id,
}),
};
return span;
},
});
}
// Meter setup
const prometheusExporter = meterDrivers.includes(MeterDriver.Prometheus)
? new PrometheusExporter({ port: 9464 })
: null;
const meterProvider = new MeterProvider({
readers: [
...(meterDrivers.includes(MeterDriver.Console)
? [
new PeriodicExportingMetricReader({
exporter: new ConsoleMetricExporter(),
exportIntervalMillis: 10000,
}),
]
: []),
...(meterDrivers.includes(MeterDriver.OpenTelemetry)
? [
new PeriodicExportingMetricReader({
exporter: new OTLPMetricExporter({
url: process.env.OTLP_COLLECTOR_METRICS_ENDPOINT_URL,
temporalityPreference: AggregationTemporality.DELTA,
}),
exportIntervalMillis: 10000,
}),
]
: []),
...(prometheusExporter ? [prometheusExporter] : []),
],
});
opentelemetry.metrics.setGlobalMeterProvider(meterProvider);