875795cc30
## Summary After the 2026-04-26 token-usage incident, identifying the responsible workspace from a Sentry trace required a Postgres scavenger hunt — Vercel AI SDK auto-instrumentation captures token counts and model name but no twenty-specific identifiers, and that same gap exists for every other auto-instrumented span (HTTP outbound, Postgres queries, GraphQL resolvers, Redis, etc.). This PR plugs that gap globally, not just for AI: - A small utility (`packages/twenty-server/src/engine/core-modules/sentry/utils/sentry-workspace-context.util.ts`) that writes workspace identifiers onto Sentry's active isolation scope as a `twenty` context block plus filterable tags and a `Sentry.setUser` call. - Two hook points covering all server traffic: - **`WorkspaceAuthContextMiddleware`** — already runs after token hydration on the GraphQL, metadata, admin-panel, and REST routes. It now calls the utility once per authenticated request, before delegating to `withWorkspaceAuthContext`. - **`BullMQDriver.work` and `SyncDriver.processJob`** — every queue job now runs inside `Sentry.withIsolationScope` and applies workspace context from `job.data.workspaceId` (skipping silently for system jobs that don't carry one). - A `beforeSendSpan` hook in `instrument.ts` that reads the scope's `twenty` context block back and projects it onto every span as `twenty.workspace.id` and (when available) `twenty.user_workspace.id` — dotted-namespace naming consistent with OTel/Sentry conventions like `user.id` and `http.response.status_code`. Spans without a workspace context (unauthenticated traffic) pass through untouched. ## Why this shape Sentry's docs position `beforeSendSpan` as a per-span hook. The previous iteration set context only at AI-specific call sites, which left non-AI spans (DB queries, outbound HTTP, regular GraphQL queries, workflow steps not touching AI) entirely unenriched. Hooking the two existing global boundaries — auth middleware for HTTP/GraphQL/REST, and the queue driver `work()` callback for background jobs — covers every authenticated span across the app with no per-handler instrumentation. ## What's not in this PR AI-specific identifiers (`twenty.agent.id`, `twenty.thread.id`, `twenty.turn.id`, `twenty.workflow_run.id`) are out of scope here. They're useful additions but require either propagating the IDs through the call stack or a more fine-grained scope (per-step, per-turn) than the request/job boundary, which is best handled in follow-up PRs that target those specific call sites. ## Test plan - [ ] Make any authenticated GraphQL request locally and confirm the resulting span(s) in Sentry carry `twenty.workspace.id` and (for user-authenticated routes) `twenty.user_workspace.id`. - [ ] Make any authenticated REST request and confirm the same. - [ ] Trigger a queue job (chat stream, agent turn evaluation, workflow run, etc.) and confirm spans produced inside the worker carry `twenty.workspace.id`. - [ ] Confirm that DB and outbound HTTP spans produced under the request/job also carry the workspace tag — these previously had no twenty-specific identifiers. - [ ] In the Sentry UI, filter events by the `twenty.workspace.id` tag and confirm matching events appear. ## Notes for review - Sentry init lives in `instrument.ts`, loaded before Nest bootstraps, so `beforeSendSpan` runs outside Nest DI and reads context off the isolation scope rather than holding a service reference. - The middleware change is three lines; the BullMQ wrap is a single `Sentry.withIsolationScope` around the existing job handler body; the SyncDriver wrap mirrors it for the dev/test path. No new modules or DI providers. - The previous iteration's `AiCallContextService` and per-handler `setContext` / `withContext` calls have been removed in favor of these two hooks. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
106 lines
3.1 KiB
TypeScript
106 lines
3.1 KiB
TypeScript
import process from 'process';
|
|
|
|
import opentelemetry from '@opentelemetry/api';
|
|
import { OTLPMetricExporter } from '@opentelemetry/exporter-metrics-otlp-http';
|
|
import { PrometheusExporter } from '@opentelemetry/exporter-prometheus';
|
|
import {
|
|
AggregationTemporality,
|
|
ConsoleMetricExporter,
|
|
MeterProvider,
|
|
PeriodicExportingMetricReader,
|
|
} from '@opentelemetry/sdk-metrics';
|
|
import * as Sentry from '@sentry/node';
|
|
import { nodeProfilingIntegration } from '@sentry/profiling-node';
|
|
|
|
import { NodeEnvironment } from 'src/engine/core-modules/twenty-config/interfaces/node-environment.interface';
|
|
|
|
import { ExceptionHandlerDriver } from 'src/engine/core-modules/exception-handler/interfaces';
|
|
import { MeterDriver } from 'src/engine/core-modules/metrics/types/meter-driver.type';
|
|
import { parseArrayEnvVar } from 'src/utils/parse-array-env-var';
|
|
|
|
const meterDrivers = parseArrayEnvVar(
|
|
process.env.METER_DRIVER,
|
|
Object.values(MeterDriver),
|
|
[],
|
|
);
|
|
|
|
if (process.env.EXCEPTION_HANDLER_DRIVER === ExceptionHandlerDriver.SENTRY) {
|
|
Sentry.init({
|
|
environment: process.env.SENTRY_ENVIRONMENT,
|
|
release: process.env.APP_VERSION,
|
|
dsn: process.env.SENTRY_DSN,
|
|
integrations: [
|
|
Sentry.redisIntegration(),
|
|
Sentry.httpIntegration(),
|
|
Sentry.expressIntegration(),
|
|
Sentry.graphqlIntegration(),
|
|
Sentry.postgresIntegration(),
|
|
Sentry.vercelAIIntegration({
|
|
recordInputs: true,
|
|
recordOutputs: true,
|
|
}),
|
|
nodeProfilingIntegration(),
|
|
],
|
|
tracesSampleRate: 0.1,
|
|
profilesSampleRate: 0.3,
|
|
sendDefaultPii: true,
|
|
debug: process.env.NODE_ENV === NodeEnvironment.DEVELOPMENT,
|
|
beforeSendSpan: (span) => {
|
|
const twentyContext = Sentry.getIsolationScope().getScopeData().contexts
|
|
?.twenty as
|
|
| {
|
|
workspace_id?: string;
|
|
user_workspace_id?: string;
|
|
}
|
|
| undefined;
|
|
|
|
if (!twentyContext?.workspace_id) {
|
|
return span;
|
|
}
|
|
|
|
span.data = {
|
|
...span.data,
|
|
'twenty.workspace.id': twentyContext.workspace_id,
|
|
...(twentyContext.user_workspace_id && {
|
|
'twenty.user_workspace.id': twentyContext.user_workspace_id,
|
|
}),
|
|
};
|
|
|
|
return span;
|
|
},
|
|
});
|
|
}
|
|
|
|
// Meter setup
|
|
|
|
const prometheusExporter = meterDrivers.includes(MeterDriver.Prometheus)
|
|
? new PrometheusExporter({ port: 9464 })
|
|
: null;
|
|
|
|
const meterProvider = new MeterProvider({
|
|
readers: [
|
|
...(meterDrivers.includes(MeterDriver.Console)
|
|
? [
|
|
new PeriodicExportingMetricReader({
|
|
exporter: new ConsoleMetricExporter(),
|
|
exportIntervalMillis: 10000,
|
|
}),
|
|
]
|
|
: []),
|
|
...(meterDrivers.includes(MeterDriver.OpenTelemetry)
|
|
? [
|
|
new PeriodicExportingMetricReader({
|
|
exporter: new OTLPMetricExporter({
|
|
url: process.env.OTLP_COLLECTOR_METRICS_ENDPOINT_URL,
|
|
temporalityPreference: AggregationTemporality.DELTA,
|
|
}),
|
|
exportIntervalMillis: 10000,
|
|
}),
|
|
]
|
|
: []),
|
|
...(prometheusExporter ? [prometheusExporter] : []),
|
|
],
|
|
});
|
|
|
|
opentelemetry.metrics.setGlobalMeterProvider(meterProvider);
|