feat(ai) - add light AI chat turn instrumentation (metrics + Sentry correlation) (#22692)
## Summary Adds minimal server-side observability for AI chat turns: lifecycle counters to measure success/failure rates, Sentry scope tags to correlate API and worker traces, and per-LLM-call telemetry metadata for turn/stream correlation. - Add turn lifecycle metrics: `ai-chat/turn-started`, `ai-chat/turn-completed`, `ai-chat/turn-failed` (with `failure_phase` and `error_code` attributes) - Emit counters at key points: job start, clean completion, execution failures, enqueue failures, interrupted streams, and empty completions - Tag Sentry scope with `streamId`, `turnId`, `threadId`, and `workspaceId` at API entry points (`sendChatMessage`, `retryChatMessage`, `answerAgentChatQuestion`) and worker entry (`StreamAgentChatJob`) - Enrich LLM `experimental_telemetry` metadata with stream/turn/thread/workspace IDs - Return `turnId` from streaming service methods so resolvers can tag the scope - Remove granular tool-learned/skill-loaded metrics in favor of the turn-level counters ## Test plan - [ ] Send a chat message and verify `ai-chat/turn-started` and `ai-chat/turn-completed` increment - [ ] Trigger a stream failure (e.g. interrupted/dead stream) and verify `ai-chat/turn-failed` with correct `failure_phase` - [ ] Retry a failed turn and confirm a new `turn-started` is emitted for the retry attempt - [ ] Answer an `ask_questions` prompt and confirm Sentry tags include `streamId` and `turnId` - [ ] Check Sentry spans for LLM calls include `streamId`, `turnId`, `threadId`, `workspaceId` in telemetry metadata - [ ] Run unit tests: - `npx jest packages/twenty-server/src/engine/metadata-modules/ai/ai-chat/jobs/__tests__/stream-agent-chat.job.spec.ts` - `npx jest packages/twenty-server/src/engine/metadata-modules/ai/ai-chat/services/__tests__/agent-chat-streaming.service.claim.spec.ts` - `npx jest packages/twenty-server/src/engine/metadata-modules/ai/ai-chat/services/__tests__/agent-chat-streaming.service.retry.spec.ts` <!-- This is an auto-generated description by cubic. --> <a href="https://cubic.dev/pr/twentyhq/twenty/pull/22692?utm_source=github" target="_blank" rel="noopener noreferrer" data-no-image-dialog="true"><picture><source media="(prefers-color-scheme: dark)" srcset="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"><source media="(prefers-color-scheme: light)" srcset="https://www.cubic.dev/buttons/review-in-cubic-light.svg"><img alt="Review in cubic" src="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"></picture></a> <!-- End of auto-generated description by cubic. -->
This commit is contained in:
@@ -25,10 +25,6 @@ export enum MetricsKeys {
|
||||
WorkflowRunSystemError = 'workflow-run/system-error',
|
||||
AiChatToolExecutionSucceeded = 'ai-chat/tool-execution-succeeded',
|
||||
AiChatToolExecutionFailed = 'ai-chat/tool-execution-failed',
|
||||
AiChatToolLearnedSucceeded = 'ai-chat/tool-learned-succeeded',
|
||||
AiChatToolLearnedFailed = 'ai-chat/tool-learned-failed',
|
||||
AiChatSkillLoadedSucceeded = 'ai-chat/skill-loaded-succeeded',
|
||||
AiChatSkillLoadedFailed = 'ai-chat/skill-loaded-failed',
|
||||
WorkflowAgentToolExecutionSucceeded = 'workflow-agent/tool-execution-succeeded',
|
||||
WorkflowAgentToolExecutionFailed = 'workflow-agent/tool-execution-failed',
|
||||
McpToolExecutionSucceeded = 'mcp/tool-execution-succeeded',
|
||||
@@ -53,5 +49,8 @@ export enum MetricsKeys {
|
||||
AiChatTurnLatencyMs = 'ai-chat/turn-latency-ms',
|
||||
AiChatStepLatencyMs = 'ai-chat/step-latency-ms',
|
||||
AiChatTtftMs = 'ai-chat/ttft-ms',
|
||||
AiChatTurnStarted = 'ai-chat/turn-started',
|
||||
AiChatTurnCompleted = 'ai-chat/turn-completed',
|
||||
AiChatTurnFailed = 'ai-chat/turn-failed',
|
||||
WorkspaceMetadataCacheLocalEviction = 'workspace-metadata-cache/local-eviction',
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user