Webhook deliveries from Google and Microsoft were invisible at the app
level: successful notifications produced no logs and no metrics, so
webhook-triggered syncs could not be told apart from cron polling.
Add two counters, connected-account-sync-webhook/received/messaging and
/received/calendar, mirroring the sync-job metric umbrellas, and log a
line whenever a notification triggers a sync. Unmatched subscriptions
keep their existing warn logs.
<!-- This is an auto-generated description by cubic. -->
<a
href="https://cubic.dev/pr/twentyhq/twenty/pull/22853?utm_source=github"
target="_blank" rel="noopener noreferrer"
data-no-image-dialog="true"><picture><source
media="(prefers-color-scheme: dark)"
srcset="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"><source
media="(prefers-color-scheme: light)"
srcset="https://www.cubic.dev/buttons/review-in-cubic-light.svg"><img
alt="Review in cubic"
src="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"></picture></a>
<!-- End of auto-generated description by cubic. -->
## Summary
Adds minimal server-side observability for AI chat turns: lifecycle
counters to measure success/failure rates, Sentry scope tags to
correlate API and worker traces, and per-LLM-call telemetry metadata for
turn/stream correlation.
- Add turn lifecycle metrics: `ai-chat/turn-started`,
`ai-chat/turn-completed`, `ai-chat/turn-failed` (with `failure_phase`
and `error_code` attributes)
- Emit counters at key points: job start, clean completion, execution
failures, enqueue failures, interrupted streams, and empty completions
- Tag Sentry scope with `streamId`, `turnId`, `threadId`, and
`workspaceId` at API entry points (`sendChatMessage`,
`retryChatMessage`, `answerAgentChatQuestion`) and worker entry
(`StreamAgentChatJob`)
- Enrich LLM `experimental_telemetry` metadata with
stream/turn/thread/workspace IDs
- Return `turnId` from streaming service methods so resolvers can tag
the scope
- Remove granular tool-learned/skill-loaded metrics in favor of the
turn-level counters
## Test plan
- [ ] Send a chat message and verify `ai-chat/turn-started` and
`ai-chat/turn-completed` increment
- [ ] Trigger a stream failure (e.g. interrupted/dead stream) and verify
`ai-chat/turn-failed` with correct `failure_phase`
- [ ] Retry a failed turn and confirm a new `turn-started` is emitted
for the retry attempt
- [ ] Answer an `ask_questions` prompt and confirm Sentry tags include
`streamId` and `turnId`
- [ ] Check Sentry spans for LLM calls include `streamId`, `turnId`,
`threadId`, `workspaceId` in telemetry metadata
- [ ] Run unit tests:
- `npx jest
packages/twenty-server/src/engine/metadata-modules/ai/ai-chat/jobs/__tests__/stream-agent-chat.job.spec.ts`
- `npx jest
packages/twenty-server/src/engine/metadata-modules/ai/ai-chat/services/__tests__/agent-chat-streaming.service.claim.spec.ts`
- `npx jest
packages/twenty-server/src/engine/metadata-modules/ai/ai-chat/services/__tests__/agent-chat-streaming.service.retry.spec.ts`
<!-- This is an auto-generated description by cubic. -->
<a
href="https://cubic.dev/pr/twentyhq/twenty/pull/22692?utm_source=github"
target="_blank" rel="noopener noreferrer"
data-no-image-dialog="true"><picture><source
media="(prefers-color-scheme: dark)"
srcset="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"><source
media="(prefers-color-scheme: light)"
srcset="https://www.cubic.dev/buttons/review-in-cubic-light.svg"><img
alt="Review in cubic"
src="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"></picture></a>
<!-- End of auto-generated description by cubic. -->
## Context
The per-pod in-process workspace metadata cache
(`WorkspaceCacheService`) evicts by a fixed **1,000-entry count**. Each
workspace's cached metadata is ~1 MB (dominated by the flat
`field-metadata` map) over ~10–13 entries, so 1,000 entries ≈ only a few
dozen workspaces per pod. On a multi-tenant instance with far more
active workspaces, the L1 cache thrashes — LRU-evicting and re-fetching
the ~1 MB of maps from Redis on misses — and since the cache sits in one
AZ while pods span both, ~half of that transfer is billed cross-AZ. (In
prod this cache node serves ~2.6 TB/day.)
## What this does
- **Raise `MAX_LOCAL_CACHE_ENTRIES` 1,000 → 7,500** (~500 workspaces at
~1 MB each; server pods are 4 GiB / `--max-old-space-size=3500`, so this
stays well within the heap).
- **Add a `workspace-metadata-cache/local-eviction` counter**
(incremented by the number of entries dropped each time the cache hits
capacity) so we can see capacity-driven evictions in metrics and tune
the limit from real data rather than guessing.
Eviction stays **batched** (`MIN_EVICT_KEYS`), so the sort runs about
once per 100 inserts at steady state rather than on every write.
### Why count, not bytes
An earlier iteration bounded by measured bytes, but that required
`JSON.stringify`-ing every cached value (incl. the ~1 MB field-metadata
maps) on every write — meaningful CPU/GC overhead on the fill path. A
raised count cap avoids that entirely; the new eviction metric gives us
the signal to right-size it.
No change to cache semantics, hashing, or the Redis format.
## Summary
- The `twenty_queue_jobs_waiting_total` gauge was summing all queues
into a single value without a `queue` label, making the Grafana "Jobs
Waiting by Queue" panel show a single aggregated line instead of
per-queue breakdown.
- Uses `getMeter()` directly to call `observableResult.observe(count, {
queue: queueName })` per queue, matching the `by (queue)` grouping the
dashboard already expects.
## Test plan
- [x] Deploy and verify the Grafana "Jobs Waiting by Queue" panel
displays separate series per queue
- [x] Confirm Prometheus scrape returns
`twenty_queue_jobs_waiting_total{queue="..."}` with distinct queue
labels
## Summary
- Records the time each job spends waiting in the queue (enqueue →
processing start) as an OpenTelemetry histogram
- Metric is broken down by `queue` and `job_name` attributes, enabling
per-queue p50/p95/p99 latency analysis
- Uses BullMQ's native `job.timestamp` for accurate measurement
## Test plan
- [x] Deploy to staging and verify `job/latency-ms` metric appears in
ClickHouse `otel_metrics_histogram` table
- [x] Confirm Grafana dashboard can query the histogram data
Adds `twenty_upgrade_instance_info` — a new "info"-style gauge that
carries the inferred instance version (derived from the last applied
upgrade migration) as the `version` label.
This follows the standard Prometheus pattern for surfacing string-valued
metadata: value is always `1` (load-bearing for PromQL `group_left`
joins), the data lives on the label. Same shape as `node_uname_info`,
`go_info`, `kube_pod_info`, etc. — see [Prometheus naming
conventions](https://prometheus.io/docs/practices/naming/#metric-names).
On `/metrics`:
```
# HELP twenty_upgrade_instance_info Inferred instance version (semver-ish, derived from the last applied upgrade migration), carried as the `version` attribute
# TYPE twenty_upgrade_instance_info gauge
twenty_upgrade_instance_info{version="2.7.3"} 1
```
Also adds `MetricsService.createInfoGauge` as the helper for the pattern
(auto-suffixes `_info`, enforces value=1). Consumed by
twentyhq/twenty-eng#65.
**AI Chat - Tool Executions (counters, tagged with model)**
ai-chat/tool-execution-succeeded: number of tool calls invoked by the AI
that completed without error
ai-chat/tool-execution-failed: number of tool calls invoked by the AI
that threw an error
**AI Chat - Token Usage (counters, tagged with model)**
ai-chat/input-tokens: total input tokens sent to the model across all
turns
ai-chat/output-tokens: total output tokens generated by the model
ai-chat/cache-read-tokens: input tokens served from the model's prompt
cache (cheaper)
ai-chat/cache-write-tokens: input tokens written into the prompt cache
for future reuse
**AI Chat - Latency (histograms in ms, tagged with model)**
ai-chat/turn-latency-ms: total duration of a full chat turn (from stream
start to stream end)
ai-chat/step-latency-ms: duration of a single reasoning/tool-call step
within a turn
ai-chat/ttft-ms: time-to-first-token, i.e. how long until the model
starts streaming output
**MCP - Tool Executions (counters)**
mcp/tool-execution-succeeded: number of MCP tool calls that completed
successfully
mcp/tool-execution-failed: number of MCP tool calls that threw an error
## Summary
The "AI" acronym was rendered inconsistently across the codebase. The
backend AI module had settled on PascalCase `Ai` (`AiAgentModule`,
`AiBillingService`, `AiChatModule`, `AiModelRegistryService`, etc.),
while frontend components, several DTOs, a few types, and shared
identifiers still used all-caps `AI` (`AIChatTab`,
`AISystemPromptPreviewDTO`, `SettingsPath.AIPrompts`, ...). CLAUDE.md
specifies PascalCase for classes; this PR normalizes everything internal
to `Ai`.
**This is a pure internal rename.** The GraphQL schema is untouched —
`@ObjectType` decorator string arguments, resolver method names (which
become Query/Mutation field names), gql template contents, and the
`generated-metadata/graphql.ts` file are preserved verbatim. The only
visible change is TypeScript identifiers and file names.
## Also folded in (adjacent cleanups)
- **`AgentModelConfigService` → `AiModelConfigService`**. Lives in
`ai-models/` and is used by multiple AI code paths, not just the Agent
entity. The "Agent" prefix was misleading.
- **`generate-text-input.dto.ts` → `generate-text.input.ts`**. The
`ai-agent/dtos/` folder already uses `<entity>.input.ts` convention for
Input classes (`create-agent.input.ts` etc.); the old path mixed
`.dto.ts` file extension with a class that has no DTO suffix. File
rename only; class stays `GenerateTextInput`.
- **Removed stale TODO** in `ai-model-config.type.ts` that asked for the
`AiModelConfig` rename that this PR performs.
## Rename methodology
Bulk rename via perl with anchored regex
`(?<!['"])(?<![A-Z.])AI([A-Z])(?=[a-z])/Ai$1/g`:
- **Lookbehind for non-uppercase** skips adjacent acronyms (`MOSAIC`,
`OIDCSSO`) and leaves `AIRBNB_ID` alone.
- **Lookbehind for non-quote** protects most string literals.
- **Lookahead for lowercase** restricts matches to PascalCase
identifiers (`AIChatTab`), leaving SCREAMING_SNAKE constants untouched.
Strict file-scope exclusions: `generated-metadata/**`, `generated/**`,
`locales/**`, `migrations/**`, `illustrations/**`, `halftone/**`, and
the two gql template files (`queries/getAISystemPromptPreview.ts`,
`mutations/uploadAIChatFile.ts`).
Post-rename reverts for identifiers where the regex was too eager:
- Backend resolver method names kept: `getAISystemPromptPreview`,
`uploadAIChatFile` (they are GraphQL field names).
- `@ObjectType('AdminAIModels')` / `('AISystemPromptPreview')` /
`('AISystemPromptSection')` kept as-is.
- Backend classes `ClientAIModelConfig` / `AdminAIModelConfig` kept
as-is (they use `@ObjectType()` with no argument, so the class name IS
the schema name).
- External-library symbols restored: `OpenAIProvider`,
`createOpenAICompatible`, `vercelAIIntegration`.
File renames use a two-step rename to work on macOS case-insensitive
filesystems: `git mv X.tsx X.tsx.tmp && git mv X.tsx.tmp renamed.tsx`.
## Diff audit
- 0 changes to migrations
- 0 changes to locale `.po` / `.ts` files
- 0 changes to `generated-metadata/graphql.ts`
- 0 changes to website illustration files (base64 blobs preserved)
- 0 renames inside user-facing translation strings (`t\`…\``,
`msg\`…\``, `<Trans>…</Trans>`)
## Test plan
- [x] `npx nx typecheck twenty-server` — PASS
- [x] `npx nx typecheck twenty-front` — PASS
- [x] `npx jest ai-model admin agent-role` — 79/79 PASS
- [x] `npx oxlint --type-aware` on 118 changed files — 0 errors
- [x] `npx prettier --check` on 118 changed files — clean
- [ ] CI
### Context
GraphQL introspection queries (__schema, __type) were going through the
full Yoga server pipeline, which forces a complete workspace schema
build, loading flat metadata maps, building all GraphQL types, wiring
all resolver factories, and calling makeExecutableSchema. This is
expensive, even though introspection only needs the type structure and
zero resolver execution.
A new WorkspaceGraphqlSchemaSDLService extracts the SDL computation that
was previously embedded inside WorkspaceSchemaFactory.
From `direct-execution.service.ts` :
- `buildSchema(sdl)` reconstructs a resolver-free GraphQLSchema from the
SDL
- pure CPU, not cached, not sure it worths it ?
- `execute({ schema, document, variableValues })` from graphql-js,
introspection is answered entirely by the graphql-js runtime from type
metadata, no resolver execution needed
#### Nice to do ?
- Use new cache service for typeDefs ?
### Renaming bonus: `typeDefs` → `sdl`
`typeDefs` is an Apollo/graphql-tools convention. It's the parameter
name in
`makeExecutableSchema({ typeDefs, resolvers })`, not a native GraphQL
spec term.
In proper GraphQL semantics, what this service produces is the **SDL**
(Schema
Definition Language): the official term for the string representation of
a schema.
`printSchema()` produces it, `buildSchema()` consumes it.
##### Why the distinction matters
- **Type definitions** implies partial type declarations (objects,
scalars, enums…)
- **Schema SDL** conveys a *complete* schema document: all types
**plus** the root
operation types (`Query`, `Mutation`) which is exactly what
`printSchema(schema)`
produces
## Summary
- Deletes the entire `1-19` upgrade version command directory (7 files,
~1500 lines) and removes all references from the upgrade module and
command runner.
- Removes `IS_NAVIGATION_MENU_ITEM_ENABLED` and
`IS_NAVIGATION_MENU_ITEM_EDITING_ENABLED` feature flags from the enum,
seed data, generated schema files, and test mocks. These flags had no
remaining feature-gated code — navigation menu items are now always
enabled.
- if >5000 workflows per hour, new ones should failed
- if >100 workflow per min, new ones should be set as not started.
Except manual trigger
- when enqueued, we check if there a not started workflows that may be
queued. If yes, we call the associated job
---------
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
- new status `ENQUEUED` added. With a command to backfill
- counter in cache per workspace, managed by a new service
[workflow-run-queue.workspace-service.ts](https://github.com/twentyhq/twenty/compare/tt-improve-workflow-run-queueing?expand=1#diff-1e2de2a48cd482a3bd7e8dedf1150a19d0b200afbd9282181a24ecddddb56927)
- cron added that will run every minute to look for not started
workflows
Here is the new flow:
- When executing a workflow, we check if the queue is not full. If not,
run is created as `ENQUEUED` and the run workflow job is triggered as
usual. If full, create the run as NOT_STARTED and do not trigger the job
- Cron will look for NOT_STARTED workflows and queue some if there is
some place again
- Only MANUAL and Form submit skip the queue limit
## Context
Added to the existing useGraphQLErrorHandlerHook yoga hook to increment
metrics after all query executions based on their error codes. I
originally wanted to create a new useMetrics hook but most of the error
handling was done in useGraphQLErrorHandlerHook so we decided to keep it
there for now.
<img width="1310" alt="Screenshot 2025-06-24 at 15 58 26"
src="https://github.com/user-attachments/assets/498d3754-851a-4051-a5c2-23ac8253aa6a"
/>
Done :
- move metrics and health cache services from health module to metrics
module
- refactor metrics counter from specific method to set up from enum keys
- add OpenTelemetry (Otel) instrumentation for metrics
- set up Otel SDK to send metrics to Otel collector
To do later :
- implement Otel instrumentation for traces + plug Sentry on top
neo773
