## Summary
- Records the time each job spends waiting in the queue (enqueue →
processing start) as an OpenTelemetry histogram
- Metric is broken down by `queue` and `job_name` attributes, enabling
per-queue p50/p95/p99 latency analysis
- Uses BullMQ's native `job.timestamp` for accurate measurement
## Test plan
- [x] Deploy to staging and verify `job/latency-ms` metric appears in
ClickHouse `otel_metrics_histogram` table
- [x] Confirm Grafana dashboard can query the histogram data
Adds `twenty_upgrade_instance_info` — a new "info"-style gauge that
carries the inferred instance version (derived from the last applied
upgrade migration) as the `version` label.
This follows the standard Prometheus pattern for surfacing string-valued
metadata: value is always `1` (load-bearing for PromQL `group_left`
joins), the data lives on the label. Same shape as `node_uname_info`,
`go_info`, `kube_pod_info`, etc. — see [Prometheus naming
conventions](https://prometheus.io/docs/practices/naming/#metric-names).
On `/metrics`:
```
# HELP twenty_upgrade_instance_info Inferred instance version (semver-ish, derived from the last applied upgrade migration), carried as the `version` attribute
# TYPE twenty_upgrade_instance_info gauge
twenty_upgrade_instance_info{version="2.7.3"} 1
```
Also adds `MetricsService.createInfoGauge` as the helper for the pattern
(auto-suffixes `_info`, enforces value=1). Consumed by
twentyhq/twenty-eng#65.
**AI Chat - Tool Executions (counters, tagged with model)**
ai-chat/tool-execution-succeeded: number of tool calls invoked by the AI
that completed without error
ai-chat/tool-execution-failed: number of tool calls invoked by the AI
that threw an error
**AI Chat - Token Usage (counters, tagged with model)**
ai-chat/input-tokens: total input tokens sent to the model across all
turns
ai-chat/output-tokens: total output tokens generated by the model
ai-chat/cache-read-tokens: input tokens served from the model's prompt
cache (cheaper)
ai-chat/cache-write-tokens: input tokens written into the prompt cache
for future reuse
**AI Chat - Latency (histograms in ms, tagged with model)**
ai-chat/turn-latency-ms: total duration of a full chat turn (from stream
start to stream end)
ai-chat/step-latency-ms: duration of a single reasoning/tool-call step
within a turn
ai-chat/ttft-ms: time-to-first-token, i.e. how long until the model
starts streaming output
**MCP - Tool Executions (counters)**
mcp/tool-execution-succeeded: number of MCP tool calls that completed
successfully
mcp/tool-execution-failed: number of MCP tool calls that threw an error
## Summary
The "AI" acronym was rendered inconsistently across the codebase. The
backend AI module had settled on PascalCase `Ai` (`AiAgentModule`,
`AiBillingService`, `AiChatModule`, `AiModelRegistryService`, etc.),
while frontend components, several DTOs, a few types, and shared
identifiers still used all-caps `AI` (`AIChatTab`,
`AISystemPromptPreviewDTO`, `SettingsPath.AIPrompts`, ...). CLAUDE.md
specifies PascalCase for classes; this PR normalizes everything internal
to `Ai`.
**This is a pure internal rename.** The GraphQL schema is untouched —
`@ObjectType` decorator string arguments, resolver method names (which
become Query/Mutation field names), gql template contents, and the
`generated-metadata/graphql.ts` file are preserved verbatim. The only
visible change is TypeScript identifiers and file names.
## Also folded in (adjacent cleanups)
- **`AgentModelConfigService` → `AiModelConfigService`**. Lives in
`ai-models/` and is used by multiple AI code paths, not just the Agent
entity. The "Agent" prefix was misleading.
- **`generate-text-input.dto.ts` → `generate-text.input.ts`**. The
`ai-agent/dtos/` folder already uses `<entity>.input.ts` convention for
Input classes (`create-agent.input.ts` etc.); the old path mixed
`.dto.ts` file extension with a class that has no DTO suffix. File
rename only; class stays `GenerateTextInput`.
- **Removed stale TODO** in `ai-model-config.type.ts` that asked for the
`AiModelConfig` rename that this PR performs.
## Rename methodology
Bulk rename via perl with anchored regex
`(?<!['"])(?<![A-Z.])AI([A-Z])(?=[a-z])/Ai$1/g`:
- **Lookbehind for non-uppercase** skips adjacent acronyms (`MOSAIC`,
`OIDCSSO`) and leaves `AIRBNB_ID` alone.
- **Lookbehind for non-quote** protects most string literals.
- **Lookahead for lowercase** restricts matches to PascalCase
identifiers (`AIChatTab`), leaving SCREAMING_SNAKE constants untouched.
Strict file-scope exclusions: `generated-metadata/**`, `generated/**`,
`locales/**`, `migrations/**`, `illustrations/**`, `halftone/**`, and
the two gql template files (`queries/getAISystemPromptPreview.ts`,
`mutations/uploadAIChatFile.ts`).
Post-rename reverts for identifiers where the regex was too eager:
- Backend resolver method names kept: `getAISystemPromptPreview`,
`uploadAIChatFile` (they are GraphQL field names).
- `@ObjectType('AdminAIModels')` / `('AISystemPromptPreview')` /
`('AISystemPromptSection')` kept as-is.
- Backend classes `ClientAIModelConfig` / `AdminAIModelConfig` kept
as-is (they use `@ObjectType()` with no argument, so the class name IS
the schema name).
- External-library symbols restored: `OpenAIProvider`,
`createOpenAICompatible`, `vercelAIIntegration`.
File renames use a two-step rename to work on macOS case-insensitive
filesystems: `git mv X.tsx X.tsx.tmp && git mv X.tsx.tmp renamed.tsx`.
## Diff audit
- 0 changes to migrations
- 0 changes to locale `.po` / `.ts` files
- 0 changes to `generated-metadata/graphql.ts`
- 0 changes to website illustration files (base64 blobs preserved)
- 0 renames inside user-facing translation strings (`t\`…\``,
`msg\`…\``, `<Trans>…</Trans>`)
## Test plan
- [x] `npx nx typecheck twenty-server` — PASS
- [x] `npx nx typecheck twenty-front` — PASS
- [x] `npx jest ai-model admin agent-role` — 79/79 PASS
- [x] `npx oxlint --type-aware` on 118 changed files — 0 errors
- [x] `npx prettier --check` on 118 changed files — clean
- [ ] CI
### Context
GraphQL introspection queries (__schema, __type) were going through the
full Yoga server pipeline, which forces a complete workspace schema
build, loading flat metadata maps, building all GraphQL types, wiring
all resolver factories, and calling makeExecutableSchema. This is
expensive, even though introspection only needs the type structure and
zero resolver execution.
A new WorkspaceGraphqlSchemaSDLService extracts the SDL computation that
was previously embedded inside WorkspaceSchemaFactory.
From `direct-execution.service.ts` :
- `buildSchema(sdl)` reconstructs a resolver-free GraphQLSchema from the
SDL
- pure CPU, not cached, not sure it worths it ?
- `execute({ schema, document, variableValues })` from graphql-js,
introspection is answered entirely by the graphql-js runtime from type
metadata, no resolver execution needed
#### Nice to do ?
- Use new cache service for typeDefs ?
### Renaming bonus: `typeDefs` → `sdl`
`typeDefs` is an Apollo/graphql-tools convention. It's the parameter
name in
`makeExecutableSchema({ typeDefs, resolvers })`, not a native GraphQL
spec term.
In proper GraphQL semantics, what this service produces is the **SDL**
(Schema
Definition Language): the official term for the string representation of
a schema.
`printSchema()` produces it, `buildSchema()` consumes it.
##### Why the distinction matters
- **Type definitions** implies partial type declarations (objects,
scalars, enums…)
- **Schema SDL** conveys a *complete* schema document: all types
**plus** the root
operation types (`Query`, `Mutation`) which is exactly what
`printSchema(schema)`
produces
## Summary
- Deletes the entire `1-19` upgrade version command directory (7 files,
~1500 lines) and removes all references from the upgrade module and
command runner.
- Removes `IS_NAVIGATION_MENU_ITEM_ENABLED` and
`IS_NAVIGATION_MENU_ITEM_EDITING_ENABLED` feature flags from the enum,
seed data, generated schema files, and test mocks. These flags had no
remaining feature-gated code — navigation menu items are now always
enabled.
- if >5000 workflows per hour, new ones should failed
- if >100 workflow per min, new ones should be set as not started.
Except manual trigger
- when enqueued, we check if there a not started workflows that may be
queued. If yes, we call the associated job
---------
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
- new status `ENQUEUED` added. With a command to backfill
- counter in cache per workspace, managed by a new service
[workflow-run-queue.workspace-service.ts](https://github.com/twentyhq/twenty/compare/tt-improve-workflow-run-queueing?expand=1#diff-1e2de2a48cd482a3bd7e8dedf1150a19d0b200afbd9282181a24ecddddb56927)
- cron added that will run every minute to look for not started
workflows
Here is the new flow:
- When executing a workflow, we check if the queue is not full. If not,
run is created as `ENQUEUED` and the run workflow job is triggered as
usual. If full, create the run as NOT_STARTED and do not trigger the job
- Cron will look for NOT_STARTED workflows and queue some if there is
some place again
- Only MANUAL and Form submit skip the queue limit
## Context
Added to the existing useGraphQLErrorHandlerHook yoga hook to increment
metrics after all query executions based on their error codes. I
originally wanted to create a new useMetrics hook but most of the error
handling was done in useGraphQLErrorHandlerHook so we decided to keep it
there for now.
<img width="1310" alt="Screenshot 2025-06-24 at 15 58 26"
src="https://github.com/user-attachments/assets/498d3754-851a-4051-a5c2-23ac8253aa6a"
/>
Done :
- move metrics and health cache services from health module to metrics
module
- refactor metrics counter from specific method to set up from enum keys
- add OpenTelemetry (Otel) instrumentation for metrics
- set up Otel SDK to send metrics to Otel collector
To do later :
- implement Otel instrumentation for traces + plug Sentry on top
Thomas Trompette