8da69e0f77ea820a6845a4c3c025b6af3861d523
27 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Charles Bochet
|
c407341912 |
feat: optimize hot database queries with multi-layer caching (#19068)
## Summary Introduces multi-layer caching for the 5 most frequent database queries identified in production (Sentry data), targeting the JWT authentication hot path and cron job logic. ### Problem Our database is under heavy load from uncached queries on the auth hot path: - `WorkspaceEntity` lookups: **638 queries/min** - `ApiKeyEntity` lookups: **491 queries/min** - `UserEntity` lookups: **147 queries/min** - `UserWorkspaceEntity` lookups: **143 queries/min** - `LogicFunctionEntity` lookups: **1800 queries/min** (cron job) ### Solution **1. New `CoreEntityCacheService`** for non-workspace-scoped entities (Workspace, User, UserWorkspace): - Mirrors `WorkspaceCacheService` architecture (in-process Map + Redis with hash validation) - Provider pattern with `@CoreEntityCache` decorator - Keyed by entity primary key (not workspaceId) - 100ms local TTL, Redis-backed hash validation for cross-instance consistency - Three providers: `WorkspaceEntityCacheProviderService`, `UserEntityCacheProviderService`, `UserWorkspaceEntityCacheProviderService` **2. New `apiKeyMap` WorkspaceCache** for workspace-scoped API key lookups: - `WorkspaceApiKeyMapCacheService` loads all API keys for a workspace into a map by ID - Leverages existing `WorkspaceCacheService` infrastructure - Cache invalidation on API key create/update/revoke **3. `CronTriggerCronJob` refactored** to use existing `flatLogicFunctionMaps` workspace cache: - Eliminates per-workspace `LogicFunctionEntity` repository queries (~1800/min) - Filters cached data in-memory instead **4. `JwtAuthStrategy` refactored** to use caches for all entity lookups: - Workspace, User, UserWorkspace → `CoreEntityCacheService` - ApiKey → `WorkspaceCacheService` (`apiKeyMap`) - Impersonation queries kept as direct DB queries (rare path, requires relations) **5. Cache invalidation** wired into mutation paths: - `WorkspaceService` → invalidates `workspaceEntity` on save/update/delete - `ApiKeyService` → invalidates `apiKeyMap` on create/update/revoke ### Architecture ``` Request → JwtAuthStrategy ├── Workspace lookup → CoreEntityCacheService (in-process → Redis → DB) ├── User lookup → CoreEntityCacheService (in-process → Redis → DB) ├── UserWorkspace lookup → CoreEntityCacheService (in-process → Redis → DB) └── ApiKey lookup → WorkspaceCacheService (in-process → Redis → DB) CronTriggerCronJob └── LogicFunction lookup → WorkspaceCacheService (flatLogicFunctionMaps) ``` ### Expected Impact | Query | Before | After | |-------|--------|-------| | WorkspaceEntity | 638/min | ~0 (cached) | | ApiKeyEntity | 491/min | ~0 (cached) | | UserEntity | 147/min | ~0 (cached) | | UserWorkspaceEntity | 143/min | ~0 (cached) | | LogicFunctionEntity | 1800/min | ~0 (cached) | ### Not included (ongoing separately) - DataSourceEntity query optimization (IS_DATASOURCE_MIGRATED migration) - ObjectMetadataEntity query optimization (already partially cached) |
||
|
Charles Bochet
|
9cb21e71fa |
feat: secure and user-scope metadata resolvers for messaging infrastructure (#18787)
## Summary Builds on the messaging infrastructure migration (#18784) by securing and user-scoping all 4 metadata resolvers: ### DTOs secured - **ConnectedAccountDTO**: `@HideField()` on `accessToken`, `refreshToken`, `connectionParameters`, `oidcTokenClaims` - **MessageChannelDTO / CalendarChannelDTO**: `@HideField()` on `syncCursor` - **MessageFolderDTO**: `@HideField()` on `syncCursor`, `externalId` - **UpdateMessageFolderInputUpdates**: stripped to only `isSynced` (removed `name`, `syncCursor`, `pendingSyncAction`) ### Resolvers user-scoped via `@AuthUserWorkspaceId()` - `myConnectedAccounts` — returns only the calling user's accounts (no permission guard) - `myMessageChannels(connectedAccountId?)` — returns channels for the user's connected accounts - `myCalendarChannels(connectedAccountId?)` — same pattern - `myMessageFolders(messageChannelId?)` — returns folders through the ownership chain ### Admin-only listing with permission guard - `connectedAccounts` query retained with `SettingsPermissionGuard(CONNECTED_ACCOUNTS)` for admin listing of all workspace accounts ### Unsafe mutations removed - Removed `createConnectedAccount`, `updateConnectedAccount` (OAuth/IMAP flows create/refresh tokens server-side) - Removed `create*`/`delete*` mutations from MessageChannel, CalendarChannel, MessageFolder (managed by sync engine) ### Update mutations restricted with ownership verification - `deleteConnectedAccount(id)` — verifies `entity.userWorkspaceId === currentUserWorkspaceId` - `updateMessageChannel` / `updateCalendarChannel` / `updateMessageFolder` — verify ownership through connected account chain - New `OWNERSHIP_VIOLATION` exception codes map to `ForbiddenError` in GraphQL ### `@AuthUserWorkspaceId` decorator hardened - Added `allowUndefined` option (default: `false`) — throws `ForbiddenException` if `userWorkspaceId` is undefined (e.g. API key auth) - Existing callers updated to `@AuthUserWorkspaceId({ allowUndefined: true })` where needed - New user-scoped resolvers enforce non-undefined `userWorkspaceId` at decorator level ### Exception handler chaining - `MessageFolderGraphqlApiExceptionInterceptor`, `MessageChannelGraphqlApiExceptionInterceptor`, `CalendarChannelGraphqlApiExceptionInterceptor` chain upstream exception handling (ConnectedAccountException, MessageChannelException) for correct `ForbiddenError` propagation ### Metadata services enhanced - `findByUserWorkspaceId()`, `getUserConnectedAccountIds()`, `findByConnectedAccountIds()`, `findByMessageChannelIds()` - `findBy*ForUser()` methods encapsulate ownership checks before querying - `verifyOwnership()` on all 4 services with proper chain validation - Named parameters throughout for clarity ### Dev seeds for both schemas - Added JANE to connected account, message channel, calendar channel workspace seeds - Created message folder workspace seeds (TIM, JONY, JANE) - New `seed-metadata-entities.util.ts` seeds core schema tables (connectedAccount, messageChannel, calendarChannel, messageFolder) with same IDs as workspace seeds, mapping `accountOwnerId` → `userWorkspaceId` ### Integration tests (using seeds, not raw SQL) - 4 test suites (`connected-account`, `message-channel`, `calendar-channel`, `message-folder`) - Tests use seeded data IDs from seed constants — no raw SQL inserts/deletes - Tests read via GraphQL resolvers - Tests cover: user scoping, admin permission checks, sensitive field exclusion, ownership enforcement on mutations ### Frontend migration - Feature-flag-gated hooks (`useMyConnectedAccounts`, `useMyMessageChannels`, `useMyCalendarChannels`, `useMyMessageFolders`) - When `IS_CONNECTED_ACCOUNT_MIGRATED` is on: hooks use metadata API (`POST /metadata`) - When flag is off: hooks use existing workspace API (`POST /graphql`, current behavior) - Settings account pages updated to use new hooks - `useEffect` extracted to `SettingsAccountsSelectedMessageChannelEffect` component per project conventions - Error messages translated with Lingui ## Test plan - [x] Server typecheck passes - [x] Server lint passes - [x] Server unit tests pass (477 suites, 4269 tests) - [x] Frontend typecheck passes - [x] Frontend lint passes - [x] Integration tests verify user-scoping, ownership enforcement, hidden fields - [ ] CI green --------- Co-authored-by: neo773 <neo773@protonmail.com> |
||
|
Sri Hari Haran Sharma
|
a07337fea0 |
fix: return method-specific MCP responses (#18671)
## Summary Fixes #18524 Fixes the MCP response contract for non-`initialize` methods. Previously, `/mcp` returned initialize-style metadata for methods like `tools/list`, which caused strict MCP clients to reject the response shape. The endpoint also returned `201 Created` for RPC calls even though no resource was being created. ## Changes - return only method-specific payloads for MCP list methods - `tools/list` -> `{ tools: [...] }` - `prompts/list` -> `{ prompts: [] }` - `resources/list` -> `{ resources: [] }` - keep MCP server metadata only on `initialize` - make `/mcp` return `200 OK` instead of `201 Created` - add regression tests for: - `tools/list` response shape - `prompts/list` response shape - `resources/list` response shape ## Why Strict MCP clients expect: - standard RPC transport semantics over HTTP - method-specific JSON-RPC result payloads Returning initialize metadata for non-`initialize` methods breaks that expectation and can cause client deserialization or protocol validation failures. ## Verification - reproduced the issue locally against `/mcp` - verified `tools/list` was previously returning initialize-style fields - verified `tools/list` now returns only `result.tools` - verified `/mcp` now returns `200 OK` - ran targeted Jest tests: ```bash cd /Users/apple/MyProjects/OpenSource/twenty/packages/twenty-server npx jest --runInBand src/engine/api/mcp/services/__tests__/mcp-protocol.service.spec.ts src/engine/api/mcp/services/__tests__/mcp-tool-executor.service.spec.ts --------- Co-authored-by: Félix Malfait <felix@twenty.com> |
||
|
Félix Malfait
|
95a35f8a1d |
Implement OAuth 2.0 Dynamic Client Registration (RFC 7591) (#18608)
## Summary This PR implements OAuth 2.0 Dynamic Client Registration (RFC 7591) and OAuth 2.0 Protected Resource Metadata (RFC 9728) support, enabling third-party applications to dynamically register as OAuth clients without manual configuration. ## Key Changes ### OAuth Dynamic Client Registration - **New Controller**: `OAuthRegistrationController` at `POST /oauth/register` endpoint - Validates client metadata according to RFC 7591 specifications - Enforces PKCE-only public client model (no client secrets) - Supports only `authorization_code` grant type and `code` response type - Rate limits registrations to 10 per hour per IP address - Returns `client_id` and registration metadata in response - **Input Validation**: `OAuthRegisterInput` DTO with constraints on: - Client name (max 256 chars) - Redirect URIs (max 20, validated for security) - Grant types, response types, scopes, and auth methods - Logo and client URIs (max 2048 chars) - **Discovery Endpoint Update**: Added `registration_endpoint` to OAuth discovery metadata ### Stale Registration Cleanup - **Cleanup Service**: Automatically removes OAuth-only registrations older than 30 days that have no active installations - **Cron Job**: Runs daily at 02:30 AM UTC with batch processing (100 records per batch) - **CLI Command**: `cron:stale-registration-cleanup` to manually trigger cleanup ### MCP (Model Context Protocol) Authentication - **New Guard**: `McpAuthGuard` implements RFC 9728 compliance - Wraps JWT authentication with proper error responses - Returns `WWW-Authenticate` header with protected resource metadata URL on 401 - Enables OAuth-protected MCP endpoints ### Protected Resource Metadata - **New Endpoint**: `GET /.well-known/oauth-protected-resource` (RFC 9728) - Advertises MCP resource as OAuth-protected - Lists supported scopes and bearer token methods - Enables OAuth clients to discover authorization requirements ### Application Registration Updates - **New Source Type**: `OAUTH_ONLY` enum value for OAuth-only registrations - **Install Service**: Skips artifact installation for OAuth-only apps (no code artifacts) ### Frontend Updates - **Authorization Page**: Support both snake_case (standard OAuth) and camelCase (legacy) query parameters - `client_id` / `clientId` - `code_challenge` / `codeChallenge` - `redirect_uri` / `redirectUrl` ## Implementation Details - **Rate Limiting**: Uses token bucket algorithm with 10 registrations per 3,600,000ms window per IP - **Scope Validation**: Requested scopes are capped to allowed OAuth scopes; defaults to all scopes if not specified - **Redirect URI Validation**: Uses existing `validateRedirectUri` utility for security - **Cache Headers**: Registration responses include `Cache-Control: no-store` and `Pragma: no-cache` - **Batch Processing**: Cleanup operations process 100 records at a time to avoid memory issues - **Grace Period**: 30-day grace period before cleanup to allow time for client activation https://claude.ai/code/session_01PxcuWFFRuXMASMaMGTLYk2 --------- Co-authored-by: Claude <noreply@anthropic.com> Co-authored-by: claude[bot] <41898282+claude[bot]@users.noreply.github.com> Co-authored-by: github-actions <github-actions@twenty.com> Co-authored-by: cubic-dev-ai[bot] <191113872+cubic-dev-ai[bot]@users.noreply.github.com> |
||
|
Raphaël Bosi
|
b11f77df2a |
[FRONT COMPONENTS] Introduce conditionalAvailabilityExpression to command menu items (#18319)
## PR Description - Uses `expr-eval` to enable front components (SDK plugins) to define conditional availability as declarative expressions. - Moves shared types and constants to `twenty-shared` - Introduces a `conditionalAvailabilityExpression` field on `CommandMenuItemEntity`, allowing command menu items to store an `expr-eval` compatible expression string that is evaluated against a CommandMenuContext to determine if the item should be shown. - Creates an esbuild transform plugin `conditional-availability-transform-plugin` in `twenty-sdk` that converts TypeScript conditional availability expressions into `expr-eval` compatible syntax at build time, so SDK developers can write natural TS expressions that get transformed to evaluable strings. - Removes deprecated `forceRegisteredActionsByKey` state and its usage. - Creates `useCommandMenuContext` hook that builds the full `CommandMenuContext` object from React state, which is then passed to `useCommandMenuItemFrontComponentActions` for evaluating conditional availability expressions. |
||
|
Abdullah.
|
9107f5bbc7 |
feat: upgrade ai package to version six and the corresponding @ai-sdk/* packages to compatible versions (#18172)
Used the migration guide to carry out this upgrade: https://ai-sdk.dev/docs/migration-guides/migration-guide-6-0 I have not been able to test locally due to credits. <img width="220" height="450" alt="image" src="https://github.com/user-attachments/assets/050b34b9-3239-4010-8c47-b43d44571994" /> --------- Co-authored-by: Félix Malfait <felix.malfait@gmail.com> |
||
|
Félix Malfait
|
66da296799 |
Unify MCP into single endpoint with lazy tool discovery (#18113)
## Summary - Consolidates the MCP server from two endpoints (`/mcp` + `/mcp/metadata`) into a **single `POST /mcp`** endpoint exposing five high-level tools: `get_tool_catalog`, `learn_tools`, `execute_tool`, `load_skills`, and `search_help_center` - Fixes **DATABASE_CRUD tools not accessible via API key auth** by removing an unnecessary `userId`/`userWorkspaceId` guard in `DatabaseToolProvider` and threading `ApiKeyWorkspaceAuthContext` through the tool context chain - Improves tool descriptions with **STEP 1/2/3 workflow guidance** so AI clients follow the correct discovery flow (catalog → learn → execute) instead of guessing tool names - Simplifies the **frontend AI settings** by removing the schema picker dropdown (no more "Core Schema" vs "Metadata Schema" choice) ## Changes ### Backend - **Deleted**: `mcp-metadata.controller.ts`, `mcp-metadata.service.ts` (merged into core) - **New**: `get-tool-catalog.tool.ts` — browsable, categorized tool discovery - **Fixed**: `DatabaseToolProvider.generateDescriptors` — removed guard that blocked API key access to CRUD tools - **Fixed**: `ToolContext` type + `ToolRegistryService` — `authContext` now flows through so API key CRUD execution works end-to-end - **Updated**: `McpProtocolService` — builds `ApiKeyWorkspaceAuthContext` for API key requests - **Updated**: All MCP tool descriptions with explicit step numbering ### Frontend - **Simplified**: `SettingsAIMCP.tsx` — removed schema selector dropdown, shows single MCP config ## Test plan - [x] All 19 MCP unit tests pass (controller + protocol service) - [x] Server and frontend lint clean - [x] Server and frontend typecheck pass - [x] Live-tested on localhost:3000 with API key: `get_tool_catalog` returns 236 tools including 196 DATABASE_CRUD tools - [x] Live-tested `execute_tool` with `find_companies` via API key — returns real data - [x] Tested MCP connection from Cursor IDE via project-level `.cursor/mcp.json` Made with [Cursor](https://cursor.com) --------- Co-authored-by: Cursor <cursoragent@cursor.com> |
||
|
martmull
|
da064d5e88 |
Support define is tool logic function (#17926)
- supports isTool and timeout settings in defineLogicFunction in apps and in setting tabs definition - compute for all toolInputSchema for logic funciton, in settings and in code steps <img width="991" height="802" alt="image" src="https://github.com/user-attachments/assets/05dc1221-cac9-45a3-87b0-3b13161446fd" /> |
||
|
Weiko
|
fc908e9d87 |
Refactor WorkspaceAuthContext to use discriminated union types (#17491)
## Context The previous WorkspaceAuthContext was a single interface with many optional fields, making it unclear which fields are available in different authentication scenarios. This made the code harder to reason about and required runtime checks scattered throughout the codebase. ## Changes - Introduced a discriminated union type for WorkspaceAuthContext with four specific variants: -> UserWorkspaceAuthContext - for authenticated users -> ApiKeyWorkspaceAuthContext - for API key authentication -> ApplicationWorkspaceAuthContext - for application-based auth -> SystemWorkspaceAuthContext - for system/internal operations - Added type guard functions (isUserAuthContext, isApiKeyAuthContext, etc.) for safe type narrowing - Added builder utilities (buildUserAuthContext, buildApiKeyAuthContext, etc.) to construct each context variant with proper type safety - Refactored WorkspaceAuthContextMiddleware to use the new builders instead of constructing a loosely-typed object - Moved the type definition from twenty-orm/interfaces/ to core-modules/auth/types/ for better organization - Updated all consumers across query runners, tool providers, and modules to use the new type location ## Notes - I had to query User and WorkspaceMember in some parts of tool module that were expecting userWorkspaceId but not the rest of UserWorkspaceAuthContext (that should be required with the new proper type otherwise it would break a lot of logic and mostly permissions with the newly added RLS -> This is what we expect from UserWorkspaceAuthContext and how it's done in the "normal" path in HTTP middleware) - WorkspaceMember is in the cache already but ideally we should move User (And Workspace?) in the cache as well to avoid querying the DB after each request (this is also valid for HTTP middleware when we hydrate the Request object btw) |
||
|
Félix Malfait
|
0173e40a20 |
feat: Serverless Functions as AI Tools (#16919)
## Summary This PR enables serverless functions to be exposed as AI tools, allowing them to be used by AI agents. ### Changes - Added new `SERVERLESS_FUNCTION` tool category - Added `toolDescription`, `toolInputSchema`, and `toolOutputSchema` fields to serverless functions - Created database migration for the new schema columns - Added tool index query and resolver for fetching available tools - Added Settings AI page tabs (Skills, Tools, Settings) with new tools table - Added utility to convert tool schema to JSON schema format - Updated frontend to display tools in the settings page ### Implementation Details - Serverless functions can now define tool metadata (description, input/output schemas) - These functions are automatically registered in the tool registry - The tool index endpoint allows querying available tools with their schemas - Settings page now has a dedicated Tools tab showing all available tools |
||
|
Félix Malfait
|
009e7e05f2 |
feat(workflow): use authContext in CRUD services for Common API migration (#16857)
## Summary This PR migrates workflow CRUD operations to properly use the Common API layer's authentication context, addressing the issues from the reverted PR #15875. The original PR was reverted because the Common API required passing either a User or an API Key for authentication, which was problematic for workflows. Since then, the "Application" concept was introduced in the Common API layer, allowing for token injection in serverless functions. This PR leverages the "Twenty Standard Application" concept for non-manual workflow triggers, providing a clean authentication path without the issues of user impersonation. ## Changes ### Core Infrastructure - **RecordCrudExecutionContext**: Replace `workspaceId` with full `authContext` - **WorkflowExecutionContext**: Add `authContext` field to carry authentication info - **ToolGeneratorContext/ToolSpecification**: Add optional `authContext` support for tool generation ### Authentication Flow - **WorkflowExecutionContextService**: Build appropriate auth context based on trigger type: - **Manual triggers**: Use user's workspace auth context with their role permissions - **Non-manual triggers**: Use Twenty Standard Application auth context (bypasses permission checks or uses default serverless function role) - **ApplicationService**: Add `findTwentyStandardApplicationOrThrow` method to retrieve the system application - **UserWorkspaceService**: Make relations configurable in `getUserWorkspaceForUserOrThrow` to load only what's needed ### CRUD Services Migration All 5 record CRUD services now receive `authContext` instead of `workspaceId`: - `CreateRecordService` - `UpdateRecordService` - `DeleteRecordService` - `FindRecordsService` - `UpsertRecordService` ### Workflow Actions All record CRUD workflow actions pass `executionContext.authContext` to the services: - `CreateRecordWorkflowAction` - `UpdateRecordWorkflowAction` - `DeleteRecordWorkflowAction` - `FindRecordsWorkflowAction` - `UpsertRecordWorkflowAction` ### AI Agent Integration - AI agent workflow action passes auth context to agent executor - Tool provider and MCP protocol service support auth context propagation ## Benefits - ✅ Proper authentication for workflow CRUD operations via Common API - ✅ Non-manual triggers use system application context (no user impersonation issues) - ✅ Manual triggers preserve user permissions correctly - ✅ Foundation for better permission handling in automated workflows - ✅ Cleaner separation between user-initiated and system-initiated operations ## Related - Reverted PR: #15875 |
||
|
Weiko
|
039a3d99a8 |
Fix APIKey typing (#16541)
Following this https://github.com/twentyhq/twenty/pull/16540 Those were not failing (yet) but were wrongly typed and error prone --------- Co-authored-by: Guillim <guillim@users.noreply.github.com> |
||
|
Félix Malfait
|
2e104c8e76 |
feat(ai): add code interpreter for AI data analysis (#16559)
## Summary - Add code interpreter tool that enables AI to execute Python code for data analysis, CSV processing, and chart generation - Support for both local (development) and E2B (sandboxed production) execution drivers - Real-time streaming of stdout/stderr and generated files - Frontend components for displaying code execution results with expandable sections ## Code Quality Improvements - Extract `getMimeType` to shared utility to reduce code duplication between drivers - Fix security issue: escape single quotes/backslashes in E2B driver env variable injection - Add `buildExecutionState` helper to reduce duplicated state object construction - Add `DEFAULT_CODE_INTERPRETER_TIMEOUT_MS` constant for consistency - Fix lingui linting warning and TypeScript theme errors in frontend ## Test Plan - [ ] Test code interpreter with local driver in development - [ ] Test code interpreter with E2B driver in production environment - [ ] Verify streaming output displays correctly in chat UI - [ ] Verify generated files (charts, CSVs) are uploaded and downloadable - [ ] Test file upload flow (CSV, Excel) triggers code interpreter <!-- CURSOR_SUMMARY --> --- > [!NOTE] > Updates generated i18n catalogs for Polish and pseudo-English, adding strings for code execution/output (code interpreter) and various UI messages, with minor text adjustments. > > - **Localization**: > - **Generated catalogs**: Refresh `locales/generated/pl-PL.ts` and `locales/generated/pseudo-en.ts`. > - Add strings for code execution/output (e.g., code, copy code/output, running/waiting states, download files, generated files, Python code execution). > - Include new UI texts (errors, prompts, menus) and minor text corrections. > - No changes to `pt-BR`; other files unchanged functionally. > > <sup>Written by [Cursor Bugbot](https://cursor.com/dashboard?tab=bugbot) for commit befc13d02c21e5a6647bc1aa6daa2a89f60b7ef8. This will update automatically on new commits. Configure [here](https://cursor.com/dashboard?tab=bugbot).</sup> <!-- /CURSOR_SUMMARY --> |
||
|
Félix Malfait
|
223082a4da |
refactor(twenty-server): consolidate AI tool provider architecture (#16355)
## Summary Consolidates the AI tool provider architecture by creating a single `ToolProviderService` as the entry point for all tool generation. This removes multiple intermediate services and simplifies the codebase. ## Changes ### New Architecture - **`ToolProviderService`**: Single service for all tool generation with: - `getTools(spec)` - Get tools by category with permissions - `getToolByType(type)` - Get specific tool for workflow execution - **`ToolCategory` enum**: Declarative specification of tool types: - `DATABASE_CRUD` - Record CRUD operations - `ACTION` - HTTP requests, email sending, article search - `WORKFLOW` - Workflow management tools - `METADATA` - Object/field metadata tools - `NATIVE_MODEL` - Model-specific tools (e.g., web search) - **`ToolSpecification` type**: Clean API for requesting tools with permissions ### Removed - `AiToolsModule` - No longer needed - `ToolService` - Logic inlined into ToolProviderService - `ToolAdapterService` - Logic inlined into ToolProviderService - `ToolRegistryService` - Logic inlined into ToolProviderService ### Updated - All consumers (agents, chat, MCP, workflows) now use `ToolProviderService` - Test files updated accordingly ## Stats - **547 insertions, 1146 deletions** (net ~600 lines removed) - 4 services deleted - 1 module deleted ## Testing - [x] Typecheck passes - [x] Lint passes |
||
|
Félix Malfait
|
34d7d82099 |
refactor(mcp): call metadata services directly instead of REST layer (#16349)
## Summary Refactors MCP metadata tools to call underlying services directly instead of going through the REST layer. This makes MCP a pure presentation layer. ### Changes **Created:** - `packages/twenty-server/src/engine/metadata-modules/metadata-tools/metadata-tools.module.ts` - Module that exports MetadataToolsFactory - `packages/twenty-server/src/engine/metadata-modules/metadata-tools/services/metadata-tools.factory.ts` - Factory that generates 8 metadata tools using Zod schemas: - `get-object-metadata`, `create-object-metadata`, `update-object-metadata`, `delete-object-metadata` - `get-field-metadata`, `create-field-metadata`, `update-field-metadata`, `delete-field-metadata` **Modified:** - `packages/twenty-server/src/engine/api/mcp/services/mcp-metadata.service.ts` - Uses new factory instead of REST-based services - `packages/twenty-server/src/engine/api/mcp/mcp.module.ts` - Imports MetadataToolsModule, removes old service imports **Deleted:** - `packages/twenty-server/src/engine/api/mcp/services/tools/create.tools.service.ts` - `packages/twenty-server/src/engine/api/mcp/services/tools/update.tools.service.ts` - `packages/twenty-server/src/engine/api/mcp/services/tools/delete.tools.service.ts` - `packages/twenty-server/src/engine/api/mcp/services/tools/get.tools.service.ts` - `packages/twenty-server/src/engine/api/mcp/services/tools/mcp-metadata-tools.service.ts` ### Architecture Improvement **Before:** ``` MCP Tool → MetadataQueryBuilderFactory → RestApiService → GraphQL API → Service ``` **After:** ``` MCP Tool → Service (ObjectMetadataService / FieldMetadataService) ``` This follows the pattern established by `direct-record-tools.factory.ts` and workflow tools. |
||
|
Weiko
|
3d95d6ca00 |
Add local only cache to cache service and cache typeorm entity metadata (#16287)
## Problem buildEntityMetadatas in GlobalWorkspaceOrmManager is computationally expensive and was running on every executeInWorkspaceContext call. This method uses TypeORM's EntitySchemaTransformer and EntityMetadataBuilder to build metadata for all workspace entities (30-50+ objects with many fields each). The resulting EntityMetadata[] is not serialisable which means it cannot be cached in Redis because they contain: - Circular references - Functions/methods - References to the DataSource instance ## Solution Extended the workspace cache system to support local-only caching, then created a cache provider for entityMetadatas. ## Implementation details Updated @WorkspaceCache decorator (workspace-cache.decorator.ts) - Added localOnly?: boolean option to skip Redis storage for non-serializable data Created WorkspaceEntityMetadatasCacheService - Computes entity metadatas from DB to avoid race condition, this is acceptable Simplified GlobalWorkspaceOrmManager - Now fetches entityMetadatas from cache instead of rebuilding on every call Updated Workspace migration runner - the only entry point where metadata can change - Now invalidate the new 'entityMetadata' local cache when shouldIncrementMetadataGraphqlSchemaVersion is true (== field/object mutations) |
||
|
Félix Malfait
|
4f20fd35c5 |
feat: Add Agent Evaluation System and Refactor AI Modules (#16111)
## Summary This PR introduces a comprehensive agent evaluation system and refactors the AI module structure for better organization. ## Key Changes ### 🎯 Agent Evaluation System - Added **Agent Turn Evaluation** entities, DTOs, and database schema - New GraphQL mutations: `evaluateAgentTurn` and `runEvaluationInput` - Added `evaluationInputs` field to Agent entity for storing test inputs - New `AgentTurnGraderService` for automatic turn evaluation - Added evaluation UI with new **Evals** and **Logs** tabs in agent detail pages ### 🏗️ Entity & Module Refactoring - Renamed `AgentChatMessage` → `AgentMessage` for clarity - Consolidated chat entities: `AgentMessage`, `AgentTurn`, and `AgentChatThread` - Reorganized AI modules under `ai/` subdirectory structure - Updated imports across codebase to reflect new module paths ### 🤖 New Agents & Roles - Added **Dashboard Builder Agent** for dashboard creation and management - Added **Dashboard Manager Role** with appropriate permissions - Updated role permissions to be more granular (users vs agents vs API keys) ### 🔐 Permission System Updates - Added `HTTP_REQUEST_TOOL` permission flag - Updated Workflow Manager role permissions (restricted tool access) - Enhanced permission flag types to differentiate between user/agent/API key contexts - Added `isRelevantForAgents`, `isRelevantForApiKeys`, `isRelevantForUsers` to permission flags ### 📨 Message Role Enhancement - Added `system` role to `AgentMessageRole` enum (alongside user/assistant) - Updated message handling to support system prompts ### 🎨 UI/UX Improvements - New tabs in agent detail: **Evals** and **Logs** - Added turn detail page: `/ai/agents/:agentId/turns/:turnId` - Fixed text overflow in `SettingsListItemCardContent` - Updated role applicability labels ("Assignable to Workspace Members") ### 🛠️ Technical Improvements - Fixed Zod schema validation for UUID and Date fields (use string validators) - Updated `ToolRegistryService` to properly register HTTP tool with permission flag - Enhanced error handling in agent execution services - Updated database migrations for new entity schema ## Database Migrations - `1764210000000-add-system-role-to-agent-message.ts` - `1764220000000-add-evaluation-inputs-to-agent.ts` - `1764200000000-add-agent-turn-evaluation.ts` - `1764100000000-refactor-agent-chat-entities.ts` ## Testing - [ ] Agent evaluation flow tested - [ ] Dashboard Builder agent tested - [ ] Permission system validated - [ ] UI tabs and navigation tested - [ ] Database migrations run successfully ## Breaking Changes ⚠️ **Entity Rename**: `AgentChatMessage` renamed to `AgentMessage` - GraphQL queries need updating ## Related Issues <!-- Link any related issues here --> ## Screenshots <!-- Add screenshots if applicable --> |
||
|
Félix Malfait
|
e7ebf51e50 |
Replace agent handoff system with planning-based router (#16003)
## Overview This PR replaces the dynamic agent handoff system with a more predictable planning-based router that decides upfront how to handle multi-agent coordination. ## Major Changes ### 🔄 Architecture Shift: Handoffs → Planning **Removed:** - `AgentHandoffEntity` and handoff tracking system - `AgentHandoffService` and `AgentHandoffExecutorService` - Dynamic agent-to-agent transfers during execution - Handoff tool generation and description templates **Added:** - `AiRouterService` with two strategies: `simple` (single agent) and `planned` (multi-agent) - `AgentPlanExecutorService` for executing multi-step plans - Plan validation (cycle detection, dependency resolution) - `UnifiedRouterResult` type with discriminated union ### 🤖 New Standard Agents Added two new specialized agents: - **Researcher Agent**: Web search, fact-finding, competitive intelligence - **Code Agent**: TypeScript function generation for serverless workflows ### 🏗️ Router Refactoring (Latest) Split router responsibilities into focused services: - `AiRouterStrategyDeciderService`: Decides simple vs planned strategy - `AiRouterPlanGeneratorService`: Generates and validates execution plans - `AiRouterService`: Coordinates between services (reduced from 426→275 lines) ### ⚙️ Configuration Improvements - Added `outputStrategy` to agent definitions (`direct` vs `synthesize`) - Removed hardcoded special cases for workflow-builder - Added `plannerModel` field to workspace entity - Increased `MAX_STEPS` from 10 to 25 for complex workflows ### 📝 Agent Prompt Refinements Significantly simplified prompts for better clarity: - Workflow Builder: 51→36 lines - Helper: 49→28 lines - Data Manipulator: Enhanced with sorting guidance ### 🔍 Enhanced Debugging - Plan reasoning and step count in data message parts - Router debug info with token usage tracking - Better logging throughout execution pipeline ## Benefits 1. **Simpler Mental Model**: Router decides upfront vs dynamic transfers 2. **Better Predictability**: Users see the plan before execution 3. **Cleaner Architecture**: SRP with focused services 4. **Configuration Over Code**: Agent behavior via config, not hardcoded logic 5. **Plan Validation**: Catches invalid dependencies and cycles ## Migration Notes - Database migration removes `agentHandoff` table - Adds `plannerModel` column to workspace table - No API breaking changes (agent endpoints unchanged) ## Testing - Integration tests updated to remove handoff dependencies - Agent tool test utilities simplified - Plan validation covered by new logic ## Next Steps (Future PRs) - Parallel execution of independent plan steps - Dynamic re-planning based on results - Plan caching for common routing patterns - Error recovery strategies in plan executor |
||
|
Félix Malfait
|
9a80164cf3 |
Add comprehensive permission guard coverage across GraphQL and REST endpoints (#15739)
This PR enhances our security model by ensuring all GraphQL resolvers and REST API endpoints have appropriate permission guards. ## Changes ### ESLint Rules - Enhanced `graphql-resolvers-should-be-guarded` to require permission guards on all resolvers (Query, Mutation, Subscription), not just mutations - Enhanced `rest-api-methods-should-be-guarded` to require permission guards on all REST endpoints (GET, POST, PUT, PATCH, DELETE), not just mutating methods - Both rules now enforce consistent security: authentication guards + permission guards for all endpoints ### Permission Guards Added **Public Endpoints** - Added `NoPermissionGuard`: - Auth-related queries (checkUserExists, findWorkspaceFromInviteHash, validatePasswordResetToken) - Billing webhooks (Stripe callbacks) - SSO callbacks (SAML authentication) - Workflow webhooks - Cloudflare webhooks - Route trigger endpoints - GraphQL subscriptions - Current workspace queries - Geo-map address autocomplete - View-related read operations (view-field, view-filter, view-group, view-sort, view-filter-group) **Settings Permission Guards** - Added `SettingsPermissionGuard`: - API Keys management: `PermissionFlagType.API_KEYS_AND_WEBHOOKS` - Webhooks management: `PermissionFlagType.API_KEYS_AND_WEBHOOKS` - Page Layouts (write operations): `PermissionFlagType.LAYOUTS` - REST Metadata API: `PermissionFlagType.DATA_MODEL` - Agent operations: `PermissionFlagType.AI` - Remote servers: `PermissionFlagType.DATA_MODEL` - Remote tables: `PermissionFlagType.DATA_MODEL` - Serverless functions: `PermissionFlagType.WORKFLOWS` **Custom Permission Guards** - Added `CustomPermissionGuard`: - REST Core API (permissions checked at query execution layer) - Timeline calendar events (permission checks in service layer) - Timeline messaging (permission checks in service layer) - Search operations (permission checks in service layer) - View operations (permission checks via dedicated view permission guards) ### View Permission Guards - Created dedicated `FindManyViewsPermissionGuard` and `FindOneViewPermissionGuard` for reading views - Created `CreateViewPermissionGuard` for view creation with visibility-based permission checks - All view child entities (view-field, view-filter, view-sort, view-group, view-filter-group) use `NoPermissionGuard` for reads - Write operations on view child entities use dedicated permission guards that check parent view access ### Page Layout Permissions - Read operations (GET/Query) now use `NoPermissionGuard` - users can view layouts without LAYOUTS permission - Write operations (POST/PATCH/DELETE/Mutation) require `SettingsPermissionGuard(PermissionFlagType.LAYOUTS)` - Applied consistently across page-layout, page-layout-tab, and page-layout-widget endpoints ## Security Model All endpoints now follow a consistent pattern: 1. **Authentication**: `UserAuthGuard`, `WorkspaceAuthGuard`, or `PublicEndpointGuard` 2. **Authorization**: One of: - `SettingsPermissionGuard(PermissionFlagType.XXX)` - for settings/admin operations - `CustomPermissionGuard` - when permissions are checked in service/data layer - `NoPermissionGuard` - for public or non-sensitive read operations The ESLint rules automatically enforce this pattern going forward. ## Stats - 47 files changed - 603 insertions, 163 deletions - 3 new guard files created |
||
|
Félix Malfait
|
c5564d9bd0 |
[BREAKING CHANGE] refactor: Add Entity suffix to TypeORM entity classes (#15239)
## Summary This PR refactors all TypeORM entity classes in the Twenty codebase to include an 'Entity' suffix (e.g., User → UserEntity, Workspace → WorkspaceEntity) to improve code clarity and follow TypeORM naming conventions. ## Changes ### Entity Renaming - ✅ Renamed **57 core TypeORM entities** with 'Entity' suffix - ✅ Updated all related imports, decorators, and type references - ✅ Fixed Repository<T>, @InjectRepository(), and TypeOrmModule.forFeature() patterns - ✅ Fixed @ManyToOne/@OneToMany/@OneToOne decorator references ### Backward Compatibility - ✅ Preserved GraphQL schema names using @ObjectType('OriginalName') decorators - ✅ **No breaking changes** to GraphQL API - ✅ **No database migrations** required - ✅ File names unchanged (user.entity.ts remains as-is) ### Code Quality - ✅ Fixed **497 TypeScript errors** (82% reduction from 606 to 109) - ✅ **All linter checks passing** - ✅ Improved type safety across the codebase ## Entities Renamed ``` User → UserEntity Workspace → WorkspaceEntity ApiKey → ApiKeyEntity AppToken → AppTokenEntity UserWorkspace → UserWorkspaceEntity Webhook → WebhookEntity FeatureFlag → FeatureFlagEntity ApprovedAccessDomain → ApprovedAccessDomainEntity TwoFactorAuthenticationMethod → TwoFactorAuthenticationMethodEntity WorkspaceSSOIdentityProvider → WorkspaceSSOIdentityProviderEntity EmailingDomain → EmailingDomainEntity KeyValuePair → KeyValuePairEntity PublicDomain → PublicDomainEntity PostgresCredentials → PostgresCredentialsEntity ...and 43 more entities ``` ## Impact ### Files Changed - **400 files** modified - **2,575 insertions**, **2,191 deletions** ### Progress - ✅ **82% complete** (497/606 errors fixed) - ⚠️ **109 TypeScript errors** remain (18% of original) ## Remaining Work The 109 remaining TypeScript errors are primarily: 1. **Function signature mismatches** (~15 errors) - Test mocks with incorrect parameter counts 2. **Entity type mismatches** (~25 errors) - UserEntity vs UserWorkspaceEntity confusion 3. **Pre-existing issues** (~50 errors) - Null safety and DTO compatibility (unrelated to refactoring) 4. **Import type issues** (~10 errors) - Entities imported with 'import type' but used as values 5. **Minor decorator issues** (~9 errors) - onDelete property configurations These can be addressed in follow-up PRs without blocking this refactoring. ## Testing Checklist - [x] Linter passing - [ ] Unit tests should be run (CI will verify) - [ ] Integration tests should be run (CI will verify) - [ ] Manual testing recommended for critical user flows ## Breaking Changes **None** - This is a pure refactoring with full backward compatibility: - GraphQL API unchanged (uses original entity names) - Database schema unchanged - External APIs unchanged ## Notes - Created comprehensive `REFACTORING_STATUS.md` documenting the entire process - All temporary scripts have been cleaned up - Branch: `refactor/add-entity-suffix-to-typeorm-entities` ## Reviewers Please review especially: - Entity renaming patterns - GraphQL backward compatibility - Any areas where entity types are confused (UserEntity vs UserWorkspaceEntity) --------- Co-authored-by: Charles Bochet <charles@twenty.com> |
||
|
Abdullah.
|
793dc3d6fc |
Remove dependency on lodash.pick. (#15213)
Fixes [Dependabot Alert 85](https://github.com/twentyhq/twenty/security/dependabot/85) - prototype pollution in lodash. Added a shared pick helper (with unit tests) in twenty-shared and refactored front-end/server code to import { pick } from the shared barrel instead of lodash.pick. --------- Co-authored-by: Félix Malfait <felix.malfait@gmail.com> Co-authored-by: martmull <martmull@hotmail.fr> |
||
|
Weiko
|
533d7fe49a | Deprecate legacy core datasource token (#14096) | ||
|
Antoine Moreaux
|
809fe5699f |
feat(ai): enhance metadata handling and add support for tools, prompt… (#14002)
…s, and resources - Removed unused `ObjectMetadataStandardIdToIdMap`. - Updated Vite config with additional allowed hosts. - Improved MCP service to handle `ping` method and lists for tools, prompts, and resources. - Refactored utility function `isFieldMetadataEntityOfType` for improved type handling. - Expanded MCP metadata service to include tools, prompts, and resources support. |
||
|
Antoine Moreaux
|
dcdf675000 |
fix(ai): improve mcp metadata logic (#13991)
Fix #13801 |
||
|
Félix Malfait
|
8b4b9ef8da |
Change type import rule (#13751)
Forcing "type" to be explicit, works best will rollup on the frontend to exclude depdendencies |
||
|
Félix Malfait
|
b44b46109c |
Update try/catch syntax (#13732)
Update try/catch syntax to prepare for eslint migration |
||
|
Antoine Moreaux
|
11abe5440b |
feat(ai): add mcp-metadata (#13150)
Co-authored-by: Félix Malfait <felix@twenty.com> |