feat: Add Agent Evaluation System and Refactor AI Modules (#16111)

## Summary

This PR introduces a comprehensive agent evaluation system and refactors
the AI module structure for better organization.

## Key Changes

### 🎯 Agent Evaluation System
- Added **Agent Turn Evaluation** entities, DTOs, and database schema
- New GraphQL mutations: `evaluateAgentTurn` and `runEvaluationInput`
- Added `evaluationInputs` field to Agent entity for storing test inputs
- New `AgentTurnGraderService` for automatic turn evaluation
- Added evaluation UI with new **Evals** and **Logs** tabs in agent
detail pages

### 🏗️ Entity & Module Refactoring
- Renamed `AgentChatMessage` → `AgentMessage` for clarity
- Consolidated chat entities: `AgentMessage`, `AgentTurn`, and
`AgentChatThread`
- Reorganized AI modules under `ai/` subdirectory structure
- Updated imports across codebase to reflect new module paths

### 🤖 New Agents & Roles
- Added **Dashboard Builder Agent** for dashboard creation and
management
- Added **Dashboard Manager Role** with appropriate permissions
- Updated role permissions to be more granular (users vs agents vs API
keys)

### 🔐 Permission System Updates
- Added `HTTP_REQUEST_TOOL` permission flag
- Updated Workflow Manager role permissions (restricted tool access)
- Enhanced permission flag types to differentiate between user/agent/API
key contexts
- Added `isRelevantForAgents`, `isRelevantForApiKeys`,
`isRelevantForUsers` to permission flags

### 📨 Message Role Enhancement
- Added `system` role to `AgentMessageRole` enum (alongside
user/assistant)
- Updated message handling to support system prompts

### 🎨 UI/UX Improvements
- New tabs in agent detail: **Evals** and **Logs**
- Added turn detail page: `/ai/agents/:agentId/turns/:turnId`
- Fixed text overflow in `SettingsListItemCardContent`
- Updated role applicability labels ("Assignable to Workspace Members")

### 🛠️ Technical Improvements
- Fixed Zod schema validation for UUID and Date fields (use string
validators)
- Updated `ToolRegistryService` to properly register HTTP tool with
permission flag
- Enhanced error handling in agent execution services
- Updated database migrations for new entity schema

## Database Migrations
- `1764210000000-add-system-role-to-agent-message.ts`
- `1764220000000-add-evaluation-inputs-to-agent.ts`
- `1764200000000-add-agent-turn-evaluation.ts`
- `1764100000000-refactor-agent-chat-entities.ts`

## Testing
- [ ] Agent evaluation flow tested
- [ ] Dashboard Builder agent tested
- [ ] Permission system validated
- [ ] UI tabs and navigation tested
- [ ] Database migrations run successfully

## Breaking Changes
⚠️ **Entity Rename**: `AgentChatMessage` renamed to `AgentMessage` -
GraphQL queries need updating

## Related Issues
<!-- Link any related issues here -->

## Screenshots
<!-- Add screenshots if applicable -->
This commit is contained in:
Félix Malfait
2025-11-27 08:25:40 +01:00
committed by GitHub
parent 35f81805b8
commit 4f20fd35c5
158 changed files with 2954 additions and 556 deletions
@@ -0,0 +1,55 @@
import {
Body,
Controller,
Post,
Res,
UseFilters,
UseGuards,
} from '@nestjs/common';
import { Response } from 'express';
import { type ExtendedUIMessage } from 'twenty-shared/ai';
import { RestApiExceptionFilter } from 'src/engine/api/rest/rest-api-exception.filter';
import { WorkspaceEntity } from 'src/engine/core-modules/workspace/workspace.entity';
import { AuthUserWorkspaceId } from 'src/engine/decorators/auth/auth-user-workspace-id.decorator';
import { AuthWorkspace } from 'src/engine/decorators/auth/auth-workspace.decorator';
import { JwtAuthGuard } from 'src/engine/guards/jwt-auth.guard';
import { SettingsPermissionGuard } from 'src/engine/guards/settings-permission.guard';
import { WorkspaceAuthGuard } from 'src/engine/guards/workspace-auth.guard';
import { type RecordIdsByObjectMetadataNameSingularType } from 'src/engine/metadata-modules/ai/ai-agent/types/recordIdsByObjectMetadataNameSingular.type';
import { AgentChatStreamingService } from 'src/engine/metadata-modules/ai/ai-chat/services/agent-chat-streaming.service';
import { PermissionFlagType } from 'src/engine/metadata-modules/permissions/constants/permission-flag-type.constants';
@Controller('rest/agent-chat')
@UseGuards(JwtAuthGuard, WorkspaceAuthGuard)
@UseFilters(RestApiExceptionFilter)
export class AgentChatController {
constructor(
private readonly agentStreamingService: AgentChatStreamingService,
) {}
@Post('stream')
@UseGuards(SettingsPermissionGuard(PermissionFlagType.AI))
async streamAgentChat(
@Body()
body: {
threadId: string;
messages: ExtendedUIMessage[];
recordIdsByObjectMetadataNameSingular?: RecordIdsByObjectMetadataNameSingularType;
},
@AuthUserWorkspaceId() userWorkspaceId: string,
@AuthWorkspace() workspace: WorkspaceEntity,
@Res() response: Response,
) {
this.agentStreamingService.streamAgentChat({
threadId: body.threadId,
messages: body.messages,
userWorkspaceId,
workspace,
recordIdsByObjectMetadataNameSingular:
body.recordIdsByObjectMetadataNameSingular || [],
response,
});
}
}