feat: Add Agent Evaluation System and Refactor AI Modules (#16111)
## Summary This PR introduces a comprehensive agent evaluation system and refactors the AI module structure for better organization. ## Key Changes ### 🎯 Agent Evaluation System - Added **Agent Turn Evaluation** entities, DTOs, and database schema - New GraphQL mutations: `evaluateAgentTurn` and `runEvaluationInput` - Added `evaluationInputs` field to Agent entity for storing test inputs - New `AgentTurnGraderService` for automatic turn evaluation - Added evaluation UI with new **Evals** and **Logs** tabs in agent detail pages ### 🏗️ Entity & Module Refactoring - Renamed `AgentChatMessage` → `AgentMessage` for clarity - Consolidated chat entities: `AgentMessage`, `AgentTurn`, and `AgentChatThread` - Reorganized AI modules under `ai/` subdirectory structure - Updated imports across codebase to reflect new module paths ### 🤖 New Agents & Roles - Added **Dashboard Builder Agent** for dashboard creation and management - Added **Dashboard Manager Role** with appropriate permissions - Updated role permissions to be more granular (users vs agents vs API keys) ### 🔐 Permission System Updates - Added `HTTP_REQUEST_TOOL` permission flag - Updated Workflow Manager role permissions (restricted tool access) - Enhanced permission flag types to differentiate between user/agent/API key contexts - Added `isRelevantForAgents`, `isRelevantForApiKeys`, `isRelevantForUsers` to permission flags ### 📨 Message Role Enhancement - Added `system` role to `AgentMessageRole` enum (alongside user/assistant) - Updated message handling to support system prompts ### 🎨 UI/UX Improvements - New tabs in agent detail: **Evals** and **Logs** - Added turn detail page: `/ai/agents/:agentId/turns/:turnId` - Fixed text overflow in `SettingsListItemCardContent` - Updated role applicability labels ("Assignable to Workspace Members") ### 🛠️ Technical Improvements - Fixed Zod schema validation for UUID and Date fields (use string validators) - Updated `ToolRegistryService` to properly register HTTP tool with permission flag - Enhanced error handling in agent execution services - Updated database migrations for new entity schema ## Database Migrations - `1764210000000-add-system-role-to-agent-message.ts` - `1764220000000-add-evaluation-inputs-to-agent.ts` - `1764200000000-add-agent-turn-evaluation.ts` - `1764100000000-refactor-agent-chat-entities.ts` ## Testing - [ ] Agent evaluation flow tested - [ ] Dashboard Builder agent tested - [ ] Permission system validated - [ ] UI tabs and navigation tested - [ ] Database migrations run successfully ## Breaking Changes ⚠️ **Entity Rename**: `AgentChatMessage` renamed to `AgentMessage` - GraphQL queries need updating ## Related Issues <!-- Link any related issues here --> ## Screenshots <!-- Add screenshots if applicable -->
This commit is contained in:
+55
@@ -0,0 +1,55 @@
|
||||
import {
|
||||
Body,
|
||||
Controller,
|
||||
Post,
|
||||
Res,
|
||||
UseFilters,
|
||||
UseGuards,
|
||||
} from '@nestjs/common';
|
||||
|
||||
import { Response } from 'express';
|
||||
import { type ExtendedUIMessage } from 'twenty-shared/ai';
|
||||
|
||||
import { RestApiExceptionFilter } from 'src/engine/api/rest/rest-api-exception.filter';
|
||||
import { WorkspaceEntity } from 'src/engine/core-modules/workspace/workspace.entity';
|
||||
import { AuthUserWorkspaceId } from 'src/engine/decorators/auth/auth-user-workspace-id.decorator';
|
||||
import { AuthWorkspace } from 'src/engine/decorators/auth/auth-workspace.decorator';
|
||||
import { JwtAuthGuard } from 'src/engine/guards/jwt-auth.guard';
|
||||
import { SettingsPermissionGuard } from 'src/engine/guards/settings-permission.guard';
|
||||
import { WorkspaceAuthGuard } from 'src/engine/guards/workspace-auth.guard';
|
||||
import { type RecordIdsByObjectMetadataNameSingularType } from 'src/engine/metadata-modules/ai/ai-agent/types/recordIdsByObjectMetadataNameSingular.type';
|
||||
import { AgentChatStreamingService } from 'src/engine/metadata-modules/ai/ai-chat/services/agent-chat-streaming.service';
|
||||
import { PermissionFlagType } from 'src/engine/metadata-modules/permissions/constants/permission-flag-type.constants';
|
||||
|
||||
@Controller('rest/agent-chat')
|
||||
@UseGuards(JwtAuthGuard, WorkspaceAuthGuard)
|
||||
@UseFilters(RestApiExceptionFilter)
|
||||
export class AgentChatController {
|
||||
constructor(
|
||||
private readonly agentStreamingService: AgentChatStreamingService,
|
||||
) {}
|
||||
|
||||
@Post('stream')
|
||||
@UseGuards(SettingsPermissionGuard(PermissionFlagType.AI))
|
||||
async streamAgentChat(
|
||||
@Body()
|
||||
body: {
|
||||
threadId: string;
|
||||
messages: ExtendedUIMessage[];
|
||||
recordIdsByObjectMetadataNameSingular?: RecordIdsByObjectMetadataNameSingularType;
|
||||
},
|
||||
@AuthUserWorkspaceId() userWorkspaceId: string,
|
||||
@AuthWorkspace() workspace: WorkspaceEntity,
|
||||
@Res() response: Response,
|
||||
) {
|
||||
this.agentStreamingService.streamAgentChat({
|
||||
threadId: body.threadId,
|
||||
messages: body.messages,
|
||||
userWorkspaceId,
|
||||
workspace,
|
||||
recordIdsByObjectMetadataNameSingular:
|
||||
body.recordIdsByObjectMetadataNameSingular || [],
|
||||
response,
|
||||
});
|
||||
}
|
||||
}
|
||||
Reference in New Issue
Block a user