Provide additional logsobservability to workflow runs (per node) (#21142)
Surfaces per-step "Logs" tabs in the workflow run side panel so users can see what each step actually did (model + tokens + tool calls for AI, console output for serverless functions, request/response for HTTP, recipients/body for Email). <img width="546" height="501" alt="ai_agent_without_websearch" src="https://github.com/user-attachments/assets/c6ca3518-9489-4484-a570-3d0569ff3b03" /> ## Storage - New `stepLogs` JSONB column on the `workflowRun` workspace entity, typed as `Record<string, WorkflowRunStepLog>` (keyed by step id). - Schema lives in `twenty-shared`: `workflowRunStepLogSchema` with a discriminated `details.type` union for `AI_AGENT | CODE | HTTP_REQUEST | EMAIL` — frontends and backends consume the same Zod-inferred type. - Field is added to existing workspaces via a workspace upgrade command (`2-9 add-workflow-run-step-logs-field`); the standard-object metadata declares it for new workspaces. - Writes happen atomically per step in `WorkflowRunStepLogWorkspaceService.setStepLog` using `jsonb_set`. That lets concurrent steps in the same run write their own keys without contending with the existing lock around `workflowRun.state`. - Per-step payload is hard-capped at 256 KB; anything larger is dropped with a `logger.warn`, so a pathological tool call can never bloat a row. See below for more information. ## How logs are produced **Aalmost everything was already being collected; this PR mostly persists and renders it.** - **AI agent** — `AgentAsyncExecutorService` already tracked token usage, model id, native web-search count, and the AI SDK's `steps[]`. We map those into the log via `mapAiStepsToToolCallLogs` (`searchVector` stripped from record outputs, per-call input/output capped at 32/64 KB, max 200 tool calls per step). The only new measurement is a wall-clock `durationMs` taken around `executeAgent`, and we now fold native web-search cost into the displayed `totalCostInDollars` (it was already billed, just not shown). - **Code / serverless function** — reuses the `console.log` output the function runner already returns (`logsByLevel`); `build-code-step-log.util` only repackages it. - **HTTP request** — built from the action's existing input/output via `build-http-request-step-log.util`. No new signals collected. - **Email (send / draft)** — added `sanitizedHtmlBody` + `plainTextBody` to the existing tool outputs (a small additive change), then `build-email-step-log.util` consumes them. No additional AI inference or external calls are made for logging — the cost is a small CPU overhead per step plus the JSONB write. ## Security The log surface intentionally shows whatever the workflow touched, which made redaction and sanitization the main design concern. - **HTTP — secrets in headers**: existing `SENSITIVE_HEADER_NAMES` set (Authorization, Cookie, …) replaced with `[redacted]` in both request and response. - **HTTP — secrets in URLs**: `SENSITIVE_URL_PARAM_NAMES` (e.g. `api_key`, `token`, `access_token`) replaced in the query string via `URL`-based parsing. - **HTTP — secrets in bodies**: `SENSITIVE_BODY_KEY_REGEX` deep-walks JSON request/response bodies (object input or stringified JSON) and redacts matching keys. Applied to the `error` field too, since transport-layer errors sometimes embed structured payloads. - **Email — XSS risk in body preview**: tool outputs now expose a server-side `sanitizedHtmlBody`; the log builder prefers it over the raw user-authored `input.body`, with `plainTextBody` as a second fallback. The original raw body is only used if sanitization didn't happen (e.g. tool failed before composing). - **AI — internal/noisy data**: `searchVector` (Postgres tsvector strings) is stripped from record outputs returned by Twenty tools to avoid leaking internal full-text-search payloads. - **DB bloat / runaway agents**: 256 KB per-step cap + 32 KB / 64 KB per-tool-call input/output cap + 200 tool calls per step. <img width="547" height="307" alt="logic_function" src="https://github.com/user-attachments/assets/dd4a3d16-67f2-434b-95b3-bdcaf9ed053d" /> ## More details on Log size & truncation Logs are stored in `workflowRun.stepLogs` (JSONB), keyed by `stepId`. ### Per-step cap Each step's log is hard-capped at **256 KB** (`MAX_STEP_LOG_BYTES` in `WorkflowRunStepLogWorkspaceService.setStepLog`). For ~99% of workflows this is roomy — typical real-world sizes: - Code / serverless function: 1–20 KB - HTTP request: 5–70 KB - Email: 5–30 KB - AI agent (a handful of tool calls): 5–50 KB ### Two layers of bounding 1. **Per-field truncation** in each builder (before writing): - **Code**: ≤ 500 entries, ≤ 4 KB per message, ≤ 8 KB stack trace - **HTTP**: ≤ 32 KB per body (request + response), UTF-8 byte-aware - **Email**: ≤ 8 KB body preview, UTF-8 byte-aware - **AI agent**: ≤ 32 KB tool input, ≤ 64 KB tool output, ≤ 200 tool calls/step 2. **Global per-step safety net** at write time: if the assembled `stepLog` still exceeds 256 KB, the write is **dropped entirely** with a `logger.warn`. The workflow itself keeps running unaffected. ### What this means in practice - **Safe**: workflow execution, step results, downstream steps — never blocked by log size. - **Safe**: iterators (each iteration overwrites the previous log for that `stepId`, so they can't accumulate). - **Safe**: step retries (same `stepId` is overwritten, not appended). - **Possible**: an AI agent step with many large tool outputs (e.g., 50+ heavy `web_search` calls) can exceed 256 KB → the **entire** step's log is dropped, side panel shows "No logs were recorded for this step". The user has no explicit signal that the log was dropped due to size (only server-side warn). - **Possible** (theoretical): a workflow with hundreds of distinct steps could push the row toward Postgres's internal ~256 MB jsonb limit. Beyond that, individual `jsonb_set` writes would error and be swallowed by the action's try/catch — workflow still completes. ### Possible future hardening (not in this PR) - Replace "drop entire log" with a stub that preserves the summary card (cost, duration, status) and marks `truncated.reason = 'size_cap'`. - Surface size-drops in the UI (similar to the existing `<StyledTruncatedNotice>`). - Emit a metric so dropped logs are observable in dashboards.
This commit is contained in:
+98
-11
@@ -1,12 +1,14 @@
|
||||
import { Test, type TestingModule } from '@nestjs/testing';
|
||||
|
||||
import { getRepositoryToken } from '@nestjs/typeorm';
|
||||
import { generateText } from 'ai';
|
||||
|
||||
import { BillingUsageService } from 'src/engine/core-modules/billing/services/billing-usage.service';
|
||||
import { ToolRegistryService } from 'src/engine/core-modules/tool-provider/services/tool-registry.service';
|
||||
import { WorkspaceEntity } from 'src/engine/core-modules/workspace/workspace.entity';
|
||||
import { AgentAsyncExecutorService } from 'src/engine/metadata-modules/ai/ai-agent-execution/services/agent-async-executor.service';
|
||||
import { type AgentEntity } from 'src/engine/metadata-modules/ai/ai-agent/entities/agent.entity';
|
||||
import { NATIVE_WEB_SEARCH_COST_PER_CALL_DOLLARS } from 'src/engine/metadata-modules/ai/ai-billing/constants/native-web-search-cost-per-call-dollars';
|
||||
import { AiBillingService } from 'src/engine/metadata-modules/ai/ai-billing/services/ai-billing.service';
|
||||
import { AiModelConfigService } from 'src/engine/metadata-modules/ai/ai-models/services/ai-model-config.service';
|
||||
import { AiModelRegistryService } from 'src/engine/metadata-modules/ai/ai-models/services/ai-model-registry.service';
|
||||
@@ -33,10 +35,20 @@ jest.mock('ai', () => ({
|
||||
}),
|
||||
}));
|
||||
|
||||
const generateTextMock = generateText as jest.MockedFunction<
|
||||
typeof generateText
|
||||
>;
|
||||
|
||||
describe('AgentAsyncExecutorService — workflow agent role-scoped tool resolution', () => {
|
||||
let service: AgentAsyncExecutorService;
|
||||
let toolRegistry: { getToolsByCategories: jest.Mock };
|
||||
let roleTargetRepository: { findOne: jest.Mock };
|
||||
let aiBillingService: {
|
||||
decrementAndCheckAvailableCredits: jest.Mock;
|
||||
calculateCost: jest.Mock;
|
||||
emitAiTokenUsageEvent: jest.Mock;
|
||||
billNativeWebSearchUsage: jest.Mock;
|
||||
};
|
||||
|
||||
const agentId = 'agent-1';
|
||||
const workspaceId = 'workspace-1';
|
||||
@@ -54,6 +66,16 @@ describe('AgentAsyncExecutorService — workflow agent role-scoped tool resoluti
|
||||
beforeEach(async () => {
|
||||
toolRegistry = { getToolsByCategories: jest.fn().mockResolvedValue({}) };
|
||||
roleTargetRepository = { findOne: jest.fn() };
|
||||
aiBillingService = {
|
||||
decrementAndCheckAvailableCredits: jest
|
||||
.fn()
|
||||
.mockResolvedValue({ hasNoMoreAvailableCredits: false }),
|
||||
calculateCost: jest.fn().mockReturnValue(0),
|
||||
emitAiTokenUsageEvent: jest.fn(),
|
||||
billNativeWebSearchUsage: jest.fn(),
|
||||
};
|
||||
|
||||
generateTextMock.mockClear();
|
||||
|
||||
const module: TestingModule = await Test.createTestingModule({
|
||||
providers: [
|
||||
@@ -82,17 +104,7 @@ describe('AgentAsyncExecutorService — workflow agent role-scoped tool resoluti
|
||||
bind: jest.fn().mockReturnValue({}),
|
||||
},
|
||||
},
|
||||
{
|
||||
provide: AiBillingService,
|
||||
useValue: {
|
||||
decrementAndCheckAvailableCredits: jest
|
||||
.fn()
|
||||
.mockResolvedValue({ hasNoMoreAvailableCredits: false }),
|
||||
calculateCost: jest.fn().mockReturnValue(0),
|
||||
emitAiTokenUsageEvent: jest.fn(),
|
||||
billNativeWebSearchUsage: jest.fn(),
|
||||
},
|
||||
},
|
||||
{ provide: AiBillingService, useValue: aiBillingService },
|
||||
{
|
||||
provide: BillingUsageService,
|
||||
useValue: {
|
||||
@@ -144,4 +156,79 @@ describe('AgentAsyncExecutorService — workflow agent role-scoped tool resoluti
|
||||
|
||||
expect(toolRegistry.getToolsByCategories).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
describe('cost folding', () => {
|
||||
const baseUsage = {
|
||||
inputTokens: 100,
|
||||
outputTokens: 50,
|
||||
totalTokens: 150,
|
||||
inputTokenDetails: {
|
||||
noCacheTokens: 100,
|
||||
cacheReadTokens: 0,
|
||||
cacheWriteTokens: 0,
|
||||
},
|
||||
outputTokenDetails: { textTokens: 50, reasoningTokens: 0 },
|
||||
};
|
||||
|
||||
it('returns token cost only when no native web searches happened', async () => {
|
||||
roleTargetRepository.findOne.mockResolvedValueOnce({
|
||||
roleId: agentRoleId,
|
||||
});
|
||||
aiBillingService.calculateCost.mockReturnValue(0.0042);
|
||||
generateTextMock.mockResolvedValueOnce({
|
||||
text: '',
|
||||
steps: [{ toolCalls: [] }],
|
||||
usage: baseUsage,
|
||||
} as unknown as Awaited<ReturnType<typeof generateText>>);
|
||||
|
||||
const result = await service.executeAgent({
|
||||
agent: buildAgent(),
|
||||
userPrompt: 'test',
|
||||
workspaceId,
|
||||
});
|
||||
|
||||
expect(result.nativeWebSearchCallCount).toBe(0);
|
||||
expect(result.totalCostInDollars).toBeCloseTo(0.0042, 6);
|
||||
// credits = dollars * 1_000_000
|
||||
expect(result.creditsUsedMicro).toBe(4200);
|
||||
});
|
||||
|
||||
it('folds native web search dollars into totalCostInDollars and creditsUsedMicro', async () => {
|
||||
roleTargetRepository.findOne.mockResolvedValueOnce({
|
||||
roleId: agentRoleId,
|
||||
});
|
||||
aiBillingService.calculateCost.mockReturnValue(0.01);
|
||||
generateTextMock.mockResolvedValueOnce({
|
||||
text: '',
|
||||
steps: [
|
||||
{
|
||||
toolCalls: [
|
||||
{ toolName: 'web_search' },
|
||||
{ toolName: 'web_search' },
|
||||
{ toolName: 'some_other_tool' },
|
||||
],
|
||||
},
|
||||
{ toolCalls: [{ toolName: 'web_search' }] },
|
||||
],
|
||||
usage: baseUsage,
|
||||
} as unknown as Awaited<ReturnType<typeof generateText>>);
|
||||
|
||||
const result = await service.executeAgent({
|
||||
agent: buildAgent(),
|
||||
userPrompt: 'test',
|
||||
workspaceId,
|
||||
});
|
||||
|
||||
const expectedSearchCost = 3 * NATIVE_WEB_SEARCH_COST_PER_CALL_DOLLARS;
|
||||
|
||||
expect(result.nativeWebSearchCallCount).toBe(3);
|
||||
expect(result.totalCostInDollars).toBeCloseTo(
|
||||
0.01 + expectedSearchCost,
|
||||
6,
|
||||
);
|
||||
expect(result.creditsUsedMicro).toBe(
|
||||
Math.round((0.01 + expectedSearchCost) * 1_000_000),
|
||||
);
|
||||
});
|
||||
});
|
||||
});
|
||||
|
||||
+59
-42
@@ -7,6 +7,7 @@ import {
|
||||
type LanguageModelUsage,
|
||||
Output,
|
||||
stepCountIs,
|
||||
type StepResult,
|
||||
type ToolSet,
|
||||
} from 'ai';
|
||||
import { AUTO_SELECT_SMART_MODEL_ID } from 'twenty-shared/constants';
|
||||
@@ -27,6 +28,7 @@ import { AGENT_CONFIG } from 'src/engine/metadata-modules/ai/ai-agent/constants/
|
||||
import { WORKFLOW_SYSTEM_PROMPTS } from 'src/engine/metadata-modules/ai/ai-agent/constants/agent-system-prompts.const';
|
||||
import { type AgentEntity } from 'src/engine/metadata-modules/ai/ai-agent/entities/agent.entity';
|
||||
import { repairToolCall } from 'src/engine/metadata-modules/ai/ai-agent/utils/repair-tool-call.util';
|
||||
import { NATIVE_WEB_SEARCH_COST_PER_CALL_DOLLARS } from 'src/engine/metadata-modules/ai/ai-billing/constants/native-web-search-cost-per-call-dollars';
|
||||
import { AiBillingService } from 'src/engine/metadata-modules/ai/ai-billing/services/ai-billing.service';
|
||||
import { convertDollarsToBillingCredits } from 'src/engine/metadata-modules/ai/ai-billing/utils/convert-dollars-to-billing-credits.util';
|
||||
import { countNativeWebSearchCallsFromSteps } from 'src/engine/metadata-modules/ai/ai-billing/utils/count-native-web-search-calls-from-steps.util';
|
||||
@@ -120,6 +122,7 @@ export class AgentAsyncExecutorService {
|
||||
let accumulatedUsage: LanguageModelUsage = EMPTY_USAGE;
|
||||
let cacheCreationTokens = 0;
|
||||
let nativeWebSearchCallCount = 0;
|
||||
let executionSteps: StepResult<ToolSet>[] = [];
|
||||
|
||||
try {
|
||||
if (agent) {
|
||||
@@ -257,69 +260,83 @@ export class AgentAsyncExecutorService {
|
||||
nativeWebSearchCallCount = countNativeWebSearchCallsFromSteps(
|
||||
textResponse.steps,
|
||||
);
|
||||
executionSteps = textResponse.steps;
|
||||
|
||||
const agentSchema =
|
||||
agent?.responseFormat?.type === 'json'
|
||||
? agent.responseFormat.schema
|
||||
: undefined;
|
||||
|
||||
if (!agentSchema) {
|
||||
return {
|
||||
result: { response: textResponse.text },
|
||||
usage: textResponse.usage,
|
||||
cacheCreationTokens,
|
||||
nativeWebSearchCallCount,
|
||||
hasNoMoreAvailableCredits,
|
||||
};
|
||||
}
|
||||
let result: object = { response: textResponse.text };
|
||||
|
||||
const structuredResult = await generateText({
|
||||
system: WORKFLOW_SYSTEM_PROMPTS.OUTPUT_GENERATOR,
|
||||
model: registeredModel.model,
|
||||
prompt: `Based on the following execution results, generate the structured output according to the schema:
|
||||
if (agentSchema) {
|
||||
const structuredResult = await generateText({
|
||||
system: WORKFLOW_SYSTEM_PROMPTS.OUTPUT_GENERATOR,
|
||||
model: registeredModel.model,
|
||||
prompt: `Based on the following execution results, generate the structured output according to the schema:
|
||||
|
||||
Execution Results: ${textResponse.text}
|
||||
|
||||
Please generate the structured output based on the execution results and context above.`,
|
||||
output: Output.object({ schema: jsonSchema(agentSchema) }),
|
||||
experimental_telemetry: AI_TELEMETRY_CONFIG,
|
||||
onStepFinish: async (step) => {
|
||||
const { hasNoMoreAvailableCredits: stepHasNoMoreAvailableCredits } =
|
||||
await this.aiBillingService.decrementAndCheckAvailableCredits(
|
||||
registeredModel.modelId,
|
||||
{
|
||||
usage: step.usage,
|
||||
cacheCreationTokens: extractCacheCreationTokens(
|
||||
step.providerMetadata,
|
||||
),
|
||||
},
|
||||
workspaceId,
|
||||
);
|
||||
output: Output.object({ schema: jsonSchema(agentSchema) }),
|
||||
experimental_telemetry: AI_TELEMETRY_CONFIG,
|
||||
onStepFinish: async (step) => {
|
||||
const { hasNoMoreAvailableCredits: stepHasNoMoreAvailableCredits } =
|
||||
await this.aiBillingService.decrementAndCheckAvailableCredits(
|
||||
registeredModel.modelId,
|
||||
{
|
||||
usage: step.usage,
|
||||
cacheCreationTokens: extractCacheCreationTokens(
|
||||
step.providerMetadata,
|
||||
),
|
||||
},
|
||||
workspaceId,
|
||||
);
|
||||
|
||||
if (stepHasNoMoreAvailableCredits) {
|
||||
hasNoMoreAvailableCredits = true;
|
||||
}
|
||||
},
|
||||
});
|
||||
if (stepHasNoMoreAvailableCredits) {
|
||||
hasNoMoreAvailableCredits = true;
|
||||
}
|
||||
},
|
||||
});
|
||||
|
||||
accumulatedUsage = mergeLanguageModelUsage(
|
||||
textResponse.usage,
|
||||
structuredResult.usage,
|
||||
);
|
||||
|
||||
if (structuredResult.output == null) {
|
||||
throw new AiException(
|
||||
'Failed to generate structured output from execution results',
|
||||
AiExceptionCode.AGENT_EXECUTION_FAILED,
|
||||
accumulatedUsage = mergeLanguageModelUsage(
|
||||
textResponse.usage,
|
||||
structuredResult.usage,
|
||||
);
|
||||
executionSteps = [...textResponse.steps, ...structuredResult.steps];
|
||||
|
||||
if (structuredResult.output == null) {
|
||||
throw new AiException(
|
||||
'Failed to generate structured output from execution results',
|
||||
AiExceptionCode.AGENT_EXECUTION_FAILED,
|
||||
);
|
||||
}
|
||||
|
||||
result = structuredResult.output as object;
|
||||
}
|
||||
|
||||
const resolvedModelId = registeredModel.modelId;
|
||||
const tokenCostInDollars = this.aiBillingService.calculateCost(
|
||||
resolvedModelId,
|
||||
{ usage: accumulatedUsage, cacheCreationTokens },
|
||||
);
|
||||
const totalCostInDollars =
|
||||
tokenCostInDollars +
|
||||
nativeWebSearchCallCount * NATIVE_WEB_SEARCH_COST_PER_CALL_DOLLARS;
|
||||
const creditsUsedMicro = Math.round(
|
||||
convertDollarsToBillingCredits(totalCostInDollars),
|
||||
);
|
||||
|
||||
return {
|
||||
result: structuredResult.output as object,
|
||||
result,
|
||||
usage: accumulatedUsage,
|
||||
cacheCreationTokens,
|
||||
nativeWebSearchCallCount,
|
||||
hasNoMoreAvailableCredits,
|
||||
steps: executionSteps,
|
||||
modelId: resolvedModelId,
|
||||
totalCostInDollars,
|
||||
creditsUsedMicro,
|
||||
};
|
||||
} catch (error) {
|
||||
if (error instanceof AiException) {
|
||||
|
||||
+5
-1
@@ -1,4 +1,4 @@
|
||||
import { type LanguageModelUsage } from 'ai';
|
||||
import { type LanguageModelUsage, type StepResult, type ToolSet } from 'ai';
|
||||
|
||||
export interface AgentExecutionResult {
|
||||
result: object;
|
||||
@@ -6,4 +6,8 @@ export interface AgentExecutionResult {
|
||||
cacheCreationTokens: number;
|
||||
nativeWebSearchCallCount: number;
|
||||
hasNoMoreAvailableCredits: boolean;
|
||||
steps?: StepResult<ToolSet>[];
|
||||
modelId?: string;
|
||||
totalCostInDollars?: number;
|
||||
creditsUsedMicro?: number;
|
||||
}
|
||||
|
||||
+226
@@ -0,0 +1,226 @@
|
||||
import { type StepResult, type ToolSet } from 'ai';
|
||||
|
||||
import { mapAiStepsToToolCallLogs } from 'src/engine/metadata-modules/ai/ai-agent-execution/utils/map-ai-steps-to-tool-call-logs.util';
|
||||
|
||||
type StepContentPart = StepResult<ToolSet>['content'][number];
|
||||
|
||||
const buildStep = (content: StepContentPart[]): StepResult<ToolSet> =>
|
||||
({ content }) as unknown as StepResult<ToolSet>;
|
||||
|
||||
describe('mapAiStepsToToolCallLogs', () => {
|
||||
it('returns an empty array when there are no steps', () => {
|
||||
expect(mapAiStepsToToolCallLogs([])).toEqual([]);
|
||||
});
|
||||
|
||||
it('pairs a tool-call with its tool-result into a single success entry', () => {
|
||||
const steps = [
|
||||
buildStep([
|
||||
{
|
||||
type: 'tool-call',
|
||||
toolName: 'findRecords',
|
||||
toolCallId: 'call_1',
|
||||
input: { limit: 10 },
|
||||
} as StepContentPart,
|
||||
{
|
||||
type: 'tool-result',
|
||||
toolName: 'findRecords',
|
||||
toolCallId: 'call_1',
|
||||
input: { limit: 10 },
|
||||
output: { totalCount: 2 },
|
||||
} as StepContentPart,
|
||||
]),
|
||||
];
|
||||
|
||||
const result = mapAiStepsToToolCallLogs(steps);
|
||||
|
||||
expect(result).toHaveLength(1);
|
||||
expect(result[0]).toMatchObject({
|
||||
toolName: 'findRecords',
|
||||
toolCallId: 'call_1',
|
||||
state: 'success',
|
||||
output: { totalCount: 2 },
|
||||
});
|
||||
});
|
||||
|
||||
it('marks a tool-call followed by tool-error as error and records the message', () => {
|
||||
const steps = [
|
||||
buildStep([
|
||||
{
|
||||
type: 'tool-call',
|
||||
toolName: 'createNote',
|
||||
toolCallId: 'call_2',
|
||||
input: { title: 'x' },
|
||||
} as StepContentPart,
|
||||
{
|
||||
type: 'tool-error',
|
||||
toolName: 'createNote',
|
||||
toolCallId: 'call_2',
|
||||
input: { title: 'x' },
|
||||
error: new Error('Validation failed'),
|
||||
} as StepContentPart,
|
||||
]),
|
||||
];
|
||||
|
||||
const result = mapAiStepsToToolCallLogs(steps);
|
||||
|
||||
expect(result).toHaveLength(1);
|
||||
expect(result[0].state).toBe('error');
|
||||
expect(result[0].errorMessage).toContain('Validation failed');
|
||||
});
|
||||
|
||||
it('truncates oversized tool input and output', () => {
|
||||
const longString = 'x'.repeat(50_000);
|
||||
|
||||
const steps = [
|
||||
buildStep([
|
||||
{
|
||||
type: 'tool-call',
|
||||
toolName: 'fetchUrl',
|
||||
toolCallId: 'call_3',
|
||||
input: { html: longString },
|
||||
} as StepContentPart,
|
||||
{
|
||||
type: 'tool-result',
|
||||
toolName: 'fetchUrl',
|
||||
toolCallId: 'call_3',
|
||||
input: { html: longString },
|
||||
output: { body: longString },
|
||||
} as StepContentPart,
|
||||
]),
|
||||
];
|
||||
|
||||
const result = mapAiStepsToToolCallLogs(steps, {
|
||||
maxToolInputBytes: 100,
|
||||
maxToolOutputBytes: 100,
|
||||
});
|
||||
|
||||
const serializedInput = JSON.stringify(result[0].input);
|
||||
const serializedOutput = JSON.stringify(result[0].output);
|
||||
|
||||
expect(serializedInput.length).toBeLessThan(200);
|
||||
expect(serializedInput).toContain('truncated');
|
||||
expect(serializedOutput.length).toBeLessThan(200);
|
||||
expect(serializedOutput).toContain('truncated');
|
||||
});
|
||||
|
||||
it('stops collecting tool calls past the per-step cap', () => {
|
||||
const content: StepContentPart[] = [];
|
||||
|
||||
for (let i = 0; i < 10; i++) {
|
||||
content.push({
|
||||
type: 'tool-call',
|
||||
toolName: 'noop',
|
||||
toolCallId: `call_${i}`,
|
||||
input: {},
|
||||
} as StepContentPart);
|
||||
}
|
||||
|
||||
const steps = [buildStep(content)];
|
||||
|
||||
const result = mapAiStepsToToolCallLogs(steps, {
|
||||
maxToolCallsPerStep: 3,
|
||||
});
|
||||
|
||||
expect(result).toHaveLength(3);
|
||||
});
|
||||
|
||||
it('preserves all web_search sources in tool output', () => {
|
||||
const manySources = Array.from({ length: 25 }, (_, index) => ({
|
||||
url: `https://example.com/${index}`,
|
||||
type: 'url',
|
||||
}));
|
||||
|
||||
const steps = [
|
||||
buildStep([
|
||||
{
|
||||
type: 'tool-call',
|
||||
toolName: 'web_search',
|
||||
toolCallId: 'call_search',
|
||||
input: {},
|
||||
} as StepContentPart,
|
||||
{
|
||||
type: 'tool-result',
|
||||
toolName: 'web_search',
|
||||
toolCallId: 'call_search',
|
||||
input: {},
|
||||
output: {
|
||||
action: { type: 'search', query: 'twenty crm' },
|
||||
sources: manySources,
|
||||
},
|
||||
} as StepContentPart,
|
||||
]),
|
||||
];
|
||||
|
||||
const result = mapAiStepsToToolCallLogs(steps);
|
||||
const output = result[0].output as {
|
||||
sources: unknown[];
|
||||
sourcesDroppedCount?: number;
|
||||
};
|
||||
|
||||
expect(output.sources).toHaveLength(25);
|
||||
expect(output.sourcesDroppedCount).toBeUndefined();
|
||||
});
|
||||
|
||||
it('strips searchVector from nested record outputs', () => {
|
||||
const steps = [
|
||||
buildStep([
|
||||
{
|
||||
type: 'tool-call',
|
||||
toolName: 'find_companies',
|
||||
toolCallId: 'call_find',
|
||||
input: {},
|
||||
} as StepContentPart,
|
||||
{
|
||||
type: 'tool-result',
|
||||
toolName: 'find_companies',
|
||||
toolCallId: 'call_find',
|
||||
input: {},
|
||||
output: {
|
||||
result: {
|
||||
count: '1',
|
||||
records: [
|
||||
{
|
||||
id: 'abc',
|
||||
name: 'Apple',
|
||||
searchVector: "'apple':1 'inc':2",
|
||||
},
|
||||
],
|
||||
},
|
||||
},
|
||||
} as StepContentPart,
|
||||
]),
|
||||
];
|
||||
|
||||
const result = mapAiStepsToToolCallLogs(steps);
|
||||
const output = result[0].output as {
|
||||
result: { records: Array<Record<string, unknown>> };
|
||||
};
|
||||
|
||||
expect(output.result.records[0]).not.toHaveProperty('searchVector');
|
||||
expect(output.result.records[0].name).toBe('Apple');
|
||||
});
|
||||
|
||||
it('ignores text / reasoning / source parts', () => {
|
||||
const steps = [
|
||||
buildStep([
|
||||
{ type: 'text', text: 'hello' } as StepContentPart,
|
||||
{
|
||||
type: 'reasoning',
|
||||
text: 'thinking…',
|
||||
state: 'done',
|
||||
} as StepContentPart,
|
||||
{
|
||||
type: 'tool-call',
|
||||
toolName: 'foo',
|
||||
toolCallId: 'call_only',
|
||||
input: {},
|
||||
} as StepContentPart,
|
||||
]),
|
||||
];
|
||||
|
||||
const result = mapAiStepsToToolCallLogs(steps);
|
||||
|
||||
expect(result).toHaveLength(1);
|
||||
expect(result[0].toolName).toBe('foo');
|
||||
});
|
||||
});
|
||||
+142
@@ -0,0 +1,142 @@
|
||||
import { type StepResult, type ToolSet } from 'ai';
|
||||
|
||||
import { type AiToolCallLog } from 'twenty-shared/workflow';
|
||||
|
||||
import {
|
||||
TRUNCATION_SENTINEL,
|
||||
truncateStringToUtf8ByteBudget,
|
||||
} from 'src/utils/truncate-string-to-utf8-byte-budget.util';
|
||||
|
||||
const DEFAULT_MAX_TOOL_INPUT_BYTES = 32_000;
|
||||
const DEFAULT_MAX_TOOL_OUTPUT_BYTES = 64_000;
|
||||
const DEFAULT_MAX_TOOL_CALLS_PER_STEP = 200;
|
||||
const MAX_ERROR_MESSAGE_LENGTH = 2_000;
|
||||
|
||||
const NOISY_RECORD_KEYS = new Set(['searchVector']);
|
||||
|
||||
const stripNoisyKeysDeep = (value: unknown): unknown => {
|
||||
if (Array.isArray(value)) {
|
||||
return value.map(stripNoisyKeysDeep);
|
||||
}
|
||||
|
||||
if (value !== null && typeof value === 'object') {
|
||||
const sanitized: Record<string, unknown> = {};
|
||||
|
||||
for (const [key, nested] of Object.entries(value)) {
|
||||
if (NOISY_RECORD_KEYS.has(key)) {
|
||||
continue;
|
||||
}
|
||||
|
||||
sanitized[key] = stripNoisyKeysDeep(nested);
|
||||
}
|
||||
|
||||
return sanitized;
|
||||
}
|
||||
|
||||
return value;
|
||||
};
|
||||
|
||||
const truncateUnknownForLog = (value: unknown, maxBytes: number): unknown => {
|
||||
if (value === undefined || value === null) {
|
||||
return value;
|
||||
}
|
||||
|
||||
if (typeof value === 'string') {
|
||||
const { value: truncatedValue, truncated } = truncateStringToUtf8ByteBudget(
|
||||
value,
|
||||
maxBytes,
|
||||
);
|
||||
|
||||
return truncated ? truncatedValue : value;
|
||||
}
|
||||
|
||||
let serialized: string;
|
||||
|
||||
try {
|
||||
serialized = JSON.stringify(value);
|
||||
} catch {
|
||||
return TRUNCATION_SENTINEL;
|
||||
}
|
||||
|
||||
const { value: truncatedValue, truncated } = truncateStringToUtf8ByteBudget(
|
||||
serialized,
|
||||
maxBytes,
|
||||
);
|
||||
|
||||
return truncated ? truncatedValue : value;
|
||||
};
|
||||
|
||||
export type MapAiStepsToToolCallLogsOptions = {
|
||||
maxToolInputBytes?: number;
|
||||
maxToolOutputBytes?: number;
|
||||
maxToolCallsPerStep?: number;
|
||||
};
|
||||
|
||||
export const mapAiStepsToToolCallLogs = (
|
||||
steps: StepResult<ToolSet>[],
|
||||
options: MapAiStepsToToolCallLogsOptions = {},
|
||||
): AiToolCallLog[] => {
|
||||
const maxToolInputBytes =
|
||||
options.maxToolInputBytes ?? DEFAULT_MAX_TOOL_INPUT_BYTES;
|
||||
const maxToolOutputBytes =
|
||||
options.maxToolOutputBytes ?? DEFAULT_MAX_TOOL_OUTPUT_BYTES;
|
||||
const maxToolCallsPerStep =
|
||||
options.maxToolCallsPerStep ?? DEFAULT_MAX_TOOL_CALLS_PER_STEP;
|
||||
|
||||
const ordered: AiToolCallLog[] = [];
|
||||
const openByCallId = new Map<string, AiToolCallLog>();
|
||||
|
||||
for (const step of steps) {
|
||||
if (ordered.length >= maxToolCallsPerStep) {
|
||||
break;
|
||||
}
|
||||
|
||||
for (const part of step.content) {
|
||||
if (ordered.length >= maxToolCallsPerStep) {
|
||||
break;
|
||||
}
|
||||
|
||||
if (part.type === 'tool-call') {
|
||||
const entry: AiToolCallLog = {
|
||||
toolName: part.toolName,
|
||||
toolCallId: part.toolCallId,
|
||||
input: truncateUnknownForLog(part.input, maxToolInputBytes),
|
||||
state: 'started',
|
||||
providerExecuted:
|
||||
'providerExecuted' in part && part.providerExecuted === true,
|
||||
};
|
||||
openByCallId.set(part.toolCallId, entry);
|
||||
ordered.push(entry);
|
||||
continue;
|
||||
}
|
||||
|
||||
if (part.type === 'tool-result') {
|
||||
const entry = openByCallId.get(part.toolCallId);
|
||||
|
||||
if (entry) {
|
||||
entry.output = truncateUnknownForLog(
|
||||
stripNoisyKeysDeep(part.output),
|
||||
maxToolOutputBytes,
|
||||
);
|
||||
entry.state = 'success';
|
||||
}
|
||||
continue;
|
||||
}
|
||||
|
||||
if (part.type === 'tool-error') {
|
||||
const entry = openByCallId.get(part.toolCallId);
|
||||
|
||||
if (entry) {
|
||||
entry.errorMessage = String(part.error).slice(
|
||||
0,
|
||||
MAX_ERROR_MESSAGE_LENGTH,
|
||||
);
|
||||
entry.state = 'error';
|
||||
}
|
||||
continue;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
return ordered;
|
||||
};
|
||||
Reference in New Issue
Block a user