Provide additional logsobservability to workflow runs (per node) (#21142)

Surfaces per-step "Logs" tabs in the workflow run side panel so users
can see what each step actually did (model + tokens + tool calls for AI,
console output for serverless functions, request/response for HTTP,
recipients/body for Email).

<img width="546" height="501" alt="ai_agent_without_websearch"
src="https://github.com/user-attachments/assets/c6ca3518-9489-4484-a570-3d0569ff3b03"
/>

## Storage

- New `stepLogs` JSONB column on the `workflowRun` workspace entity,
typed as `Record<string, WorkflowRunStepLog>` (keyed by step id).
- Schema lives in `twenty-shared`: `workflowRunStepLogSchema` with a
discriminated `details.type` union for `AI_AGENT | CODE | HTTP_REQUEST |
EMAIL` — frontends and backends consume the same Zod-inferred type.
- Field is added to existing workspaces via a workspace upgrade command
(`2-9 add-workflow-run-step-logs-field`); the standard-object metadata
declares it for new workspaces.
- Writes happen atomically per step in
`WorkflowRunStepLogWorkspaceService.setStepLog` using `jsonb_set`. That
lets concurrent steps in the same run write their own keys without
contending with the existing lock around `workflowRun.state`.
- Per-step payload is hard-capped at 256 KB; anything larger is dropped
with a `logger.warn`, so a pathological tool call can never bloat a row.
See below for more information.

## How logs are produced

**Aalmost everything was already being collected; this PR mostly
persists and renders it.**

- **AI agent** — `AgentAsyncExecutorService` already tracked token
usage, model id, native web-search count, and the AI SDK's `steps[]`. We
map those into the log via `mapAiStepsToToolCallLogs` (`searchVector`
stripped from record outputs, per-call input/output capped at 32/64 KB,
max 200 tool calls per step). The only new measurement is a wall-clock
`durationMs` taken around `executeAgent`, and we now fold native
web-search cost into the displayed `totalCostInDollars` (it was already
billed, just not shown).
- **Code / serverless function** — reuses the `console.log` output the
function runner already returns (`logsByLevel`);
`build-code-step-log.util` only repackages it.
- **HTTP request** — built from the action's existing input/output via
`build-http-request-step-log.util`. No new signals collected.
- **Email (send / draft)** — added `sanitizedHtmlBody` + `plainTextBody`
to the existing tool outputs (a small additive change), then
`build-email-step-log.util` consumes them.

No additional AI inference or external calls are made for logging — the
cost is a small CPU overhead per step plus the JSONB write.

## Security

The log surface intentionally shows whatever the workflow touched, which
made redaction and sanitization the main design concern.

- **HTTP — secrets in headers**: existing `SENSITIVE_HEADER_NAMES` set
(Authorization, Cookie, …) replaced with `[redacted]` in both request
and response.
- **HTTP — secrets in URLs**: `SENSITIVE_URL_PARAM_NAMES` (e.g.
`api_key`, `token`, `access_token`) replaced in the query string via
`URL`-based parsing.
- **HTTP — secrets in bodies**: `SENSITIVE_BODY_KEY_REGEX` deep-walks
JSON request/response bodies (object input or stringified JSON) and
redacts matching keys. Applied to the `error` field too, since
transport-layer errors sometimes embed structured payloads.
- **Email — XSS risk in body preview**: tool outputs now expose a
server-side `sanitizedHtmlBody`; the log builder prefers it over the raw
user-authored `input.body`, with `plainTextBody` as a second fallback.
The original raw body is only used if sanitization didn't happen (e.g.
tool failed before composing).
- **AI — internal/noisy data**: `searchVector` (Postgres tsvector
strings) is stripped from record outputs returned by Twenty tools to
avoid leaking internal full-text-search payloads.
- **DB bloat / runaway agents**: 256 KB per-step cap + 32 KB / 64 KB
per-tool-call input/output cap + 200 tool calls per step.

<img width="547" height="307" alt="logic_function"
src="https://github.com/user-attachments/assets/dd4a3d16-67f2-434b-95b3-bdcaf9ed053d"
/>

## More details on Log size & truncation

Logs are stored in `workflowRun.stepLogs` (JSONB), keyed by `stepId`.

### Per-step cap

Each step's log is hard-capped at **256 KB** (`MAX_STEP_LOG_BYTES` in
`WorkflowRunStepLogWorkspaceService.setStepLog`).

For ~99% of workflows this is roomy — typical real-world sizes:
- Code / serverless function: 1–20 KB
- HTTP request: 5–70 KB
- Email: 5–30 KB
- AI agent (a handful of tool calls): 5–50 KB

### Two layers of bounding

1. **Per-field truncation** in each builder (before writing):
   - **Code**: ≤ 500 entries, ≤ 4 KB per message, ≤ 8 KB stack trace
   - **HTTP**: ≤ 32 KB per body (request + response), UTF-8 byte-aware
   - **Email**: ≤ 8 KB body preview, UTF-8 byte-aware
- **AI agent**: ≤ 32 KB tool input, ≤ 64 KB tool output, ≤ 200 tool
calls/step

2. **Global per-step safety net** at write time: if the assembled
`stepLog` still exceeds 256 KB, the write is **dropped entirely** with a
`logger.warn`. The workflow itself keeps running unaffected.

### What this means in practice

- **Safe**: workflow execution, step results, downstream steps — never
blocked by log size.
- **Safe**: iterators (each iteration overwrites the previous log for
that `stepId`, so they can't accumulate).
- **Safe**: step retries (same `stepId` is overwritten, not appended).
- **Possible**: an AI agent step with many large tool outputs (e.g., 50+
heavy `web_search` calls) can exceed 256 KB → the **entire** step's log
is dropped, side panel shows "No logs were recorded for this step". The
user has no explicit signal that the log was dropped due to size (only
server-side warn).
- **Possible** (theoretical): a workflow with hundreds of distinct steps
could push the row toward Postgres's internal ~256 MB jsonb limit.
Beyond that, individual `jsonb_set` writes would error and be swallowed
by the action's try/catch — workflow still completes.

### Possible future hardening (not in this PR)

- Replace "drop entire log" with a stub that preserves the summary card
(cost, duration, status) and marks `truncated.reason = 'size_cap'`.
- Surface size-drops in the UI (similar to the existing
`<StyledTruncatedNotice>`).
- Emit a metric so dropped logs are observable in dashboards.
This commit is contained in:
Marie
2026-06-03 18:53:47 +02:00
committed by GitHub
parent 0671ff3de5
commit 4b15b949f3
74 changed files with 4834 additions and 445 deletions
@@ -2327,7 +2327,7 @@ exports[`getStandardObjectMetadataRelatedEntityIds should return standard object
"id": "00000000-0000-0000-0000-000000000697",
},
"searchVector": {
"id": "00000000-0000-0000-0000-000000000702",
"id": "00000000-0000-0000-0000-000000000703",
},
"startedAt": {
"id": "00000000-0000-0000-0000-000000000694",
@@ -2338,9 +2338,12 @@ exports[`getStandardObjectMetadataRelatedEntityIds should return standard object
"status": {
"id": "00000000-0000-0000-0000-000000000696",
},
"timelineActivities": {
"stepLogs": {
"id": "00000000-0000-0000-0000-000000000701",
},
"timelineActivities": {
"id": "00000000-0000-0000-0000-000000000702",
},
"updatedAt": {
"id": "00000000-0000-0000-0000-000000000688",
},
@@ -2354,80 +2357,80 @@ exports[`getStandardObjectMetadataRelatedEntityIds should return standard object
"id": "00000000-0000-0000-0000-000000000691",
},
},
"id": "00000000-0000-0000-0000-000000000725",
"id": "00000000-0000-0000-0000-000000000726",
"views": {
"allWorkflowRuns": {
"id": "00000000-0000-0000-0000-000000000709",
"id": "00000000-0000-0000-0000-000000000710",
"viewFieldGroups": {},
"viewFields": {
"createdBy": {
"id": "00000000-0000-0000-0000-000000000707",
"id": "00000000-0000-0000-0000-000000000708",
},
"name": {
"id": "00000000-0000-0000-0000-000000000703",
},
"startedAt": {
"id": "00000000-0000-0000-0000-000000000706",
},
"status": {
"id": "00000000-0000-0000-0000-000000000705",
},
"workflow": {
"id": "00000000-0000-0000-0000-000000000704",
},
"startedAt": {
"id": "00000000-0000-0000-0000-000000000707",
},
"status": {
"id": "00000000-0000-0000-0000-000000000706",
},
"workflow": {
"id": "00000000-0000-0000-0000-000000000705",
},
"workflowVersion": {
"id": "00000000-0000-0000-0000-000000000708",
"id": "00000000-0000-0000-0000-000000000709",
},
},
"viewGroups": {},
},
"workflowRunRecordPageFields": {
"id": "00000000-0000-0000-0000-000000000724",
"id": "00000000-0000-0000-0000-000000000725",
"viewFieldGroups": {
"general": {
"id": "00000000-0000-0000-0000-000000000722",
"id": "00000000-0000-0000-0000-000000000723",
},
"system": {
"id": "00000000-0000-0000-0000-000000000723",
"id": "00000000-0000-0000-0000-000000000724",
},
},
"viewFields": {
"createdAt": {
"id": "00000000-0000-0000-0000-000000000715",
},
"createdBy": {
"id": "00000000-0000-0000-0000-000000000716",
},
"endedAt": {
"id": "00000000-0000-0000-0000-000000000714",
},
"enqueuedAt": {
"createdBy": {
"id": "00000000-0000-0000-0000-000000000717",
},
"startedAt": {
"id": "00000000-0000-0000-0000-000000000713",
"endedAt": {
"id": "00000000-0000-0000-0000-000000000715",
},
"state": {
"enqueuedAt": {
"id": "00000000-0000-0000-0000-000000000718",
},
"status": {
"id": "00000000-0000-0000-0000-000000000710",
"startedAt": {
"id": "00000000-0000-0000-0000-000000000714",
},
"timelineActivities": {
"id": "00000000-0000-0000-0000-000000000721",
},
"updatedAt": {
"state": {
"id": "00000000-0000-0000-0000-000000000719",
},
"updatedBy": {
"id": "00000000-0000-0000-0000-000000000720",
},
"workflow": {
"status": {
"id": "00000000-0000-0000-0000-000000000711",
},
"workflowVersion": {
"timelineActivities": {
"id": "00000000-0000-0000-0000-000000000722",
},
"updatedAt": {
"id": "00000000-0000-0000-0000-000000000720",
},
"updatedBy": {
"id": "00000000-0000-0000-0000-000000000721",
},
"workflow": {
"id": "00000000-0000-0000-0000-000000000712",
},
"workflowVersion": {
"id": "00000000-0000-0000-0000-000000000713",
},
},
"viewGroups": {},
},
@@ -2436,115 +2439,115 @@ exports[`getStandardObjectMetadataRelatedEntityIds should return standard object
"workflowVersion": {
"fields": {
"createdAt": {
"id": "00000000-0000-0000-0000-000000000727",
},
"createdBy": {
"id": "00000000-0000-0000-0000-000000000739",
},
"deletedAt": {
"id": "00000000-0000-0000-0000-000000000729",
},
"id": {
"id": "00000000-0000-0000-0000-000000000726",
},
"name": {
"id": "00000000-0000-0000-0000-000000000730",
},
"position": {
"id": "00000000-0000-0000-0000-000000000734",
},
"runs": {
"id": "00000000-0000-0000-0000-000000000735",
},
"searchVector": {
"id": "00000000-0000-0000-0000-000000000738",
},
"status": {
"id": "00000000-0000-0000-0000-000000000733",
},
"steps": {
"id": "00000000-0000-0000-0000-000000000736",
},
"timelineActivities": {
"id": "00000000-0000-0000-0000-000000000737",
},
"trigger": {
"id": "00000000-0000-0000-0000-000000000732",
},
"updatedAt": {
"id": "00000000-0000-0000-0000-000000000728",
},
"updatedBy": {
"createdBy": {
"id": "00000000-0000-0000-0000-000000000740",
},
"workflow": {
"deletedAt": {
"id": "00000000-0000-0000-0000-000000000730",
},
"id": {
"id": "00000000-0000-0000-0000-000000000727",
},
"name": {
"id": "00000000-0000-0000-0000-000000000731",
},
"position": {
"id": "00000000-0000-0000-0000-000000000735",
},
"runs": {
"id": "00000000-0000-0000-0000-000000000736",
},
"searchVector": {
"id": "00000000-0000-0000-0000-000000000739",
},
"status": {
"id": "00000000-0000-0000-0000-000000000734",
},
"steps": {
"id": "00000000-0000-0000-0000-000000000737",
},
"timelineActivities": {
"id": "00000000-0000-0000-0000-000000000738",
},
"trigger": {
"id": "00000000-0000-0000-0000-000000000733",
},
"updatedAt": {
"id": "00000000-0000-0000-0000-000000000729",
},
"updatedBy": {
"id": "00000000-0000-0000-0000-000000000741",
},
"workflow": {
"id": "00000000-0000-0000-0000-000000000732",
},
},
"id": "00000000-0000-0000-0000-000000000760",
"id": "00000000-0000-0000-0000-000000000761",
"views": {
"allWorkflowVersions": {
"id": "00000000-0000-0000-0000-000000000746",
"id": "00000000-0000-0000-0000-000000000747",
"viewFieldGroups": {},
"viewFields": {
"name": {
"id": "00000000-0000-0000-0000-000000000741",
"id": "00000000-0000-0000-0000-000000000742",
},
"runs": {
"id": "00000000-0000-0000-0000-000000000745",
"id": "00000000-0000-0000-0000-000000000746",
},
"status": {
"id": "00000000-0000-0000-0000-000000000743",
},
"updatedAt": {
"id": "00000000-0000-0000-0000-000000000744",
},
"updatedAt": {
"id": "00000000-0000-0000-0000-000000000745",
},
"workflow": {
"id": "00000000-0000-0000-0000-000000000742",
"id": "00000000-0000-0000-0000-000000000743",
},
},
"viewGroups": {},
},
"workflowVersionRecordPageFields": {
"id": "00000000-0000-0000-0000-000000000759",
"id": "00000000-0000-0000-0000-000000000760",
"viewFieldGroups": {
"general": {
"id": "00000000-0000-0000-0000-000000000757",
"id": "00000000-0000-0000-0000-000000000758",
},
"system": {
"id": "00000000-0000-0000-0000-000000000758",
"id": "00000000-0000-0000-0000-000000000759",
},
},
"viewFields": {
"createdAt": {
"id": "00000000-0000-0000-0000-000000000750",
},
"createdBy": {
"id": "00000000-0000-0000-0000-000000000752",
},
"runs": {
"id": "00000000-0000-0000-0000-000000000755",
},
"status": {
"id": "00000000-0000-0000-0000-000000000747",
},
"steps": {
"id": "00000000-0000-0000-0000-000000000751",
},
"timelineActivities": {
"id": "00000000-0000-0000-0000-000000000756",
},
"trigger": {
"id": "00000000-0000-0000-0000-000000000749",
},
"updatedAt": {
"createdBy": {
"id": "00000000-0000-0000-0000-000000000753",
},
"updatedBy": {
"runs": {
"id": "00000000-0000-0000-0000-000000000756",
},
"status": {
"id": "00000000-0000-0000-0000-000000000748",
},
"steps": {
"id": "00000000-0000-0000-0000-000000000752",
},
"timelineActivities": {
"id": "00000000-0000-0000-0000-000000000757",
},
"trigger": {
"id": "00000000-0000-0000-0000-000000000750",
},
"updatedAt": {
"id": "00000000-0000-0000-0000-000000000754",
},
"updatedBy": {
"id": "00000000-0000-0000-0000-000000000755",
},
"workflow": {
"id": "00000000-0000-0000-0000-000000000748",
"id": "00000000-0000-0000-0000-000000000749",
},
},
"viewGroups": {},
@@ -2554,123 +2557,123 @@ exports[`getStandardObjectMetadataRelatedEntityIds should return standard object
"workspaceMember": {
"fields": {
"accountOwnerForCompanies": {
"id": "00000000-0000-0000-0000-000000000774",
},
"assignedTasks": {
"id": "00000000-0000-0000-0000-000000000772",
},
"avatarUrl": {
"id": "00000000-0000-0000-0000-000000000769",
},
"blocklist": {
"id": "00000000-0000-0000-0000-000000000776",
},
"calendarEventParticipants": {
"id": "00000000-0000-0000-0000-000000000777",
},
"calendarStartDay": {
"id": "00000000-0000-0000-0000-000000000783",
},
"colorScheme": {
"id": "00000000-0000-0000-0000-000000000767",
},
"createdAt": {
"id": "00000000-0000-0000-0000-000000000762",
},
"createdBy": {
"id": "00000000-0000-0000-0000-000000000785",
},
"dateFormat": {
"id": "00000000-0000-0000-0000-000000000780",
},
"deletedAt": {
"id": "00000000-0000-0000-0000-000000000764",
},
"id": {
"id": "00000000-0000-0000-0000-000000000761",
},
"locale": {
"id": "00000000-0000-0000-0000-000000000768",
},
"messageParticipants": {
"id": "00000000-0000-0000-0000-000000000775",
},
"name": {
"id": "00000000-0000-0000-0000-000000000766",
},
"numberFormat": {
"id": "00000000-0000-0000-0000-000000000784",
},
"ownedOpportunities": {
"assignedTasks": {
"id": "00000000-0000-0000-0000-000000000773",
},
"position": {
"id": "00000000-0000-0000-0000-000000000765",
},
"searchVector": {
"id": "00000000-0000-0000-0000-000000000782",
},
"timeFormat": {
"id": "00000000-0000-0000-0000-000000000781",
},
"timeZone": {
"id": "00000000-0000-0000-0000-000000000779",
},
"timelineActivities": {
"id": "00000000-0000-0000-0000-000000000778",
},
"updatedAt": {
"id": "00000000-0000-0000-0000-000000000763",
},
"updatedBy": {
"id": "00000000-0000-0000-0000-000000000786",
},
"userEmail": {
"avatarUrl": {
"id": "00000000-0000-0000-0000-000000000770",
},
"userId": {
"blocklist": {
"id": "00000000-0000-0000-0000-000000000777",
},
"calendarEventParticipants": {
"id": "00000000-0000-0000-0000-000000000778",
},
"calendarStartDay": {
"id": "00000000-0000-0000-0000-000000000784",
},
"colorScheme": {
"id": "00000000-0000-0000-0000-000000000768",
},
"createdAt": {
"id": "00000000-0000-0000-0000-000000000763",
},
"createdBy": {
"id": "00000000-0000-0000-0000-000000000786",
},
"dateFormat": {
"id": "00000000-0000-0000-0000-000000000781",
},
"deletedAt": {
"id": "00000000-0000-0000-0000-000000000765",
},
"id": {
"id": "00000000-0000-0000-0000-000000000762",
},
"locale": {
"id": "00000000-0000-0000-0000-000000000769",
},
"messageParticipants": {
"id": "00000000-0000-0000-0000-000000000776",
},
"name": {
"id": "00000000-0000-0000-0000-000000000767",
},
"numberFormat": {
"id": "00000000-0000-0000-0000-000000000785",
},
"ownedOpportunities": {
"id": "00000000-0000-0000-0000-000000000774",
},
"position": {
"id": "00000000-0000-0000-0000-000000000766",
},
"searchVector": {
"id": "00000000-0000-0000-0000-000000000783",
},
"timeFormat": {
"id": "00000000-0000-0000-0000-000000000782",
},
"timeZone": {
"id": "00000000-0000-0000-0000-000000000780",
},
"timelineActivities": {
"id": "00000000-0000-0000-0000-000000000779",
},
"updatedAt": {
"id": "00000000-0000-0000-0000-000000000764",
},
"updatedBy": {
"id": "00000000-0000-0000-0000-000000000787",
},
"userEmail": {
"id": "00000000-0000-0000-0000-000000000771",
},
"userId": {
"id": "00000000-0000-0000-0000-000000000772",
},
},
"id": "00000000-0000-0000-0000-000000000799",
"id": "00000000-0000-0000-0000-000000000800",
"views": {
"allWorkspaceMembers": {
"id": "00000000-0000-0000-0000-000000000798",
"id": "00000000-0000-0000-0000-000000000799",
"viewFieldGroups": {},
"viewFields": {
"assignedTasks": {
"id": "00000000-0000-0000-0000-000000000797",
"id": "00000000-0000-0000-0000-000000000798",
},
"avatarUrl": {
"id": "00000000-0000-0000-0000-000000000789",
},
"colorScheme": {
"id": "00000000-0000-0000-0000-000000000790",
},
"createdAt": {
"id": "00000000-0000-0000-0000-000000000795",
},
"dateFormat": {
"id": "00000000-0000-0000-0000-000000000793",
},
"locale": {
"colorScheme": {
"id": "00000000-0000-0000-0000-000000000791",
},
"name": {
"id": "00000000-0000-0000-0000-000000000787",
},
"ownedOpportunities": {
"createdAt": {
"id": "00000000-0000-0000-0000-000000000796",
},
"timeFormat": {
"dateFormat": {
"id": "00000000-0000-0000-0000-000000000794",
},
"timeZone": {
"locale": {
"id": "00000000-0000-0000-0000-000000000792",
},
"userEmail": {
"name": {
"id": "00000000-0000-0000-0000-000000000788",
},
"ownedOpportunities": {
"id": "00000000-0000-0000-0000-000000000797",
},
"timeFormat": {
"id": "00000000-0000-0000-0000-000000000795",
},
"timeZone": {
"id": "00000000-0000-0000-0000-000000000793",
},
"userEmail": {
"id": "00000000-0000-0000-0000-000000000789",
},
},
"viewGroups": {},
},
@@ -308,6 +308,26 @@ export const buildWorkflowRunStandardFlatFieldMetadatas = ({
twentyStandardApplicationId,
now,
}),
stepLogs: createStandardFieldFlatMetadata({
objectName,
workspaceId,
context: {
fieldName: 'stepLogs',
type: FieldMetadataType.RAW_JSON,
label: i18nLabel(msg`Step logs`),
description: i18nLabel(
msg`Per-step observability payload (token usage, tool calls, log entries)`,
),
icon: 'IconTerminal2',
isSystem: true,
isNullable: true,
isUIReadOnly: true,
},
standardObjectMetadataRelatedEntityIds,
dependencyFlatEntityMaps,
twentyStandardApplicationId,
now,
}),
position: createStandardFieldFlatMetadata({
objectName,
workspaceId,