Provide additional logsobservability to workflow runs (per node) (#21142)

Surfaces per-step "Logs" tabs in the workflow run side panel so users can see what each step actually did (model + tokens + tool calls for AI, console output for serverless functions, request/response for HTTP, recipients/body for Email). <img width="546" height="501" alt="ai_agent_without_websearch" src="https://github.com/user-attachments/assets/c6ca3518-9489-4484-a570-3d0569ff3b03" /> ## Storage - New `stepLogs` JSONB column on the `workflowRun` workspace entity, typed as `Record<string, WorkflowRunStepLog>` (keyed by step id). - Schema lives in `twenty-shared`: `workflowRunStepLogSchema` with a discriminated `details.type` union for `AI_AGENT | CODE | HTTP_REQUEST | EMAIL` — frontends and backends consume the same Zod-inferred type. - Field is added to existing workspaces via a workspace upgrade command (`2-9 add-workflow-run-step-logs-field`); the standard-object metadata declares it for new workspaces. - Writes happen atomically per step in `WorkflowRunStepLogWorkspaceService.setStepLog` using `jsonb_set`. That lets concurrent steps in the same run write their own keys without contending with the existing lock around `workflowRun.state`. - Per-step payload is hard-capped at 256 KB; anything larger is dropped with a `logger.warn`, so a pathological tool call can never bloat a row. See below for more information. ## How logs are produced **Aalmost everything was already being collected; this PR mostly persists and renders it.** - **AI agent** — `AgentAsyncExecutorService` already tracked token usage, model id, native web-search count, and the AI SDK's `steps[]`. We map those into the log via `mapAiStepsToToolCallLogs` (`searchVector` stripped from record outputs, per-call input/output capped at 32/64 KB, max 200 tool calls per step). The only new measurement is a wall-clock `durationMs` taken around `executeAgent`, and we now fold native web-search cost into the displayed `totalCostInDollars` (it was already billed, just not shown). - **Code / serverless function** — reuses the `console.log` output the function runner already returns (`logsByLevel`); `build-code-step-log.util` only repackages it. - **HTTP request** — built from the action's existing input/output via `build-http-request-step-log.util`. No new signals collected. - **Email (send / draft)** — added `sanitizedHtmlBody` + `plainTextBody` to the existing tool outputs (a small additive change), then `build-email-step-log.util` consumes them. No additional AI inference or external calls are made for logging — the cost is a small CPU overhead per step plus the JSONB write. ## Security The log surface intentionally shows whatever the workflow touched, which made redaction and sanitization the main design concern. - **HTTP — secrets in headers**: existing `SENSITIVE_HEADER_NAMES` set (Authorization, Cookie, …) replaced with `[redacted]` in both request and response. - **HTTP — secrets in URLs**: `SENSITIVE_URL_PARAM_NAMES` (e.g. `api_key`, `token`, `access_token`) replaced in the query string via `URL`-based parsing. - **HTTP — secrets in bodies**: `SENSITIVE_BODY_KEY_REGEX` deep-walks JSON request/response bodies (object input or stringified JSON) and redacts matching keys. Applied to the `error` field too, since transport-layer errors sometimes embed structured payloads. - **Email — XSS risk in body preview**: tool outputs now expose a server-side `sanitizedHtmlBody`; the log builder prefers it over the raw user-authored `input.body`, with `plainTextBody` as a second fallback. The original raw body is only used if sanitization didn't happen (e.g. tool failed before composing). - **AI — internal/noisy data**: `searchVector` (Postgres tsvector strings) is stripped from record outputs returned by Twenty tools to avoid leaking internal full-text-search payloads. - **DB bloat / runaway agents**: 256 KB per-step cap + 32 KB / 64 KB per-tool-call input/output cap + 200 tool calls per step. <img width="547" height="307" alt="logic_function" src="https://github.com/user-attachments/assets/dd4a3d16-67f2-434b-95b3-bdcaf9ed053d" /> ## More details on Log size & truncation Logs are stored in `workflowRun.stepLogs` (JSONB), keyed by `stepId`. ### Per-step cap Each step's log is hard-capped at **256 KB** (`MAX_STEP_LOG_BYTES` in `WorkflowRunStepLogWorkspaceService.setStepLog`). For ~99% of workflows this is roomy — typical real-world sizes: - Code / serverless function: 1–20 KB - HTTP request: 5–70 KB - Email: 5–30 KB - AI agent (a handful of tool calls): 5–50 KB ### Two layers of bounding 1. **Per-field truncation** in each builder (before writing): - **Code**: ≤ 500 entries, ≤ 4 KB per message, ≤ 8 KB stack trace - **HTTP**: ≤ 32 KB per body (request + response), UTF-8 byte-aware - **Email**: ≤ 8 KB body preview, UTF-8 byte-aware - **AI agent**: ≤ 32 KB tool input, ≤ 64 KB tool output, ≤ 200 tool calls/step 2. **Global per-step safety net** at write time: if the assembled `stepLog` still exceeds 256 KB, the write is **dropped entirely** with a `logger.warn`. The workflow itself keeps running unaffected. ### What this means in practice - **Safe**: workflow execution, step results, downstream steps — never blocked by log size. - **Safe**: iterators (each iteration overwrites the previous log for that `stepId`, so they can't accumulate). - **Safe**: step retries (same `stepId` is overwritten, not appended). - **Possible**: an AI agent step with many large tool outputs (e.g., 50+ heavy `web_search` calls) can exceed 256 KB → the **entire** step's log is dropped, side panel shows "No logs were recorded for this step". The user has no explicit signal that the log was dropped due to size (only server-side warn). - **Possible** (theoretical): a workflow with hundreds of distinct steps could push the row toward Postgres's internal ~256 MB jsonb limit. Beyond that, individual `jsonb_set` writes would error and be swallowed by the action's try/catch — workflow still completes. ### Possible future hardening (not in this PR) - Replace "drop entire log" with a stub that preserves the summary card (cost, duration, status) and marks `truncated.reason = 'size_cap'`. - Surface size-drops in the UI (similar to the existing `<StyledTruncatedNotice>`). - Emit a metric so dropped logs are observable in dashboards.
2026-06-03 18:53:47 +02:00
parent 0671ff3de5
commit 4b15b949f3
74 changed files with 4834 additions and 445 deletions
@@ -2,9 +2,12 @@ import { Module } from '@nestjs/common';
 import { TypeOrmModule } from '@nestjs/typeorm';

 import { WorkspaceIteratorModule } from 'src/database/commands/command-runners/workspace-iterator.module';
-import { BackfillFieldsWidgetNewFieldDefaultVisibilityCommand } from 'src/database/commands/upgrade-version-command/2-9/2-9-workspace-command-1799000030000-backfill-fields-widget-new-field-default-visibility.command';
 import { MigrateAiModelPreferencesCommand } from 'src/database/commands/upgrade-version-command/2-9/2-9-workspace-command-1799000000000-migrate-ai-model-preferences.command';
+import { BackfillFieldsWidgetNewFieldDefaultVisibilityCommand } from 'src/database/commands/upgrade-version-command/2-9/2-9-workspace-command-1799000030000-backfill-fields-widget-new-field-default-visibility.command';
+import { AddWorkflowRunStepLogsFieldCommand } from 'src/database/commands/upgrade-version-command/2-9/2-9-workspace-command-1800000000000-add-workflow-run-step-logs-field.command';
+import { ApplicationModule } from 'src/engine/core-modules/application/application.module';
 import { KeyValuePairEntity } from 'src/engine/core-modules/key-value-pair/key-value-pair.entity';
+import { FieldMetadataModule } from 'src/engine/metadata-modules/field-metadata/field-metadata.module';
 import { WorkspaceCacheModule } from 'src/engine/workspace-cache/workspace-cache.module';
 import { WorkspaceMigrationModule } from 'src/engine/workspace-manager/workspace-migration/workspace-migration.module';

@@ -13,10 +16,14 @@ import { WorkspaceMigrationModule } from 'src/engine/workspace-manager/workspace
    TypeOrmModule.forFeature([KeyValuePairEntity]),
    WorkspaceCacheModule,
    WorkspaceIteratorModule,
+    ApplicationModule,
+    FieldMetadataModule,
+    WorkspaceCacheModule,
    WorkspaceMigrationModule,
  ],
  providers: [
    MigrateAiModelPreferencesCommand,
+    AddWorkflowRunStepLogsFieldCommand,
    BackfillFieldsWidgetNewFieldDefaultVisibilityCommand,
  ],
 })
@@ -0,0 +1,129 @@
+import { Command } from 'nest-commander';
+import { STANDARD_OBJECTS } from 'twenty-shared/metadata';
+import { FieldMetadataType } from 'twenty-shared/types';
+import { isDefined } from 'twenty-shared/utils';
+
+import { ActiveOrSuspendedWorkspaceCommandRunner } from 'src/database/commands/command-runners/active-or-suspended-workspace.command-runner';
+import { WorkspaceIteratorService } from 'src/database/commands/command-runners/workspace-iterator.service';
+import { type RunOnWorkspaceArgs } from 'src/database/commands/command-runners/workspace.command-runner';
+import { ApplicationService } from 'src/engine/core-modules/application/application.service';
+import { RegisteredWorkspaceCommand } from 'src/engine/core-modules/upgrade/decorators/registered-workspace-command.decorator';
+import { type CreateFieldInput } from 'src/engine/metadata-modules/field-metadata/dtos/create-field.input';
+import { FieldMetadataService } from 'src/engine/metadata-modules/field-metadata/services/field-metadata.service';
+import { findFlatEntityByUniversalIdentifier } from 'src/engine/metadata-modules/flat-entity/utils/find-flat-entity-by-universal-identifier.util';
+import { type FlatFieldMetadata } from 'src/engine/metadata-modules/flat-field-metadata/types/flat-field-metadata.type';
+import { type FlatObjectMetadata } from 'src/engine/metadata-modules/flat-object-metadata/types/flat-object-metadata.type';
+import { WorkspaceCacheService } from 'src/engine/workspace-cache/services/workspace-cache.service';
+
+const WORKFLOW_RUN_UNIVERSAL_IDENTIFIER =
+  STANDARD_OBJECTS.workflowRun.universalIdentifier;
+
+const STEP_LOGS_FIELD_UNIVERSAL_IDENTIFIER =
+  STANDARD_OBJECTS.workflowRun.fields.stepLogs.universalIdentifier;
+
+@RegisteredWorkspaceCommand('2.9.0', 1800000000000)
+@Command({
+  name: 'upgrade:2-9:add-workflow-run-step-logs-field',
+  description:
+    'Add stepLogs JSONB field to the workflowRun standard object for existing workspaces. Per-step observability payload (token usage, tool calls, log entries) is written here.',
+})
+export class AddWorkflowRunStepLogsFieldCommand extends ActiveOrSuspendedWorkspaceCommandRunner {
+  constructor(
+    protected readonly workspaceIteratorService: WorkspaceIteratorService,
+    private readonly applicationService: ApplicationService,
+    private readonly workspaceCacheService: WorkspaceCacheService,
+    private readonly fieldMetadataService: FieldMetadataService,
+  ) {
+    super(workspaceIteratorService);
+  }
+
+  override async runOnWorkspace({
+    workspaceId,
+    options,
+  }: RunOnWorkspaceArgs): Promise<void> {
+    const isDryRun = options.dryRun ?? false;
+
+    const { flatObjectMetadataMaps, flatFieldMetadataMaps } =
+      await this.workspaceCacheService.getOrRecompute(workspaceId, [
+        'flatObjectMetadataMaps',
+        'flatFieldMetadataMaps',
+      ]);
+
+    const workflowRunObject =
+      findFlatEntityByUniversalIdentifier<FlatObjectMetadata>({
+        flatEntityMaps: flatObjectMetadataMaps,
+        universalIdentifier: WORKFLOW_RUN_UNIVERSAL_IDENTIFIER,
+      });
+
+    if (!isDefined(workflowRunObject)) {
+      this.logger.log(
+        `workflowRun object not found for workspace ${workspaceId}, skipping`,
+      );
+
+      return;
+    }
+
+    const existingStepLogsField =
+      findFlatEntityByUniversalIdentifier<FlatFieldMetadata>({
+        flatEntityMaps: flatFieldMetadataMaps,
+        universalIdentifier: STEP_LOGS_FIELD_UNIVERSAL_IDENTIFIER,
+      });
+
+    if (isDefined(existingStepLogsField)) {
+      this.logger.log(
+        `stepLogs field already present on workflowRun for workspace ${workspaceId}, skipping`,
+      );
+
+      return;
+    }
+
+    const createFieldInput: Omit<CreateFieldInput, 'workspaceId'> = {
+      objectMetadataId: workflowRunObject.id,
+      name: 'stepLogs',
+      type: FieldMetadataType.RAW_JSON,
+      label: 'Step logs',
+      description:
+        'Per-step observability payload (token usage, tool calls, log entries)',
+      icon: 'IconTerminal2',
+      isNullable: true,
+      isUIReadOnly: true,
+      isCustom: false,
+      isSystem: true,
+      isActive: true,
+      universalIdentifier: STEP_LOGS_FIELD_UNIVERSAL_IDENTIFIER,
+    };
+
+    if (isDryRun) {
+      this.logger.log(
+        `[DRY RUN] Would create stepLogs field on workflowRun for workspace ${workspaceId}`,
+      );
+
+      return;
+    }
+
+    const { twentyStandardFlatApplication } =
+      await this.applicationService.findWorkspaceTwentyStandardAndCustomApplicationOrThrow(
+        { workspaceId },
+      );
+
+    try {
+      await this.fieldMetadataService.createManyFields({
+        createFieldInputs: [createFieldInput],
+        workspaceId,
+        ownerFlatApplication: twentyStandardFlatApplication,
+        isSystemBuild: true,
+      });
+    } catch (error) {
+      this.logger.error(
+        `Failed to add stepLogs field on workflowRun for workspace ${workspaceId}:\n${
+          error instanceof Error ? error.stack : JSON.stringify(error, null, 2)
+        }`,
+      );
+      throw error;
+    }
+
+    this.logger.log(
+      `Added stepLogs field on workflowRun for workspace ${workspaceId}`,
+    );
+  }
+}