Files
twenty/packages/twenty-server/src/engine/twenty-orm/entity-manager/workspace-entity-manager.spec.ts
T
Marie 4b15b949f3 Provide additional logsobservability to workflow runs (per node) (#21142)
Surfaces per-step "Logs" tabs in the workflow run side panel so users
can see what each step actually did (model + tokens + tool calls for AI,
console output for serverless functions, request/response for HTTP,
recipients/body for Email).

<img width="546" height="501" alt="ai_agent_without_websearch"
src="https://github.com/user-attachments/assets/c6ca3518-9489-4484-a570-3d0569ff3b03"
/>

## Storage

- New `stepLogs` JSONB column on the `workflowRun` workspace entity,
typed as `Record<string, WorkflowRunStepLog>` (keyed by step id).
- Schema lives in `twenty-shared`: `workflowRunStepLogSchema` with a
discriminated `details.type` union for `AI_AGENT | CODE | HTTP_REQUEST |
EMAIL` — frontends and backends consume the same Zod-inferred type.
- Field is added to existing workspaces via a workspace upgrade command
(`2-9 add-workflow-run-step-logs-field`); the standard-object metadata
declares it for new workspaces.
- Writes happen atomically per step in
`WorkflowRunStepLogWorkspaceService.setStepLog` using `jsonb_set`. That
lets concurrent steps in the same run write their own keys without
contending with the existing lock around `workflowRun.state`.
- Per-step payload is hard-capped at 256 KB; anything larger is dropped
with a `logger.warn`, so a pathological tool call can never bloat a row.
See below for more information.

## How logs are produced

**Aalmost everything was already being collected; this PR mostly
persists and renders it.**

- **AI agent** — `AgentAsyncExecutorService` already tracked token
usage, model id, native web-search count, and the AI SDK's `steps[]`. We
map those into the log via `mapAiStepsToToolCallLogs` (`searchVector`
stripped from record outputs, per-call input/output capped at 32/64 KB,
max 200 tool calls per step). The only new measurement is a wall-clock
`durationMs` taken around `executeAgent`, and we now fold native
web-search cost into the displayed `totalCostInDollars` (it was already
billed, just not shown).
- **Code / serverless function** — reuses the `console.log` output the
function runner already returns (`logsByLevel`);
`build-code-step-log.util` only repackages it.
- **HTTP request** — built from the action's existing input/output via
`build-http-request-step-log.util`. No new signals collected.
- **Email (send / draft)** — added `sanitizedHtmlBody` + `plainTextBody`
to the existing tool outputs (a small additive change), then
`build-email-step-log.util` consumes them.

No additional AI inference or external calls are made for logging — the
cost is a small CPU overhead per step plus the JSONB write.

## Security

The log surface intentionally shows whatever the workflow touched, which
made redaction and sanitization the main design concern.

- **HTTP — secrets in headers**: existing `SENSITIVE_HEADER_NAMES` set
(Authorization, Cookie, …) replaced with `[redacted]` in both request
and response.
- **HTTP — secrets in URLs**: `SENSITIVE_URL_PARAM_NAMES` (e.g.
`api_key`, `token`, `access_token`) replaced in the query string via
`URL`-based parsing.
- **HTTP — secrets in bodies**: `SENSITIVE_BODY_KEY_REGEX` deep-walks
JSON request/response bodies (object input or stringified JSON) and
redacts matching keys. Applied to the `error` field too, since
transport-layer errors sometimes embed structured payloads.
- **Email — XSS risk in body preview**: tool outputs now expose a
server-side `sanitizedHtmlBody`; the log builder prefers it over the raw
user-authored `input.body`, with `plainTextBody` as a second fallback.
The original raw body is only used if sanitization didn't happen (e.g.
tool failed before composing).
- **AI — internal/noisy data**: `searchVector` (Postgres tsvector
strings) is stripped from record outputs returned by Twenty tools to
avoid leaking internal full-text-search payloads.
- **DB bloat / runaway agents**: 256 KB per-step cap + 32 KB / 64 KB
per-tool-call input/output cap + 200 tool calls per step.

<img width="547" height="307" alt="logic_function"
src="https://github.com/user-attachments/assets/dd4a3d16-67f2-434b-95b3-bdcaf9ed053d"
/>

## More details on Log size & truncation

Logs are stored in `workflowRun.stepLogs` (JSONB), keyed by `stepId`.

### Per-step cap

Each step's log is hard-capped at **256 KB** (`MAX_STEP_LOG_BYTES` in
`WorkflowRunStepLogWorkspaceService.setStepLog`).

For ~99% of workflows this is roomy — typical real-world sizes:
- Code / serverless function: 1–20 KB
- HTTP request: 5–70 KB
- Email: 5–30 KB
- AI agent (a handful of tool calls): 5–50 KB

### Two layers of bounding

1. **Per-field truncation** in each builder (before writing):
   - **Code**: ≤ 500 entries, ≤ 4 KB per message, ≤ 8 KB stack trace
   - **HTTP**: ≤ 32 KB per body (request + response), UTF-8 byte-aware
   - **Email**: ≤ 8 KB body preview, UTF-8 byte-aware
- **AI agent**: ≤ 32 KB tool input, ≤ 64 KB tool output, ≤ 200 tool
calls/step

2. **Global per-step safety net** at write time: if the assembled
`stepLog` still exceeds 256 KB, the write is **dropped entirely** with a
`logger.warn`. The workflow itself keeps running unaffected.

### What this means in practice

- **Safe**: workflow execution, step results, downstream steps — never
blocked by log size.
- **Safe**: iterators (each iteration overwrites the previous log for
that `stepId`, so they can't accumulate).
- **Safe**: step retries (same `stepId` is overwritten, not appended).
- **Possible**: an AI agent step with many large tool outputs (e.g., 50+
heavy `web_search` calls) can exceed 256 KB → the **entire** step's log
is dropped, side panel shows "No logs were recorded for this step". The
user has no explicit signal that the log was dropped due to size (only
server-side warn).
- **Possible** (theoretical): a workflow with hundreds of distinct steps
could push the row toward Postgres's internal ~256 MB jsonb limit.
Beyond that, individual `jsonb_set` writes would error and be swallowed
by the action's try/catch — workflow still completes.

### Possible future hardening (not in this PR)

- Replace "drop entire log" with a stub that preserves the summary card
(cost, duration, status) and marks `truncated.reason = 'size_cap'`.
- Surface size-drops in the UI (similar to the existing
`<StyledTruncatedNotice>`).
- Emit a metric so dropped logs are observable in dashboards.
2026-06-03 16:53:47 +00:00

513 lines
18 KiB
TypeScript

import {
type FieldMetadataType,
type ObjectsPermissions,
} from 'twenty-shared/types';
import { EntityManager } from 'typeorm';
import { EntityPersistExecutor } from 'typeorm/persistence/EntityPersistExecutor';
import { PlainObjectToDatabaseEntityTransformer } from 'typeorm/query-builder/transformer/PlainObjectToDatabaseEntityTransformer';
import { type WorkspaceInternalContext } from 'src/engine/twenty-orm/interfaces/workspace-internal-context.interface';
import { type WorkspaceAuthContext } from 'src/engine/core-modules/auth/types/workspace-auth-context.type';
import { type FlatEntityMaps } from 'src/engine/metadata-modules/flat-entity/types/flat-entity-maps.type';
import { type FlatFieldMetadata } from 'src/engine/metadata-modules/flat-field-metadata/types/flat-field-metadata.type';
import { type FlatObjectMetadata } from 'src/engine/metadata-modules/flat-object-metadata/types/flat-object-metadata.type';
import { type GlobalWorkspaceDataSource } from 'src/engine/twenty-orm/global-workspace-datasource/global-workspace-datasource';
import { validateOperationIsPermittedOrThrow } from 'src/engine/twenty-orm/repository/permissions.utils';
import {
setWorkspaceContext,
withWorkspaceContext,
type ORMWorkspaceContext,
} from 'src/engine/twenty-orm/storage/orm-workspace-context.storage';
import { getObjectMetadataFromEntityTarget } from 'src/engine/twenty-orm/utils/get-object-metadata-from-entity-target.util';
import { WorkspaceEntityManager } from './workspace-entity-manager';
jest.mock('src/engine/twenty-orm/repository/permissions.utils', () => ({
validateOperationIsPermittedOrThrow: jest.fn(),
}));
jest.mock(
'src/engine/twenty-orm/utils/get-object-metadata-from-entity-target.util',
() => ({
getObjectMetadataFromEntityTarget: jest.fn(),
}),
);
jest.mock('src/engine/twenty-orm/utils/format-data.util', () => ({
formatData: jest.fn().mockReturnValue([]),
}));
jest.mock('src/engine/twenty-orm/utils/format-result.util', () => ({
formatResult: jest.fn().mockReturnValue([]),
}));
jest.mock(
'src/engine/twenty-orm/entity-manager/workspace-entity-manager',
() => ({
...jest.requireActual(
'src/engine/twenty-orm/entity-manager/workspace-entity-manager',
),
}),
);
const mockedWorkspaceUpdateQueryBuilder = {
set: jest.fn().mockImplementation(() => ({
where: jest.fn().mockReturnThis(),
whereInIds: jest.fn().mockReturnThis(),
execute: jest
.fn()
.mockResolvedValue({ affected: 1, raw: [], generatedMaps: [] }),
returning: jest.fn().mockReturnThis(),
})),
execute: jest
.fn()
.mockResolvedValue({ affected: 1, raw: [], generatedMaps: [] }),
};
jest.mock('../repository/workspace-select-query-builder', () => ({
WorkspaceSelectQueryBuilder: jest.fn().mockImplementation(() => ({
where: jest.fn().mockReturnThis(),
getMany: jest.fn().mockResolvedValue([]),
getOne: jest.fn().mockResolvedValue(null),
getManyAndCount: jest.fn().mockResolvedValue([[], 0]),
execute: jest
.fn()
.mockResolvedValue({ affected: 1, raw: [], generatedMaps: [] }),
setFindOptions: jest.fn().mockReturnThis(),
returning: jest.fn().mockReturnThis(),
update: jest.fn().mockReturnValue(mockedWorkspaceUpdateQueryBuilder),
insert: jest.fn().mockReturnThis(),
})),
}));
describe('WorkspaceEntityManager', () => {
let entityManager: WorkspaceEntityManager;
let mockDataSource: GlobalWorkspaceDataSource;
let mockPermissionOptions: {
shouldBypassPermissionChecks: boolean;
objectRecordsPermissions?: ObjectsPermissions;
};
let mockInternalContext: WorkspaceInternalContext;
let mockWorkspaceContext: ORMWorkspaceContext;
beforeEach(() => {
const mockFlatObjectMetadata: FlatObjectMetadata = {
id: 'test-entity-id',
nameSingular: 'test-entity',
namePlural: 'test-entities',
labelSingular: 'Test Entity',
labelPlural: 'Test Entities',
workspaceId: 'test-workspace-id',
icon: 'test-icon',
color: null,
isCustom: false,
isRemote: false,
isAuditLogged: false,
isSearchable: false,
isSystem: false,
isActive: true,
targetTableName: 'test_entity',
fieldIds: ['field-id'],
indexMetadataIds: [],
objectPermissionIds: [],
fieldPermissionIds: [],
viewIds: [],
universalIdentifier: 'test-entity-id',
description: null,
imageIdentifierFieldMetadataId: null,
labelIdentifierFieldMetadataId: null,
shortcut: null,
standardOverrides: null,
applicationId: 'test-application-id',
isLabelSyncedWithName: false,
isUIReadOnly: false,
duplicateCriteria: null,
createdAt: new Date().toISOString(),
updatedAt: new Date().toISOString(),
applicationUniversalIdentifier: 'test-application-id',
fieldUniversalIdentifiers: ['field-id'],
objectPermissionUniversalIdentifiers: [],
fieldPermissionUniversalIdentifiers: [],
viewUniversalIdentifiers: [],
indexMetadataUniversalIdentifiers: [],
labelIdentifierFieldMetadataUniversalIdentifier: null,
imageIdentifierFieldMetadataUniversalIdentifier: null,
};
(getObjectMetadataFromEntityTarget as jest.Mock).mockReturnValue(
mockFlatObjectMetadata,
);
const mockFlatFieldMetadata: FlatFieldMetadata = {
id: 'field-id',
type: 'TEXT' as FieldMetadataType,
name: 'fieldName',
label: 'Field Name',
objectMetadataId: 'test-entity-id',
isNullable: true,
isLabelSyncedWithName: false,
createdAt: new Date().toISOString(),
updatedAt: new Date().toISOString(),
universalIdentifier: 'field-id',
defaultValue: null,
description: null,
icon: null,
isActive: true,
isCustom: false,
isSystem: false,
isUIReadOnly: false,
isUnique: false,
options: null,
settings: null,
standardOverrides: null,
workspaceId: 'test-workspace-id',
viewFieldIds: [],
viewFilterIds: [],
fieldPermissionIds: [],
kanbanAggregateOperationViewIds: [],
calendarViewIds: [],
mainGroupByFieldMetadataViewIds: [],
relationTargetFieldMetadataId: null,
relationTargetObjectMetadataId: null,
morphId: null,
applicationId: 'application-id',
applicationUniversalIdentifier: 'application-id',
objectMetadataUniversalIdentifier: 'test-entity-id',
relationTargetObjectMetadataUniversalIdentifier: null,
relationTargetFieldMetadataUniversalIdentifier: null,
viewFilterUniversalIdentifiers: [],
viewFieldUniversalIdentifiers: [],
kanbanAggregateOperationViewUniversalIdentifiers: [],
calendarViewUniversalIdentifiers: [],
mainGroupByFieldMetadataViewUniversalIdentifiers: [],
fieldPermissionUniversalIdentifiers: [],
viewSortIds: [],
viewSortUniversalIdentifiers: [],
universalSettings: null,
};
const flatObjectMetadataMaps: FlatEntityMaps<FlatObjectMetadata> = {
byUniversalIdentifier: {
'test-entity-id': mockFlatObjectMetadata,
},
universalIdentifierById: {
'test-entity-id': 'test-entity-id',
},
universalIdentifiersByApplicationId: {},
};
const flatFieldMetadataMaps: FlatEntityMaps<FlatFieldMetadata> = {
byUniversalIdentifier: {
'field-id': mockFlatFieldMetadata,
},
universalIdentifierById: {
'field-id': 'field-id',
},
universalIdentifiersByApplicationId: {},
};
mockInternalContext = {
workspaceId: 'test-workspace-id',
flatObjectMetadataMaps,
flatFieldMetadataMaps,
flatIndexMaps: {
byUniversalIdentifier: {},
universalIdentifierById: {},
universalIdentifiersByApplicationId: {},
},
flatRowLevelPermissionPredicateMaps: {
byUniversalIdentifier: {},
universalIdentifierById: {},
universalIdentifiersByApplicationId: {},
},
flatRowLevelPermissionPredicateGroupMaps: {
byUniversalIdentifier: {},
universalIdentifierById: {},
universalIdentifiersByApplicationId: {},
},
objectIdByNameSingular: {
'test-entity': 'test-entity-id',
},
featureFlagsMap: {
IS_UNIQUE_INDEXES_ENABLED: false,
IS_JSON_FILTER_ENABLED: false,
IS_MARKETPLACE_SETTING_TAB_VISIBLE: false,
IS_PUBLIC_DOMAIN_ENABLED: false,
IS_EMAIL_GROUP_ENABLED: false,
IS_JUNCTION_RELATIONS_ENABLED: false,
IS_REST_METADATA_API_NEW_FORMAT_DIRECT: false,
IS_LOGIC_FUNCTION_PREBUILT_MODE_ENABLED: false,
IS_SETTINGS_DISCOVERY_HERO_ENABLED: false,
IS_WORKFLOW_RUN_STEP_LOGS_ENABLED: false,
},
userWorkspaceRoleMap: {},
eventEmitterService: {
emitMutationEvent: jest.fn(),
emitDatabaseBatchEvent: jest.fn(),
emitCustomBatchEvent: jest.fn(),
} as any,
coreDataSource: {
getRepository: jest.fn(() => ({
find: jest.fn(),
softDelete: jest.fn(),
})),
} as any,
} as WorkspaceInternalContext;
mockDataSource = {
featureFlagMap: {
IS_UNIQUE_INDEXES_ENABLED: false,
IS_JSON_FILTER_ENABLED: false,
IS_PUBLIC_DOMAIN_ENABLED: false,
},
permissionsPerRoleId: {},
eventEmitterService: mockInternalContext.eventEmitterService,
coreDataSource: mockInternalContext.coreDataSource,
} as GlobalWorkspaceDataSource;
mockPermissionOptions = {
shouldBypassPermissionChecks: false,
objectRecordsPermissions: {
'test-entity': {
canReadObjectRecords: true,
canUpdateObjectRecords: false,
canSoftDeleteObjectRecords: false,
canDestroyObjectRecords: false,
restrictedFields: {},
rowLevelPermissionPredicates: [],
rowLevelPermissionPredicateGroups: [],
},
},
};
const mockAuthContext = {
user: { id: 'user-id' },
workspace: { id: 'test-workspace-id' },
workspaceMemberId: 'workspace-member-id',
userWorkspaceId: 'user-workspace-id',
apiKey: null,
} as unknown as WorkspaceAuthContext;
mockWorkspaceContext = {
authContext: mockAuthContext,
flatObjectMetadataMaps,
flatFieldMetadataMaps,
flatIndexMaps: mockInternalContext.flatIndexMaps,
flatRowLevelPermissionPredicateMaps:
mockInternalContext.flatRowLevelPermissionPredicateMaps,
flatRowLevelPermissionPredicateGroupMaps:
mockInternalContext.flatRowLevelPermissionPredicateGroupMaps,
objectIdByNameSingular: mockInternalContext.objectIdByNameSingular,
featureFlagsMap: mockInternalContext.featureFlagsMap,
permissionsPerRoleId: mockDataSource.permissionsPerRoleId,
entityMetadatas: [],
userWorkspaceRoleMap: {
'user-workspace-id': 'role-id',
},
apiKeyRoleMap: {},
};
setWorkspaceContext(mockWorkspaceContext);
// Mock TypeORM connection methods
const mockWorkspaceDataSource = {
getMetadata: jest.fn().mockReturnValue({
name: 'test-entity',
columns: [],
relations: [],
findInheritanceMetadata: jest.fn(),
findColumnWithPropertyPath: jest.fn(),
}),
eventEmitterService: mockInternalContext.eventEmitterService,
coreDataSource: mockInternalContext.coreDataSource,
createQueryBuilder: jest.fn().mockReturnValue({
delete: jest.fn().mockReturnThis(),
from: jest.fn().mockReturnThis(),
where: jest.fn().mockReturnThis(),
execute: jest
.fn()
.mockResolvedValue({ affected: 1, raw: [], generatedMaps: [] }),
getMany: jest.fn().mockResolvedValue([]),
getOne: jest.fn().mockResolvedValue(null),
getManyAndCount: jest.fn().mockResolvedValue([[], 0]),
update: jest.fn().mockReturnThis(),
softDelete: jest.fn().mockReturnThis(),
restore: jest.fn().mockReturnThis(),
}),
createQueryRunner: jest.fn().mockReturnValue({
connect: jest.fn(),
startTransaction: jest.fn(),
commitTransaction: jest.fn(),
rollbackTransaction: jest.fn(),
release: jest.fn(),
clearTable: jest.fn(),
}),
createQueryRunnerForEntityPersistExecutor: jest.fn().mockReturnValue({
connect: jest.fn(),
startTransaction: jest.fn(),
commitTransaction: jest.fn(),
rollbackTransaction: jest.fn(),
release: jest.fn(),
clearTable: jest.fn(),
}),
};
entityManager = new WorkspaceEntityManager(mockDataSource);
Object.defineProperty(entityManager, 'connection', {
get: () => mockWorkspaceDataSource,
});
jest.spyOn(entityManager as any, 'validatePermissions');
jest.spyOn(entityManager as any, 'createQueryBuilder');
jest
.spyOn(entityManager as any, 'getFormattedResultWithoutNonReadableFields')
.mockImplementation(
({ formattedResult }: { formattedResult: string[] }) => formattedResult,
);
jest
.spyOn(entityManager as any, 'extractTargetNameSingularFromEntityTarget')
.mockImplementation((entityName: string) => {
return entityName;
});
// Mock typeORM's EntityManager methods
jest
.spyOn(EntityManager.prototype, 'save')
.mockImplementation(() => Promise.resolve({}));
jest
.spyOn(EntityManager.prototype, 'update')
.mockImplementation(() =>
Promise.resolve({ affected: 1, raw: [], generatedMaps: [] }),
);
jest
.spyOn(EntityManager.prototype, 'restore')
.mockImplementation(() =>
Promise.resolve({ affected: 1, raw: [], generatedMaps: [] }),
);
jest
.spyOn(EntityManager.prototype, 'clear')
.mockImplementation(() => Promise.resolve());
jest
.spyOn(EntityManager.prototype, 'preload')
.mockImplementation(() => Promise.resolve({}));
jest
.spyOn(EntityPersistExecutor.prototype, 'execute')
.mockImplementation(() => Promise.resolve());
jest
.spyOn(PlainObjectToDatabaseEntityTransformer.prototype, 'transform')
.mockImplementation(() => Promise.resolve({}));
// Mock metadata methods
const mockMetadata = {
hasAllPrimaryKeys: jest.fn().mockReturnValue(true),
columns: [],
relations: [],
findInheritanceMetadata: jest.fn(),
findColumnWithPropertyPath: jest.fn(),
};
// Update mockWorkspaceDataSource to include metadata
mockWorkspaceDataSource.getMetadata = jest
.fn()
.mockReturnValue(mockMetadata);
// Reset the mock before each test
jest.clearAllMocks();
});
describe('Query Method', () => {
it('should call validatePermissions and validateOperationIsPermittedOrThrow for find', async () => {
await withWorkspaceContext(mockWorkspaceContext, () =>
entityManager.find('test-entity', {}, mockPermissionOptions),
);
expect(entityManager.createQueryBuilder).toHaveBeenCalledWith(
'test-entity',
undefined,
undefined,
mockPermissionOptions,
);
});
it('should throw error for query', async () => {
expect(() => entityManager.query('SELECT * FROM test')).toThrow(
'Method not allowed.',
);
});
});
describe('Save Methods', () => {
it('should call validatePermissions and validateOperationIsPermittedOrThrow for save', async () => {
await withWorkspaceContext(mockWorkspaceContext, () =>
entityManager.save(
'test-entity',
{},
{ reload: false },
mockPermissionOptions,
),
);
expect(entityManager['validatePermissions']).toHaveBeenCalledWith({
target: 'test-entity',
operationType: 'update',
permissionOptions: mockPermissionOptions,
selectedColumns: [],
updatedColumns: [],
});
expect(validateOperationIsPermittedOrThrow).toHaveBeenCalledWith({
entityName: 'test-entity',
operationType: 'update',
flatObjectMetadataMaps: mockInternalContext.flatObjectMetadataMaps,
flatFieldMetadataMaps: mockInternalContext.flatFieldMetadataMaps,
objectIdByNameSingular: mockInternalContext.objectIdByNameSingular,
objectsPermissions: mockPermissionOptions.objectRecordsPermissions,
selectedColumns: [],
allFieldsSelected: false,
updatedColumns: [],
});
});
});
describe('Update Methods', () => {
it('should call createQueryBuilder with permissionOptions for update', async () => {
await withWorkspaceContext(mockWorkspaceContext, () =>
entityManager.update('test-entity', {}, {}, mockPermissionOptions),
);
expect(entityManager['createQueryBuilder']).toHaveBeenCalledWith(
'test-entity',
undefined,
undefined,
mockPermissionOptions,
);
});
});
describe('Other Methods', () => {
it('should call validatePermissions and validateOperationIsPermittedOrThrow for clear', async () => {
await withWorkspaceContext(mockWorkspaceContext, () =>
entityManager.clear('test-entity', mockPermissionOptions),
);
expect(entityManager['validatePermissions']).toHaveBeenCalledWith({
target: 'test-entity',
operationType: 'delete',
permissionOptions: mockPermissionOptions,
selectedColumns: [],
});
expect(validateOperationIsPermittedOrThrow).toHaveBeenCalledWith({
entityName: 'test-entity',
operationType: 'delete',
flatObjectMetadataMaps: mockInternalContext.flatObjectMetadataMaps,
flatFieldMetadataMaps: mockInternalContext.flatFieldMetadataMaps,
objectIdByNameSingular: mockInternalContext.objectIdByNameSingular,
objectsPermissions: mockPermissionOptions.objectRecordsPermissions,
selectedColumns: [],
allFieldsSelected: false,
updatedColumns: [],
});
});
});
});