feat(server): unify workspace-event ingestion behind one EventSink pipeline (#21197)

## Why

The five event-log streams (`workspaceEvent`, `pageview`, `objectEvent`,
`usageEvent`, `applicationLog`) each wrote to ClickHouse through their
own fire-and-forget writer (`AuditService`, `UsageEventWriterService`,
and the `application-logs` driver), with the per-type knowledge (table
names, normalization, access rules) spread across several modules. Three
of them reimplemented the same ClickHouse insert, and the read side, the
live stream, and the producers lived in different modules under two
different names.

This consolidates them into one `core-modules/event-logs/` subsystem
(emit, write, live, read), with the per-type config in a single registry
so adding an event type is roughly one file.

The base Logs settings tab and free application logs shipped separately
in #21180 (merged). This PR adds the unified backend, the registry, and
the viewer's live mode and entitlement gating.

## Pipeline

```mermaid
flowchart TB
    subgraph PROD["Producers"]
      A["auth, billing, impersonation,<br/>webhook, custom-domain"]
      U["usage listener"]
      F["logic-function executor (app logs)"]
      R["record CRUD (entity events)"]
    end
    EM["EventLogEmitterService<br/>createContext().insert* / dispatch()"]
    EQ(["entityEventsToDbQueue<br/>(existing, shared with timeline)"])
    CIE["CreateEventLogFromInternalEvent"]
    SINK["WorkspaceEventSinkService.ingest()"]
    C1["ClickHouseEventSink"]
    C2["ConsoleEventSink"]
    LIVE["EventLogLiveService.publishWatched()<br/>(presence-gated)"]
    CH[("ClickHouse, 5 tables, async_insert")]
    CHAN(["WORKSPACE_EVENTS_CHANNEL"])
    RS["EventLogsService (registry-driven read)"]
    LR["EventLogsLiveResolver"]
    UI["Settings > Logs"]

    A --> EM
    U --> EM
    F --> EM
    EM -->|direct| SINK
    R --> EQ --> CIE -->|ingest| SINK
    SINK --> C1 --> CH
    SINK --> C2
    SINK --> LIVE -.->|if a viewer is watching| CHAN --> LR --> UI
    CH --> RS --> UI
```

## What it does

- Producers call `EventLogEmitterService.createContext().insert*()`,
which builds a typed `WorkspaceEventEnvelope` and writes it through
`WorkspaceEventSinkService` to the configured sinks (ClickHouse,
Console) plus a presence-gated live fan-out. Record/CRUD events reach
the same sink through the existing `entityEventsToDbQueue`. There is no
dedicated queue; ClickHouse `async_insert` batches server-side. Writes
are best-effort, as on main today.
- `EVENT_LOG_TYPES[table]` is the per-type source of truth: the
ClickHouse table, the required entitlement, the free-text filter column,
and the row-to-GraphQL mapping. Read row shapes derive from the write
rows.
- Four modules along their dependency boundaries:
`EventLogEmitterModule` (producer API), `EventLogIngestionModule` (sink
layer), `EventLogLiveModule` (fan-out), and `EventLogsViewerModule` (the
entitlement-gated GraphQL read, which is where
billing/enterprise/permissions stay so producers stay light).
- Logs viewer: per-table columns, filters (text, date, record), live
mode, and an upgrade card that points to Billing on Cloud or the Admin
Panel on self-hosted. Application logs are free on every plan; the other
four require the `AUDIT_LOGS` entitlement (with a `NO_ENTITLEMENT`
fallback to the upgrade card).
- Renames `AuditService` to `EventLogEmitterService`, and the generic
`Monitoring` event to a typed `Impersonation` event (`level` +
`action`).
- Removes `UsageEventWriterService`, the `application-logs`
driver/module, and `AuditService`'s direct inserts.

## Durability

Writes are best-effort, the same as main today (the old writers were
fire-and-forget). A dedicated queue was tried mid-PR and removed:
`async_insert` already batches server-side, so the queue only added
durability, which isn't a requirement right now. The `EventSink` seam
keeps a durable transport (e.g. a Redis-Streams buffer) easy to add
later without touching producers.

## Out of scope

S3 peer sink (seam only), Postgres or any second read path,
`ReplicatedMergeTree`, ClickHouse table-schema changes, and the
record-data `EVENT_STREAM_CHANNEL` (unchanged, separate concern).

## Testing

Unit tests cover the registry definitions and row normalization, the
entitlement gating, the envelope builders, and the producers.
Integration tests cover the write paths (record create produces an
`objectEvent`; the track mutation produces a `workspaceEvent`) and the
read/query path across all five tables. Verified with typecheck, lint, a
server boot, and GraphQL/SDK codegen.
This commit is contained in:
Félix Malfait
2026-06-06 10:32:56 +02:00
committed by GitHub
parent 6c65d26ced
commit 91f2f08995
148 changed files with 2919 additions and 2132 deletions
@@ -6,14 +6,7 @@ import { EventLogTable } from 'twenty-shared/types';
import { ClickHouseService } from 'src/database/clickHouse/clickHouse.service';
import { formatDateTimeForClickHouse } from 'src/database/clickHouse/clickHouse.util';
const CLICKHOUSE_TABLE_NAMES: Record<EventLogTable, string> = {
[EventLogTable.WORKSPACE_EVENT]: 'workspaceEvent',
[EventLogTable.PAGEVIEW]: 'pageview',
[EventLogTable.OBJECT_EVENT]: 'objectEvent',
[EventLogTable.USAGE_EVENT]: 'usageEvent',
[EventLogTable.APPLICATION_LOG]: 'applicationLog',
};
import { getClickHouseTableName } from 'src/engine/core-modules/event-logs/registry/event-log-registry';
export type EventLogCleanupParams = {
workspaceId: string;
@@ -43,11 +36,9 @@ export class EventLogCleanupService {
cutoffDate.setDate(cutoffDate.getDate() - retentionDays);
for (const table of Object.values(EventLogTable)) {
const tableName = CLICKHOUSE_TABLE_NAMES[table];
const tableName = getClickHouseTableName(table);
try {
// ClickHouse ALTER TABLE DELETE is async by default
// We use lightweight deletes (mutations) which are efficient
const success = await this.clickHouseService.executeCommand(
`ALTER TABLE ${tableName} DELETE WHERE "workspaceId" = {workspaceId:String} AND "timestamp" < {cutoffDate:DateTime64(3)}`,
{
@@ -7,5 +7,3 @@ export const registerEventLogTableEnum = () => {
name: 'EventLogTable',
});
};
export { EventLogTable };
@@ -0,0 +1,53 @@
import { makePageview } from './analytics.utils';
describe('makePageview', () => {
it('should create a pageview with default properties when none are provided', () => {
const result = makePageview('test-page');
expect(result.name).toBe('test-page');
expect(result.type).toBe('page');
expect(result.properties).toEqual({
href: '',
locale: '',
pathname: '',
referrer: '',
sessionId: '',
timeZone: '',
userAgent: '',
});
expect(result.timestamp).toBeDefined();
expect(result.version).toBe('1');
});
it('should create a pageview with provided properties and fill in defaults for missing ones', () => {
const providedProperties = {
href: 'https://example.com',
sessionId: 'test-session-id',
};
const result = makePageview('test-page', providedProperties);
expect(result.name).toBe('test-page');
expect(result.type).toBe('page');
expect(result.properties).toEqual({
href: 'https://example.com',
locale: '',
pathname: '',
referrer: '',
sessionId: 'test-session-id',
timeZone: '',
userAgent: '',
});
expect(result.timestamp).toBeDefined();
expect(result.version).toBe('1');
});
it('should handle empty properties object', () => {
const result = makePageview('test-page', {});
expect(result.name).toBe('test-page');
expect(result.type).toBe('page');
expect(result.properties.sessionId).toBe('');
expect(result.properties.href).toBe('');
});
});
@@ -0,0 +1,49 @@
import { formatDateTimeForClickHouse } from 'src/database/clickHouse/clickHouse.util';
import { type EventCommonPropertiesType } from 'src/engine/core-modules/event-logs/emit/common.type';
import {
type TrackEventName,
type TrackEventProperties,
} from 'src/engine/core-modules/event-logs/emit/events.type';
import {
type PageviewProperties,
pageviewSchema,
} from 'src/engine/core-modules/event-logs/emit/events/pageview/pageview';
import {
eventsRegistry,
type GenericTrackEvent,
} from 'src/engine/core-modules/event-logs/emit/events/workspace-event/track';
const common = (): Record<EventCommonPropertiesType, string> => ({
timestamp: formatDateTimeForClickHouse(new Date()),
version: '1',
});
export function makePageview(
name: string,
properties: Partial<PageviewProperties> = {},
) {
return pageviewSchema.parse({
type: 'page',
name,
...common(),
properties,
});
}
export function makeTrackEvent<T extends TrackEventName>(
event: T,
properties: TrackEventProperties<T>,
): GenericTrackEvent<T> {
const schema = eventsRegistry.get(event);
if (!schema) {
throw new Error(`Schema for event ${event} is not implemented`);
}
return schema.parse({
type: 'track',
event,
properties,
...common(),
});
}
@@ -0,0 +1,36 @@
import {
buildPageviewEnvelope,
computeEventContextFields,
} from 'src/engine/core-modules/event-logs/emit/build-event-envelope';
describe('build-event-envelope', () => {
describe('computeEventContextFields', () => {
it('keeps defined ids and drops null/undefined', () => {
expect(
computeEventContextFields({ workspaceId: 'w', userId: 'u' }),
).toEqual({ workspaceId: 'w', userId: 'u' });
expect(
computeEventContextFields({ workspaceId: 'w', userId: null }),
).toEqual({ workspaceId: 'w' });
expect(computeEventContextFields()).toEqual({});
});
});
describe('buildPageviewEnvelope', () => {
it('tags the envelope with the pageview table and merges context', () => {
const envelope = buildPageviewEnvelope(
{ workspaceId: 'w', userId: 'u' },
'home',
{},
);
expect(envelope.table).toBe('pageview');
expect(envelope.row).toMatchObject({
workspaceId: 'w',
userId: 'u',
type: 'page',
name: 'home',
});
});
});
});
@@ -0,0 +1,65 @@
import {
type TrackEventName,
type TrackEventProperties,
} from 'src/engine/core-modules/event-logs/emit/events.type';
import {
type EventContextFields,
type WorkspaceEventEnvelope,
} from 'src/engine/core-modules/event-logs/types/workspace-event-envelope.type';
import {
makePageview,
makeTrackEvent,
} from 'src/engine/core-modules/event-logs/emit/analytics.utils';
import { type PageviewProperties } from 'src/engine/core-modules/event-logs/emit/events/pageview/pageview';
export const computeEventContextFields = (
context?: EventContextFields,
): EventContextFields => ({
...(context?.workspaceId ? { workspaceId: context.workspaceId } : {}),
...(context?.userId ? { userId: context.userId } : {}),
});
export const buildWorkspaceEventEnvelope = <T extends TrackEventName>(
contextFields: EventContextFields,
event: T,
properties: TrackEventProperties<T>,
): WorkspaceEventEnvelope => ({
table: 'workspaceEvent',
row: { ...contextFields, ...makeTrackEvent(event, properties) },
});
export const buildObjectEventEnvelope = <T extends TrackEventName>(
contextFields: EventContextFields,
event: T,
properties: TrackEventProperties<T> & {
recordId: string;
objectMetadataId: string;
isCustom?: boolean;
},
): WorkspaceEventEnvelope => {
const { recordId, objectMetadataId, isCustom, ...restProperties } =
properties;
return {
table: 'objectEvent',
row: {
...contextFields,
...makeTrackEvent(
event,
restProperties as unknown as TrackEventProperties<T>,
),
recordId,
objectMetadataId,
isCustom,
},
};
};
export const buildPageviewEnvelope = (
contextFields: EventContextFields,
name: string,
properties: Partial<PageviewProperties>,
): WorkspaceEventEnvelope => ({
table: 'pageview',
row: { ...contextFields, ...makePageview(name, properties) },
});
@@ -0,0 +1,2 @@
export type EventCommonPropertiesType = 'timestamp' | 'version';
export type IdentifierType = 'workspaceId' | 'userId';
@@ -0,0 +1,9 @@
import { ObjectType, Field } from '@nestjs/graphql';
@ObjectType()
export class Analytics {
@Field(() => Boolean, {
description: 'Boolean that confirms query was dispatched',
})
success: boolean;
}
@@ -0,0 +1,50 @@
import { ArgsType, Field, registerEnumType } from '@nestjs/graphql';
import { IsEnum, IsObject, IsOptional, IsString } from 'class-validator';
import GraphQLJSON from 'graphql-type-json';
import { TrackEventName } from 'src/engine/core-modules/event-logs/emit/events.type';
import { type PageviewProperties } from 'src/engine/core-modules/event-logs/emit/events/pageview/pageview';
enum AnalyticsType {
PAGEVIEW = 'pageview',
TRACK = 'track',
}
registerEnumType(AnalyticsType, {
name: 'AnalyticsType',
});
@ArgsType()
export class CreateAnalyticsInputV2 {
@Field(() => AnalyticsType)
@IsEnum(AnalyticsType)
type: 'pageview' | 'track';
@Field(() => String, { nullable: true })
@IsOptional()
@IsString()
name?: string;
@Field(() => String, { nullable: true })
@IsOptional()
@IsString()
event?: TrackEventName;
@Field(() => GraphQLJSON, { nullable: true })
@IsOptional()
@IsObject()
properties?: PageviewProperties | Record<string, unknown>;
}
export function isPageviewAnalyticsInput(
input: CreateAnalyticsInputV2,
): input is CreateAnalyticsInputV2 & { name: string } {
return input.type === 'pageview' && !!input.name;
}
export function isTrackAnalyticsInput(
input: CreateAnalyticsInputV2,
): input is CreateAnalyticsInputV2 & { event: TrackEventName } {
return input.type === 'track' && !!input.event;
}
@@ -0,0 +1,37 @@
import { ArgsType, Field } from '@nestjs/graphql';
import {
IsNotEmpty,
IsObject,
IsOptional,
IsString,
IsUUID,
} from 'class-validator';
import GraphQLJSON from 'graphql-type-json';
import { UUIDScalarType } from 'src/engine/api/graphql/workspace-schema-builder/graphql-types/scalars';
import { TrackEventName } from 'src/engine/core-modules/event-logs/emit/events.type';
@ArgsType()
export class CreateObjectEventInput {
@Field(() => String)
@IsNotEmpty()
@IsString()
event: TrackEventName;
@Field(() => UUIDScalarType)
@IsNotEmpty()
@IsUUID()
recordId: string;
@Field(() => UUIDScalarType)
@IsNotEmpty()
@IsUUID()
objectMetadataId: string;
@Field(() => GraphQLJSON, { nullable: true })
@IsObject()
@IsOptional()
// oxlint-disable-next-line typescript/no-explicit-any
properties?: Record<string, any>;
}
@@ -0,0 +1,23 @@
import { Catch, type ExceptionFilter } from '@nestjs/common';
import { assertUnreachable } from 'twenty-shared/utils';
import {
EventLogEmitterException,
EventLogEmitterExceptionCode,
} from 'src/engine/core-modules/event-logs/emit/event-log-emitter.exception';
import { UserInputError } from 'src/engine/core-modules/graphql/utils/graphql-errors.util';
@Catch(EventLogEmitterException)
export class EventLogEmitterExceptionFilter implements ExceptionFilter {
catch(exception: EventLogEmitterException) {
switch (exception.code) {
case EventLogEmitterExceptionCode.INVALID_TYPE:
case EventLogEmitterExceptionCode.INVALID_INPUT:
throw new UserInputError(exception);
default: {
assertUnreachable(exception.code);
}
}
}
}
@@ -0,0 +1,37 @@
import { type MessageDescriptor } from '@lingui/core';
import { msg } from '@lingui/core/macro';
import { assertUnreachable } from 'twenty-shared/utils';
import { CustomException } from 'src/utils/custom-exception';
export enum EventLogEmitterExceptionCode {
INVALID_TYPE = 'INVALID_TYPE',
INVALID_INPUT = 'INVALID_INPUT',
}
const getEventLogEmitterExceptionUserFriendlyMessage = (
code: EventLogEmitterExceptionCode,
) => {
switch (code) {
case EventLogEmitterExceptionCode.INVALID_TYPE:
return msg`Invalid event type.`;
case EventLogEmitterExceptionCode.INVALID_INPUT:
return msg`Invalid event input.`;
default:
assertUnreachable(code);
}
};
export class EventLogEmitterException extends CustomException<EventLogEmitterExceptionCode> {
constructor(
message: string,
code: EventLogEmitterExceptionCode,
{ userFriendlyMessage }: { userFriendlyMessage?: MessageDescriptor } = {},
) {
super(message, code, {
userFriendlyMessage:
userFriendlyMessage ??
getEventLogEmitterExceptionUserFriendlyMessage(code),
});
}
}
@@ -0,0 +1,11 @@
import { Module } from '@nestjs/common';
import { EventLogEmitterService } from 'src/engine/core-modules/event-logs/emit/event-log-emitter.service';
import { EventLogIngestionModule } from 'src/engine/core-modules/event-logs/ingest/event-log-ingestion.module';
@Module({
imports: [EventLogIngestionModule],
providers: [EventLogEmitterService],
exports: [EventLogEmitterService],
})
export class EventLogEmitterModule {}
@@ -0,0 +1,173 @@
import { Test, type TestingModule } from '@nestjs/testing';
import {
EventLogEmitterException,
EventLogEmitterExceptionCode,
} from 'src/engine/core-modules/event-logs/emit/event-log-emitter.exception';
import { type UserEntity } from 'src/engine/core-modules/user/user.entity';
import { type WorkspaceEntity } from 'src/engine/core-modules/workspace/workspace.entity';
import { EventLogEmitterResolver } from './event-log-emitter.resolver';
import { EventLogEmitterService } from './event-log-emitter.service';
describe('EventLogEmitterResolver', () => {
let resolver: EventLogEmitterResolver;
let auditService: jest.Mocked<EventLogEmitterService>;
beforeEach(async () => {
auditService = {
createContext: jest.fn(),
} as any;
const module: TestingModule = await Test.createTestingModule({
providers: [
EventLogEmitterResolver,
{
provide: EventLogEmitterService,
useValue: auditService,
},
],
}).compile();
resolver = module.get<EventLogEmitterResolver>(EventLogEmitterResolver);
});
it('should be defined', () => {
expect(resolver).toBeDefined();
});
it('should handle a valid pageview input', async () => {
const mockInsertPageviewEvent = jest
.fn()
.mockResolvedValue('Pageview created');
auditService.createContext.mockReturnValue({
createPageviewEvent: mockInsertPageviewEvent,
insertWorkspaceEvent: jest.fn(),
createObjectEvent: jest.fn(),
});
const input = {
type: 'pageview' as const,
name: 'Test Page',
properties: {},
};
const result = await resolver.trackAnalytics(
input,
{ id: 'workspace-1' } as WorkspaceEntity,
{ id: 'user-1' } as UserEntity,
);
expect(auditService.createContext).toHaveBeenCalledWith({
workspaceId: 'workspace-1',
userId: 'user-1',
});
expect(mockInsertPageviewEvent).toHaveBeenCalledWith('Test Page', {});
expect(result).toBe('Pageview created');
});
it('should handle a valid track input', async () => {
const mockInsertWorkspaceEvent = jest
.fn()
.mockResolvedValue('Track created');
auditService.createContext.mockReturnValue({
insertWorkspaceEvent: mockInsertWorkspaceEvent,
createObjectEvent: jest.fn(),
createPageviewEvent: jest.fn(),
});
const input = {
type: 'track' as const,
event: 'Custom Domain Activated' as const,
properties: {},
};
const result = await resolver.trackAnalytics(
input,
{ id: 'workspace-2' } as WorkspaceEntity,
{ id: 'user-2' } as UserEntity,
);
expect(auditService.createContext).toHaveBeenCalledWith({
workspaceId: 'workspace-2',
userId: 'user-2',
});
expect(mockInsertWorkspaceEvent).toHaveBeenCalledWith(
'Custom Domain Activated',
{},
);
expect(result).toBe('Track created');
});
it('should handle object event creation', async () => {
const mockInsertObjectEvent = jest
.fn()
.mockResolvedValue('Object event created');
auditService.createContext.mockReturnValue({
insertWorkspaceEvent: jest.fn(),
createObjectEvent: mockInsertObjectEvent,
createPageviewEvent: jest.fn(),
});
const input = {
event: 'Object Record Created' as const,
recordId: 'test-record-id',
objectMetadataId: 'test-object-metadata-id',
properties: { additionalData: 'test-data' },
};
const result = await resolver.createObjectEvent(
input,
{ id: 'workspace-3' } as WorkspaceEntity,
{ id: 'user-3' } as UserEntity,
);
expect(auditService.createContext).toHaveBeenCalledWith({
workspaceId: 'workspace-3',
userId: 'user-3',
});
expect(mockInsertObjectEvent).toHaveBeenCalledWith(
'Object Record Created',
{
additionalData: 'test-data',
recordId: 'test-record-id',
objectMetadataId: 'test-object-metadata-id',
isCustom: true,
},
);
expect(result).toBe('Object event created');
});
it('should throw an EventLogEmitterException for invalid input', async () => {
const invalidInput = { type: 'invalid' };
await expect(
resolver.trackAnalytics(invalidInput as any, undefined, undefined),
).rejects.toThrowError(
new EventLogEmitterException(
'Invalid analytics input',
EventLogEmitterExceptionCode.INVALID_TYPE,
),
);
});
it('should throw an EventLogEmitterException when workspace is missing for createObjectEvent', async () => {
const input = {
event: 'Object Record Created' as const,
recordId: 'test-record-id',
objectMetadataId: 'test-object-metadata-id',
};
await expect(
resolver.createObjectEvent(input, undefined, undefined),
).rejects.toThrowError(
new EventLogEmitterException(
'Missing workspace',
EventLogEmitterExceptionCode.INVALID_INPUT,
),
);
});
});
@@ -0,0 +1,101 @@
import { UseFilters, UseGuards, UsePipes } from '@nestjs/common';
import { Args, Mutation } from '@nestjs/graphql';
import { MetadataResolver } from 'src/engine/api/graphql/graphql-config/decorators/metadata-resolver.decorator';
import { EventLogEmitterExceptionFilter } from 'src/engine/core-modules/event-logs/emit/event-log-emitter-exception.filter';
import {
EventLogEmitterException,
EventLogEmitterExceptionCode,
} from 'src/engine/core-modules/event-logs/emit/event-log-emitter.exception';
import { CreateObjectEventInput } from 'src/engine/core-modules/event-logs/emit/dtos/create-object-event.input';
import { PreventNestToAutoLogGraphqlErrorsFilter } from 'src/engine/core-modules/graphql/filters/prevent-nest-to-auto-log-graphql-errors.filter';
import { ResolverValidationPipe } from 'src/engine/core-modules/graphql/pipes/resolver-validation.pipe';
import { UserEntity } from 'src/engine/core-modules/user/user.entity';
import { WorkspaceEntity } from 'src/engine/core-modules/workspace/workspace.entity';
import { AuthUser } from 'src/engine/decorators/auth/auth-user.decorator';
import { AuthWorkspace } from 'src/engine/decorators/auth/auth-workspace.decorator';
import { NoPermissionGuard } from 'src/engine/guards/no-permission.guard';
import { PublicEndpointGuard } from 'src/engine/guards/public-endpoint.guard';
import { WorkspaceAuthGuard } from 'src/engine/guards/workspace-auth.guard';
import { Analytics } from './dtos/analytics.dto';
import {
CreateAnalyticsInputV2,
isPageviewAnalyticsInput,
isTrackAnalyticsInput,
} from './dtos/create-analytics.input';
import { EventLogEmitterService } from './event-log-emitter.service';
@MetadataResolver(() => Analytics)
@UsePipes(ResolverValidationPipe)
@UseFilters(
EventLogEmitterExceptionFilter,
PreventNestToAutoLogGraphqlErrorsFilter,
)
export class EventLogEmitterResolver {
constructor(
private readonly eventLogEmitterService: EventLogEmitterService,
) {}
@Mutation(() => Analytics)
@UseGuards(WorkspaceAuthGuard, NoPermissionGuard)
async createObjectEvent(
@Args()
createObjectEventInput: CreateObjectEventInput,
@AuthWorkspace() workspace: WorkspaceEntity | undefined,
@AuthUser({ allowUndefined: true }) user: UserEntity | undefined,
) {
if (!workspace) {
throw new EventLogEmitterException(
'Missing workspace',
EventLogEmitterExceptionCode.INVALID_INPUT,
);
}
const eventLogContext = this.eventLogEmitterService.createContext({
workspaceId: workspace.id,
userId: user?.id,
});
return eventLogContext.createObjectEvent(createObjectEventInput.event, {
...createObjectEventInput.properties,
recordId: createObjectEventInput.recordId,
objectMetadataId: createObjectEventInput.objectMetadataId,
isCustom: true,
});
}
@Mutation(() => Analytics)
@UseGuards(PublicEndpointGuard, NoPermissionGuard)
async trackAnalytics(
@Args()
createAnalyticsInput: CreateAnalyticsInputV2,
@AuthWorkspace({ allowUndefined: true })
workspace: WorkspaceEntity | undefined,
@AuthUser({ allowUndefined: true }) user: UserEntity | undefined,
) {
const eventLogContext = this.eventLogEmitterService.createContext({
workspaceId: workspace?.id,
userId: user?.id,
});
if (isPageviewAnalyticsInput(createAnalyticsInput)) {
return eventLogContext.createPageviewEvent(
createAnalyticsInput.name,
createAnalyticsInput.properties ?? {},
);
}
if (isTrackAnalyticsInput(createAnalyticsInput)) {
return eventLogContext.insertWorkspaceEvent(
createAnalyticsInput.event,
createAnalyticsInput.properties ?? {},
);
}
throw new EventLogEmitterException(
'Invalid analytics input',
EventLogEmitterExceptionCode.INVALID_TYPE,
);
}
}
@@ -0,0 +1,149 @@
import { Test, type TestingModule } from '@nestjs/testing';
import { AuditContextMock } from 'test/utils/audit-context.mock';
import { ClickHouseService } from 'src/database/clickHouse/clickHouse.service';
import { OBJECT_RECORD_CREATED_EVENT } from 'src/engine/core-modules/event-logs/emit/events/object-event/object-record-created';
import { CUSTOM_DOMAIN_ACTIVATED_EVENT } from 'src/engine/core-modules/event-logs/emit/events/workspace-event/custom-domain/custom-domain-activated';
import { ExceptionHandlerService } from 'src/engine/core-modules/exception-handler/exception-handler.service';
import { TwentyConfigService } from 'src/engine/core-modules/twenty-config/twenty-config.service';
import { EventLogEmitterService } from './event-log-emitter.service';
describe('EventLogEmitterService', () => {
let service: EventLogEmitterService;
beforeEach(async () => {
const module: TestingModule = await Test.createTestingModule({
providers: [
{
provide: EventLogEmitterService,
useValue: {
createContext: AuditContextMock,
},
},
{
provide: TwentyConfigService,
useValue: {
get: jest.fn().mockReturnValue(true),
},
},
{
provide: ClickHouseService,
useValue: {
pushEvent: jest.fn(),
},
},
{
provide: ExceptionHandlerService,
useValue: {
captureExceptions: jest.fn(),
},
},
],
}).compile();
service = module.get<EventLogEmitterService>(EventLogEmitterService);
});
it('should be defined', () => {
expect(service).toBeDefined();
});
describe('createContext', () => {
const mockUserIdAndWorkspaceId = {
userId: 'test-user-id',
workspaceId: 'test-workspace-id',
};
it('should create a valid context object', () => {
const context = service.createContext(mockUserIdAndWorkspaceId);
expect(context).toHaveProperty('insertWorkspaceEvent');
expect(context).toHaveProperty('createObjectEvent');
expect(context).toHaveProperty('createPageviewEvent');
});
it('should call insertWorkspaceEvent with correct parameters', async () => {
const insertWorkspaceEventSpy = jest
.fn()
.mockResolvedValue({ success: true });
const mockContext = AuditContextMock({
insertWorkspaceEvent: insertWorkspaceEventSpy,
});
jest.spyOn(service, 'createContext').mockReturnValue(mockContext);
const context = service.createContext(mockUserIdAndWorkspaceId);
await context.insertWorkspaceEvent(CUSTOM_DOMAIN_ACTIVATED_EVENT, {});
expect(insertWorkspaceEventSpy).toHaveBeenCalledWith(
CUSTOM_DOMAIN_ACTIVATED_EVENT,
{},
);
});
it('should call createPageviewEvent with correct parameters', async () => {
const createPageviewEventSpy = jest
.fn()
.mockResolvedValue({ success: true });
const mockContext = AuditContextMock({
createPageviewEvent: createPageviewEventSpy,
});
jest.spyOn(service, 'createContext').mockReturnValue(mockContext);
const context = service.createContext(mockUserIdAndWorkspaceId);
const testPageviewProperties = {
href: '/test-url',
locale: '',
pathname: '',
referrer: '',
sessionId: '',
timeZone: '',
userAgent: '',
};
await context.createPageviewEvent('page-view', testPageviewProperties);
expect(createPageviewEventSpy).toHaveBeenCalledWith(
'page-view',
testPageviewProperties,
);
});
it('should return success when insertWorkspaceEvent is called', async () => {
const context = service.createContext(mockUserIdAndWorkspaceId);
const result = await context.insertWorkspaceEvent(
CUSTOM_DOMAIN_ACTIVATED_EVENT,
{},
);
expect(result).toEqual({ success: true });
});
it('should return success when createPageviewEvent is called', async () => {
const context = service.createContext(mockUserIdAndWorkspaceId);
const result = await context.createPageviewEvent('page-view', {});
expect(result).toEqual({ success: true });
});
it('should return success when createObjectEvent is called', async () => {
const context = service.createContext(mockUserIdAndWorkspaceId);
const result = await context.createObjectEvent(
OBJECT_RECORD_CREATED_EVENT,
{
recordId: 'test-record-id',
objectMetadataId: 'test-object-metadata-id',
},
);
expect(result).toEqual({ success: true });
});
});
});
@@ -0,0 +1,83 @@
import { Injectable, Logger } from '@nestjs/common';
import {
buildObjectEventEnvelope,
buildPageviewEnvelope,
buildWorkspaceEventEnvelope,
computeEventContextFields,
} from 'src/engine/core-modules/event-logs/emit/build-event-envelope';
import { type PageviewProperties } from 'src/engine/core-modules/event-logs/emit/events/pageview/pageview';
import {
type TrackEventName,
type TrackEventProperties,
} from 'src/engine/core-modules/event-logs/emit/events.type';
import { WorkspaceEventSinkService } from 'src/engine/core-modules/event-logs/ingest/workspace-event-sink.service';
import {
type EventContextFields,
type WorkspaceEventEnvelope,
} from 'src/engine/core-modules/event-logs/types/workspace-event-envelope.type';
@Injectable()
export class EventLogEmitterService {
private readonly logger = new Logger(EventLogEmitterService.name);
constructor(
private readonly workspaceEventSinkService: WorkspaceEventSinkService,
) {}
isEnabled(): boolean {
return this.workspaceEventSinkService.isEnabled();
}
async dispatch(events: WorkspaceEventEnvelope[]): Promise<void> {
if (events.length === 0 || !this.isEnabled()) {
return;
}
await this.workspaceEventSinkService.ingest(events);
}
createContext(context?: EventContextFields) {
const contextFields = computeEventContextFields(context);
return {
insertWorkspaceEvent: <T extends TrackEventName>(
event: T,
properties: TrackEventProperties<T>,
) =>
this.emit(() =>
buildWorkspaceEventEnvelope(contextFields, event, properties),
),
createObjectEvent: <T extends TrackEventName>(
event: T,
properties: TrackEventProperties<T> & {
recordId: string;
objectMetadataId: string;
isCustom?: boolean;
},
) =>
this.emit(() =>
buildObjectEventEnvelope(contextFields, event, properties),
),
createPageviewEvent: (
name: string,
properties: Partial<PageviewProperties>,
) =>
this.emit(() => buildPageviewEnvelope(contextFields, name, properties)),
};
}
private async emit(
buildEnvelope: () => WorkspaceEventEnvelope,
): Promise<{ success: boolean }> {
try {
await this.dispatch([buildEnvelope()]);
return { success: true };
} catch (error) {
this.logger.error('Failed to emit workspace event', error);
return { success: false };
}
}
}
@@ -0,0 +1,82 @@
import {
type OBJECT_RECORD_CREATED_EVENT,
type ObjectRecordCreatedTrackEvent,
} from 'src/engine/core-modules/event-logs/emit/events/object-event/object-record-created';
import {
type OBJECT_RECORD_DELETED_EVENT,
type ObjectRecordDeletedTrackEvent,
} from 'src/engine/core-modules/event-logs/emit/events/object-event/object-record-delete';
import {
type OBJECT_RECORD_UPDATED_EVENT,
type ObjectRecordUpdatedTrackEvent,
} from 'src/engine/core-modules/event-logs/emit/events/object-event/object-record-updated';
import {
type OBJECT_RECORD_UPSERTED_EVENT,
type ObjectRecordUpsertedTrackEvent,
} from 'src/engine/core-modules/event-logs/emit/events/object-event/object-record-upserted';
import {
type CUSTOM_DOMAIN_ACTIVATED_EVENT,
type CustomDomainActivatedTrackEvent,
} from 'src/engine/core-modules/event-logs/emit/events/workspace-event/custom-domain/custom-domain-activated';
import {
type CUSTOM_DOMAIN_DEACTIVATED_EVENT,
type CustomDomainDeactivatedTrackEvent,
} from 'src/engine/core-modules/event-logs/emit/events/workspace-event/custom-domain/custom-domain-deactivated';
import {
type LOGIC_FUNCTION_EXECUTED_EVENT,
type LogicFunctionExecutedTrackEvent,
} from 'src/engine/core-modules/event-logs/emit/events/workspace-event/logic-function/logic-function-executed';
import {
type IMPERSONATION_EVENT,
type ImpersonationTrackEvent,
} from 'src/engine/core-modules/event-logs/emit/events/workspace-event/impersonation/impersonation';
import {
type USER_SIGNUP_EVENT,
type UserSignupTrackEvent,
} from 'src/engine/core-modules/event-logs/emit/events/workspace-event/user/user-signup';
import {
type WEBHOOK_RESPONSE_EVENT,
type WebhookResponseTrackEvent,
} from 'src/engine/core-modules/event-logs/emit/events/workspace-event/webhook/webhook-response';
import {
type PAYMENT_RECEIVED_EVENT,
type PaymentReceivedTrackEvent,
} from 'src/engine/core-modules/event-logs/emit/events/workspace-event/billing/payment-received';
import {
type WORKSPACE_CREATED_EVENT,
type WorkspaceCreatedTrackEvent,
} from 'src/engine/core-modules/event-logs/emit/events/workspace-event/workspace/workspace-created';
export type TrackEventName =
| typeof CUSTOM_DOMAIN_ACTIVATED_EVENT
| typeof CUSTOM_DOMAIN_DEACTIVATED_EVENT
| typeof LOGIC_FUNCTION_EXECUTED_EVENT
| typeof WEBHOOK_RESPONSE_EVENT
| typeof IMPERSONATION_EVENT
| typeof OBJECT_RECORD_CREATED_EVENT
| typeof OBJECT_RECORD_UPDATED_EVENT
| typeof OBJECT_RECORD_DELETED_EVENT
| typeof OBJECT_RECORD_UPSERTED_EVENT
| typeof USER_SIGNUP_EVENT
| typeof WORKSPACE_CREATED_EVENT
| typeof PAYMENT_RECEIVED_EVENT;
export interface TrackEvents {
[CUSTOM_DOMAIN_ACTIVATED_EVENT]: CustomDomainActivatedTrackEvent;
[CUSTOM_DOMAIN_DEACTIVATED_EVENT]: CustomDomainDeactivatedTrackEvent;
[LOGIC_FUNCTION_EXECUTED_EVENT]: LogicFunctionExecutedTrackEvent;
[WEBHOOK_RESPONSE_EVENT]: WebhookResponseTrackEvent;
[USER_SIGNUP_EVENT]: UserSignupTrackEvent;
[IMPERSONATION_EVENT]: ImpersonationTrackEvent;
[OBJECT_RECORD_DELETED_EVENT]: ObjectRecordDeletedTrackEvent;
[OBJECT_RECORD_CREATED_EVENT]: ObjectRecordCreatedTrackEvent;
[OBJECT_RECORD_UPDATED_EVENT]: ObjectRecordUpdatedTrackEvent;
[OBJECT_RECORD_UPSERTED_EVENT]: ObjectRecordUpsertedTrackEvent;
[WORKSPACE_CREATED_EVENT]: WorkspaceCreatedTrackEvent;
[PAYMENT_RECEIVED_EVENT]: PaymentReceivedTrackEvent;
}
export type TrackEventProperties<T extends TrackEventName> =
T extends keyof TrackEvents
? TrackEvents[T]['properties']
: Record<string, unknown>;
@@ -0,0 +1,8 @@
import { z } from 'zod';
export const baseEventSchema = z.strictObject({
timestamp: z.string(),
userId: z.string().nullish(),
workspaceId: z.string().nullish(),
version: z.string(),
});
@@ -0,0 +1,15 @@
import { z } from 'zod';
import { registerEvent } from 'src/engine/core-modules/event-logs/emit/events/workspace-event/track';
export const OBJECT_RECORD_CREATED_EVENT = 'Object Record Created' as const;
export const objectRecordCreatedSchema = z.object({
event: z.literal(OBJECT_RECORD_CREATED_EVENT),
properties: z.looseObject({}),
});
export type ObjectRecordCreatedTrackEvent = z.infer<
typeof objectRecordCreatedSchema
>;
registerEvent(OBJECT_RECORD_CREATED_EVENT, objectRecordCreatedSchema);
@@ -0,0 +1,15 @@
import { z } from 'zod';
import { registerEvent } from 'src/engine/core-modules/event-logs/emit/events/workspace-event/track';
export const OBJECT_RECORD_DELETED_EVENT = 'Object Record Deleted' as const;
export const objectRecordDeletedSchema = z.object({
event: z.literal(OBJECT_RECORD_DELETED_EVENT),
properties: z.looseObject({}),
});
export type ObjectRecordDeletedTrackEvent = z.infer<
typeof objectRecordDeletedSchema
>;
registerEvent(OBJECT_RECORD_DELETED_EVENT, objectRecordDeletedSchema);
@@ -0,0 +1,15 @@
import { z } from 'zod';
import { registerEvent } from 'src/engine/core-modules/event-logs/emit/events/workspace-event/track';
export const OBJECT_RECORD_UPDATED_EVENT = 'Object Record Updated' as const;
export const objectRecordUpdatedSchema = z.object({
event: z.literal(OBJECT_RECORD_UPDATED_EVENT),
properties: z.looseObject({}),
});
export type ObjectRecordUpdatedTrackEvent = z.infer<
typeof objectRecordUpdatedSchema
>;
registerEvent(OBJECT_RECORD_UPDATED_EVENT, objectRecordUpdatedSchema);
@@ -0,0 +1,15 @@
import { z } from 'zod';
import { registerEvent } from 'src/engine/core-modules/event-logs/emit/events/workspace-event/track';
export const OBJECT_RECORD_UPSERTED_EVENT = 'Object Record Upserted' as const;
export const objectRecordUpsertedSchema = z.object({
event: z.literal(OBJECT_RECORD_UPSERTED_EVENT),
properties: z.looseObject({}),
});
export type ObjectRecordUpsertedTrackEvent = z.infer<
typeof objectRecordUpsertedSchema
>;
registerEvent(OBJECT_RECORD_UPSERTED_EVENT, objectRecordUpsertedSchema);
@@ -0,0 +1,19 @@
import { z } from 'zod';
import { baseEventSchema } from 'src/engine/core-modules/event-logs/emit/events/common/base-schemas';
export const pageviewSchema = baseEventSchema.extend({
type: z.literal('page'),
name: z.string(),
properties: z.object({
href: z.string().optional().default(''),
locale: z.string().optional().default(''),
pathname: z.string().optional().default(''),
referrer: z.string().optional().default(''),
sessionId: z.string().optional().default(''),
timeZone: z.string().optional().default(''),
userAgent: z.string().optional().default(''),
}),
});
export type PageviewProperties = z.infer<typeof pageviewSchema>['properties'];
@@ -0,0 +1,15 @@
import { z } from 'zod';
import { registerEvent } from 'src/engine/core-modules/event-logs/emit/events/workspace-event/track';
export const PAYMENT_RECEIVED_EVENT = 'Payment Received' as const;
export const paymentReceivedSchema = z.strictObject({
event: z.literal(PAYMENT_RECEIVED_EVENT),
properties: z.strictObject({
amountPaid: z.number(),
}),
});
export type PaymentReceivedTrackEvent = z.infer<typeof paymentReceivedSchema>;
registerEvent(PAYMENT_RECEIVED_EVENT, paymentReceivedSchema);
@@ -0,0 +1,15 @@
import { z } from 'zod';
import { registerEvent } from 'src/engine/core-modules/event-logs/emit/events/workspace-event/track';
export const CUSTOM_DOMAIN_ACTIVATED_EVENT = 'Custom Domain Activated' as const;
export const customDomainActivatedSchema = z.strictObject({
event: z.literal(CUSTOM_DOMAIN_ACTIVATED_EVENT),
properties: z.strictObject({}),
});
export type CustomDomainActivatedTrackEvent = z.infer<
typeof customDomainActivatedSchema
>;
registerEvent(CUSTOM_DOMAIN_ACTIVATED_EVENT, customDomainActivatedSchema);
@@ -0,0 +1,16 @@
import { z } from 'zod';
import { registerEvent } from 'src/engine/core-modules/event-logs/emit/events/workspace-event/track';
export const CUSTOM_DOMAIN_DEACTIVATED_EVENT =
'Custom Domain Deactivated' as const;
export const customDomainDeactivatedSchema = z.strictObject({
event: z.literal(CUSTOM_DOMAIN_DEACTIVATED_EVENT),
properties: z.strictObject({}),
});
export type CustomDomainDeactivatedTrackEvent = z.infer<
typeof customDomainDeactivatedSchema
>;
registerEvent(CUSTOM_DOMAIN_DEACTIVATED_EVENT, customDomainDeactivatedSchema);
@@ -0,0 +1,28 @@
import { z } from 'zod';
import { registerEvent } from 'src/engine/core-modules/event-logs/emit/events/workspace-event/track';
export const IMPERSONATION_EVENT = 'Impersonation' as const;
export const impersonationSchema = z.strictObject({
event: z.literal(IMPERSONATION_EVENT),
properties: z.strictObject({
level: z.enum(['server', 'workspace']),
action: z.enum([
'attempt',
'attempted',
'issued',
'login_token_attempt',
'login_token_generated',
'login_token_failed',
'token_exchange_attempt',
'token_exchange_success',
'token_exchange_failed',
]),
message: z.string().optional(),
}),
});
export type ImpersonationTrackEvent = z.infer<typeof impersonationSchema>;
registerEvent(IMPERSONATION_EVENT, impersonationSchema);
@@ -0,0 +1,21 @@
import { z } from 'zod';
import { registerEvent } from 'src/engine/core-modules/event-logs/emit/events/workspace-event/track';
export const LOGIC_FUNCTION_EXECUTED_EVENT = 'Logic Function Executed' as const;
export const logicFunctionExecutedSchema = z.strictObject({
event: z.literal(LOGIC_FUNCTION_EXECUTED_EVENT),
properties: z.strictObject({
duration: z.number(),
status: z.enum(['IDLE', 'SUCCESS', 'ERROR']),
errorType: z.string().optional(),
functionId: z.string(),
functionName: z.string(),
}),
});
export type LogicFunctionExecutedTrackEvent = z.infer<
typeof logicFunctionExecutedSchema
>;
registerEvent(LOGIC_FUNCTION_EXECUTED_EVENT, logicFunctionExecutedSchema);
@@ -0,0 +1,31 @@
import { z } from 'zod';
import { baseEventSchema } from 'src/engine/core-modules/event-logs/emit/events/common/base-schemas';
export const genericTrackSchema = baseEventSchema.extend({
type: z.literal('track'),
event: z.string(),
properties: z.any(),
});
export type GenericTrackEvent<E extends string = string> = {
type: 'track';
event: E;
// oxlint-disable-next-line typescript/no-explicit-any
properties: any;
timestamp: string;
version: string;
userId?: string;
workspaceId?: string;
};
// oxlint-disable-next-line typescript/no-explicit-any
export const eventsRegistry = new Map<string, z.ZodSchema<any>>();
// oxlint-disable-next-line typescript/no-explicit-any
export function registerEvent<E extends string, S extends z.ZodObject<any>>(
event: E,
schema: S,
): void {
eventsRegistry.set(event, genericTrackSchema.merge(schema));
}
@@ -0,0 +1,13 @@
import { z } from 'zod';
import { registerEvent } from 'src/engine/core-modules/event-logs/emit/events/workspace-event/track';
export const USER_SIGNUP_EVENT = 'User Signup' as const;
export const userSignupSchema = z.strictObject({
event: z.literal(USER_SIGNUP_EVENT),
properties: z.strictObject({}),
});
export type UserSignupTrackEvent = z.infer<typeof userSignupSchema>;
registerEvent(USER_SIGNUP_EVENT, userSignupSchema);
@@ -0,0 +1,20 @@
import { z } from 'zod';
import { registerEvent } from 'src/engine/core-modules/event-logs/emit/events/workspace-event/track';
export const WEBHOOK_RESPONSE_EVENT = 'Webhook Response' as const;
export const webhookResponseSchema = z.strictObject({
event: z.literal(WEBHOOK_RESPONSE_EVENT),
properties: z.strictObject({
status: z.number().optional(),
success: z.boolean(),
url: z.string(),
webhookId: z.string(),
eventName: z.string(),
error: z.string().optional(),
}),
});
export type WebhookResponseTrackEvent = z.infer<typeof webhookResponseSchema>;
registerEvent(WEBHOOK_RESPONSE_EVENT, webhookResponseSchema);
@@ -0,0 +1,13 @@
import { z } from 'zod';
import { registerEvent } from 'src/engine/core-modules/event-logs/emit/events/workspace-event/track';
export const WORKSPACE_CREATED_EVENT = 'Workspace Created' as const;
export const workspaceCreatedSchema = z.strictObject({
event: z.literal(WORKSPACE_CREATED_EVENT),
properties: z.strictObject({}),
});
export type WorkspaceCreatedTrackEvent = z.infer<typeof workspaceCreatedSchema>;
registerEvent(WORKSPACE_CREATED_EVENT, workspaceCreatedSchema);
@@ -0,0 +1,96 @@
import { UseFilters, UseGuards, UsePipes } from '@nestjs/common';
import { Args, Subscription } from '@nestjs/graphql';
import { PermissionFlagType } from 'twenty-shared/constants';
import { EventLogTable } from 'twenty-shared/types';
import { MetadataResolver } from 'src/engine/api/graphql/graphql-config/decorators/metadata-resolver.decorator';
import { AuthGraphqlApiExceptionFilter } from 'src/engine/core-modules/auth/filters/auth-graphql-api-exception.filter';
import { EventLogsGraphqlApiExceptionFilter } from 'src/engine/core-modules/event-logs/filters/event-logs-graphql-api-exception.filter';
import { ForbiddenExceptionGraphqlFilter } from 'src/engine/core-modules/event-logs/filters/forbidden-exception-graphql.filter';
import { PreventNestToAutoLogGraphqlErrorsFilter } from 'src/engine/core-modules/graphql/filters/prevent-nest-to-auto-log-graphql-errors.filter';
import { ResolverValidationPipe } from 'src/engine/core-modules/graphql/pipes/resolver-validation.pipe';
import { WorkspaceEntity } from 'src/engine/core-modules/workspace/workspace.entity';
import { AuthWorkspace } from 'src/engine/decorators/auth/auth-workspace.decorator';
import { SettingsPermissionGuard } from 'src/engine/guards/settings-permission.guard';
import { WorkspaceAuthGuard } from 'src/engine/guards/workspace-auth.guard';
import { PermissionsGraphqlApiExceptionFilter } from 'src/engine/metadata-modules/permissions/utils/permissions-graphql-api-exception.filter';
import { APPLICATION_KEEPALIVE_INTERVAL_MS } from 'src/engine/subscriptions/constants/application-keepalive-interval-ms.constant';
import { SubscriptionChannel } from 'src/engine/subscriptions/enums/subscription-channel.enum';
import { SubscriptionService } from 'src/engine/subscriptions/subscription.service';
import { wrapAsyncIteratorWithLifecycle } from 'src/engine/subscriptions/utils/wrap-async-iterator-with-lifecycle';
import { EventLogLiveService } from 'src/engine/core-modules/event-logs/live/event-log-live.service';
import { EventLogsService } from './event-logs.service';
import { EventLogRecord } from './dtos/event-log-result.dto';
import { getClickHouseTableName } from './registry/event-log-registry';
import { normalizeEventLogRecords } from './utils/normalize-event-log-records';
type WorkspaceEventLivePayload = {
table: string;
rows: Record<string, unknown>[];
};
@MetadataResolver()
@UseFilters(
ForbiddenExceptionGraphqlFilter,
AuthGraphqlApiExceptionFilter,
EventLogsGraphqlApiExceptionFilter,
PermissionsGraphqlApiExceptionFilter,
PreventNestToAutoLogGraphqlErrorsFilter,
)
@UsePipes(ResolverValidationPipe)
export class EventLogsLiveResolver {
constructor(
private readonly eventLogsService: EventLogsService,
private readonly subscriptionService: SubscriptionService,
private readonly workspaceEventLiveService: EventLogLiveService,
) {}
@UseGuards(
WorkspaceAuthGuard,
SettingsPermissionGuard(PermissionFlagType.SECURITY),
)
@Subscription(() => [EventLogRecord], {
nullable: true,
filter: (
payload: WorkspaceEventLivePayload,
variables: { table: EventLogTable },
) => getClickHouseTableName(variables.table) === payload.table,
resolve: (
payload: WorkspaceEventLivePayload,
variables: { table: EventLogTable },
) => normalizeEventLogRecords(payload.rows, variables.table),
})
async eventLogsLive(
@Args('table', { type: () => EventLogTable }) table: EventLogTable,
@AuthWorkspace() workspace: WorkspaceEntity,
) {
await this.eventLogsService.validateAccess(workspace.id, table);
const clickHouseTable = getClickHouseTableName(table);
await this.workspaceEventLiveService.markWatched(
workspace.id,
clickHouseTable,
);
const iterator = await this.subscriptionService.subscribe({
channel: SubscriptionChannel.WORKSPACE_EVENTS_CHANNEL,
workspaceId: workspace.id,
});
return wrapAsyncIteratorWithLifecycle(iterator, {
onHeartbeat: async () => {
await this.workspaceEventLiveService.markWatched(
workspace.id,
clickHouseTable,
);
return true;
},
heartbeatIntervalMs: APPLICATION_KEEPALIVE_INTERVAL_MS,
});
}
}
@@ -3,13 +3,19 @@
import { Module } from '@nestjs/common';
import { TypeOrmModule } from '@nestjs/typeorm';
import { ClickHouseModule } from 'src/database/clickHouse/clickHouse.module';
import { BillingModule } from 'src/engine/core-modules/billing/billing.module';
import { EnterpriseModule } from 'src/engine/core-modules/enterprise/enterprise.module';
import { EventLogEmitterModule } from 'src/engine/core-modules/event-logs/emit/event-log-emitter.module';
import { EventLogEmitterResolver } from 'src/engine/core-modules/event-logs/emit/event-log-emitter.resolver';
import { EventLogLiveModule } from 'src/engine/core-modules/event-logs/live/event-log-live.module';
import { GuardRedirectModule } from 'src/engine/core-modules/guard-redirect/guard-redirect.module';
import { JwtModule } from 'src/engine/core-modules/jwt/jwt.module';
import { ClickHouseModule } from 'src/database/clickHouse/clickHouse.module';
import { UserWorkspaceEntity } from 'src/engine/core-modules/user-workspace/user-workspace.entity';
import { PermissionsModule } from 'src/engine/metadata-modules/permissions/permissions.module';
import { SubscriptionsModule } from 'src/engine/subscriptions/subscriptions.module';
import { EventLogsLiveResolver } from './event-logs-live.resolver';
import { EventLogsResolver } from './event-logs.resolver';
import { EventLogsService } from './event-logs.service';
@@ -20,9 +26,18 @@ import { EventLogsService } from './event-logs.service';
BillingModule,
EnterpriseModule,
GuardRedirectModule,
JwtModule,
EventLogLiveModule,
EventLogEmitterModule,
SubscriptionsModule,
TypeOrmModule.forFeature([UserWorkspaceEntity]),
],
providers: [EventLogsResolver, EventLogsService],
providers: [
EventLogsResolver,
EventLogsLiveResolver,
EventLogsService,
EventLogEmitterResolver,
],
exports: [EventLogsService],
})
export class EventLogsModule {}
export class EventLogsViewerModule {}
@@ -0,0 +1,79 @@
import { EventLogTable } from 'twenty-shared/types';
import { ClickHouseService } from 'src/database/clickHouse/clickHouse.service';
import { BillingEntitlementKey } from 'src/engine/core-modules/billing/enums/billing-entitlement-key.enum';
import { BillingService } from 'src/engine/core-modules/billing/services/billing.service';
import { EnterprisePlanService } from 'src/engine/core-modules/enterprise/services/enterprise-plan.service';
import { EventLogsExceptionCode } from 'src/engine/core-modules/event-logs/event-logs.exception';
import { EventLogsService } from 'src/engine/core-modules/event-logs/event-logs.service';
describe('EventLogsService.validateAccess', () => {
let service: EventLogsService;
let getMainClient: jest.Mock;
let hasEntitlement: jest.Mock;
let isValid: jest.Mock;
beforeEach(() => {
getMainClient = jest.fn().mockReturnValue({});
hasEntitlement = jest.fn().mockResolvedValue(true);
isValid = jest.fn().mockReturnValue(true);
service = new EventLogsService(
{ getMainClient } as unknown as ClickHouseService,
{ hasEntitlement } as unknown as BillingService,
{ isValid } as unknown as EnterprisePlanService,
{} as never,
);
});
const validateAccessError = async (table: EventLogTable) =>
service.validateAccess('ws-1', table).then(
() => undefined,
(error) => error,
);
it('throws CLICKHOUSE_NOT_CONFIGURED when ClickHouse is unavailable', async () => {
getMainClient.mockReturnValue(undefined);
const error = await validateAccessError(EventLogTable.WORKSPACE_EVENT);
expect(error?.code).toBe(EventLogsExceptionCode.CLICKHOUSE_NOT_CONFIGURED);
});
it('allows application logs with no entitlement (free on every plan)', async () => {
isValid.mockReturnValue(false);
hasEntitlement.mockResolvedValue(false);
await expect(
service.validateAccess('ws-1', EventLogTable.APPLICATION_LOG),
).resolves.toBeUndefined();
expect(hasEntitlement).not.toHaveBeenCalled();
});
it('throws NO_ENTITLEMENT for a gated table when the Enterprise plan is invalid (skips the billing call)', async () => {
isValid.mockReturnValue(false);
const error = await validateAccessError(EventLogTable.WORKSPACE_EVENT);
expect(error?.code).toBe(EventLogsExceptionCode.NO_ENTITLEMENT);
expect(hasEntitlement).not.toHaveBeenCalled();
});
it('throws NO_ENTITLEMENT for a gated table when the AUDIT_LOGS entitlement is missing', async () => {
hasEntitlement.mockResolvedValue(false);
const error = await validateAccessError(EventLogTable.USAGE_EVENT);
expect(error?.code).toBe(EventLogsExceptionCode.NO_ENTITLEMENT);
expect(hasEntitlement).toHaveBeenCalledWith(
'ws-1',
BillingEntitlementKey.AUDIT_LOGS,
);
});
it('allows a gated table when the plan is valid and the entitlement is held', async () => {
await expect(
service.validateAccess('ws-1', EventLogTable.OBJECT_EVENT),
).resolves.toBeUndefined();
});
});
@@ -9,7 +9,6 @@ import { Repository } from 'typeorm';
import { ClickHouseService } from 'src/database/clickHouse/clickHouse.service';
import { formatDateTimeForClickHouse } from 'src/database/clickHouse/clickHouse.util';
import { BillingEntitlementKey } from 'src/engine/core-modules/billing/enums/billing-entitlement-key.enum';
import { BillingService } from 'src/engine/core-modules/billing/services/billing.service';
import { EnterprisePlanService } from 'src/engine/core-modules/enterprise/services/enterprise-plan.service';
import { UserWorkspaceEntity } from 'src/engine/core-modules/user-workspace/user-workspace.entity';
@@ -21,57 +20,16 @@ import {
import { EventLogFiltersInput } from './dtos/event-log-filters.input';
import { EventLogQueryInput } from './dtos/event-log-query.input';
import { EventLogQueryResult } from './dtos/event-log-result.dto';
import {
EventLogQueryResult,
EventLogRecord,
} from './dtos/event-log-result.dto';
type ClickHouseEventRecord = {
event?: string;
name?: string;
timestamp: string;
userId?: string;
properties?: Record<string, unknown>;
recordId?: string;
objectMetadataId?: string;
isCustom?: boolean;
};
type ClickHouseUsageEventRecord = {
timestamp: string;
userWorkspaceId?: string;
resourceType?: string;
operationType?: string;
quantity?: number;
unit?: string;
creditsUsedMicro?: number;
resourceId?: string;
resourceContext?: string;
metadata?: Record<string, unknown>;
};
type ClickHouseApplicationLogRecord = {
timestamp: string;
applicationId?: string;
logicFunctionId?: string;
logicFunctionName?: string;
executionId?: string;
level?: string;
message?: string;
properties?: Record<string, unknown>;
};
EVENT_LOG_TYPES,
getClickHouseTableName,
} from './registry/event-log-registry';
import { normalizeEventLogRecords } from './utils/normalize-event-log-records';
const ALLOWED_TABLES = Object.values(EventLogTable);
const MAX_LIMIT = 10000;
const CLICKHOUSE_TABLE_NAMES: Record<EventLogTable, string> = {
[EventLogTable.WORKSPACE_EVENT]: 'workspaceEvent',
[EventLogTable.PAGEVIEW]: 'pageview',
[EventLogTable.OBJECT_EVENT]: 'objectEvent',
[EventLogTable.USAGE_EVENT]: 'usageEvent',
[EventLogTable.APPLICATION_LOG]: 'applicationLog',
};
@Injectable()
export class EventLogsService {
constructor(
@@ -93,15 +51,8 @@ export class EventLogsService {
}
const limit = Math.min(input.first ?? 100, MAX_LIMIT);
const tableName = CLICKHOUSE_TABLE_NAMES[input.table];
const eventFieldName =
input.table === EventLogTable.USAGE_EVENT
? 'resourceType'
: input.table === EventLogTable.PAGEVIEW
? 'name'
: input.table === EventLogTable.APPLICATION_LOG
? 'logicFunctionName'
: 'event';
const tableName = getClickHouseTableName(input.table);
const eventFieldName = EVENT_LOG_TYPES[input.table].eventFieldName;
const whereClauses: string[] = ['"workspaceId" = {workspaceId:String}'];
const params: Record<string, unknown> = { workspaceId };
@@ -145,7 +96,7 @@ export class EventLogsService {
params.limit = limit + 1;
const [records, countResult] = await Promise.all([
this.clickHouseService.select<ClickHouseEventRecord>(query, params),
this.clickHouseService.select<Record<string, unknown>>(query, params),
this.clickHouseService.select<{ totalCount: number }>(countQuery, params),
]);
@@ -156,7 +107,7 @@ export class EventLogsService {
records.pop();
}
const normalizedRecords = this.normalizeRecords(records, input.table);
const normalizedRecords = normalizeEventLogRecords(records, input.table);
const lastRecord = normalizedRecords[normalizedRecords.length - 1];
const endCursor =
hasNextPage && lastRecord
@@ -173,7 +124,7 @@ export class EventLogsService {
};
}
private async validateAccess(
async validateAccess(
workspaceId: string,
table: EventLogTable,
): Promise<void> {
@@ -184,23 +135,20 @@ export class EventLogsService {
);
}
if (table === EventLogTable.APPLICATION_LOG) {
const requiredEntitlement = EVENT_LOG_TYPES[table].requiresEntitlement;
if (requiredEntitlement === null) {
return;
}
if (!this.enterprisePlanService.isValid()) {
throw new EventLogsException(
'Audit logs require an Enterprise subscription.',
EventLogsExceptionCode.NO_ENTITLEMENT,
);
}
const hasAccess =
this.enterprisePlanService.isValid() &&
(await this.billingService.hasEntitlement(
workspaceId,
requiredEntitlement,
));
const hasEntitlement = await this.billingService.hasEntitlement(
workspaceId,
BillingEntitlementKey.AUDIT_LOGS,
);
if (!hasEntitlement) {
if (!hasAccess) {
throw new EventLogsException(
'Audit logs require an Enterprise subscription.',
EventLogsExceptionCode.NO_ENTITLEMENT,
@@ -226,10 +174,7 @@ export class EventLogsService {
params.eventTypePattern = `%${filters.eventType.toLowerCase()}%`;
}
// TODO: Legacy event tables (workspaceEvent, pageview, objectEvent) use
// userId because some actions are logged out. Usage events use
// userWorkspaceId directly which is more relevant in a workspace context.
// Consider migrating all event tables to userWorkspaceId for consistency.
// TODO: non-usage tables filter by userId (some actions are logged out) while usageEvent uses userWorkspaceId; migrate all to userWorkspaceId for consistency.
if (isDefined(filters.userWorkspaceId)) {
if (table === EventLogTable.APPLICATION_LOG) {
// Application logs don't have a user column
@@ -279,61 +224,4 @@ export class EventLogsService {
private decodeCursor(cursor: string): number {
return parseInt(Buffer.from(cursor, 'base64').toString('utf-8'), 10);
}
private normalizeRecords(
records:
| ClickHouseEventRecord[]
| ClickHouseUsageEventRecord[]
| ClickHouseApplicationLogRecord[],
table: EventLogTable,
): EventLogRecord[] {
if (table === EventLogTable.USAGE_EVENT) {
return (records as ClickHouseUsageEventRecord[]).map((record) => ({
event: record.resourceType ?? '',
timestamp: new Date(record.timestamp),
userId: record.userWorkspaceId,
properties: {
operationType: record.operationType,
quantity: record.quantity,
unit: record.unit,
creditsUsedMicro: record.creditsUsedMicro,
resourceId: record.resourceId,
resourceContext: record.resourceContext,
...(record.metadata ?? {}),
},
}));
}
if (table === EventLogTable.APPLICATION_LOG) {
return (records as ClickHouseApplicationLogRecord[]).map((record) => ({
event: record.logicFunctionName ?? '',
timestamp: new Date(record.timestamp),
properties: {
level: record.level,
message: record.message,
executionId: record.executionId,
logicFunctionId: record.logicFunctionId,
applicationId: record.applicationId,
...(record.properties ?? {}),
},
}));
}
return (records as ClickHouseEventRecord[]).map((record) => {
const eventName =
table === EventLogTable.PAGEVIEW
? (record.name ?? '')
: (record.event ?? '');
return {
event: eventName,
timestamp: new Date(record.timestamp),
userId: record.userId,
properties: record.properties,
recordId: record.recordId,
objectMetadataId: record.objectMetadataId,
isCustom: record.isCustom,
};
});
}
}
@@ -0,0 +1,69 @@
import { ClickHouseService } from 'src/database/clickHouse/clickHouse.service';
import { ClickHouseEventSink } from 'src/engine/core-modules/event-logs/ingest/clickhouse-event.sink';
import { type WorkspaceEventEnvelope } from 'src/engine/core-modules/event-logs/types/workspace-event-envelope.type';
const makePageview = (name: string): WorkspaceEventEnvelope => ({
table: 'pageview',
row: { type: 'page', name, properties: {}, timestamp: 't', version: '1' },
});
const applicationLog: WorkspaceEventEnvelope = {
table: 'applicationLog',
row: {
timestamp: 't',
workspaceId: 'w',
applicationId: '',
logicFunctionId: '',
logicFunctionName: 'fn',
executionId: 'e',
level: 'INFO',
message: 'm',
},
};
describe('ClickHouseEventSink', () => {
let sink: ClickHouseEventSink;
let insert: jest.Mock;
let getMainClient: jest.Mock;
beforeEach(() => {
insert = jest.fn().mockResolvedValue({ success: true });
getMainClient = jest.fn().mockReturnValue({});
sink = new ClickHouseEventSink({
insert,
getMainClient,
} as unknown as ClickHouseService);
});
it('groups envelopes by table and inserts each group once', async () => {
const first = makePageview('a');
const second = makePageview('b');
await sink.write([first, second, applicationLog]);
expect(insert).toHaveBeenCalledTimes(2);
expect(insert).toHaveBeenCalledWith('pageview', [first.row, second.row]);
expect(insert).toHaveBeenCalledWith('applicationLog', [applicationLog.row]);
});
it('no-ops when ClickHouse is not configured', async () => {
getMainClient.mockReturnValue(undefined);
await sink.write([makePageview('a')]);
expect(insert).not.toHaveBeenCalled();
});
it('no-ops on an empty batch', async () => {
await sink.write([]);
expect(insert).not.toHaveBeenCalled();
});
it('throws when a ClickHouse insert fails so the consumer retries', async () => {
insert.mockResolvedValue({ success: false });
await expect(sink.write([makePageview('a')])).rejects.toThrow();
});
});
@@ -0,0 +1,37 @@
import { Injectable } from '@nestjs/common';
import { ClickHouseService } from 'src/database/clickHouse/clickHouse.service';
import { type EventSink } from 'src/engine/core-modules/event-logs/ingest/event-sink';
import { type WorkspaceEventEnvelope } from 'src/engine/core-modules/event-logs/types/workspace-event-envelope.type';
@Injectable()
export class ClickHouseEventSink implements EventSink {
constructor(private readonly clickHouseService: ClickHouseService) {}
async write(events: WorkspaceEventEnvelope[]): Promise<void> {
if (events.length === 0 || !this.clickHouseService.getMainClient()) {
return;
}
const rowsByTable = new Map<string, Record<string, unknown>[]>();
for (const event of events) {
const rows = rowsByTable.get(event.table) ?? [];
rows.push(event.row);
rowsByTable.set(event.table, rows);
}
await Promise.all(
[...rowsByTable.entries()].map(async ([table, rows]) => {
const result = await this.clickHouseService.insert(table, rows);
if (!result.success) {
throw new Error(
`Failed to insert ${rows.length} ${table} row(s) into ClickHouse`,
);
}
}),
);
}
}
@@ -0,0 +1,34 @@
import { Injectable, Logger } from '@nestjs/common';
import { type EventSink } from 'src/engine/core-modules/event-logs/ingest/event-sink';
import { type WorkspaceEventEnvelope } from 'src/engine/core-modules/event-logs/types/workspace-event-envelope.type';
@Injectable()
export class ConsoleEventSink implements EventSink {
private readonly logger = new Logger(ConsoleEventSink.name);
async write(events: WorkspaceEventEnvelope[]): Promise<void> {
for (const event of events) {
if (event.table === 'applicationLog') {
const context = `${event.row.logicFunctionName}:${event.row.executionId}`;
switch (event.row.level) {
case 'ERROR':
this.logger.error(event.row.message, undefined, context);
break;
case 'WARN':
this.logger.warn(event.row.message, context);
break;
case 'DEBUG':
this.logger.debug(event.row.message, context);
break;
default:
this.logger.log(event.row.message, context);
break;
}
} else {
this.logger.log(JSON.stringify(event.row), event.table);
}
}
}
}
@@ -0,0 +1,30 @@
import { type ObjectRecordEvent } from 'twenty-shared/database-events';
import { CreateEventLogFromInternalEvent } from 'src/engine/core-modules/event-logs/ingest/create-event-log-from-internal-event';
import { WorkspaceEventSinkService } from 'src/engine/core-modules/event-logs/ingest/workspace-event-sink.service';
import { type WorkspaceEventBatch } from 'src/engine/workspace-event-emitter/types/workspace-event-batch.type';
describe('CreateEventLogFromInternalEvent', () => {
it('ingests object events (persist + live fan-out) through the sink pipeline', async () => {
const ingest = jest.fn().mockResolvedValue(undefined);
const handler = new CreateEventLogFromInternalEvent({
isEnabled: () => true,
ingest,
} as unknown as WorkspaceEventSinkService);
const batch = {
name: 'company.created',
workspaceId: 'workspace-1',
objectMetadata: { id: 'object-metadata-1' },
events: [{ recordId: 'record-1', userId: 'user-1', properties: {} }],
} as unknown as WorkspaceEventBatch<ObjectRecordEvent>;
await handler.handle(batch);
const ingestedEnvelopes = ingest.mock.calls[0]?.[0];
expect(ingestedEnvelopes).toHaveLength(1);
expect(ingestedEnvelopes[0].table).toBe('objectEvent');
});
});
@@ -0,0 +1,84 @@
import { type ObjectRecordEvent } from 'twenty-shared/database-events';
import { isDefined } from 'twenty-shared/utils';
import { WorkspaceEventSinkService } from 'src/engine/core-modules/event-logs/ingest/workspace-event-sink.service';
import { type WorkspaceEventEnvelope } from 'src/engine/core-modules/event-logs/types/workspace-event-envelope.type';
import {
buildObjectEventEnvelope,
computeEventContextFields,
} from 'src/engine/core-modules/event-logs/emit/build-event-envelope';
import { OBJECT_RECORD_CREATED_EVENT } from 'src/engine/core-modules/event-logs/emit/events/object-event/object-record-created';
import { OBJECT_RECORD_DELETED_EVENT } from 'src/engine/core-modules/event-logs/emit/events/object-event/object-record-delete';
import { OBJECT_RECORD_UPDATED_EVENT } from 'src/engine/core-modules/event-logs/emit/events/object-event/object-record-updated';
import { OBJECT_RECORD_UPSERTED_EVENT } from 'src/engine/core-modules/event-logs/emit/events/object-event/object-record-upserted';
import { Process } from 'src/engine/core-modules/message-queue/decorators/process.decorator';
import { Processor } from 'src/engine/core-modules/message-queue/decorators/processor.decorator';
import { MessageQueue } from 'src/engine/core-modules/message-queue/message-queue.constants';
import { WorkspaceEventBatch } from 'src/engine/workspace-event-emitter/types/workspace-event-batch.type';
const OBJECT_EVENT_BY_SUFFIX = {
'.created': OBJECT_RECORD_CREATED_EVENT,
'.updated': OBJECT_RECORD_UPDATED_EVENT,
'.deleted': OBJECT_RECORD_DELETED_EVENT,
'.upserted': OBJECT_RECORD_UPSERTED_EVENT,
} as const;
@Processor(MessageQueue.entityEventsToDbQueue)
export class CreateEventLogFromInternalEvent {
constructor(
private readonly workspaceEventSinkService: WorkspaceEventSinkService,
) {}
@Process(CreateEventLogFromInternalEvent.name)
async handle(batch: WorkspaceEventBatch<ObjectRecordEvent>): Promise<void> {
if (!this.workspaceEventSinkService.isEnabled()) {
return;
}
const envelopes = this.toEnvelopes(batch);
if (envelopes.length === 0) {
return;
}
await this.workspaceEventSinkService.ingest(envelopes);
}
private toEnvelopes(
batch: WorkspaceEventBatch<ObjectRecordEvent>,
): WorkspaceEventEnvelope[] {
const suffix = (
Object.keys(
OBJECT_EVENT_BY_SUFFIX,
) as (keyof typeof OBJECT_EVENT_BY_SUFFIX)[]
).find((candidate) => batch.name.endsWith(candidate));
if (!isDefined(suffix)) {
return [];
}
const event = OBJECT_EVENT_BY_SUFFIX[suffix];
return batch.events.map((eventData) =>
buildObjectEventEnvelope(
computeEventContextFields({
workspaceId: batch.workspaceId,
userId: eventData.userId,
}),
event,
this.objectProperties(batch, eventData),
),
);
}
private objectProperties(
batch: WorkspaceEventBatch<ObjectRecordEvent>,
eventData: ObjectRecordEvent,
) {
return {
...eventData.properties,
recordId: eventData.recordId,
objectMetadataId: batch.objectMetadata.id,
};
}
}
@@ -0,0 +1,65 @@
import { Logger, Module } from '@nestjs/common';
import { isDefined } from 'twenty-shared/utils';
import { ClickHouseModule } from 'src/database/clickHouse/clickHouse.module';
import { ClickHouseEventSink } from 'src/engine/core-modules/event-logs/ingest/clickhouse-event.sink';
import { ConsoleEventSink } from 'src/engine/core-modules/event-logs/ingest/console-event.sink';
import { CreateEventLogFromInternalEvent } from 'src/engine/core-modules/event-logs/ingest/create-event-log-from-internal-event';
import {
getAvailableSinkNames,
KNOWN_SINK_NAMES,
} from 'src/engine/core-modules/event-logs/ingest/event-sink-availability';
import {
EVENT_SINKS,
type EventSink,
} from 'src/engine/core-modules/event-logs/ingest/event-sink';
import { WorkspaceEventSinkService } from 'src/engine/core-modules/event-logs/ingest/workspace-event-sink.service';
import { EventLogLiveModule } from 'src/engine/core-modules/event-logs/live/event-log-live.module';
import { TwentyConfigService } from 'src/engine/core-modules/twenty-config/twenty-config.service';
const eventSinksProvider = {
provide: EVENT_SINKS,
useFactory: (
twentyConfigService: TwentyConfigService,
clickHouseEventSink: ClickHouseEventSink,
consoleEventSink: ConsoleEventSink,
): EventSink[] => {
const sinkByName: Record<string, EventSink> = {
clickhouse: clickHouseEventSink,
console: consoleEventSink,
};
const configuredSinkNames = twentyConfigService.get('EVENT_SINKS');
const unknownSinkNames = configuredSinkNames.filter(
(name) => !KNOWN_SINK_NAMES.includes(name.toLowerCase() as never),
);
if (unknownSinkNames.length > 0) {
new Logger('WorkspaceEventSinks').warn(
`Ignoring unknown EVENT_SINKS: ${unknownSinkNames.join(', ')}`,
);
}
return getAvailableSinkNames(configuredSinkNames, {
hasClickhouseUrl: Boolean(twentyConfigService.get('CLICKHOUSE_URL')),
})
.map((name) => sinkByName[name.toLowerCase()])
.filter(isDefined);
},
inject: [TwentyConfigService, ClickHouseEventSink, ConsoleEventSink],
};
@Module({
imports: [ClickHouseModule, EventLogLiveModule],
providers: [
ClickHouseEventSink,
ConsoleEventSink,
eventSinksProvider,
WorkspaceEventSinkService,
CreateEventLogFromInternalEvent,
],
exports: [WorkspaceEventSinkService],
})
export class EventLogIngestionModule {}
@@ -0,0 +1,19 @@
export const KNOWN_SINK_NAMES = ['clickhouse', 'console'] as const;
export const getAvailableSinkNames = (
configuredSinkNames: string[],
{ hasClickhouseUrl }: { hasClickhouseUrl: boolean },
): string[] =>
configuredSinkNames.filter((name) => {
const lowerCasedName = name.toLowerCase();
if (lowerCasedName === 'clickhouse') {
return hasClickhouseUrl;
}
if (lowerCasedName === 'console') {
return true;
}
return false;
});
@@ -0,0 +1,7 @@
import { type WorkspaceEventEnvelope } from 'src/engine/core-modules/event-logs/types/workspace-event-envelope.type';
export type EventSink = {
write(events: WorkspaceEventEnvelope[]): Promise<void>;
};
export const EVENT_SINKS = Symbol('EVENT_SINKS');
@@ -0,0 +1,30 @@
import { Inject, Injectable } from '@nestjs/common';
import {
EVENT_SINKS,
type EventSink,
} from 'src/engine/core-modules/event-logs/ingest/event-sink';
import { EventLogLiveService } from 'src/engine/core-modules/event-logs/live/event-log-live.service';
import { type WorkspaceEventEnvelope } from 'src/engine/core-modules/event-logs/types/workspace-event-envelope.type';
@Injectable()
export class WorkspaceEventSinkService {
constructor(
@Inject(EVENT_SINKS)
private readonly sinks: EventSink[],
private readonly workspaceEventLiveService: EventLogLiveService,
) {}
isEnabled(): boolean {
return this.sinks.length > 0;
}
async ingest(events: WorkspaceEventEnvelope[]): Promise<void> {
await this.persist(events);
await this.workspaceEventLiveService.publishWatched(events);
}
private async persist(events: WorkspaceEventEnvelope[]): Promise<void> {
await Promise.all(this.sinks.map((sink) => sink.write(events)));
}
}
@@ -0,0 +1 @@
export const EVENT_LOG_LIVE_TTL_MS = 60 * 1_000;
@@ -0,0 +1,12 @@
import { Module } from '@nestjs/common';
import { CacheStorageModule } from 'src/engine/core-modules/cache-storage/cache-storage.module';
import { EventLogLiveService } from 'src/engine/core-modules/event-logs/live/event-log-live.service';
import { SubscriptionsModule } from 'src/engine/subscriptions/subscriptions.module';
@Module({
imports: [CacheStorageModule, SubscriptionsModule],
providers: [EventLogLiveService],
exports: [EventLogLiveService],
})
export class EventLogLiveModule {}
@@ -0,0 +1,93 @@
import { Injectable, Logger } from '@nestjs/common';
import { isDefined } from 'twenty-shared/utils';
import { type WorkspaceEventEnvelope } from 'src/engine/core-modules/event-logs/types/workspace-event-envelope.type';
import { InjectCacheStorage } from 'src/engine/core-modules/cache-storage/decorators/cache-storage.decorator';
import { CacheStorageService } from 'src/engine/core-modules/cache-storage/services/cache-storage.service';
import { CacheStorageNamespace } from 'src/engine/core-modules/cache-storage/types/cache-storage-namespace.enum';
import { EVENT_LOG_LIVE_TTL_MS } from 'src/engine/core-modules/event-logs/live/event-log-live-ttl.constant';
import { SubscriptionChannel } from 'src/engine/subscriptions/enums/subscription-channel.enum';
import { SubscriptionService } from 'src/engine/subscriptions/subscription.service';
type WatchedGroup = {
workspaceId: string;
table: string;
rows: Record<string, unknown>[];
};
@Injectable()
export class EventLogLiveService {
private readonly logger = new Logger(EventLogLiveService.name);
constructor(
@InjectCacheStorage(CacheStorageNamespace.EngineSubscriptions)
private readonly cacheStorageService: CacheStorageService,
private readonly subscriptionService: SubscriptionService,
) {}
private getPresenceKey(workspaceId: string, key: string): string {
return `workspaceEventLive:${workspaceId}:${key}`;
}
async markWatched(workspaceId: string, key: string): Promise<void> {
await this.cacheStorageService.set<boolean>(
this.getPresenceKey(workspaceId, key),
true,
EVENT_LOG_LIVE_TTL_MS,
);
}
async isWatched(workspaceId: string, key: string): Promise<boolean> {
const value = await this.cacheStorageService.get<boolean>(
this.getPresenceKey(workspaceId, key),
);
return isDefined(value);
}
async publishWatched(events: WorkspaceEventEnvelope[]): Promise<void> {
const groups = new Map<string, WatchedGroup>();
for (const event of events) {
const workspaceId = event.row.workspaceId;
if (!isDefined(workspaceId)) {
continue;
}
const key = `${workspaceId}:${event.table}`;
const group = groups.get(key) ?? {
workspaceId,
table: event.table,
rows: [],
};
group.rows.push(event.row);
groups.set(key, group);
}
const results = await Promise.allSettled(
[...groups.values()].map(async ({ workspaceId, table, rows }) => {
if (!(await this.isWatched(workspaceId, table))) {
return;
}
await this.subscriptionService.publish({
channel: SubscriptionChannel.WORKSPACE_EVENTS_CHANNEL,
workspaceId,
payload: { table, rows },
});
}),
);
for (const result of results) {
if (result.status === 'rejected') {
this.logger.error(
'Failed to publish live workspace events',
result.reason,
);
}
}
}
}
@@ -0,0 +1,10 @@
export type ApplicationLogEntry = {
timestamp: Date;
workspaceId: string;
applicationId: string;
logicFunctionId: string;
logicFunctionName: string;
executionId: string;
level: string;
message: string;
};
@@ -0,0 +1,20 @@
import { formatDateTimeForClickHouse } from 'src/database/clickHouse/clickHouse.util';
import { type ApplicationLogEntry } from 'src/engine/core-modules/event-logs/producers/application-log/application-log-entry.interface';
import { type WorkspaceEventEnvelope } from 'src/engine/core-modules/event-logs/types/workspace-event-envelope.type';
export const buildApplicationLogEnvelopes = (
entries: ApplicationLogEntry[],
): WorkspaceEventEnvelope[] =>
entries.map((entry) => ({
table: 'applicationLog',
row: {
timestamp: formatDateTimeForClickHouse(entry.timestamp),
workspaceId: entry.workspaceId,
applicationId: entry.applicationId,
logicFunctionId: entry.logicFunctionId,
logicFunctionName: entry.logicFunctionName,
executionId: entry.executionId,
level: entry.level,
message: entry.message,
},
}));
@@ -0,0 +1,145 @@
import { parseApplicationLogLines } from './parse-application-log-lines';
describe('parseApplicationLogLines', () => {
it('should return an empty array for empty string', () => {
expect(parseApplicationLogLines('')).toEqual([]);
});
it('should return an empty array for undefined-ish input', () => {
expect(parseApplicationLogLines(undefined as unknown as string)).toEqual(
[],
);
});
it('should parse a structured INFO log line', () => {
const raw = '2024-06-15T10:30:00.123Z INFO Hello world';
expect(parseApplicationLogLines(raw)).toEqual([
{
timestamp: new Date('2024-06-15T10:30:00.123Z'),
level: 'INFO',
message: 'Hello world',
},
]);
});
it('should parse all supported log levels', () => {
const raw = [
'2024-01-01T00:00:00.000Z INFO info message',
'2024-01-01T00:00:01.000Z ERROR error message',
'2024-01-01T00:00:02.000Z WARN warn message',
'2024-01-01T00:00:03.000Z DEBUG debug message',
].join('\n');
expect(parseApplicationLogLines(raw)).toEqual([
{
timestamp: new Date('2024-01-01T00:00:00.000Z'),
level: 'INFO',
message: 'info message',
},
{
timestamp: new Date('2024-01-01T00:00:01.000Z'),
level: 'ERROR',
message: 'error message',
},
{
timestamp: new Date('2024-01-01T00:00:02.000Z'),
level: 'WARN',
message: 'warn message',
},
{
timestamp: new Date('2024-01-01T00:00:03.000Z'),
level: 'DEBUG',
message: 'debug message',
},
]);
});
it('should default unstructured lines to INFO with current timestamp', () => {
const now = Date.now();
const raw = 'some plain text without timestamp or level';
const result = parseApplicationLogLines(raw);
expect(result).toHaveLength(1);
expect(result[0].level).toBe('INFO');
expect(result[0].message).toBe(
'some plain text without timestamp or level',
);
expect(result[0].timestamp.getTime()).toBeGreaterThanOrEqual(now);
expect(result[0].timestamp.getTime()).toBeLessThanOrEqual(now + 1000);
});
it('should skip empty lines', () => {
const raw =
'2024-01-01T00:00:00.000Z INFO first\n\n\n2024-01-01T00:00:01.000Z ERROR second\n';
expect(parseApplicationLogLines(raw)).toEqual([
{
timestamp: new Date('2024-01-01T00:00:00.000Z'),
level: 'INFO',
message: 'first',
},
{
timestamp: new Date('2024-01-01T00:00:01.000Z'),
level: 'ERROR',
message: 'second',
},
]);
});
it('should handle a mix of structured and unstructured lines', () => {
const raw = [
'2024-01-01T00:00:00.000Z INFO structured line',
'plain unstructured line',
'2024-01-01T00:00:01.000Z ERROR another structured',
].join('\n');
const result = parseApplicationLogLines(raw);
expect(result).toHaveLength(3);
expect(result[0]).toEqual({
timestamp: new Date('2024-01-01T00:00:00.000Z'),
level: 'INFO',
message: 'structured line',
});
expect(result[1].level).toBe('INFO');
expect(result[1].message).toBe('plain unstructured line');
expect(result[2]).toEqual({
timestamp: new Date('2024-01-01T00:00:01.000Z'),
level: 'ERROR',
message: 'another structured',
});
});
it('should preserve message content including special characters', () => {
const raw = '2024-01-01T00:00:00.000Z INFO {"key": "value", "count": 42}';
expect(parseApplicationLogLines(raw)).toEqual([
{
timestamp: new Date('2024-01-01T00:00:00.000Z'),
level: 'INFO',
message: '{"key": "value", "count": 42}',
},
]);
});
it('strips ANSI color escapes from structured messages (chalk-style)', () => {
const raw = '2024-01-01T00:00:00.000Z INFO \u001B[33m4 \u001B[39m';
expect(parseApplicationLogLines(raw)).toEqual([
{
timestamp: new Date('2024-01-01T00:00:00.000Z'),
level: 'INFO',
message: '4 ',
},
]);
});
it('strips ANSI color escapes from unstructured lines too', () => {
const raw = '\u001B[1;31mfatal\u001B[0m something bad';
const result = parseApplicationLogLines(raw);
expect(result).toHaveLength(1);
expect(result[0].message).toBe('fatal something bad');
});
});
@@ -0,0 +1,32 @@
import { type ParsedLogLine } from 'src/engine/core-modules/event-logs/producers/application-log/parsed-log-line.type';
import { stripAnsiEscapes } from 'src/engine/core-modules/event-logs/producers/application-log/strip-ansi-escapes.util';
// Matches: 2024-01-01T00:00:00.000Z INFO some message
const LOG_LINE_REGEX =
/^(\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}\.\d{3}Z)\s+(INFO|ERROR|WARN|DEBUG)\s+(.*)$/;
export const parseApplicationLogLines = (rawLogs: string): ParsedLogLine[] => {
if (!rawLogs) {
return [];
}
const lines = rawLogs.split('\n').filter(Boolean);
return lines.map((line) => {
const match = line.match(LOG_LINE_REGEX);
if (match) {
return {
timestamp: new Date(match[1]),
level: match[2],
message: stripAnsiEscapes(match[3]),
};
}
return {
timestamp: new Date(),
level: 'INFO',
message: stripAnsiEscapes(line),
};
});
};
@@ -0,0 +1,5 @@
export type ParsedLogLine = {
timestamp: Date;
level: string;
message: string;
};
@@ -0,0 +1,48 @@
import { stripAnsiEscapes } from './strip-ansi-escapes.util';
describe('stripAnsiEscapes', () => {
it('returns plain ASCII strings unchanged', () => {
expect(stripAnsiEscapes('hello world')).toBe('hello world');
});
it('strips SGR color codes from a yellow value', () => {
// What `console.log(chalk.yellow('4 '))` emits.
expect(stripAnsiEscapes('\u001B[33m4 \u001B[39m')).toBe('4 ');
});
it('strips compound SGR codes (bold + red, then reset)', () => {
expect(stripAnsiEscapes('\u001B[1;31merror\u001B[0m')).toBe('error');
});
it('strips 256-color and truecolor SGR sequences', () => {
expect(stripAnsiEscapes('\u001B[38;5;208mwarn\u001B[39m')).toBe('warn');
expect(stripAnsiEscapes('\u001B[38;2;0;128;255mblue\u001B[0m')).toBe(
'blue',
);
});
it('strips cursor-movement CSI sequences', () => {
expect(stripAnsiEscapes('a\u001B[2Jb\u001B[Hc')).toBe('abc');
});
it('strips OSC sequences (e.g. terminal hyperlinks)', () => {
const link =
'\u001B]8;;https://twenty.com\u0007Twenty\u001B]8;;\u0007 rocks';
expect(stripAnsiEscapes(link)).toBe('Twenty rocks');
});
it('handles mixed colored output across multiple chunks', () => {
const raw =
'\u001B[32mOK\u001B[39m \u001B[2mready\u001B[22m: \u001B[1mdone\u001B[0m';
expect(stripAnsiEscapes(raw)).toBe('OK ready: done');
});
it('leaves untouched the bracket text that survived a missing ESC', () => {
// Defensive: if the ESC byte was already stripped upstream, we should not
// try to "fix" the bracketed remnants (we cannot tell them apart from real
// user text).
expect(stripAnsiEscapes('[33m4 [39m')).toBe('[33m4 [39m');
});
});
@@ -0,0 +1,6 @@
const ANSI_CSI_REGEX = /\u001B\[[0-?]*[ -/]*[@-~]/g;
const ANSI_OSC_REGEX = /\u001B\][^\u0007\u001B]*(?:\u0007|\u001B\\)/g;
export const stripAnsiEscapes = (value: string): string =>
value.replace(ANSI_CSI_REGEX, '').replace(ANSI_OSC_REGEX, '');
@@ -0,0 +1,104 @@
import { EventLogTable } from 'twenty-shared/types';
import {
EVENT_LOG_TYPES,
getClickHouseTableName,
} from 'src/engine/core-modules/event-logs/registry/event-log-registry';
import { normalizeEventLogRecords } from 'src/engine/core-modules/event-logs/utils/normalize-event-log-records';
describe('event-log registry', () => {
it('has a complete definition for every EventLogTable', () => {
for (const table of Object.values(EventLogTable)) {
const definition = EVENT_LOG_TYPES[table];
expect(definition).toBeDefined();
expect(definition.clickHouseTable).toBeTruthy();
expect(definition.eventFieldName).toBeTruthy();
}
});
it('exposes the ClickHouse table name', () => {
expect(getClickHouseTableName(EventLogTable.WORKSPACE_EVENT)).toBe(
'workspaceEvent',
);
expect(getClickHouseTableName(EventLogTable.APPLICATION_LOG)).toBe(
'applicationLog',
);
});
it('gates every table except application logs behind an entitlement', () => {
expect(
EVENT_LOG_TYPES[EventLogTable.APPLICATION_LOG].requiresEntitlement,
).toBeNull();
expect(
EVENT_LOG_TYPES[EventLogTable.WORKSPACE_EVENT].requiresEntitlement,
).not.toBeNull();
expect(
EVENT_LOG_TYPES[EventLogTable.USAGE_EVENT].requiresEntitlement,
).not.toBeNull();
});
describe('normalize', () => {
const timestamp = '2026-01-01 00:00:00.000';
it('maps a workspace event row', () => {
const record = EVENT_LOG_TYPES[EventLogTable.WORKSPACE_EVENT].normalize({
event: 'user.signup',
timestamp,
userId: 'user-1',
properties: { a: 1 },
});
expect(record.event).toBe('user.signup');
expect(record.userId).toBe('user-1');
expect(record.properties).toEqual({ a: 1 });
});
it('parses the ClickHouse timestamp as UTC, not server-local time', () => {
const [record] = normalizeEventLogRecords(
[{ event: 'user.signup', timestamp }],
EventLogTable.WORKSPACE_EVENT,
);
expect(record.timestamp.toISOString()).toBe('2026-01-01T00:00:00.000Z');
});
it('maps a pageview row from the name column', () => {
const record = EVENT_LOG_TYPES[EventLogTable.PAGEVIEW].normalize({
name: '/settings',
timestamp,
});
expect(record.event).toBe('/settings');
});
it('maps a usage event row (resourceType, userWorkspaceId, folded metadata)', () => {
const record = EVENT_LOG_TYPES[EventLogTable.USAGE_EVENT].normalize({
resourceType: 'WORKFLOW_NODE_RUN',
userWorkspaceId: 'uw-1',
timestamp,
quantity: 3,
metadata: { extra: true },
});
expect(record.event).toBe('WORKFLOW_NODE_RUN');
expect(record.userId).toBe('uw-1');
expect(record.properties).toMatchObject({ quantity: 3, extra: true });
});
it('maps an application log row (logicFunctionName, level, message)', () => {
const record = EVENT_LOG_TYPES[EventLogTable.APPLICATION_LOG].normalize({
logicFunctionName: 'myFn',
timestamp,
level: 'INFO',
message: 'hello',
});
expect(record.event).toBe('myFn');
expect(record.properties).toMatchObject({
level: 'INFO',
message: 'hello',
});
});
});
});
@@ -0,0 +1,109 @@
/* @license Enterprise */
import { EventLogTable } from 'twenty-shared/types';
import { BillingEntitlementKey } from 'src/engine/core-modules/billing/enums/billing-entitlement-key.enum';
import { type EventLogRecord } from 'src/engine/core-modules/event-logs/dtos/event-log-result.dto';
import {
type ApplicationLogRow,
type ObjectEventRow,
type PageviewRow,
type UsageEventRow,
} from 'src/engine/core-modules/event-logs/types/workspace-event-envelope.type';
type StoredRow<TRow> = Partial<Omit<TRow, 'type' | 'version'>> & {
timestamp: string;
};
type StoredEventRow = StoredRow<ObjectEventRow & Pick<PageviewRow, 'name'>>;
export type EventLogTypeDefinition = {
clickHouseTable: string;
// null = free on every plan; otherwise the required billing entitlement
requiresEntitlement: BillingEntitlementKey | null;
eventFieldName: string;
// The shared dispatcher stamps `timestamp`; each type maps only its own fields.
normalize: (
row: Record<string, unknown>,
) => Omit<EventLogRecord, 'timestamp'>;
};
const normalizeGenericEvent =
(eventFieldName: 'event' | 'name') =>
(row: Record<string, unknown>): Omit<EventLogRecord, 'timestamp'> => {
const record = row as StoredEventRow;
return {
event: record[eventFieldName] ?? '',
userId: record.userId ?? undefined,
properties: record.properties,
recordId: record.recordId,
objectMetadataId: record.objectMetadataId,
isCustom: record.isCustom,
};
};
export const EVENT_LOG_TYPES: Record<EventLogTable, EventLogTypeDefinition> = {
[EventLogTable.WORKSPACE_EVENT]: {
clickHouseTable: 'workspaceEvent',
requiresEntitlement: BillingEntitlementKey.AUDIT_LOGS,
eventFieldName: 'event',
normalize: normalizeGenericEvent('event'),
},
[EventLogTable.PAGEVIEW]: {
clickHouseTable: 'pageview',
requiresEntitlement: BillingEntitlementKey.AUDIT_LOGS,
eventFieldName: 'name',
normalize: normalizeGenericEvent('name'),
},
[EventLogTable.OBJECT_EVENT]: {
clickHouseTable: 'objectEvent',
requiresEntitlement: BillingEntitlementKey.AUDIT_LOGS,
eventFieldName: 'event',
normalize: normalizeGenericEvent('event'),
},
[EventLogTable.USAGE_EVENT]: {
clickHouseTable: 'usageEvent',
requiresEntitlement: BillingEntitlementKey.AUDIT_LOGS,
eventFieldName: 'resourceType',
normalize: (row) => {
const record = row as StoredRow<UsageEventRow>;
return {
event: record.resourceType ?? '',
userId: record.userWorkspaceId,
properties: {
...(record.metadata ?? {}),
operationType: record.operationType,
quantity: record.quantity,
unit: record.unit,
creditsUsedMicro: record.creditsUsedMicro,
resourceId: record.resourceId,
resourceContext: record.resourceContext,
},
};
},
},
[EventLogTable.APPLICATION_LOG]: {
clickHouseTable: 'applicationLog',
requiresEntitlement: null,
eventFieldName: 'logicFunctionName',
normalize: (row) => {
const record = row as StoredRow<ApplicationLogRow>;
return {
event: record.logicFunctionName ?? '',
properties: {
level: record.level,
message: record.message,
executionId: record.executionId,
logicFunctionId: record.logicFunctionId,
applicationId: record.applicationId,
},
};
},
},
};
export const getClickHouseTableName = (table: EventLogTable): string =>
EVENT_LOG_TYPES[table].clickHouseTable;
@@ -0,0 +1,62 @@
export type EventContextFields = {
workspaceId?: string | null;
userId?: string | null;
};
type AnalyticsEventRow = EventContextFields & {
type: 'track';
event: string;
properties: Record<string, unknown>;
timestamp: string;
version: string;
};
export type PageviewRow = EventContextFields & {
type: 'page';
name: string;
properties: Record<string, unknown>;
timestamp: string;
version: string;
};
export type ObjectEventRow = AnalyticsEventRow & {
recordId: string;
objectMetadataId: string;
isCustom?: boolean;
};
export type UsageEventRow = {
timestamp: string;
workspaceId: string;
periodStart?: string;
userWorkspaceId: string;
resourceType: string;
operationType: string;
quantity: number;
unit: string;
creditsUsedMicro: number;
resourceId: string;
resourceContext: string;
metadata: Record<string, unknown>;
};
export type ApplicationLogRow = {
timestamp: string;
workspaceId: string;
applicationId: string;
logicFunctionId: string;
logicFunctionName: string;
executionId: string;
level: string;
message: string;
};
// The `table` literals must match the registry's clickHouseTable values (EVENT_LOG_TYPES).
export type WorkspaceEventEnvelope =
| { table: 'workspaceEvent'; row: AnalyticsEventRow }
| { table: 'pageview'; row: PageviewRow }
| { table: 'objectEvent'; row: ObjectEventRow }
| { table: 'usageEvent'; row: UsageEventRow }
| { table: 'applicationLog'; row: ApplicationLogRow };
export type WorkspaceEventTable = WorkspaceEventEnvelope['table'];
@@ -0,0 +1,14 @@
import { EventLogTable } from 'twenty-shared/types';
import { parseClickHouseDateTime } from 'src/database/clickHouse/clickHouse.util';
import { type EventLogRecord } from 'src/engine/core-modules/event-logs/dtos/event-log-result.dto';
import { EVENT_LOG_TYPES } from 'src/engine/core-modules/event-logs/registry/event-log-registry';
export const normalizeEventLogRecords = (
records: Record<string, unknown>[],
table: EventLogTable,
): EventLogRecord[] =>
records.map((row) => ({
...EVENT_LOG_TYPES[table].normalize(row),
timestamp: parseClickHouseDateTime(row.timestamp as string),
}));