feat(app): infrastructure for pre-installed apps (#19973)

**PR 1 of 2.** Follow-up PR ships the Exa app, sets it as a default
pre-installed app, and removes the current `WebSearchTool` /
`WebSearchService` / `ExaDriver`. This PR adds the plumbing; no
user-visible change yet.

## Summary

- Server admins can declare a list of npm app packages to auto-install
on every new workspace and backfill onto existing workspaces via CLI.
- Server-level secrets (like Exa's API key) live on the
`ApplicationRegistration` (one row per server, encrypted) and are
injected into logic function execution env at runtime. No more
per-workspace storage of global secrets.
- A generic `POST /app/billing/charge` endpoint lets app logic functions
emit workspace usage events for metered features. Exa uses it in PR 2;
future apps (call recorder, etc.) reuse it.
- `LogicFunctionToolProvider` tool name prefix changes `logic_function_`
→ `app_`. Shorter, accurate (they come from installed apps).

## What's in this PR

**Logic function executor — server-level variables**
- `LogicFunctionExecutorService.getExecutionEnvVariables` now resolves
env vars in the order: hardcoded defaults →
`ApplicationRegistrationVariable[]` (server-level) →
`ApplicationVariable[]` (workspace-level override). The manifest
`serverVariables` schema has existed; this closes the loop.

**Config**
- `PRE_INSTALLED_APPS` — comma-separated list of npm packages. Default:
empty.

**\`PreInstalledAppsService\`** (new module)
- \`onApplicationBootstrap()\` — fetches each package's manifest from
the app registry CDN, upserts an \`ApplicationRegistration\`, and seeds
declared \`serverVariables\` from matching env vars (e.g.
\`EXA_API_KEY\` env → encrypted registration variable).
- \`installOnWorkspace(workspaceId)\` — installs all pre-installed apps
on a single workspace. Tolerates per-app failures.

**Auto-install on new workspace activation**
- \`WorkspaceService.prefillCreatedWorkspaceRecords\` invokes
\`installOnWorkspace\` after prefilling standard records. Non-blocking
on failure.

**Backfill CLI command**
- \`install-pre-installed-apps\` — iterates active and suspended
workspaces, installs pre-installed apps that aren't yet installed.
Idempotent. Run after changing \`PRE_INSTALLED_APPS\`.

**App billing endpoint**
- \`POST /app/billing/charge\`. Authenticated via \`APPLICATION_ACCESS\`
token (already injected into logic function execution env as
\`DEFAULT_APP_ACCESS_TOKEN\`). Body: \`{ creditsUsedMicro, quantity,
unit, operationType, resourceContext? }\`. Emits \`USAGE_RECORDED\` with
\`applicationId\` as \`resourceId\`. Generic — reusable by any app.

**Tool name prefix**
- \`LogicFunctionToolProvider.buildLogicFunctionToolName\` now produces
\`app_<name>\` instead of \`logic_function_<name>\`. Only affects tools
sourced from logic functions; other tool providers unchanged.

## Stats

- 16 files, +501 / −2
- 7 new files (1 command, 1 service × 2, 1 controller, 1 DTO, 2 modules)
- Typecheck: 7 pre-existing errors, zero new
- Prettier clean

## Behavior deltas

- **\`PRE_INSTALLED_APPS\` default = empty**: existing servers see no
change on merge.
- **\`ApplicationRegistrationVariable\` is now read by the executor**:
apps that were using manifest \`serverVariables\` but expecting them to
be ignored by the executor will now see them injected. No apps ship with
\`isTool: true\` logic functions today, so this is latent — first
consumer is Exa in PR 2.
- **Tool prefix**: currently no logic-function tools are named
\`logic_function_*\` in any production flow. The prefix change affects
only future tools emitted by \`LogicFunctionToolProvider\`.

## Risks

- **CDN unavailability at startup**: if the app registry CDN is down,
\`ensureRegistrationsExist\` logs warnings but doesn't block server
start. Installation on new workspaces during this window will find no
registrations and log a non-blocking error. Backfill command can retry
after CDN recovers.
- **Cold-start overhead**: \`ensureRegistrationsExist\` is called once
per process on bootstrap. Current configurable default is empty, so zero
overhead. When an admin sets \`PRE_INSTALLED_APPS\`, they accept one
HTTP call per package at boot.
- **Server-level variables flow**:
\`ApplicationRegistrationVariable.encryptedValue\` is shared by all
workspaces of a server. Appropriate for a single-tenant Exa key. Not
appropriate for per-tenant keys — those go in workspace-level
\`ApplicationVariable\` and override.

## Test plan

- [ ] \`npx nx typecheck twenty-server\` passes (verified: 7
pre-existing unrelated errors, zero new)
- [ ] Set \`PRE_INSTALLED_APPS=@twenty-apps/hello-world\` (or any real
npm-published app), \`HELLO_WORLD_API_KEY=xxx\`, restart server:
\`ApplicationRegistration\` row is upserted,
\`ApplicationRegistrationVariable\` for HELLO_WORLD_API_KEY is populated
(encrypted).
- [ ] Create a new workspace: the app is auto-installed,
\`ApplicationEntity\` row created, \`LogicFunctionEntity\` rows created.
- [ ] Existing workspace: run \`yarn nx run twenty-server:command
install-pre-installed-apps\`: apps install across all workspaces,
idempotent on re-run.
- [ ] Trigger a logic function that reads
\`process.env.HELLO_WORLD_API_KEY\`: value resolves from the
server-level \`ApplicationRegistrationVariable\`.
- [ ] Log a charge from the handler: \`POST /app/billing/charge\` with
\`Authorization: Bearer \$DEFAULT_APP_ACCESS_TOKEN\` body
\`{creditsUsedMicro: 1000, quantity: 1, unit: "INVOCATION",
operationType: "WEB_SEARCH"}\` → returns \`{success: true}\`,
\`USAGE_RECORDED\` event emitted with correct
\`resourceId=applicationId\`.
- [ ] Tool name generated by \`LogicFunctionToolProvider\` starts with
\`app_\`.

## What's NOT in this PR (PR 2 scope)

- The Exa app itself (\`packages/twenty-apps/...\` directory)
- Removing \`WebSearchTool\`, \`WebSearchService\`, \`ExaDriver\`,
\`web-search\` module
- Removing \`WEB_SEARCH_DRIVER\` config var
- Removing the current \`exa_web_search\` entry in
\`ActionToolProvider\`
- Chat preload list updated to \`app_exa_web_search\`
- Frontend \`getToolDisplayMessage\` branch for \`app_exa_web_search\`
- Setting \`PRE_INSTALLED_APPS\` default to include \`@twenty-apps/exa\`

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
Félix Malfait
2026-04-23 21:59:00 +02:00
committed by GitHub
parent ec98130def
commit 4f938aa097
71 changed files with 5546 additions and 648 deletions
@@ -34,6 +34,7 @@ import { MarketplaceCatalogSyncCommand } from 'src/engine/core-modules/applicati
MarketplaceCatalogSyncService,
MarketplaceQueryService,
MarketplaceCatalogSyncCronCommand,
MarketplaceService,
],
})
export class MarketplaceModule {}
@@ -124,6 +124,12 @@ export class ApplicationRegistrationEntity {
@Column({ name: 'isFeatured', type: 'boolean', default: false })
isFeatured: boolean;
// Auto-installed on every new workspace; existing workspaces are
// backfilled by the `install-pre-installed-apps` CLI command.
@Field(() => Boolean)
@Column({ type: 'boolean', default: false })
isPreInstalled: boolean;
@Column({ type: 'jsonb', nullable: true })
manifest: Manifest | null;
@@ -0,0 +1,16 @@
import { Module } from '@nestjs/common';
import { TypeOrmModule } from '@nestjs/typeorm';
import { ApplicationInstallModule } from 'src/engine/core-modules/application/application-install/application-install.module';
import { ApplicationRegistrationEntity } from 'src/engine/core-modules/application/application-registration/application-registration.entity';
import { PreInstalledAppsService } from 'src/engine/core-modules/application/pre-installed-apps/pre-installed-apps.service';
@Module({
imports: [
TypeOrmModule.forFeature([ApplicationRegistrationEntity]),
ApplicationInstallModule,
],
providers: [PreInstalledAppsService],
exports: [PreInstalledAppsService],
})
export class PreInstalledAppsModule {}
@@ -0,0 +1,48 @@
import { Injectable, Logger } from '@nestjs/common';
import { InjectRepository } from '@nestjs/typeorm';
import { Repository } from 'typeorm';
import { ApplicationInstallService } from 'src/engine/core-modules/application/application-install/application-install.service';
import { ApplicationRegistrationEntity } from 'src/engine/core-modules/application/application-registration/application-registration.entity';
@Injectable()
export class PreInstalledAppsService {
private readonly logger = new Logger(PreInstalledAppsService.name);
constructor(
private readonly applicationInstallService: ApplicationInstallService,
@InjectRepository(ApplicationRegistrationEntity)
private readonly applicationRegistrationRepository: Repository<ApplicationRegistrationEntity>,
) {}
// Per-app failures are logged but never block the other installs —
// `ApplicationInstallService` holds a per-app cache lock so parallel
// installs are safe.
async installOnWorkspace(workspaceId: string): Promise<void> {
const registrations = await this.applicationRegistrationRepository.find({
where: { isPreInstalled: true },
});
if (registrations.length === 0) {
return;
}
await Promise.allSettled(
registrations.map(async (registration) => {
try {
await this.applicationInstallService.installApplication({
appRegistrationId: registration.id,
workspaceId,
});
} catch (error) {
this.logger.error(
`Failed to install pre-installed app "${registration.name}" (${registration.id}) on workspace ${workspaceId}: ${
error instanceof Error ? error.message : String(error)
}`,
);
}
}),
);
}
}
@@ -0,0 +1,77 @@
/* @license Enterprise */
import {
Body,
Controller,
ForbiddenException,
HttpCode,
HttpStatus,
NotFoundException,
Post,
Req,
UseGuards,
UsePipes,
ValidationPipe,
} from '@nestjs/common';
import { Request } from 'express';
import { isDefined } from 'twenty-shared/utils';
import { AppBillingService } from 'src/engine/core-modules/billing/app-billing/app-billing.service';
import { ChargeDto } from 'src/engine/core-modules/billing/app-billing/dtos/charge.dto';
import { ThrottlerService } from 'src/engine/core-modules/throttler/throttler.service';
import { TwentyConfigService } from 'src/engine/core-modules/twenty-config/twenty-config.service';
import { JwtAuthGuard } from 'src/engine/guards/jwt-auth.guard';
import { NoPermissionGuard } from 'src/engine/guards/no-permission.guard';
import { WorkspaceAuthGuard } from 'src/engine/guards/workspace-auth.guard';
// Belt-and-suspenders on top of LogicFunctionExecutorService's execution
// throttle: application-access tokens are JWTs usable outside the runtime.
const APP_BILLING_CHARGE_THROTTLE_LIMIT = 1000;
const APP_BILLING_CHARGE_THROTTLE_TTL_MS = 60_000;
@Controller('app/billing')
@UseGuards(JwtAuthGuard, WorkspaceAuthGuard, NoPermissionGuard)
export class AppBillingController {
constructor(
private readonly appBillingService: AppBillingService,
private readonly throttlerService: ThrottlerService,
private readonly twentyConfigService: TwentyConfigService,
) {}
@Post('charge')
@HttpCode(HttpStatus.NO_CONTENT)
@UsePipes(new ValidationPipe({ whitelist: true, forbidNonWhitelisted: true }))
async charge(
@Req() request: Request,
@Body() charge: ChargeDto,
): Promise<void> {
// Billing disabled: no listener consumes the event — fail fast so apps
// don't silently discard charges on Community instances.
if (!this.twentyConfigService.get('IS_BILLING_ENABLED')) {
throw new NotFoundException();
}
// Reject user-access / api-key tokens — only application-access tokens
// populate `request.application`.
if (!isDefined(request.application) || !isDefined(request.workspace)) {
throw new ForbiddenException(
'App billing endpoint requires an APPLICATION_ACCESS token.',
);
}
await this.throttlerService.tokenBucketThrottleOrThrow(
`${request.workspace.id}-${request.application.id}-app-billing-charge`,
1,
APP_BILLING_CHARGE_THROTTLE_LIMIT,
APP_BILLING_CHARGE_THROTTLE_TTL_MS,
);
this.appBillingService.emitChargeEvent({
workspaceId: request.workspace.id,
applicationId: request.application.id,
userWorkspaceId: request.userWorkspaceId,
charge,
});
}
}
@@ -0,0 +1,25 @@
/* @license Enterprise */
import { Module } from '@nestjs/common';
import { AppBillingController } from 'src/engine/core-modules/billing/app-billing/app-billing.controller';
import { AppBillingService } from 'src/engine/core-modules/billing/app-billing/app-billing.service';
import { AuthModule } from 'src/engine/core-modules/auth/auth.module';
import { ThrottlerModule } from 'src/engine/core-modules/throttler/throttler.module';
import { TwentyConfigModule } from 'src/engine/core-modules/twenty-config/twenty-config.module';
import { WorkspaceCacheStorageModule } from 'src/engine/workspace-cache-storage/workspace-cache-storage.module';
import { WorkspaceEventEmitterModule } from 'src/engine/workspace-event-emitter/workspace-event-emitter.module';
@Module({
imports: [
AuthModule,
ThrottlerModule,
TwentyConfigModule,
WorkspaceCacheStorageModule,
WorkspaceEventEmitterModule,
],
controllers: [AppBillingController],
providers: [AppBillingService],
exports: [AppBillingService],
})
export class AppBillingModule {}
@@ -0,0 +1,63 @@
/* @license Enterprise */
import { Injectable, Logger } from '@nestjs/common';
import { type ChargeDto } from 'src/engine/core-modules/billing/app-billing/dtos/charge.dto';
import { USAGE_RECORDED } from 'src/engine/core-modules/usage/constants/usage-recorded.constant';
import { UsageOperationType } from 'src/engine/core-modules/usage/enums/usage-operation-type.enum';
import { UsageResourceType } from 'src/engine/core-modules/usage/enums/usage-resource-type.enum';
import { UsageUnit } from 'src/engine/core-modules/usage/enums/usage-unit.enum';
import { type UsageEvent } from 'src/engine/core-modules/usage/types/usage-event.type';
import { WorkspaceEventEmitter } from 'src/engine/workspace-event-emitter/workspace-event-emitter';
// Each operation type has one canonical counting unit — matches how
// `ai-billing.service.ts` emits native usage events.
const USAGE_UNIT_BY_OPERATION_TYPE: Record<UsageOperationType, UsageUnit> = {
[UsageOperationType.AI_CHAT_TOKEN]: UsageUnit.TOKEN,
[UsageOperationType.AI_WORKFLOW_TOKEN]: UsageUnit.TOKEN,
[UsageOperationType.WORKFLOW_EXECUTION]: UsageUnit.INVOCATION,
[UsageOperationType.CODE_EXECUTION]: UsageUnit.INVOCATION,
[UsageOperationType.WEB_SEARCH]: UsageUnit.INVOCATION,
};
// `workspaceId` + `applicationId` come from the application-access token,
// never from the body — an app can't charge a different workspace or
// masquerade as a different app.
@Injectable()
export class AppBillingService {
private readonly logger = new Logger(AppBillingService.name);
constructor(private readonly workspaceEventEmitter: WorkspaceEventEmitter) {}
emitChargeEvent(params: {
workspaceId: string;
applicationId: string;
userWorkspaceId?: string | null;
charge: ChargeDto;
}): void {
const { workspaceId, applicationId, userWorkspaceId, charge } = params;
const unit = USAGE_UNIT_BY_OPERATION_TYPE[charge.operationType];
this.logger.log(
`App charge from applicationId=${applicationId} workspaceId=${workspaceId}: ` +
`${charge.creditsUsedMicro} micro-credits (${charge.quantity} ${unit}, ${charge.operationType})`,
);
this.workspaceEventEmitter.emitCustomBatchEvent<UsageEvent>(
USAGE_RECORDED,
[
{
resourceType: UsageResourceType.APP,
operationType: charge.operationType,
creditsUsedMicro: charge.creditsUsedMicro,
quantity: charge.quantity,
unit,
resourceId: applicationId,
resourceContext: charge.resourceContext ?? null,
userWorkspaceId: userWorkspaceId ?? null,
},
],
workspaceId,
);
}
}
@@ -0,0 +1,29 @@
/* @license Enterprise */
import { IsEnum, IsInt, IsOptional, IsString, Max, Min } from 'class-validator';
import { UsageOperationType } from 'src/engine/core-modules/usage/enums/usage-operation-type.enum';
// $1000 in micro-credits (1 USD = 1_000_000 micro-credits). Bounds a single
// charge so a compromised or buggy app can't drain credits in one request.
const MAX_CREDITS_USED_MICRO_PER_CHARGE = 1_000_000_000;
const MAX_QUANTITY_PER_CHARGE = 10_000;
export class ChargeDto {
@IsInt()
@Min(0)
@Max(MAX_CREDITS_USED_MICRO_PER_CHARGE)
creditsUsedMicro!: number;
@IsInt()
@Min(1)
@Max(MAX_QUANTITY_PER_CHARGE)
quantity!: number;
@IsEnum(UsageOperationType)
operationType!: UsageOperationType;
@IsOptional()
@IsString()
resourceContext?: string;
}
@@ -16,9 +16,11 @@ import { ApplicationOAuthModule } from 'src/engine/core-modules/application/appl
import { ApplicationRegistrationModule } from 'src/engine/core-modules/application/application-registration/application-registration.module';
import { ApplicationUpgradeModule } from 'src/engine/core-modules/application/application-upgrade/application-upgrade.module';
import { ApplicationModule } from 'src/engine/core-modules/application/application.module';
import { PreInstalledAppsModule } from 'src/engine/core-modules/application/pre-installed-apps/pre-installed-apps.module';
import { ApprovedAccessDomainModule } from 'src/engine/core-modules/approved-access-domain/approved-access-domain.module';
import { AuthModule } from 'src/engine/core-modules/auth/auth.module';
import { BillingWebhookModule } from 'src/engine/core-modules/billing-webhook/billing-webhook.module';
import { AppBillingModule } from 'src/engine/core-modules/billing/app-billing/app-billing.module';
import { BillingModule } from 'src/engine/core-modules/billing/billing.module';
import { BillingGraphqlApiExceptionFilter } from 'src/engine/core-modules/billing/filters/billing-graphql-api-exception.filter';
import { CacheStorageModule } from 'src/engine/core-modules/cache-storage/cache-storage.module';
@@ -26,7 +28,6 @@ import { TimelineCalendarEventModule } from 'src/engine/core-modules/calendar/ti
import { CaptchaModule } from 'src/engine/core-modules/captcha/captcha.module';
import { CloudflareModule } from 'src/engine/core-modules/cloudflare/cloudflare.module';
import { CodeInterpreterModule } from 'src/engine/core-modules/code-interpreter/code-interpreter.module';
import { WebSearchModule } from 'src/engine/core-modules/web-search/web-search.module';
import { DnsManagerModule } from 'src/engine/core-modules/dns-manager/dns-manager.module';
import { EmailModule } from 'src/engine/core-modules/email/email.module';
import { EmailingDomainModule } from 'src/engine/core-modules/emailing-domain/emailing-domain.module';
@@ -156,7 +157,6 @@ import { FileModule } from './file/file.module';
AiBillingModule,
LogicFunctionModule.forRoot(),
CodeInterpreterModule.forRoot(),
WebSearchModule.forRoot(),
SearchModule,
ApiKeyModule,
PageLayoutModule,
@@ -164,6 +164,8 @@ import { FileModule } from './file/file.module';
TrashCleanupModule,
DashboardModule,
EventLogsModule,
PreInstalledAppsModule,
AppBillingModule,
],
providers: [
{
@@ -1,5 +1,7 @@
import { Module } from '@nestjs/common';
import { TypeOrmModule } from '@nestjs/typeorm';
import { ApplicationRegistrationVariableEntity } from 'src/engine/core-modules/application/application-registration-variable/application-registration-variable.entity';
import { LogicFunctionExecutorService } from 'src/engine/core-modules/logic-function/logic-function-executor/logic-function-executor.service';
import { ThrottlerModule } from 'src/engine/core-modules/throttler/throttler.module';
import { AuditModule } from 'src/engine/core-modules/audit/audit.module';
@@ -16,6 +18,7 @@ import { WorkspaceCacheModule } from 'src/engine/workspace-cache/workspace-cache
SecretEncryptionModule,
SubscriptionsModule,
WorkspaceCacheModule,
TypeOrmModule.forFeature([ApplicationRegistrationVariableEntity]),
],
providers: [LogicFunctionExecutorService],
exports: [LogicFunctionExecutorService],
@@ -1,4 +1,5 @@
import { Injectable, Logger } from '@nestjs/common';
import { InjectRepository } from '@nestjs/typeorm';
import {
DEFAULT_API_KEY_NAME,
@@ -6,6 +7,7 @@ import {
DEFAULT_APP_ACCESS_TOKEN_NAME,
} from 'twenty-shared/application';
import { isDefined } from 'twenty-shared/utils';
import { Not, Repository } from 'typeorm';
import { v4 } from 'uuid';
import {
@@ -16,6 +18,7 @@ import {
import { ApplicationLogsService } from 'src/engine/core-modules/application-logs/application-logs.service';
import { parseApplicationLogLines } from 'src/engine/core-modules/application-logs/utils/parse-application-log-lines';
import { ApplicationRegistrationVariableEntity } from 'src/engine/core-modules/application/application-registration-variable/application-registration-variable.entity';
import type { FlatApplicationVariable } from 'src/engine/core-modules/application/application-variable/types/flat-application-variable.type';
import { FlatApplication } from 'src/engine/core-modules/application/types/flat-application.type';
import { AuditService } from 'src/engine/core-modules/audit/services/audit.service';
@@ -69,6 +72,8 @@ export class LogicFunctionExecutorService {
private readonly auditService: AuditService,
private readonly applicationLogsService: ApplicationLogsService,
private readonly workspaceEventEmitter: WorkspaceEventEmitter,
@InjectRepository(ApplicationRegistrationVariableEntity)
private readonly applicationRegistrationVariableRepository: Repository<ApplicationRegistrationVariableEntity>,
) {}
async execute({
@@ -219,15 +224,66 @@ export class LogicFunctionExecutorService {
const baseUrl = cleanServerUrl(this.twentyConfigService.get('SERVER_URL'));
const serverVariables = await this.buildServerVariableEnvMap(
flatApplication.applicationRegistrationId,
);
const workspaceVariables = buildEnvVar(
flatApplicationVariables,
this.secretEncryptionService,
);
return {
[DEFAULT_API_URL_NAME]: baseUrl ?? '',
[DEFAULT_APP_ACCESS_TOKEN_NAME]: applicationAccessToken.token,
[DEFAULT_API_KEY_NAME]: applicationAccessToken.token,
APPLICATION_ID: flatApplication.id,
...buildEnvVar(flatApplicationVariables, this.secretEncryptionService),
// Server variables first, workspace variables override. Workspace-level
// values let a specific tenant customize a server default.
...serverVariables,
...workspaceVariables,
};
}
// Resolves encrypted server-level variables (ApplicationRegistrationVariable)
// for the application's registration. Returns an empty object when the
// application isn't linked to a registration (legacy LOCAL apps).
//
// Runs on every logic function execution — the query is indexed on
// applicationRegistrationId and filters unfilled rows server-side. Most
// apps have 0-3 server variables so the round-trip is cheap, but if this
// becomes a hot path, move to a WorkspaceCacheProvider mirroring
// WorkspaceApplicationVariableMapCacheService.
private async buildServerVariableEnvMap(
applicationRegistrationId: string | null,
): Promise<Record<string, string>> {
if (!isDefined(applicationRegistrationId)) {
return {};
}
const serverVariables =
await this.applicationRegistrationVariableRepository.find({
where: {
applicationRegistrationId,
encryptedValue: Not(''),
},
});
const envMap: Record<string, string> = {};
// ApplicationRegistrationVariable.encryptedValue is always written
// encrypted (ApplicationRegistrationVariableService.createVariable and
// .updateVariable call encrypt unconditionally), independent of
// `isSecret`. `isSecret` is display metadata — the storage contract is
// not conditional, so decryption isn't either.
for (const variable of serverVariables) {
envMap[variable.key] = this.secretEncryptionService.decrypt(
variable.encryptedValue,
);
}
return envMap;
}
private async handleExecutionResult({
result,
flatApplication,
@@ -17,10 +17,8 @@ import { SendEmailTool } from 'src/engine/core-modules/tool/tools/email-tool/sen
import { HttpTool } from 'src/engine/core-modules/tool/tools/http-tool/http-tool';
import { NavigateAppTool } from 'src/engine/core-modules/tool/tools/navigate-tool/navigate-app-tool';
import { SearchHelpCenterTool } from 'src/engine/core-modules/tool/tools/search-help-center-tool/search-help-center-tool';
import { WebSearchTool } from 'src/engine/core-modules/tool/tools/web-search-tool/web-search-tool';
import { type ToolOutput } from 'src/engine/core-modules/tool/types/tool-output.type';
import { type Tool } from 'src/engine/core-modules/tool/types/tool.type';
import { WebSearchService } from 'src/engine/core-modules/web-search/web-search.service';
import { PermissionsService } from 'src/engine/metadata-modules/permissions/permissions.service';
@Injectable()
@@ -36,9 +34,7 @@ export class ActionToolProvider implements ToolProvider {
private readonly searchHelpCenterTool: SearchHelpCenterTool,
private readonly codeInterpreterTool: CodeInterpreterTool,
private readonly navigateAppTool: NavigateAppTool,
private readonly webSearchTool: WebSearchTool,
private readonly codeInterpreterService: CodeInterpreterService,
private readonly webSearchService: WebSearchService,
private readonly permissionsService: PermissionsService,
) {
this.toolMap = new Map<string, Tool>([
@@ -48,7 +44,6 @@ export class ActionToolProvider implements ToolProvider {
['search_help_center', this.searchHelpCenterTool],
['code_interpreter', this.codeInterpreterTool],
['navigate_app', this.navigateAppTool],
['exa_web_search', this.webSearchTool],
]);
}
@@ -128,16 +123,6 @@ export class ActionToolProvider implements ToolProvider {
);
}
if (this.webSearchService.isEnabled()) {
descriptors.push(
this.buildDescriptor(
'exa_web_search',
this.webSearchTool,
includeSchemas,
),
);
}
return descriptors;
}
@@ -94,7 +94,7 @@ export class LogicFunctionToolProvider implements ToolProvider {
}
private buildLogicFunctionToolName(functionName: string): string {
return `logic_function_${functionName
return `app_${functionName
.toLowerCase()
.replace(/[^a-z0-9]+/g, '_')
.replace(/^_+|_+$/g, '')}`;
@@ -14,7 +14,6 @@ import { SendEmailTool } from 'src/engine/core-modules/tool/tools/email-tool/sen
import { HttpTool } from 'src/engine/core-modules/tool/tools/http-tool/http-tool';
import { NavigateAppTool } from 'src/engine/core-modules/tool/tools/navigate-tool/navigate-app-tool';
import { SearchHelpCenterTool } from 'src/engine/core-modules/tool/tools/search-help-center-tool/search-help-center-tool';
import { WebSearchTool } from 'src/engine/core-modules/tool/tools/web-search-tool/web-search-tool';
import { WorkspaceManyOrAllFlatEntityMapsCacheModule } from 'src/engine/metadata-modules/flat-entity/services/workspace-many-or-all-flat-entity-maps-cache.module';
import { ConnectedAccountEntity } from 'src/engine/metadata-modules/connected-account/entities/connected-account.entity';
import { NavigationMenuItemModule } from 'src/engine/metadata-modules/navigation-menu-item/navigation-menu-item.module';
@@ -46,7 +45,6 @@ import { MessagingSendManagerModule } from 'src/modules/messaging/message-outbou
SearchHelpCenterTool,
CodeInterpreterTool,
NavigateAppTool,
WebSearchTool,
],
exports: [
HttpTool,
@@ -56,7 +54,6 @@ import { MessagingSendManagerModule } from 'src/modules/messaging/message-outbou
SearchHelpCenterTool,
CodeInterpreterTool,
NavigateAppTool,
WebSearchTool,
],
})
export class ToolModule {}
@@ -1,5 +0,0 @@
import { type z } from 'zod';
import { type WebSearchInputZodSchema } from 'src/engine/core-modules/tool/tools/web-search-tool/web-search-tool.schema';
export type WebSearchInput = z.infer<typeof WebSearchInputZodSchema>;
@@ -1,29 +0,0 @@
import { z } from 'zod';
import { WEB_SEARCH_CATEGORIES } from 'src/engine/core-modules/web-search/constants/web-search-categories.const';
export const WEB_SEARCH_DEFAULT_NUM_RESULTS = 10;
export const WEB_SEARCH_MAX_NUM_RESULTS = 30;
export const WebSearchInputZodSchema = z.object({
query: z
.string()
.describe(
'The search query to look up on the web. Be specific and include relevant keywords for better results.',
),
category: z
.enum(WEB_SEARCH_CATEGORIES)
.optional()
.describe(
'Optional content category to focus the search. Use "company" for business/organization info, "people" for person profiles, "news" for recent articles, "research paper" for academic content.',
),
numResults: z
.number()
.int()
.min(1)
.max(WEB_SEARCH_MAX_NUM_RESULTS)
.optional()
.describe(
`Number of search results to return. Defaults to ${WEB_SEARCH_DEFAULT_NUM_RESULTS}, max ${WEB_SEARCH_MAX_NUM_RESULTS}. Use more results when you need comprehensive coverage.`,
),
});
@@ -1,48 +0,0 @@
import { Injectable } from '@nestjs/common';
import { type WebSearchInput } from 'src/engine/core-modules/tool/tools/web-search-tool/web-search-input.type';
import { WebSearchInputZodSchema } from 'src/engine/core-modules/tool/tools/web-search-tool/web-search-tool.schema';
import { type ToolExecutionContext } from 'src/engine/core-modules/tool/types/tool-execution-context.type';
import { type ToolInput } from 'src/engine/core-modules/tool/types/tool-input.type';
import { type ToolOutput } from 'src/engine/core-modules/tool/types/tool-output.type';
import { type Tool } from 'src/engine/core-modules/tool/types/tool.type';
import { WebSearchService } from 'src/engine/core-modules/web-search/web-search.service';
@Injectable()
export class WebSearchTool implements Tool {
description =
'Structured web search powered by Exa. Returns entity-aware results with category filtering (companies, people, research papers, news, and other content types). Prefer this when the query benefits from structured data or a specific category. For general real-time web browsing, prefer the native `web_search` tool when it is available.';
inputSchema = WebSearchInputZodSchema;
constructor(private readonly webSearchService: WebSearchService) {}
async execute(
parameters: ToolInput,
context: ToolExecutionContext,
): Promise<ToolOutput> {
const { query, category, numResults } = parameters as WebSearchInput;
try {
const results = await this.webSearchService.search(
query,
{ category, numResults },
{
workspaceId: context.workspaceId,
userWorkspaceId: context.userWorkspaceId,
},
);
return {
success: true,
message: `Found ${results.length} results for "${query}"${category ? ` (category: ${category})` : ''}`,
result: results,
};
} catch (error) {
return {
success: false,
message: `Web search failed for "${query}"`,
error: error instanceof Error ? error.message : 'Web search failed',
};
}
}
}
@@ -42,7 +42,6 @@ import {
ConfigVariableException,
ConfigVariableExceptionCode,
} from 'src/engine/core-modules/twenty-config/twenty-config.exception';
import { WebSearchDriverType } from 'src/engine/core-modules/web-search/web-search.interface';
import { type AiModelPreferences } from 'src/engine/metadata-modules/ai/ai-models/types/ai-model-preferences.type';
import { type AiProvidersConfig } from 'src/engine/metadata-modules/ai/ai-models/types/ai-providers-config.type';
import { loadDefaultModelPreferences } from 'src/engine/metadata-modules/ai/ai-models/utils/load-default-model-preferences.util';
@@ -675,26 +674,6 @@ export class ConfigVariables {
@CastToPositiveNumber()
CODE_INTERPRETER_TIMEOUT_MS = 300_000;
@ConfigVariablesMetadata({
group: ConfigVariablesGroup.LLM,
description:
'Web search driver type - EXA for Exa search, DISABLED to turn off',
type: ConfigVariableType.STRING,
options: Object.values(WebSearchDriverType),
})
@IsOptional()
@CastToUpperSnakeCase()
WEB_SEARCH_DRIVER: WebSearchDriverType = WebSearchDriverType.DISABLED;
@ConfigVariablesMetadata({
group: ConfigVariablesGroup.LLM,
description: 'Exa API key for web search',
type: ConfigVariableType.STRING,
isSensitive: true,
})
@ValidateIf((env) => env.WEB_SEARCH_DRIVER === WebSearchDriverType.EXA)
EXA_API_KEY?: string;
@ConfigVariablesMetadata({
group: ConfigVariablesGroup.ANALYTICS_CONFIG,
description: 'Enable or disable analytics for telemetry',
@@ -1,9 +0,0 @@
export const WEB_SEARCH_CATEGORIES = [
'company',
'research paper',
'news',
'pdf',
'personal site',
'financial report',
'people',
] as const;
@@ -1,19 +0,0 @@
import {
type WebSearchCostModel,
type WebSearchDriver,
} from 'src/engine/core-modules/web-search/drivers/interfaces/web-search-driver.interface';
import { type WebSearchResult } from 'src/engine/core-modules/web-search/types/web-search-result.type';
export class DisabledWebSearchDriver implements WebSearchDriver {
readonly costModel: WebSearchCostModel = {
baseResultCount: 0,
baseCostDollars: 0,
costPerAdditionalResultDollars: 0,
};
constructor(private readonly reason: string) {}
async search(): Promise<WebSearchResult[]> {
throw new Error(this.reason);
}
}
@@ -1,52 +0,0 @@
import Exa from 'exa-js';
import {
type WebSearchCostModel,
type WebSearchDriver,
} from 'src/engine/core-modules/web-search/drivers/interfaces/web-search-driver.interface';
import { type WebSearchOptions } from 'src/engine/core-modules/web-search/types/web-search-options.type';
import { type WebSearchResult } from 'src/engine/core-modules/web-search/types/web-search-result.type';
const DEFAULT_NUM_RESULTS = 10;
const MAX_HIGHLIGHT_CHARACTERS = 4000;
// Exa charges $7/1k requests for auto search type (up to 10 results)
// Additional results above 10 cost $1/1k = $0.001 each
const EXA_BASE_COST_DOLLARS = 0.007;
const EXA_COST_PER_ADDITIONAL_RESULT_DOLLARS = 0.001;
export class ExaDriver implements WebSearchDriver {
readonly costModel: WebSearchCostModel = {
baseResultCount: DEFAULT_NUM_RESULTS,
baseCostDollars: EXA_BASE_COST_DOLLARS,
costPerAdditionalResultDollars: EXA_COST_PER_ADDITIONAL_RESULT_DOLLARS,
};
private readonly client: Exa;
constructor(apiKey: string) {
this.client = new Exa(apiKey);
}
async search(
query: string,
options?: WebSearchOptions,
): Promise<WebSearchResult[]> {
const numResults = options?.numResults ?? DEFAULT_NUM_RESULTS;
const response = await this.client.search(query, {
type: 'auto',
numResults,
category: options?.category,
contents: {
highlights: { maxCharacters: MAX_HIGHLIGHT_CHARACTERS },
},
});
return response.results.map((result) => ({
title: result.title ?? '',
url: result.url,
snippet: result.highlights?.join('\n') ?? '',
}));
}
}
@@ -1,14 +0,0 @@
import { type WebSearchOptions } from 'src/engine/core-modules/web-search/types/web-search-options.type';
import { type WebSearchResult } from 'src/engine/core-modules/web-search/types/web-search-result.type';
export type WebSearchCostModel = {
baseResultCount: number;
baseCostDollars: number;
costPerAdditionalResultDollars: number;
};
export interface WebSearchDriver {
readonly costModel: WebSearchCostModel;
search(query: string, options?: WebSearchOptions): Promise<WebSearchResult[]>;
}
@@ -1,4 +0,0 @@
export type WebSearchBillingContext = {
workspaceId: string;
userWorkspaceId?: string;
};
@@ -1,3 +0,0 @@
import { type WEB_SEARCH_CATEGORIES } from 'src/engine/core-modules/web-search/constants/web-search-categories.const';
export type WebSearchCategory = (typeof WEB_SEARCH_CATEGORIES)[number];
@@ -1,6 +0,0 @@
import { type WebSearchCategory } from 'src/engine/core-modules/web-search/types/web-search-category.type';
export type WebSearchOptions = {
category?: WebSearchCategory;
numResults?: number;
};
@@ -1,5 +0,0 @@
export type WebSearchResult = {
title: string;
url: string;
snippet: string;
};
@@ -1,59 +0,0 @@
import { Injectable } from '@nestjs/common';
import { type WebSearchDriver } from 'src/engine/core-modules/web-search/drivers/interfaces/web-search-driver.interface';
import { DisabledWebSearchDriver } from 'src/engine/core-modules/web-search/drivers/disabled.driver';
import { ExaDriver } from 'src/engine/core-modules/web-search/drivers/exa.driver';
import { WebSearchDriverType } from 'src/engine/core-modules/web-search/web-search.interface';
import { DriverFactoryBase } from 'src/engine/core-modules/twenty-config/dynamic-factory.base';
import { ConfigVariablesGroup } from 'src/engine/core-modules/twenty-config/enums/config-variables-group.enum';
import { ConfigGroupHashService } from 'src/engine/core-modules/twenty-config/services/config-group-hash.service';
import { TwentyConfigService } from 'src/engine/core-modules/twenty-config/twenty-config.service';
@Injectable()
export class WebSearchDriverFactory extends DriverFactoryBase<WebSearchDriver> {
constructor(
twentyConfigService: TwentyConfigService,
configGroupHashService: ConfigGroupHashService,
) {
super(twentyConfigService, configGroupHashService);
}
protected buildConfigKey(): string {
const driverType = this.twentyConfigService.get('WEB_SEARCH_DRIVER');
if (driverType !== WebSearchDriverType.DISABLED) {
return `${driverType}|${this.configGroupHashService.computeHash(ConfigVariablesGroup.LLM)}`;
}
return driverType;
}
protected createDriver(): WebSearchDriver {
const driverType = this.twentyConfigService.get('WEB_SEARCH_DRIVER');
switch (driverType) {
case WebSearchDriverType.DISABLED:
return new DisabledWebSearchDriver(
'Web search is disabled. Set WEB_SEARCH_DRIVER to EXA and provide EXA_API_KEY to enable it.',
);
case WebSearchDriverType.EXA: {
const apiKey = this.twentyConfigService.get('EXA_API_KEY');
if (!apiKey) {
throw new Error(
'EXA_API_KEY is required when WEB_SEARCH_DRIVER is EXA',
);
}
return new ExaDriver(apiKey);
}
default:
throw new Error(
`Invalid web search driver type (${driverType}), check your .env file`,
);
}
}
}
@@ -1,4 +0,0 @@
export enum WebSearchDriverType {
EXA = 'EXA',
DISABLED = 'DISABLED',
}
@@ -1,17 +0,0 @@
import { type DynamicModule, Global } from '@nestjs/common';
import { WebSearchDriverFactory } from 'src/engine/core-modules/web-search/web-search-driver.factory';
import { WebSearchService } from 'src/engine/core-modules/web-search/web-search.service';
import { TwentyConfigModule } from 'src/engine/core-modules/twenty-config/twenty-config.module';
@Global()
export class WebSearchModule {
static forRoot(): DynamicModule {
return {
module: WebSearchModule,
imports: [TwentyConfigModule],
providers: [WebSearchDriverFactory, WebSearchService],
exports: [WebSearchService],
};
}
}
@@ -1,95 +0,0 @@
import { Injectable } from '@nestjs/common';
import {
type WebSearchCostModel,
type WebSearchDriver,
} from 'src/engine/core-modules/web-search/drivers/interfaces/web-search-driver.interface';
import { type WebSearchBillingContext } from 'src/engine/core-modules/web-search/types/web-search-billing-context.type';
import { type WebSearchOptions } from 'src/engine/core-modules/web-search/types/web-search-options.type';
import { type WebSearchResult } from 'src/engine/core-modules/web-search/types/web-search-result.type';
import { WebSearchDriverFactory } from 'src/engine/core-modules/web-search/web-search-driver.factory';
import { WebSearchDriverType } from 'src/engine/core-modules/web-search/web-search.interface';
import { USAGE_RECORDED } from 'src/engine/core-modules/usage/constants/usage-recorded.constant';
import { UsageOperationType } from 'src/engine/core-modules/usage/enums/usage-operation-type.enum';
import { UsageResourceType } from 'src/engine/core-modules/usage/enums/usage-resource-type.enum';
import { UsageUnit } from 'src/engine/core-modules/usage/enums/usage-unit.enum';
import { type UsageEvent } from 'src/engine/core-modules/usage/types/usage-event.type';
import { TwentyConfigService } from 'src/engine/core-modules/twenty-config/twenty-config.service';
import { WorkspaceEventEmitter } from 'src/engine/workspace-event-emitter/workspace-event-emitter';
import { DOLLAR_TO_CREDIT_MULTIPLIER } from 'src/engine/metadata-modules/ai/ai-billing/constants/dollar-to-credit-multiplier';
@Injectable()
export class WebSearchService {
constructor(
private readonly webSearchDriverFactory: WebSearchDriverFactory,
private readonly twentyConfigService: TwentyConfigService,
private readonly workspaceEventEmitter: WorkspaceEventEmitter,
) {}
isEnabled(): boolean {
return (
this.twentyConfigService.get('WEB_SEARCH_DRIVER') !==
WebSearchDriverType.DISABLED
);
}
async search(
query: string,
options?: WebSearchOptions,
billingContext?: WebSearchBillingContext,
): Promise<WebSearchResult[]> {
const driver = this.webSearchDriverFactory.getCurrentDriver();
const results = await driver.search(query, options);
if (billingContext) {
this.emitUsageEvent(driver, results.length, billingContext);
}
return results;
}
static computeQueryCostDollars(
costModel: WebSearchCostModel,
numResults: number,
): number {
const additionalResults = Math.max(
0,
numResults - costModel.baseResultCount,
);
return (
costModel.baseCostDollars +
additionalResults * costModel.costPerAdditionalResultDollars
);
}
private emitUsageEvent(
driver: WebSearchDriver,
numResults: number,
billingContext: WebSearchBillingContext,
): void {
const costDollars = WebSearchService.computeQueryCostDollars(
driver.costModel,
numResults,
);
const creditsUsedMicro = Math.round(
costDollars * DOLLAR_TO_CREDIT_MULTIPLIER,
);
this.workspaceEventEmitter.emitCustomBatchEvent<UsageEvent>(
USAGE_RECORDED,
[
{
resourceType: UsageResourceType.AI,
operationType: UsageOperationType.WEB_SEARCH,
creditsUsedMicro,
quantity: 1,
unit: UsageUnit.INVOCATION,
resourceContext: this.twentyConfigService.get('WEB_SEARCH_DRIVER'),
userWorkspaceId: billingContext.userWorkspaceId ?? null,
},
],
billingContext.workspaceId,
);
}
}
@@ -38,6 +38,7 @@ import { WorkspaceCacheStorageService } from 'src/engine/workspace-cache-storage
import { WorkspaceDataSourceService } from 'src/engine/workspace-datasource/workspace-datasource.service';
import { PrefillLogicFunctionService } from 'src/engine/workspace-manager/standard-objects-prefill-data/services/prefill-logic-function.service';
import { ApplicationService } from 'src/engine/core-modules/application/application.service';
import { PreInstalledAppsService } from 'src/engine/core-modules/application/pre-installed-apps/pre-installed-apps.service';
import { WorkspaceMigrationValidateBuildAndRunService } from 'src/engine/workspace-manager/workspace-migration/services/workspace-migration-validate-build-and-run-service';
import { WorkspaceManagerService } from 'src/engine/workspace-manager/workspace-manager.service';
@@ -126,6 +127,7 @@ describe('WorkspaceService', () => {
FileCorePictureService,
AiModelRegistryService,
ApplicationService,
PreInstalledAppsService,
PrefillLogicFunctionService,
WorkspaceMigrationValidateBuildAndRunService,
UpgradeMigrationService,
@@ -13,6 +13,7 @@ import { DataSource, QueryRunner, Repository } from 'typeorm';
import { CoreEntityCacheService } from 'src/engine/core-entity-cache/services/core-entity-cache.service';
import { ApiKeyEntity } from 'src/engine/core-modules/api-key/api-key.entity';
import { ApplicationService } from 'src/engine/core-modules/application/application.service';
import { PreInstalledAppsService } from 'src/engine/core-modules/application/pre-installed-apps/pre-installed-apps.service';
import { type AuthContextUser } from 'src/engine/core-modules/auth/types/auth-context.type';
import { BillingSubscriptionService } from 'src/engine/core-modules/billing/services/billing-subscription.service';
import { BillingService } from 'src/engine/core-modules/billing/services/billing.service';
@@ -119,6 +120,7 @@ export class WorkspaceService extends TypeOrmQueryService<WorkspaceEntity> {
private readonly flatEntityMapsCacheService: WorkspaceManyOrAllFlatEntityMapsCacheService,
private readonly prefillLogicFunctionService: PrefillLogicFunctionService,
private readonly applicationService: ApplicationService,
private readonly preInstalledAppsService: PreInstalledAppsService,
private readonly workspaceMigrationValidateBuildAndRunService: WorkspaceMigrationValidateBuildAndRunService,
private readonly workspaceCacheStorageService: WorkspaceCacheStorageService,
private readonly subdomainManagerService: SubdomainManagerService,
@@ -819,6 +821,16 @@ export class WorkspaceService extends TypeOrmQueryService<WorkspaceEntity> {
);
this.exceptionHandlerService.captureExceptions([error as Error]);
}
try {
await this.preInstalledAppsService.installOnWorkspace(workspaceId);
} catch (error) {
this.logger.error(
`Non-critical: failed to install pre-installed apps for workspace ${workspaceId}`,
error,
);
this.exceptionHandlerService.captureExceptions([error as Error]);
}
}
async findOneWorkspaceById(id: string) {
@@ -6,6 +6,7 @@ import { NestjsQueryTypeOrmModule } from '@ptc-org/nestjs-query-typeorm';
import { TypeORMModule } from 'src/database/typeorm/typeorm.module';
import { ApplicationModule } from 'src/engine/core-modules/application/application.module';
import { PreInstalledAppsModule } from 'src/engine/core-modules/application/pre-installed-apps/pre-installed-apps.module';
import { AuditModule } from 'src/engine/core-modules/audit/audit.module';
import { TokenModule } from 'src/engine/core-modules/auth/token/token.module';
import { BillingModule } from 'src/engine/core-modules/billing/billing.module';
@@ -82,6 +83,7 @@ import { StandardObjectsPrefillModule } from 'src/engine/workspace-manager/stand
ViewModule,
WorkspaceManyOrAllFlatEntityMapsCacheModule,
ApplicationModule,
PreInstalledAppsModule,
EnterpriseModule,
StandardObjectsPrefillModule,
WorkspaceMigrationModule,
@@ -139,10 +139,13 @@ export class ChatExecutionService {
`Built tool catalog with ${toolCatalog.length} tools, ${skillCatalog.length} skills available`,
);
// Preload Exa when the workspace has it enabled; ActionToolProvider
// only emits the exa_web_search descriptor when isEnabled() is true,
// so getToolsByName silently skips it otherwise.
const toolNamesToPreload = [...COMMON_PRELOAD_TOOLS, 'exa_web_search'];
// Preload the Exa app tool (shipped as the `twenty-exa` npm package) so chat
// has structured web search ready without discovery. getToolsByName
// silently skips the entry when the workspace doesn't have the Exa app
// installed (admin hasn't registered it + flipped `isPreInstalled`).
// TODO(app-preloading): move this list into the app manifest so any
// app can declare `preloadedInChat: true` instead of hardcoding here.
const toolNamesToPreload = [...COMMON_PRELOAD_TOOLS, 'app_exa_web_search'];
const preloadedTools = await this.toolRegistry.getToolsByName(
toolNamesToPreload,
@@ -167,8 +170,8 @@ export class ChatExecutionService {
);
// Native web_search is returned when the resolved model's SDK provider
// exposes it (Anthropic, OpenAI). Coexists with exa_web_search when both
// are available — the model picks based on tool descriptions.
// exposes it (Anthropic, OpenAI). Coexists with app_exa_web_search when
// both are available — the model picks based on tool descriptions.
const { tools: nativeSearchTools, callableToolNames: searchToolNames } =
this.getNativeWebSearchTools(registeredModel);