feat(code-interpreter): reuse a warm sandbox per conversation (E2B) (#21664)
## What
The E2B code-interpreter driver created a **fresh sandbox on every
execution** and killed it in `finally`, so every call in a conversation
paid full cold-start and started blank. This PR keeps **one warm sandbox
per conversation** and, on idle, **pauses** it rather than killing it.
## How
- **Discovery without a registry:** the sandbox is tagged with the chat
`threadId` (scoped `workspaceId:threadId`) via E2B **metadata**, found
with `Sandbox.list({ query: { state: ['running','paused'], metadata }
})` and resumed with `Sandbox.connect()` (which auto-resumes a paused
sandbox). E2B is the source of truth — no Redis/DB mapping.
- **Pause/resume (E2B 2.x):** session sandboxes are created with
`lifecycle: { onTimeout: 'pause', autoResume: true }`. When idle they
**pause** — compute billing stops, filesystem **and** kernel/memory
state are preserved — and resume in ~1s on the next call. This replaces
the earlier keepalive approach.
- **No premature pause mid-run:** the sandbox is kept alive for
`max(execution timeout, idle window)`, so a long execution is never
paused underneath itself.
- **Tenant isolation:** discovery filters by the `twentySessionId` tag
and **re-checks it client-side**, so a loose server-side match can never
hand one conversation's warm sandbox (with its files, kernel state,
token) to another.
- **Concurrency:** executions sharing a session are serialized
in-process (one active stream per thread, run as a single job — the chat
resolver queues concurrent messages), so parallel tool calls can't race
the shared kernel.
- **Output isolation:** `/home/user/output` is reset at the start of
each reused run, so a call only returns the artifacts it actually
produced; durable state lives elsewhere and persists.
## SDK upgrade
`@e2b/code-interpreter` **`^1.0.4` → `^2.6.0`** (pulls `e2b@2.x`). The
typed pause/resume API, `lifecycle`, and the `state`/`metadata` list
filter only exist in the 2.x line; 1.x exposed them only as untyped
OpenAPI internals. `Sandbox.list()` is now a paginator (handled).
## Config
| Var | Default | Purpose |
|---|---|---|
| `CODE_INTERPRETER_TIMEOUT_MS` | `300000` | Max single-execution
duration. |
| `CODE_INTERPRETER_IDLE_TIMEOUT_MS` | `300000` | Idle window before the
warm sandbox auto-pauses. |
Reuse is always-on when a session id is present (chat path). The
workflow-agent path and the dev-only `LocalDriver` are unaffected.
## ⚠️ Open item before merge: paused-sandbox GC
E2B retains paused sandboxes **indefinitely** (no TTL). Unlike the old
keepalive path (which auto-killed on idle), pause means a conversation's
sandbox persists after the chat ends — so without garbage collection,
paused sandboxes accumulate (≈ one per historical conversation) and
consume storage. A GC policy is required; the approach + retention
window are being decided (see PR discussion). Also: the E2B runtime path
can't run in CI, so this still needs a **live smoke test** (reuse hit,
idle→pause, resume) and confirmation of paused-storage pricing before
rollout.
## Tests / checks
- Resolver unit tests (`getOrCreateSessionSandbox`): reuse+extend,
create-when-absent, duplicate reaping, connect-failure fallback,
keep-first-connectable-when-earlier-dead, **ignore cross-tenant
metadata**, and **kill-on-timeout-refresh-failure**.
- `nx typecheck twenty-server` (against e2b 2.x), `oxlint --type-aware`,
`oxfmt --check` all clean.
---------
Co-authored-by: Claude <noreply@anthropic.com>
This commit is contained in:
+7
-1
@@ -64,7 +64,13 @@ export class CodeInterpreterDriverFactory extends DriverFactoryBase<CodeInterpre
|
||||
);
|
||||
}
|
||||
|
||||
return new E2BDriver({ apiKey, timeoutMs });
|
||||
return new E2BDriver({
|
||||
apiKey,
|
||||
timeoutMs,
|
||||
idleTimeoutMs: this.twentyConfigService.get(
|
||||
'CODE_INTERPRETER_IDLE_TIMEOUT_MS',
|
||||
),
|
||||
});
|
||||
}
|
||||
|
||||
default:
|
||||
|
||||
+69
-2
@@ -1,5 +1,7 @@
|
||||
import { Injectable } from '@nestjs/common';
|
||||
|
||||
import { isDefined } from 'twenty-shared/utils';
|
||||
|
||||
import { CodeInterpreterDriverFactory } from 'src/engine/core-modules/code-interpreter/code-interpreter-driver.factory';
|
||||
import { CodeInterpreterDriverType } from 'src/engine/core-modules/code-interpreter/code-interpreter.interface';
|
||||
import {
|
||||
@@ -13,6 +15,10 @@ import { TwentyConfigService } from 'src/engine/core-modules/twenty-config/twent
|
||||
|
||||
@Injectable()
|
||||
export class CodeInterpreterService implements CodeInterpreterDriver {
|
||||
// One active stream per thread (the chat resolver queues the rest), so
|
||||
// in-process chaining is enough to serialize a session — no distributed lock.
|
||||
private readonly sessionExecutionTails = new Map<string, Promise<void>>();
|
||||
|
||||
constructor(
|
||||
private readonly codeInterpreterDriverFactory: CodeInterpreterDriverFactory,
|
||||
private readonly twentyConfigService: TwentyConfigService,
|
||||
@@ -25,14 +31,75 @@ export class CodeInterpreterService implements CodeInterpreterDriver {
|
||||
);
|
||||
}
|
||||
|
||||
async releaseThreadSandbox(
|
||||
workspaceId: string,
|
||||
threadId: string,
|
||||
): Promise<void> {
|
||||
await this.codeInterpreterDriverFactory
|
||||
.getCurrentDriver()
|
||||
.releaseSession?.(`${workspaceId}:${threadId}`);
|
||||
}
|
||||
|
||||
async sweepExpiredSandboxes(): Promise<number> {
|
||||
const maxAgeMs = this.twentyConfigService.get(
|
||||
'CODE_INTERPRETER_SESSION_MAX_AGE_MS',
|
||||
);
|
||||
|
||||
return (
|
||||
(await this.codeInterpreterDriverFactory
|
||||
.getCurrentDriver()
|
||||
.sweepExpiredSessions?.(maxAgeMs)) ?? 0
|
||||
);
|
||||
}
|
||||
|
||||
execute(
|
||||
code: string,
|
||||
files?: InputFile[],
|
||||
context?: ExecutionContext,
|
||||
callbacks?: StreamCallbacks,
|
||||
): Promise<CodeExecutionResult> {
|
||||
const driver = this.codeInterpreterDriverFactory.getCurrentDriver();
|
||||
const sessionId = context?.sessionId;
|
||||
|
||||
return driver.execute(code, files, context, callbacks);
|
||||
if (isDefined(sessionId)) {
|
||||
return this.runSerializedPerSession(sessionId, () =>
|
||||
this.runOnDriver(code, files, context, callbacks),
|
||||
);
|
||||
}
|
||||
|
||||
return this.runOnDriver(code, files, context, callbacks);
|
||||
}
|
||||
|
||||
private runOnDriver(
|
||||
code: string,
|
||||
files?: InputFile[],
|
||||
context?: ExecutionContext,
|
||||
callbacks?: StreamCallbacks,
|
||||
): Promise<CodeExecutionResult> {
|
||||
return this.codeInterpreterDriverFactory
|
||||
.getCurrentDriver()
|
||||
.execute(code, files, context, callbacks);
|
||||
}
|
||||
|
||||
private async runSerializedPerSession<T>(
|
||||
sessionId: string,
|
||||
task: () => Promise<T>,
|
||||
): Promise<T> {
|
||||
const previous =
|
||||
this.sessionExecutionTails.get(sessionId) ?? Promise.resolve();
|
||||
const result = previous.then(task, task);
|
||||
const tail = result.then(
|
||||
() => undefined,
|
||||
() => undefined,
|
||||
);
|
||||
|
||||
this.sessionExecutionTails.set(sessionId, tail);
|
||||
|
||||
try {
|
||||
return await result;
|
||||
} finally {
|
||||
if (this.sessionExecutionTails.get(sessionId) === tail) {
|
||||
this.sessionExecutionTails.delete(sessionId);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
+1
@@ -0,0 +1 @@
|
||||
export const CODE_INTERPRETER_SESSION_CLEANUP_CRON_PATTERN = '0 * * * *';
|
||||
+1
@@ -0,0 +1 @@
|
||||
export const SESSION_SANDBOX_METADATA_KEY = 'twentySessionId';
|
||||
+13
@@ -0,0 +1,13 @@
|
||||
import { Module } from '@nestjs/common';
|
||||
|
||||
import { CodeInterpreterSessionCleanupCronCommand } from 'src/engine/core-modules/code-interpreter/crons/commands/code-interpreter-session-cleanup.cron.command';
|
||||
import { CodeInterpreterSessionCleanupCronJob } from 'src/engine/core-modules/code-interpreter/crons/jobs/code-interpreter-session-cleanup.cron.job';
|
||||
|
||||
@Module({
|
||||
providers: [
|
||||
CodeInterpreterSessionCleanupCronJob,
|
||||
CodeInterpreterSessionCleanupCronCommand,
|
||||
],
|
||||
exports: [CodeInterpreterSessionCleanupCronCommand],
|
||||
})
|
||||
export class CodeInterpreterSessionCleanupModule {}
|
||||
+33
@@ -0,0 +1,33 @@
|
||||
import { Command, CommandRunner } from 'nest-commander';
|
||||
|
||||
import { CODE_INTERPRETER_SESSION_CLEANUP_CRON_PATTERN } from 'src/engine/core-modules/code-interpreter/constants/code-interpreter-session-cleanup-cron-pattern.constant';
|
||||
import { CodeInterpreterSessionCleanupCronJob } from 'src/engine/core-modules/code-interpreter/crons/jobs/code-interpreter-session-cleanup.cron.job';
|
||||
import { InjectMessageQueue } from 'src/engine/core-modules/message-queue/decorators/message-queue.decorator';
|
||||
import { MessageQueue } from 'src/engine/core-modules/message-queue/message-queue.constants';
|
||||
import { MessageQueueService } from 'src/engine/core-modules/message-queue/services/message-queue.service';
|
||||
|
||||
@Command({
|
||||
name: 'cron:code-interpreter:session-cleanup',
|
||||
description:
|
||||
'Starts a cron job to reclaim expired (abandoned) code interpreter sandboxes',
|
||||
})
|
||||
export class CodeInterpreterSessionCleanupCronCommand extends CommandRunner {
|
||||
constructor(
|
||||
@InjectMessageQueue(MessageQueue.cronQueue)
|
||||
private readonly messageQueueService: MessageQueueService,
|
||||
) {
|
||||
super();
|
||||
}
|
||||
|
||||
async run(): Promise<void> {
|
||||
await this.messageQueueService.addCron<undefined>({
|
||||
jobName: CodeInterpreterSessionCleanupCronJob.name,
|
||||
data: undefined,
|
||||
options: {
|
||||
repeat: {
|
||||
pattern: CODE_INTERPRETER_SESSION_CLEANUP_CRON_PATTERN,
|
||||
},
|
||||
},
|
||||
});
|
||||
}
|
||||
}
|
||||
+36
@@ -0,0 +1,36 @@
|
||||
import { Injectable, Logger } from '@nestjs/common';
|
||||
|
||||
import { CodeInterpreterService } from 'src/engine/core-modules/code-interpreter/code-interpreter.service';
|
||||
import { CODE_INTERPRETER_SESSION_CLEANUP_CRON_PATTERN } from 'src/engine/core-modules/code-interpreter/constants/code-interpreter-session-cleanup-cron-pattern.constant';
|
||||
import { SentryCronMonitor } from 'src/engine/core-modules/cron/sentry-cron-monitor.decorator';
|
||||
import { Process } from 'src/engine/core-modules/message-queue/decorators/process.decorator';
|
||||
import { Processor } from 'src/engine/core-modules/message-queue/decorators/processor.decorator';
|
||||
import { MessageQueue } from 'src/engine/core-modules/message-queue/message-queue.constants';
|
||||
|
||||
@Injectable()
|
||||
@Processor(MessageQueue.cronQueue)
|
||||
export class CodeInterpreterSessionCleanupCronJob {
|
||||
private readonly logger = new Logger(
|
||||
CodeInterpreterSessionCleanupCronJob.name,
|
||||
);
|
||||
|
||||
constructor(
|
||||
private readonly codeInterpreterService: CodeInterpreterService,
|
||||
) {}
|
||||
|
||||
@Process(CodeInterpreterSessionCleanupCronJob.name)
|
||||
@SentryCronMonitor(
|
||||
CodeInterpreterSessionCleanupCronJob.name,
|
||||
CODE_INTERPRETER_SESSION_CLEANUP_CRON_PATTERN,
|
||||
)
|
||||
async handle(): Promise<void> {
|
||||
const reclaimedCount =
|
||||
await this.codeInterpreterService.sweepExpiredSandboxes();
|
||||
|
||||
if (reclaimedCount > 0) {
|
||||
this.logger.log(
|
||||
`Reclaimed ${reclaimedCount} expired code interpreter sandbox(es)`,
|
||||
);
|
||||
}
|
||||
}
|
||||
}
|
||||
+84
-21
@@ -2,8 +2,14 @@ import { promises as fs } from 'fs';
|
||||
import { join } from 'path';
|
||||
|
||||
import { Sandbox } from '@e2b/code-interpreter';
|
||||
import { Logger } from '@nestjs/common';
|
||||
|
||||
import { isDefined } from 'twenty-shared/utils';
|
||||
|
||||
import { DEFAULT_CODE_INTERPRETER_TIMEOUT_MS } from 'src/engine/core-modules/code-interpreter/code-interpreter.constants';
|
||||
import { getOrCreateSessionSandbox } from 'src/engine/core-modules/code-interpreter/drivers/utils/get-or-create-session-sandbox.util';
|
||||
import { releaseSessionSandboxes } from 'src/engine/core-modules/code-interpreter/drivers/utils/release-session-sandboxes.util';
|
||||
import { sweepExpiredSessionSandboxes } from 'src/engine/core-modules/code-interpreter/drivers/utils/sweep-expired-session-sandboxes.util';
|
||||
import { getMimeType } from 'src/engine/core-modules/code-interpreter/utils/get-mime-type.util';
|
||||
|
||||
import {
|
||||
@@ -18,12 +24,13 @@ import {
|
||||
export type E2BDriverOptions = {
|
||||
apiKey: string;
|
||||
timeoutMs?: number;
|
||||
idleTimeoutMs?: number;
|
||||
};
|
||||
|
||||
const SANDBOX_SCRIPTS_PATH = join(__dirname, '..', 'sandbox-scripts');
|
||||
|
||||
async function uploadDirectoryToSandbox(
|
||||
sbx: Sandbox,
|
||||
sandbox: Sandbox,
|
||||
localPath: string,
|
||||
remotePath: string,
|
||||
) {
|
||||
@@ -34,17 +41,19 @@ async function uploadDirectoryToSandbox(
|
||||
const remoteEntryPath = `${remotePath}/${entry.name}`;
|
||||
|
||||
if (entry.isDirectory()) {
|
||||
await uploadDirectoryToSandbox(sbx, localEntryPath, remoteEntryPath);
|
||||
await uploadDirectoryToSandbox(sandbox, localEntryPath, remoteEntryPath);
|
||||
} else {
|
||||
const content = await fs.readFile(localEntryPath);
|
||||
const arrayBuffer = new Uint8Array(content).buffer;
|
||||
|
||||
await sbx.files.write(remoteEntryPath, arrayBuffer);
|
||||
await sandbox.files.write(remoteEntryPath, arrayBuffer);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
export class E2BDriver implements CodeInterpreterDriver {
|
||||
private readonly logger = new Logger(E2BDriver.name);
|
||||
|
||||
constructor(private options: E2BDriverOptions) {}
|
||||
|
||||
async execute(
|
||||
@@ -53,27 +62,52 @@ export class E2BDriver implements CodeInterpreterDriver {
|
||||
context?: ExecutionContext,
|
||||
callbacks?: StreamCallbacks,
|
||||
): Promise<CodeExecutionResult> {
|
||||
const sbx = await Sandbox.create({
|
||||
apiKey: this.options.apiKey,
|
||||
timeoutMs: this.options.timeoutMs ?? DEFAULT_CODE_INTERPRETER_TIMEOUT_MS,
|
||||
});
|
||||
const { apiKey } = this.options;
|
||||
const sessionId = context?.sessionId;
|
||||
const idleTimeoutMs =
|
||||
this.options.idleTimeoutMs ?? DEFAULT_CODE_INTERPRETER_TIMEOUT_MS;
|
||||
const timeoutMs =
|
||||
this.options.timeoutMs ?? DEFAULT_CODE_INTERPRETER_TIMEOUT_MS;
|
||||
|
||||
let sandbox: Sandbox;
|
||||
let isReused = false;
|
||||
let keepWarm = false;
|
||||
|
||||
if (isDefined(sessionId)) {
|
||||
keepWarm = true;
|
||||
({ sandbox, isReused } = await getOrCreateSessionSandbox({
|
||||
sandboxApi: Sandbox,
|
||||
apiKey,
|
||||
sessionId,
|
||||
timeoutMs,
|
||||
idleTimeoutMs,
|
||||
}));
|
||||
} else {
|
||||
sandbox = await Sandbox.create({ apiKey, timeoutMs });
|
||||
}
|
||||
|
||||
try {
|
||||
// Upload pre-installed scripts to sandbox
|
||||
try {
|
||||
await uploadDirectoryToSandbox(
|
||||
sbx,
|
||||
SANDBOX_SCRIPTS_PATH,
|
||||
'/home/user/scripts',
|
||||
);
|
||||
} catch {
|
||||
// Scripts directory might not exist
|
||||
// A reused sandbox already has the scripts from its first run.
|
||||
if (!isReused) {
|
||||
try {
|
||||
await uploadDirectoryToSandbox(
|
||||
sandbox,
|
||||
SANDBOX_SCRIPTS_PATH,
|
||||
'/home/user/scripts',
|
||||
);
|
||||
} catch {
|
||||
// Scripts directory might not exist
|
||||
}
|
||||
}
|
||||
|
||||
if (isReused) {
|
||||
await this.resetOutputDirectory(sandbox);
|
||||
}
|
||||
|
||||
for (const file of files ?? []) {
|
||||
const arrayBuffer = new Uint8Array(file.content).buffer;
|
||||
|
||||
await sbx.files.write(`/home/user/${file.filename}`, arrayBuffer);
|
||||
await sandbox.files.write(`/home/user/${file.filename}`, arrayBuffer);
|
||||
}
|
||||
|
||||
const envSetup = context?.env
|
||||
@@ -93,7 +127,7 @@ export class E2BDriver implements CodeInterpreterDriver {
|
||||
const outputFiles: OutputFile[] = [];
|
||||
let chartCounter = 0;
|
||||
|
||||
const execution = await sbx.runCode(envSetup + code, {
|
||||
const execution = await sandbox.runCode(envSetup + code, {
|
||||
onStdout: (data) => callbacks?.onStdout?.(data.line),
|
||||
onStderr: (data) => callbacks?.onStderr?.(data.line),
|
||||
onResult: async (result) => {
|
||||
@@ -111,12 +145,13 @@ export class E2BDriver implements CodeInterpreterDriver {
|
||||
});
|
||||
|
||||
try {
|
||||
const outputDir = await sbx.files.list('/home/user/output');
|
||||
const outputDir = await sandbox.files.list('/home/user/output');
|
||||
|
||||
for (const file of outputDir) {
|
||||
if (file.type === 'file') {
|
||||
const content = await sbx.files.read(
|
||||
const content = await sandbox.files.read(
|
||||
`/home/user/output/${file.name}`,
|
||||
{ format: 'bytes' },
|
||||
);
|
||||
|
||||
const outputFile: OutputFile = {
|
||||
@@ -141,7 +176,35 @@ export class E2BDriver implements CodeInterpreterDriver {
|
||||
error: execution.error?.value,
|
||||
};
|
||||
} finally {
|
||||
await sbx.kill();
|
||||
if (!keepWarm) {
|
||||
await sandbox.kill();
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
private async resetOutputDirectory(sandbox: Sandbox): Promise<void> {
|
||||
const outputDirectory = '/home/user/output';
|
||||
|
||||
try {
|
||||
if (await sandbox.files.exists(outputDirectory)) {
|
||||
await sandbox.files.remove(outputDirectory);
|
||||
}
|
||||
|
||||
await sandbox.files.makeDir(outputDirectory);
|
||||
} catch (error) {
|
||||
const reason = error instanceof Error ? error.message : String(error);
|
||||
|
||||
this.logger.warn(
|
||||
`Failed to reset reused sandbox output directory; results may include stale files: ${reason}`,
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
async releaseSession(sessionId: string): Promise<void> {
|
||||
await releaseSessionSandboxes(Sandbox, this.options.apiKey, sessionId);
|
||||
}
|
||||
|
||||
async sweepExpiredSessions(maxAgeMs: number): Promise<number> {
|
||||
return sweepExpiredSessionSandboxes(Sandbox, this.options.apiKey, maxAgeMs);
|
||||
}
|
||||
}
|
||||
|
||||
+3
@@ -20,6 +20,7 @@ export type CodeExecutionResult = {
|
||||
|
||||
export type ExecutionContext = {
|
||||
env?: Record<string, string>;
|
||||
sessionId?: string;
|
||||
};
|
||||
|
||||
export type StreamCallbacks = {
|
||||
@@ -35,4 +36,6 @@ export interface CodeInterpreterDriver {
|
||||
context?: ExecutionContext,
|
||||
callbacks?: StreamCallbacks,
|
||||
): Promise<CodeExecutionResult>;
|
||||
releaseSession?(sessionId: string): Promise<void>;
|
||||
sweepExpiredSessions?(maxAgeMs: number): Promise<number>;
|
||||
}
|
||||
|
||||
+250
@@ -0,0 +1,250 @@
|
||||
import { type Sandbox } from '@e2b/code-interpreter';
|
||||
|
||||
import { SESSION_SANDBOX_METADATA_KEY } from 'src/engine/core-modules/code-interpreter/constants/session-sandbox-metadata-key.constant';
|
||||
import { getOrCreateSessionSandbox } from 'src/engine/core-modules/code-interpreter/drivers/utils/get-or-create-session-sandbox.util';
|
||||
|
||||
type SandboxApiMock = {
|
||||
list: jest.Mock;
|
||||
connect: jest.Mock;
|
||||
create: jest.Mock;
|
||||
kill: jest.Mock;
|
||||
};
|
||||
|
||||
type ListedSandbox = { sandboxId: string; metadata: Record<string, string> };
|
||||
|
||||
const apiKey = 'test-api-key';
|
||||
const sessionId = 'workspace-1:thread-1';
|
||||
const timeoutMs = 600_000;
|
||||
const idleTimeoutMs = 300_000;
|
||||
// The resolver keeps the sandbox alive for the larger of the two.
|
||||
const aliveTimeoutMs = 600_000;
|
||||
|
||||
// Sandbox.list() returns a paginator; emit the tagged sandboxes as a single page.
|
||||
const paginatorOf = (items: ListedSandbox[]) => {
|
||||
let fetched = false;
|
||||
|
||||
return {
|
||||
get hasNext() {
|
||||
return !fetched;
|
||||
},
|
||||
nextItems: async () => {
|
||||
fetched = true;
|
||||
|
||||
return items;
|
||||
},
|
||||
};
|
||||
};
|
||||
|
||||
const buildSandboxApi = (
|
||||
overrides: Partial<SandboxApiMock> = {},
|
||||
): { api: typeof Sandbox; mock: SandboxApiMock } => {
|
||||
const mock: SandboxApiMock = {
|
||||
list: jest.fn(() => paginatorOf([])),
|
||||
connect: jest.fn(),
|
||||
create: jest.fn(),
|
||||
kill: jest.fn().mockResolvedValue(true),
|
||||
...overrides,
|
||||
};
|
||||
|
||||
return { api: mock as unknown as typeof Sandbox, mock };
|
||||
};
|
||||
|
||||
const buildFakeSandbox = (): Sandbox =>
|
||||
({
|
||||
setTimeout: jest.fn().mockResolvedValue(undefined),
|
||||
kill: jest.fn().mockResolvedValue(undefined),
|
||||
}) as unknown as Sandbox;
|
||||
|
||||
const listedSandbox = (
|
||||
sandboxId: string,
|
||||
tag: string = sessionId,
|
||||
): ListedSandbox => ({
|
||||
sandboxId,
|
||||
metadata: { [SESSION_SANDBOX_METADATA_KEY]: tag },
|
||||
});
|
||||
|
||||
describe('getOrCreateSessionSandbox', () => {
|
||||
it('reuses and extends the existing sandbox for the session', async () => {
|
||||
const sandbox = buildFakeSandbox();
|
||||
const { api, mock } = buildSandboxApi({
|
||||
list: jest.fn(() => paginatorOf([listedSandbox('sbx-1')])),
|
||||
connect: jest.fn().mockResolvedValue(sandbox),
|
||||
});
|
||||
|
||||
const result = await getOrCreateSessionSandbox({
|
||||
sandboxApi: api,
|
||||
apiKey,
|
||||
sessionId,
|
||||
timeoutMs,
|
||||
idleTimeoutMs,
|
||||
});
|
||||
|
||||
expect(result).toEqual({ sandbox, isReused: true });
|
||||
expect(mock.list).toHaveBeenCalledWith({
|
||||
apiKey,
|
||||
query: {
|
||||
state: ['running', 'paused'],
|
||||
metadata: { [SESSION_SANDBOX_METADATA_KEY]: sessionId },
|
||||
},
|
||||
});
|
||||
expect(mock.connect).toHaveBeenCalledWith('sbx-1', { apiKey });
|
||||
expect(sandbox.setTimeout).toHaveBeenCalledWith(aliveTimeoutMs);
|
||||
expect(mock.create).not.toHaveBeenCalled();
|
||||
expect(mock.kill).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it('creates a new tagged auto-pausing sandbox when none exists for the session', async () => {
|
||||
const sandbox = buildFakeSandbox();
|
||||
const { api, mock } = buildSandboxApi({
|
||||
list: jest.fn(() => paginatorOf([])),
|
||||
create: jest.fn().mockResolvedValue(sandbox),
|
||||
});
|
||||
|
||||
const result = await getOrCreateSessionSandbox({
|
||||
sandboxApi: api,
|
||||
apiKey,
|
||||
sessionId,
|
||||
timeoutMs,
|
||||
idleTimeoutMs,
|
||||
});
|
||||
|
||||
expect(result).toEqual({ sandbox, isReused: false });
|
||||
expect(mock.create).toHaveBeenCalledWith({
|
||||
apiKey,
|
||||
timeoutMs: aliveTimeoutMs,
|
||||
lifecycle: { onTimeout: 'pause', autoResume: true },
|
||||
metadata: { [SESSION_SANDBOX_METADATA_KEY]: sessionId },
|
||||
});
|
||||
expect(mock.connect).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it('reaps duplicate sandboxes and reuses a single one', async () => {
|
||||
const keep = buildFakeSandbox();
|
||||
const { api, mock } = buildSandboxApi({
|
||||
list: jest.fn(() =>
|
||||
paginatorOf([
|
||||
listedSandbox('sbx-keep'),
|
||||
listedSandbox('sbx-dup-1'),
|
||||
listedSandbox('sbx-dup-2'),
|
||||
]),
|
||||
),
|
||||
connect: jest.fn().mockResolvedValue(keep),
|
||||
});
|
||||
|
||||
const result = await getOrCreateSessionSandbox({
|
||||
sandboxApi: api,
|
||||
apiKey,
|
||||
sessionId,
|
||||
timeoutMs,
|
||||
idleTimeoutMs,
|
||||
});
|
||||
|
||||
expect(result).toEqual({ sandbox: keep, isReused: true });
|
||||
expect(mock.connect).toHaveBeenCalledTimes(1);
|
||||
expect(mock.connect).toHaveBeenCalledWith('sbx-keep', { apiKey });
|
||||
expect(mock.kill).toHaveBeenCalledWith('sbx-dup-1', { apiKey });
|
||||
expect(mock.kill).toHaveBeenCalledWith('sbx-dup-2', { apiKey });
|
||||
expect(mock.kill).not.toHaveBeenCalledWith('sbx-keep', { apiKey });
|
||||
expect(mock.create).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it('creates a fresh sandbox when connecting to the existing one fails', async () => {
|
||||
const created = buildFakeSandbox();
|
||||
const { api, mock } = buildSandboxApi({
|
||||
list: jest.fn(() => paginatorOf([listedSandbox('sbx-dead')])),
|
||||
connect: jest.fn().mockRejectedValue(new Error('sandbox not found')),
|
||||
create: jest.fn().mockResolvedValue(created),
|
||||
});
|
||||
|
||||
const result = await getOrCreateSessionSandbox({
|
||||
sandboxApi: api,
|
||||
apiKey,
|
||||
sessionId,
|
||||
timeoutMs,
|
||||
idleTimeoutMs,
|
||||
});
|
||||
|
||||
expect(result).toEqual({ sandbox: created, isReused: false });
|
||||
expect(mock.create).toHaveBeenCalledTimes(1);
|
||||
});
|
||||
|
||||
it('keeps the first connectable sandbox instead of cold-creating when an earlier one is dead', async () => {
|
||||
const live = buildFakeSandbox();
|
||||
const { api, mock } = buildSandboxApi({
|
||||
list: jest.fn(() =>
|
||||
paginatorOf([listedSandbox('sbx-dead'), listedSandbox('sbx-live')]),
|
||||
),
|
||||
connect: jest.fn((sandboxId: string) =>
|
||||
sandboxId === 'sbx-live'
|
||||
? Promise.resolve(live)
|
||||
: Promise.reject(new Error('sandbox not found')),
|
||||
),
|
||||
});
|
||||
|
||||
const result = await getOrCreateSessionSandbox({
|
||||
sandboxApi: api,
|
||||
apiKey,
|
||||
sessionId,
|
||||
timeoutMs,
|
||||
idleTimeoutMs,
|
||||
});
|
||||
|
||||
expect(result).toEqual({ sandbox: live, isReused: true });
|
||||
expect(live.setTimeout).toHaveBeenCalledWith(aliveTimeoutMs);
|
||||
expect(mock.kill).toHaveBeenCalledWith('sbx-dead', { apiKey });
|
||||
expect(mock.create).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it('never reuses a sandbox whose metadata does not exactly match the session', async () => {
|
||||
const created = buildFakeSandbox();
|
||||
const { api, mock } = buildSandboxApi({
|
||||
// A sandbox from a different session/tenant that a loose server-side match
|
||||
// could surface — it must be ignored, never connected to or reaped.
|
||||
list: jest.fn(() =>
|
||||
paginatorOf([listedSandbox('sbx-foreign', 'workspace-2:thread-9')]),
|
||||
),
|
||||
create: jest.fn().mockResolvedValue(created),
|
||||
});
|
||||
|
||||
const result = await getOrCreateSessionSandbox({
|
||||
sandboxApi: api,
|
||||
apiKey,
|
||||
sessionId,
|
||||
timeoutMs,
|
||||
idleTimeoutMs,
|
||||
});
|
||||
|
||||
expect(result).toEqual({ sandbox: created, isReused: false });
|
||||
expect(mock.connect).not.toHaveBeenCalled();
|
||||
expect(mock.kill).not.toHaveBeenCalled();
|
||||
expect(mock.create).toHaveBeenCalledTimes(1);
|
||||
});
|
||||
|
||||
it('kills a connected sandbox and creates fresh when refreshing its timeout fails', async () => {
|
||||
const stale = {
|
||||
setTimeout: jest
|
||||
.fn()
|
||||
.mockRejectedValue(new Error('timeout refresh failed')),
|
||||
kill: jest.fn().mockResolvedValue(undefined),
|
||||
} as unknown as Sandbox;
|
||||
const created = buildFakeSandbox();
|
||||
const { api, mock } = buildSandboxApi({
|
||||
list: jest.fn(() => paginatorOf([listedSandbox('sbx-stale')])),
|
||||
connect: jest.fn().mockResolvedValue(stale),
|
||||
create: jest.fn().mockResolvedValue(created),
|
||||
});
|
||||
|
||||
const result = await getOrCreateSessionSandbox({
|
||||
sandboxApi: api,
|
||||
apiKey,
|
||||
sessionId,
|
||||
timeoutMs,
|
||||
idleTimeoutMs,
|
||||
});
|
||||
|
||||
expect(result).toEqual({ sandbox: created, isReused: false });
|
||||
expect(stale.setTimeout).toHaveBeenCalledWith(aliveTimeoutMs);
|
||||
expect(stale.kill).toHaveBeenCalledTimes(1);
|
||||
expect(mock.create).toHaveBeenCalledTimes(1);
|
||||
});
|
||||
});
|
||||
+75
@@ -0,0 +1,75 @@
|
||||
import { type Sandbox } from '@e2b/code-interpreter';
|
||||
|
||||
import { SESSION_SANDBOX_METADATA_KEY } from 'src/engine/core-modules/code-interpreter/constants/session-sandbox-metadata-key.constant';
|
||||
import { releaseSessionSandboxes } from 'src/engine/core-modules/code-interpreter/drivers/utils/release-session-sandboxes.util';
|
||||
|
||||
const apiKey = 'test-api-key';
|
||||
const sessionId = 'workspace-1:thread-1';
|
||||
|
||||
type ListedSandbox = {
|
||||
sandboxId: string;
|
||||
metadata: Record<string, string>;
|
||||
startedAt: string;
|
||||
};
|
||||
|
||||
const paginatorOf = (items: ListedSandbox[]) => {
|
||||
let fetched = false;
|
||||
|
||||
return {
|
||||
get hasNext() {
|
||||
return !fetched;
|
||||
},
|
||||
nextItems: async () => {
|
||||
fetched = true;
|
||||
|
||||
return items;
|
||||
},
|
||||
};
|
||||
};
|
||||
|
||||
const buildApi = (items: ListedSandbox[]) => {
|
||||
const kill = jest.fn().mockResolvedValue(true);
|
||||
const list = jest.fn(() => paginatorOf(items));
|
||||
|
||||
return { api: { list, kill } as unknown as typeof Sandbox, kill, list };
|
||||
};
|
||||
|
||||
const sandboxInfo = (
|
||||
sandboxId: string,
|
||||
{ tag = sessionId }: { tag?: string | null } = {},
|
||||
): ListedSandbox => ({
|
||||
sandboxId,
|
||||
metadata: tag === null ? {} : { [SESSION_SANDBOX_METADATA_KEY]: tag },
|
||||
startedAt: new Date().toISOString(),
|
||||
});
|
||||
|
||||
describe('releaseSessionSandboxes', () => {
|
||||
it('kills every sandbox tagged for the session', async () => {
|
||||
const { api, kill, list } = buildApi([
|
||||
sandboxInfo('sbx-1'),
|
||||
sandboxInfo('sbx-2'),
|
||||
]);
|
||||
|
||||
await releaseSessionSandboxes(api, apiKey, sessionId);
|
||||
|
||||
expect(list).toHaveBeenCalledWith({
|
||||
apiKey,
|
||||
query: {
|
||||
state: ['running', 'paused'],
|
||||
metadata: { [SESSION_SANDBOX_METADATA_KEY]: sessionId },
|
||||
},
|
||||
});
|
||||
expect(kill).toHaveBeenCalledWith('sbx-1', { apiKey });
|
||||
expect(kill).toHaveBeenCalledWith('sbx-2', { apiKey });
|
||||
});
|
||||
|
||||
it('ignores a sandbox whose tag does not match the session', async () => {
|
||||
const { api, kill } = buildApi([
|
||||
sandboxInfo('sbx-other', { tag: 'workspace-2:thread-9' }),
|
||||
]);
|
||||
|
||||
await releaseSessionSandboxes(api, apiKey, sessionId);
|
||||
|
||||
expect(kill).not.toHaveBeenCalled();
|
||||
});
|
||||
});
|
||||
+73
@@ -0,0 +1,73 @@
|
||||
import { type Sandbox } from '@e2b/code-interpreter';
|
||||
|
||||
import { SESSION_SANDBOX_METADATA_KEY } from 'src/engine/core-modules/code-interpreter/constants/session-sandbox-metadata-key.constant';
|
||||
import { sweepExpiredSessionSandboxes } from 'src/engine/core-modules/code-interpreter/drivers/utils/sweep-expired-session-sandboxes.util';
|
||||
|
||||
const apiKey = 'test-api-key';
|
||||
const sessionId = 'workspace-1:thread-1';
|
||||
const dayMs = 24 * 60 * 60 * 1000;
|
||||
|
||||
type ListedSandbox = {
|
||||
sandboxId: string;
|
||||
metadata: Record<string, string>;
|
||||
startedAt: string;
|
||||
};
|
||||
|
||||
const paginatorOf = (items: ListedSandbox[]) => {
|
||||
let fetched = false;
|
||||
|
||||
return {
|
||||
get hasNext() {
|
||||
return !fetched;
|
||||
},
|
||||
nextItems: async () => {
|
||||
fetched = true;
|
||||
|
||||
return items;
|
||||
},
|
||||
};
|
||||
};
|
||||
|
||||
const buildApi = (items: ListedSandbox[]) => {
|
||||
const kill = jest.fn().mockResolvedValue(true);
|
||||
const list = jest.fn(() => paginatorOf(items));
|
||||
|
||||
return { api: { list, kill } as unknown as typeof Sandbox, kill, list };
|
||||
};
|
||||
|
||||
const sandboxInfo = (
|
||||
sandboxId: string,
|
||||
{ tag = sessionId, ageMs = 0 }: { tag?: string | null; ageMs?: number } = {},
|
||||
): ListedSandbox => ({
|
||||
sandboxId,
|
||||
metadata: tag === null ? {} : { [SESSION_SANDBOX_METADATA_KEY]: tag },
|
||||
startedAt: new Date(Date.now() - ageMs).toISOString(),
|
||||
});
|
||||
|
||||
describe('sweepExpiredSessionSandboxes', () => {
|
||||
it('kills only session sandboxes older than the max age', async () => {
|
||||
const { api, kill } = buildApi([
|
||||
sandboxInfo('sbx-old', { ageMs: 2 * dayMs }),
|
||||
sandboxInfo('sbx-fresh', { ageMs: 60 * 1000 }),
|
||||
sandboxInfo('sbx-untagged', { tag: null, ageMs: 5 * dayMs }),
|
||||
]);
|
||||
|
||||
const killed = await sweepExpiredSessionSandboxes(api, apiKey, dayMs);
|
||||
|
||||
expect(killed).toBe(1);
|
||||
expect(kill).toHaveBeenCalledWith('sbx-old', { apiKey });
|
||||
expect(kill).not.toHaveBeenCalledWith('sbx-fresh', { apiKey });
|
||||
expect(kill).not.toHaveBeenCalledWith('sbx-untagged', { apiKey });
|
||||
});
|
||||
|
||||
it('lists both running and paused sandboxes', async () => {
|
||||
const { api, list } = buildApi([]);
|
||||
|
||||
await sweepExpiredSessionSandboxes(api, apiKey, 1000);
|
||||
|
||||
expect(list).toHaveBeenCalledWith({
|
||||
apiKey,
|
||||
query: { state: ['running', 'paused'] },
|
||||
});
|
||||
});
|
||||
});
|
||||
+144
@@ -0,0 +1,144 @@
|
||||
import { type Sandbox } from '@e2b/code-interpreter';
|
||||
|
||||
import { isDefined } from 'twenty-shared/utils';
|
||||
|
||||
import { SESSION_SANDBOX_METADATA_KEY } from 'src/engine/core-modules/code-interpreter/constants/session-sandbox-metadata-key.constant';
|
||||
|
||||
type SandboxApi = typeof Sandbox;
|
||||
|
||||
type GetOrCreateSessionSandboxArgs = {
|
||||
sandboxApi: SandboxApi;
|
||||
apiKey: string;
|
||||
sessionId: string;
|
||||
timeoutMs: number;
|
||||
idleTimeoutMs: number;
|
||||
};
|
||||
|
||||
const listSandboxesForSession = async (
|
||||
sandboxApi: SandboxApi,
|
||||
apiKey: string,
|
||||
sessionId: string,
|
||||
) => {
|
||||
const paginator = sandboxApi.list({
|
||||
apiKey,
|
||||
query: {
|
||||
state: ['running', 'paused'],
|
||||
metadata: { [SESSION_SANDBOX_METADATA_KEY]: sessionId },
|
||||
},
|
||||
});
|
||||
|
||||
const sandboxes: Awaited<ReturnType<typeof paginator.nextItems>> = [];
|
||||
|
||||
while (paginator.hasNext) {
|
||||
sandboxes.push(...(await paginator.nextItems()));
|
||||
}
|
||||
|
||||
// Re-check the tag client-side so a loose server-side match can never reuse
|
||||
// another tenant's sandbox.
|
||||
return sandboxes.filter(
|
||||
(sandbox) => sandbox.metadata?.[SESSION_SANDBOX_METADATA_KEY] === sessionId,
|
||||
);
|
||||
};
|
||||
|
||||
const connectAndKeepAlive = async (
|
||||
sandboxApi: SandboxApi,
|
||||
apiKey: string,
|
||||
sandboxId: string,
|
||||
timeoutMs: number,
|
||||
): Promise<Sandbox | undefined> => {
|
||||
const sandbox = await sandboxApi
|
||||
.connect(sandboxId, { apiKey })
|
||||
.catch(() => undefined);
|
||||
|
||||
if (!isDefined(sandbox)) {
|
||||
return undefined;
|
||||
}
|
||||
|
||||
try {
|
||||
await sandbox.setTimeout(timeoutMs);
|
||||
|
||||
return sandbox;
|
||||
} catch {
|
||||
// Couldn't refresh the timeout — kill it instead of leaking a running sandbox.
|
||||
await sandbox.kill().catch(() => undefined);
|
||||
|
||||
return undefined;
|
||||
}
|
||||
};
|
||||
|
||||
const killSandboxById = (
|
||||
sandboxApi: SandboxApi,
|
||||
apiKey: string,
|
||||
sandboxId: string,
|
||||
) => sandboxApi.kill(sandboxId, { apiKey }).catch(() => undefined);
|
||||
|
||||
const createSessionSandbox = (
|
||||
sandboxApi: SandboxApi,
|
||||
apiKey: string,
|
||||
sessionId: string,
|
||||
timeoutMs: number,
|
||||
) =>
|
||||
sandboxApi.create({
|
||||
apiKey,
|
||||
timeoutMs,
|
||||
lifecycle: { onTimeout: 'pause', autoResume: true },
|
||||
metadata: { [SESSION_SANDBOX_METADATA_KEY]: sessionId },
|
||||
});
|
||||
|
||||
export const getOrCreateSessionSandbox = async ({
|
||||
sandboxApi,
|
||||
apiKey,
|
||||
sessionId,
|
||||
timeoutMs,
|
||||
idleTimeoutMs,
|
||||
}: GetOrCreateSessionSandboxArgs): Promise<{
|
||||
sandbox: Sandbox;
|
||||
isReused: boolean;
|
||||
}> => {
|
||||
// The sandbox must outlive a single execution and the idle window before it
|
||||
// auto-pauses, so take the larger of the two.
|
||||
const aliveTimeoutMs = Math.max(timeoutMs, idleTimeoutMs);
|
||||
|
||||
const sessionSandboxes = await listSandboxesForSession(
|
||||
sandboxApi,
|
||||
apiKey,
|
||||
sessionId,
|
||||
);
|
||||
|
||||
let reusedSandbox: Sandbox | undefined;
|
||||
let reusedSandboxId: string | undefined;
|
||||
|
||||
for (const { sandboxId } of sessionSandboxes) {
|
||||
const sandbox = await connectAndKeepAlive(
|
||||
sandboxApi,
|
||||
apiKey,
|
||||
sandboxId,
|
||||
aliveTimeoutMs,
|
||||
);
|
||||
|
||||
if (isDefined(sandbox)) {
|
||||
reusedSandbox = sandbox;
|
||||
reusedSandboxId = sandboxId;
|
||||
break;
|
||||
}
|
||||
}
|
||||
|
||||
if (isDefined(reusedSandbox)) {
|
||||
await Promise.all(
|
||||
sessionSandboxes
|
||||
.filter(({ sandboxId }) => sandboxId !== reusedSandboxId)
|
||||
.map(({ sandboxId }) => killSandboxById(sandboxApi, apiKey, sandboxId)),
|
||||
);
|
||||
|
||||
return { sandbox: reusedSandbox, isReused: true };
|
||||
}
|
||||
|
||||
const sandbox = await createSessionSandbox(
|
||||
sandboxApi,
|
||||
apiKey,
|
||||
sessionId,
|
||||
aliveTimeoutMs,
|
||||
);
|
||||
|
||||
return { sandbox, isReused: false };
|
||||
};
|
||||
+37
@@ -0,0 +1,37 @@
|
||||
import { type Sandbox } from '@e2b/code-interpreter';
|
||||
|
||||
import { SESSION_SANDBOX_METADATA_KEY } from 'src/engine/core-modules/code-interpreter/constants/session-sandbox-metadata-key.constant';
|
||||
|
||||
type SandboxApi = typeof Sandbox;
|
||||
|
||||
// Kills every sandbox tagged for a session — used when a conversation is deleted.
|
||||
export const releaseSessionSandboxes = async (
|
||||
sandboxApi: SandboxApi,
|
||||
apiKey: string,
|
||||
sessionId: string,
|
||||
): Promise<void> => {
|
||||
const paginator = sandboxApi.list({
|
||||
apiKey,
|
||||
query: {
|
||||
state: ['running', 'paused'],
|
||||
metadata: { [SESSION_SANDBOX_METADATA_KEY]: sessionId },
|
||||
},
|
||||
});
|
||||
|
||||
const sandboxes: Awaited<ReturnType<typeof paginator.nextItems>> = [];
|
||||
|
||||
while (paginator.hasNext) {
|
||||
sandboxes.push(...(await paginator.nextItems()));
|
||||
}
|
||||
|
||||
await Promise.all(
|
||||
sandboxes
|
||||
.filter(
|
||||
(sandbox) =>
|
||||
sandbox.metadata?.[SESSION_SANDBOX_METADATA_KEY] === sessionId,
|
||||
)
|
||||
.map((sandbox) =>
|
||||
sandboxApi.kill(sandbox.sandboxId, { apiKey }).catch(() => undefined),
|
||||
),
|
||||
);
|
||||
};
|
||||
+42
@@ -0,0 +1,42 @@
|
||||
import { type Sandbox } from '@e2b/code-interpreter';
|
||||
|
||||
import { isDefined } from 'twenty-shared/utils';
|
||||
|
||||
import { SESSION_SANDBOX_METADATA_KEY } from 'src/engine/core-modules/code-interpreter/constants/session-sandbox-metadata-key.constant';
|
||||
|
||||
type SandboxApi = typeof Sandbox;
|
||||
|
||||
// E2B retains paused sandboxes indefinitely, so abandoned sessions need an age bound.
|
||||
export const sweepExpiredSessionSandboxes = async (
|
||||
sandboxApi: SandboxApi,
|
||||
apiKey: string,
|
||||
maxAgeMs: number,
|
||||
): Promise<number> => {
|
||||
const paginator = sandboxApi.list({
|
||||
apiKey,
|
||||
query: { state: ['running', 'paused'] },
|
||||
});
|
||||
|
||||
const sandboxes: Awaited<ReturnType<typeof paginator.nextItems>> = [];
|
||||
|
||||
while (paginator.hasNext) {
|
||||
sandboxes.push(...(await paginator.nextItems()));
|
||||
}
|
||||
|
||||
const expiredBefore = Date.now() - maxAgeMs;
|
||||
|
||||
const expiredSandboxes = sandboxes.filter(
|
||||
(sandbox) =>
|
||||
isDefined(sandbox.metadata?.[SESSION_SANDBOX_METADATA_KEY]) &&
|
||||
// startedAt is typed as Date but comes back as an ISO string at runtime.
|
||||
new Date(sandbox.startedAt).getTime() < expiredBefore,
|
||||
);
|
||||
|
||||
await Promise.all(
|
||||
expiredSandboxes.map((sandbox) =>
|
||||
sandboxApi.kill(sandbox.sandboxId, { apiKey }).catch(() => undefined),
|
||||
),
|
||||
);
|
||||
|
||||
return expiredSandboxes.length;
|
||||
};
|
||||
Reference in New Issue
Block a user