feat(files): direct-to-storage upload endpoints with pending file lifecycle (#22449)

<img width="1484" height="404" alt="image"
src="https://github.com/user-attachments/assets/b2d363bf-d9e1-49fb-9811-8cc98041aa79"
/>


## Context

Uploading large files currently OOMs the server: every upload resolver
buffers the whole file in memory (`streamToBuffer`) before writing it to
storage. This PR is the first of a series introducing direct
client-to-storage uploads. It adds the server-side endpoints and driver
support only — it is non-breaking and nothing consumes the new flow yet.
Follow-up PRs will migrate the frontend upload paths, add a
stale-pending-file cleanup cron, and cap the legacy buffered resolvers.

## What it does

**New upload flow (initiate → PUT → confirm):**

- `createFileUpload(filename, size, fileFolder, fieldMetadataId?)`
validates the request (folder allowlist: `FilesField`/`Workflow`, max
size, extension-derived mime type), creates the file record in a new
`PENDING` status, and returns an upload target:
- **S3 with presign enabled** → a presigned PUT URL with
`Content-Type`/`Content-Length` pinned in the signature, so the client
uploads straight to the bucket;
- **local storage, or S3 without presign** → a token-authenticated
streaming endpoint on the server (`PUT /file-upload/:id?token=…`, new
`FILE_UPLOAD` JWT type) that pipes the request body to the storage
driver with constant memory usage and a declared-size cap.
- `completeFileUpload(fileId)` verifies the bytes actually landed in
storage (HEAD + size match against the declared size) and flips the
record to `UPLOADED`. Idempotent.

**Pending lifecycle safety:**

- New `status` column on `core.file` (`PENDING`/`UPLOADED`, default
`UPLOADED` so all existing rows and the legacy upload path are
unaffected) + fast instance command.
- Files are refused by the serving endpoints and by FILES-field sync
while `PENDING`.

**Driver support (both drivers):**

- `getPresignedUploadUrl` (S3: presigned PUT; local: `null` →
server-endpoint fallback)
- `writeFileStream` (local: `fs` pipeline with the existing
symlink/containment hardening, partial-file cleanup on error; S3:
`@aws-sdk/lib-storage` `Upload` for bounded-memory streaming)
- `getFileMetadata` (HEAD/stat for confirm-time verification)

## Tests

- `file-upload.service.spec.ts`: initiate validation (folder allowlist,
size), presigned vs fallback target, confirm verification (missing
object, size mismatch, happy path, idempotency)
- `local.driver.spec.ts`: `writeFileStream` (content, symlink rejection,
partial-file cleanup on stream error), `getFileMetadata`
- `s3.driver.spec.ts`: `getPresignedUploadUrl` (disabled → null, PUT
command with signed content-type/content-length)
- `direct-file-upload.integration-spec.ts`: full end-to-end flow against
the local driver (initiate → PUT → complete → download), plus error
paths (complete without upload, oversized PUT → 413, invalid token →
403, unsupported folder, size above max)

## Notes for reviewers

- The upload-size ceiling for direct uploads is
`settings.storage.maxDirectUploadFileSize` (1GB), separate from the 10MB
`maxFileSize` used for pictures.
- Since content can't be sniffed before it reaches storage, the mime
type is derived from the file extension (with the existing
`TWENTY_MIME_POLICY` override) and unknown extensions fall back to
`application/octet-stream`; the serving path already forces
`Content-Disposition: attachment` for anything not on the inline-safe
allowlist.
- Self-hosters using S3 presign will need a bucket CORS policy allowing
`PUT` from the frontend origin (config variable description updated).

https://claude.ai/code/session_015UH8KWmsB9zdYaog8MFG1d

---
_Generated by [Claude
Code](https://claude.ai/code/session_015UH8KWmsB9zdYaog8MFG1d)_

<!-- This is an auto-generated description by cubic. -->
<a
href="https://cubic.dev/pr/twentyhq/twenty/pull/22449?utm_source=github"
target="_blank" rel="noopener noreferrer"
data-no-image-dialog="true"><picture><source
media="(prefers-color-scheme: dark)"
srcset="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"><source
media="(prefers-color-scheme: light)"
srcset="https://www.cubic.dev/buttons/review-in-cubic-light.svg"><img
alt="Review in cubic"
src="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"></picture></a>
<!-- End of auto-generated description by cubic. -->
This commit is contained in:
martmull
2026-07-03 15:11:53 +02:00
committed by GitHub
parent 0db4ddd46e
commit 1a85b88d38
40 changed files with 2808 additions and 613 deletions
@@ -0,0 +1,8 @@
import { type CommonPropertiesJwtPayload } from 'src/engine/core-modules/auth/types/common-properties-jwt-payload.type';
import { JwtTokenTypeEnum } from 'src/engine/core-modules/auth/types/jwt-token-type.enum';
export type FileUploadTokenJwtPayload = CommonPropertiesJwtPayload & {
type: JwtTokenTypeEnum.FILE_UPLOAD;
workspaceId: string;
fileId: string;
};
@@ -5,6 +5,7 @@ import { type ApplicationRefreshTokenJwtPayload } from 'src/engine/core-modules/
import { type AppOAuthStateJwtPayload } from 'src/engine/core-modules/auth/types/app-oauth-state-jwt-payload.type';
import { type ApprovedAccessDomainJwtPayload } from 'src/engine/core-modules/auth/types/approved-access-domain-jwt-payload.type';
import { type FileTokenJwtPayload } from 'src/engine/core-modules/auth/types/file-token-jwt-payload.type';
import { type FileUploadTokenJwtPayload } from 'src/engine/core-modules/auth/types/file-upload-token-jwt-payload.type';
import { type FileTokenJwtPayloadLegacy } from 'src/engine/core-modules/auth/types/file-token-jwt-payload-legacy.type';
import { type LoginTokenJwtPayload } from 'src/engine/core-modules/auth/types/login-token-jwt-payload.type';
import { type PlaygroundTokenJwtPayload } from 'src/engine/core-modules/auth/types/playground-token-jwt-payload.type';
@@ -23,6 +24,7 @@ export type JwtPayload =
| RefreshTokenJwtPayload
| FileTokenJwtPayload
| FileTokenJwtPayloadLegacy
| FileUploadTokenJwtPayload
| AppOAuthStateJwtPayload
| ApprovedAccessDomainJwtPayload
| PlaygroundTokenJwtPayload;
@@ -4,6 +4,7 @@ export enum JwtTokenTypeEnum {
WORKSPACE_AGNOSTIC = 'WORKSPACE_AGNOSTIC',
LOGIN = 'LOGIN',
FILE = 'FILE',
FILE_UPLOAD = 'FILE_UPLOAD',
API_KEY = 'API_KEY',
REMOTE_SERVER = 'REMOTE_SERVER',
KEY_ENCRYPTION_KEY = 'KEY_ENCRYPTION_KEY',
@@ -1,6 +1,15 @@
import { mkdtemp, mkdir, readFile, rm, symlink, writeFile } from 'fs/promises';
import {
mkdtemp,
mkdir,
readFile,
rm,
stat,
symlink,
writeFile,
} from 'fs/promises';
import { tmpdir } from 'os';
import path from 'path';
import { Readable } from 'stream';
import { FileStorageExceptionCode } from 'src/engine/core-modules/file-storage/interfaces/file-storage-exception';
@@ -78,4 +87,108 @@ describe('LocalDriver security hardening', () => {
code: FileStorageExceptionCode.ACCESS_DENIED,
});
});
describe('writeFileStream', () => {
it('should write the streamed content to disk', async () => {
const storagePath = await createTempDirectory('local-driver-storage-');
const driver = new LocalDriver({ storagePath });
await driver.writeFileStream({
filePath: 'workspace/app/streamed.txt',
stream: Readable.from([
Buffer.from('streamed-'),
Buffer.from('content'),
]),
mimeType: 'text/plain',
});
await expect(
readFile(path.join(storagePath, 'workspace/app/streamed.txt'), 'utf8'),
).resolves.toBe('streamed-content');
});
it('should reject when target is a symlink', async () => {
const storagePath = await createTempDirectory('local-driver-storage-');
const outsidePath = await createTempDirectory('local-driver-outside-');
const outsideFilePath = path.join(outsidePath, 'outside.txt');
const symlinkFolderPath = path.join(storagePath, 'workspace', 'app');
const symlinkFilePath = path.join(symlinkFolderPath, 'target.txt');
await mkdir(symlinkFolderPath, { recursive: true });
await writeFile(outsideFilePath, 'outside');
await symlink(outsideFilePath, symlinkFilePath);
const driver = new LocalDriver({ storagePath });
await expect(
driver.writeFileStream({
filePath: 'workspace/app/target.txt',
stream: Readable.from([Buffer.from('new-content')]),
mimeType: undefined,
}),
).rejects.toMatchObject({
code: FileStorageExceptionCode.ACCESS_DENIED,
});
await expect(readFile(outsideFilePath, 'utf8')).resolves.toBe('outside');
});
it('should remove the partial file when the stream errors', async () => {
const storagePath = await createTempDirectory('local-driver-storage-');
const driver = new LocalDriver({ storagePath });
const failingStream = new Readable({
read() {
this.push(Buffer.from('partial'));
this.destroy(new Error('stream interrupted'));
},
});
await expect(
driver.writeFileStream({
filePath: 'workspace/app/partial.txt',
stream: failingStream,
mimeType: undefined,
}),
).rejects.toThrow('stream interrupted');
await expect(
stat(path.join(storagePath, 'workspace/app/partial.txt')),
).rejects.toMatchObject({ code: 'ENOENT' });
});
});
describe('getFileMetadata', () => {
it('should return the file size', async () => {
const storagePath = await createTempDirectory('local-driver-storage-');
const folderPath = path.join(storagePath, 'workspace', 'app');
await mkdir(folderPath, { recursive: true });
await writeFile(path.join(folderPath, 'file.txt'), '12345');
const driver = new LocalDriver({ storagePath });
await expect(
driver.getFileMetadata({ filePath: 'workspace/app/file.txt' }),
).resolves.toEqual({ size: 5 });
});
it('should return null when the file does not exist', async () => {
const storagePath = await createTempDirectory('local-driver-storage-');
const driver = new LocalDriver({ storagePath });
await expect(
driver.getFileMetadata({ filePath: 'workspace/app/missing.txt' }),
).resolves.toBeNull();
});
});
describe('getPresignedUploadUrl', () => {
it('should return null so callers fall back to the server endpoint', async () => {
const storagePath = await createTempDirectory('local-driver-storage-');
const driver = new LocalDriver({ storagePath });
await expect(driver.getPresignedUploadUrl()).resolves.toBeNull();
});
});
});
@@ -1,4 +1,4 @@
import { GetObjectCommand } from '@aws-sdk/client-s3';
import { GetObjectCommand, PutObjectCommand } from '@aws-sdk/client-s3';
import { getSignedUrl } from '@aws-sdk/s3-request-presigner';
import { S3Driver } from 'src/engine/core-modules/file-storage/drivers/s3.driver';
@@ -120,3 +120,66 @@ describe('S3Driver.getPresignedUrl', () => {
);
});
});
describe('S3Driver.getPresignedUploadUrl', () => {
afterEach(() => {
jest.clearAllMocks();
});
it('should return null when presigning is not enabled', async () => {
const driver = new S3Driver({
bucketName: 'test-bucket',
region: 'us-east-1',
});
const result = await driver.getPresignedUploadUrl({
filePath: 'some/file.pdf',
contentType: 'application/pdf',
contentLength: 1024,
});
expect(result).toBeNull();
expect(getSignedUrl).not.toHaveBeenCalled();
});
it('should presign a PUT with content-type and content-length in the signature', async () => {
(getSignedUrl as jest.Mock).mockResolvedValue(
'https://s3.us-east-1.amazonaws.com/test-bucket/some/file.pdf?X-Amz-Signature=abc',
);
const driver = new S3Driver({
bucketName: 'test-bucket',
region: 'us-east-1',
presignEnabled: true,
});
const result = await driver.getPresignedUploadUrl({
filePath: 'some/file.pdf',
contentType: 'application/pdf',
contentLength: 1024,
expiresInSeconds: 900,
});
expect(result).toBe(
'https://s3.us-east-1.amazonaws.com/test-bucket/some/file.pdf?X-Amz-Signature=abc',
);
expect(getSignedUrl).toHaveBeenCalledWith(
expect.anything(),
expect.any(PutObjectCommand),
{
expiresIn: 900,
signableHeaders: new Set(['content-type', 'content-length']),
},
);
const command = (getSignedUrl as jest.Mock).mock
.calls[0][1] as PutObjectCommand;
expect(command.input).toMatchObject({
Bucket: 'test-bucket',
Key: 'some/file.pdf',
ContentType: 'application/pdf',
ContentLength: 1024,
});
});
});
@@ -8,6 +8,9 @@ import { ValidatedStorageDriver } from 'src/engine/core-modules/file-storage/dri
const createMockDriver = (): jest.Mocked<StorageDriver> => ({
readFile: jest.fn().mockResolvedValue(Readable.from([])),
writeFile: jest.fn().mockResolvedValue(undefined),
writeFileStream: jest.fn().mockResolvedValue(undefined),
getFileMetadata: jest.fn().mockResolvedValue(null),
getPresignedUploadUrl: jest.fn().mockResolvedValue(null),
downloadFolder: jest.fn().mockResolvedValue(undefined),
uploadFolder: jest.fn().mockResolvedValue(undefined),
downloadFile: jest.fn().mockResolvedValue(undefined),
@@ -138,6 +141,52 @@ describe('ValidatedStorageDriver', () => {
responseContentDisposition: 'inline',
});
});
it('should delegate writeFileStream', async () => {
const params = {
filePath: 'folder/file.txt',
stream: Readable.from([Buffer.from('data')]),
mimeType: 'text/plain' as string | undefined,
};
await driver.writeFileStream(params);
expect(mockDelegate.writeFileStream).toHaveBeenCalledWith(params);
});
it('should delegate getFileMetadata', async () => {
mockDelegate.getFileMetadata.mockResolvedValue({ size: 1024 });
const result = await driver.getFileMetadata({
filePath: 'folder/file.txt',
});
expect(result).toEqual({ size: 1024 });
expect(mockDelegate.getFileMetadata).toHaveBeenCalledWith({
filePath: 'folder/file.txt',
});
});
it('should delegate getPresignedUploadUrl', async () => {
mockDelegate.getPresignedUploadUrl.mockResolvedValue(
'https://s3.example.com/signed-put',
);
const result = await driver.getPresignedUploadUrl({
filePath: 'folder/file.txt',
contentType: 'application/pdf',
contentLength: 1024,
expiresInSeconds: 900,
});
expect(result).toBe('https://s3.example.com/signed-put');
expect(mockDelegate.getPresignedUploadUrl).toHaveBeenCalledWith({
filePath: 'folder/file.txt',
contentType: 'application/pdf',
contentLength: 1024,
expiresInSeconds: 900,
});
});
});
describe('rejects path traversal attempts', () => {
@@ -165,6 +214,44 @@ describe('ValidatedStorageDriver', () => {
expect(mockDelegate.writeFile).not.toHaveBeenCalled();
});
it('should reject writeFileStream with traversal', async () => {
await expect(
driver.writeFileStream({
filePath: '../../evil',
stream: Readable.from([Buffer.from('x')]),
mimeType: undefined,
}),
).rejects.toMatchObject({
code: FileStorageExceptionCode.ACCESS_DENIED,
});
expect(mockDelegate.writeFileStream).not.toHaveBeenCalled();
});
it('should reject getFileMetadata with traversal', async () => {
await expect(
driver.getFileMetadata({ filePath: '../etc/passwd' }),
).rejects.toMatchObject({
code: FileStorageExceptionCode.ACCESS_DENIED,
});
expect(mockDelegate.getFileMetadata).not.toHaveBeenCalled();
});
it('should reject getPresignedUploadUrl with traversal', async () => {
await expect(
driver.getPresignedUploadUrl({
filePath: '../etc/passwd',
contentType: 'application/pdf',
contentLength: 1024,
}),
).rejects.toMatchObject({
code: FileStorageExceptionCode.ACCESS_DENIED,
});
expect(mockDelegate.getPresignedUploadUrl).not.toHaveBeenCalled();
});
it('should reject delete with traversal in folderPath', async () => {
await expect(
driver.delete({ folderPath: '../secret' }),
@@ -8,6 +8,16 @@ export interface StorageDriver {
mimeType: string | undefined;
}): Promise<void>;
writeFileStream(params: {
filePath: string;
stream: Readable;
mimeType: string | undefined;
}): Promise<void>;
getFileMetadata(params: {
filePath: string;
}): Promise<{ size: number } | null>;
downloadFolder(params: {
onStoragePath: string;
localPath: string;
@@ -42,4 +52,11 @@ export interface StorageDriver {
responseContentDisposition?: string;
responseCacheControl?: string;
}): Promise<string | null>;
getPresignedUploadUrl(params: {
filePath: string;
contentType: string;
contentLength: number;
expiresInSeconds?: number;
}): Promise<string | null>;
}
@@ -1,7 +1,13 @@
import { createReadStream, existsSync, realpathSync } from 'fs';
import {
createReadStream,
createWriteStream,
existsSync,
realpathSync,
} from 'fs';
import * as fs from 'fs/promises';
import path, { dirname, join } from 'path';
import { type Readable } from 'stream';
import { pipeline } from 'stream/promises';
import { type StorageDriver } from 'src/engine/core-modules/file-storage/drivers/interfaces/storage-driver.interface';
import {
@@ -64,18 +70,18 @@ export class LocalDriver implements StorageDriver {
}
}
async writeFile(params: {
filePath: string;
sourceFile: Buffer | Uint8Array | string;
mimeType: string | undefined;
}): Promise<void> {
const filePath = path.resolve(this.options.storagePath, params.filePath);
const folderPath = dirname(filePath);
// Resolves the on-disk path for a write, creating the parent folder and
// enforcing storage containment + symlink rejection.
private async resolveWritableRealPathOrThrow(
filePath: string,
): Promise<string> {
const resolvedPath = path.resolve(this.options.storagePath, filePath);
const folderPath = dirname(resolvedPath);
await this.createFolder(folderPath);
const realFolderPath = realpathSync(folderPath);
const realFilePath = path.join(realFolderPath, path.basename(filePath));
const realFilePath = path.join(realFolderPath, path.basename(resolvedPath));
this.assertRealPathIsWithinStorage(realFilePath);
@@ -94,9 +100,67 @@ export class LocalDriver implements StorageDriver {
}
}
return realFilePath;
}
async writeFile(params: {
filePath: string;
sourceFile: Buffer | Uint8Array | string;
mimeType: string | undefined;
}): Promise<void> {
const realFilePath = await this.resolveWritableRealPathOrThrow(
params.filePath,
);
await fs.writeFile(realFilePath, params.sourceFile);
}
async writeFileStream(params: {
filePath: string;
stream: Readable;
mimeType: string | undefined;
}): Promise<void> {
const realFilePath = await this.resolveWritableRealPathOrThrow(
params.filePath,
);
try {
await pipeline(params.stream, createWriteStream(realFilePath));
} catch (error) {
// Remove the partial file so a failed upload can be retried cleanly
await fs.rm(realFilePath, { force: true });
throw error;
}
}
async getFileMetadata(params: {
filePath: string;
}): Promise<{ size: number } | null> {
const joinedPath = join(this.options.storagePath, params.filePath);
let filePath: string;
try {
filePath = realpathSync(path.resolve(joinedPath));
} catch {
return null;
}
this.assertRealPathIsWithinStorage(filePath);
try {
const stats = await fs.stat(filePath);
return { size: stats.size };
} catch (error) {
if ((error as NodeJS.ErrnoException).code === 'ENOENT') {
return null;
}
throw error;
}
}
async downloadFile(params: {
onStoragePath: string;
localPath: string;
@@ -304,6 +368,12 @@ export class LocalDriver implements StorageDriver {
return null;
}
// Local storage has no external endpoint to upload to: the caller falls
// back to the server-side streaming upload endpoint.
async getPresignedUploadUrl(): Promise<string | null> {
return null;
}
async checkFolderExists(params: { folderPath: string }): Promise<boolean> {
const folderFullPath = path.resolve(
this.options.storagePath,
@@ -20,6 +20,7 @@ import {
S3,
type S3ClientConfig,
} from '@aws-sdk/client-s3';
import { Upload } from '@aws-sdk/lib-storage';
import { getSignedUrl } from '@aws-sdk/s3-request-presigner';
import { isDefined } from 'twenty-shared/utils';
@@ -112,6 +113,47 @@ export class S3Driver implements StorageDriver {
await this.s3Client.send(command);
}
async writeFileStream(params: {
filePath: string;
stream: Readable;
mimeType: string | undefined;
}): Promise<void> {
// Upload streams the body with bounded memory (multipart under the hood),
// unlike PutObjectCommand which requires the whole payload upfront.
const upload = new Upload({
client: this.s3Client,
params: {
Bucket: this.bucketName,
Key: params.filePath,
Body: params.stream,
ContentType: params.mimeType,
},
});
await upload.done();
}
async getFileMetadata(params: {
filePath: string;
}): Promise<{ size: number } | null> {
try {
const head = await this.s3Client.send(
new HeadObjectCommand({
Bucket: this.bucketName,
Key: params.filePath,
}),
);
return { size: head.ContentLength ?? 0 };
} catch (error) {
if (error instanceof NotFound) {
return null;
}
throw error;
}
}
private async createFolder(path: string) {
return fs.mkdirSync(path, { recursive: true });
}
@@ -404,6 +446,31 @@ export class S3Driver implements StorageDriver {
});
}
async getPresignedUploadUrl(params: {
filePath: string;
contentType: string;
contentLength: number;
expiresInSeconds?: number;
}): Promise<string | null> {
if (!this.presignClient) {
return null;
}
const command = new PutObjectCommand({
Bucket: this.bucketName,
Key: params.filePath,
ContentType: params.contentType,
ContentLength: params.contentLength,
});
// Content-Type and Content-Length are part of the signature so the client
// cannot upload a payload of a different type or size than declared.
return getSignedUrl(this.presignClient, command, {
expiresIn: params.expiresInSeconds ?? 900,
signableHeaders: new Set(['content-type', 'content-length']),
});
}
async checkBucketExists(args: HeadBucketCommandInput) {
try {
await this.s3Client.headBucket(args);
@@ -23,6 +23,24 @@ export class ValidatedStorageDriver implements StorageDriver {
return this.delegate.writeFile(params);
}
async writeFileStream(params: {
filePath: string;
stream: Readable;
mimeType: string | undefined;
}): Promise<void> {
assertStoragePathIsSafe(params.filePath);
return this.delegate.writeFileStream(params);
}
async getFileMetadata(params: {
filePath: string;
}): Promise<{ size: number } | null> {
assertStoragePathIsSafe(params.filePath);
return this.delegate.getFileMetadata(params);
}
async downloadFolder(params: {
onStoragePath: string;
localPath: string;
@@ -111,6 +129,17 @@ export class ValidatedStorageDriver implements StorageDriver {
return this.delegate.getPresignedUrl(params);
}
async getPresignedUploadUrl(params: {
filePath: string;
contentType: string;
contentLength: number;
expiresInSeconds?: number;
}): Promise<string | null> {
assertStoragePathIsSafe(params.filePath);
return this.delegate.getPresignedUploadUrl(params);
}
async checkFileExists(params: { filePath: string }): Promise<boolean> {
assertStoragePathIsSafe(params.filePath);
@@ -19,6 +19,7 @@ import { validateFolderPath } from 'src/engine/core-modules/file-storage/utils/v
import { validateStoragePathIsWithinWorkspaceOrThrow } from 'src/engine/core-modules/file-storage/utils/validate-storage-path-is-within-workspace-or-throw.util';
import { FileEntity } from 'src/engine/core-modules/file/entities/file.entity';
import { FileSettings } from 'src/engine/core-modules/file/types/file-settings.types';
import { FILE_STATUS } from 'src/engine/core-modules/file/types/file-status.types';
import { removeFileFolderFromFileEntityPath } from 'src/engine/core-modules/file/utils/remove-file-folder-from-file-entity-path.utils';
import { InjectWorkspaceScopedRepository } from 'src/engine/twenty-orm/workspace-scoped-repository/inject-workspace-scoped-repository.decorator';
import { WorkspaceScopedRepository } from 'src/engine/twenty-orm/workspace-scoped-repository/workspace-scoped-repository';
@@ -187,6 +188,99 @@ export class FileStorageService {
);
}
// Creates the file record ahead of a direct client upload. The bytes are
// not in storage yet: the record stays PENDING until the upload is
// confirmed (completeFileUpload) or reaped by the cleanup cron.
async createPendingFile({
fileFolder,
applicationUniversalIdentifier,
workspaceId,
resourcePath,
fileId,
size,
mimeType,
settings,
}: ResourceIdentifier & {
fileId: string;
size: number;
mimeType: string;
settings: FileSettings;
}): Promise<FileEntity> {
const application = await this.applicationRepository.findOneOrFail({
where: {
universalIdentifier: applicationUniversalIdentifier,
workspaceId,
},
});
const { filePath } = this.validateAndBuildFileStoragePathOrThrow({
workspaceId,
applicationUniversalIdentifier,
fileFolder,
resourcePath,
});
return this.fileRepository.upsertAndReturnOne(
workspaceId,
{
path: filePath,
applicationId: application.id,
id: fileId,
mimeType,
size,
settings,
status: FILE_STATUS.PENDING,
},
['path', 'workspaceId', 'applicationId'],
);
}
async writeFileStream(
params: ResourceIdentifier & {
stream: Readable;
mimeType: string | undefined;
},
): Promise<void> {
const driver = this.fileStorageDriverFactory.getCurrentDriver();
const { onStorageFilePath } =
this.validateAndBuildFileStoragePathOrThrow(params);
return driver.writeFileStream({
filePath: onStorageFilePath,
stream: params.stream,
mimeType: params.mimeType,
});
}
async getFileMetadata(
params: ResourceIdentifier,
): Promise<{ size: number } | null> {
const driver = this.fileStorageDriverFactory.getCurrentDriver();
const { onStorageFilePath } =
this.validateAndBuildFileStoragePathOrThrow(params);
return driver.getFileMetadata({ filePath: onStorageFilePath });
}
async getPresignedUploadUrl(
params: ResourceIdentifier & {
contentType: string;
contentLength: number;
expiresInSeconds?: number;
},
): Promise<string | null> {
const driver = this.fileStorageDriverFactory.getCurrentDriver();
const { onStorageFilePath } =
this.validateAndBuildFileStoragePathOrThrow(params);
return driver.getPresignedUploadUrl({
filePath: onStorageFilePath,
contentType: params.contentType,
contentLength: params.contentLength,
expiresInSeconds: params.expiresInSeconds,
});
}
async getPresignedUrl(
params: ResourceIdentifier & {
expiresInSeconds?: number;
@@ -12,12 +12,19 @@ import {
UpdateDateColumn,
} from 'typeorm';
import { ADD_STATUS_TO_FILE_UPGRADE_COMMAND_NAME } from 'src/database/commands/upgrade-version-command/2-19/add-status-to-file-upgrade-command-name.constant';
import { ApplicationEntity } from 'src/engine/core-modules/application/application.entity';
import { FileSettings } from 'src/engine/core-modules/file/types/file-settings.types';
import {
FILE_STATUS,
FileStatus,
} from 'src/engine/core-modules/file/types/file-status.types';
import { WasIntroducedInUpgrade } from 'src/engine/core-modules/upgrade/decorators/was-introduced-in-upgrade.decorator';
import { WorkspaceRelatedEntity } from 'src/engine/workspace-manager/types/workspace-related-entity';
@Entity('file')
@Index('IDX_FILE_WORKSPACE_ID', ['workspaceId'])
@Index('IDX_FILE_STATUS', ['status'])
@Unique('IDX_APPLICATION_PATH_WORKSPACE_ID_APPLICATION_ID_UNIQUE', [
'workspaceId',
'applicationId',
@@ -63,4 +70,15 @@ export class FileEntity extends WorkspaceRelatedEntity {
default: 'application/octet-stream',
})
mimeType: string;
@WasIntroducedInUpgrade({
upgradeCommandName: ADD_STATUS_TO_FILE_UPGRADE_COMMAND_NAME,
})
@Column({
type: 'enum',
enum: Object.values(FILE_STATUS),
nullable: false,
default: FILE_STATUS.UPLOADED,
})
status: FileStatus;
}
@@ -0,0 +1,44 @@
import {
Controller,
Param,
Put,
Req,
Res,
UseFilters,
UseGuards,
} from '@nestjs/common';
import { Request, Response } from 'express';
import { FileUploadApiExceptionFilter } from 'src/engine/core-modules/file/file-upload/filters/file-upload-api-exception.filter';
import { FileUploadTokenGuard } from 'src/engine/core-modules/file/file-upload/guards/file-upload-token.guard';
import { FileUploadService } from 'src/engine/core-modules/file/file-upload/services/file-upload.service';
import { NoPermissionGuard } from 'src/engine/guards/no-permission.guard';
@Controller()
@UseFilters(FileUploadApiExceptionFilter)
export class FileUploadController {
constructor(private readonly fileUploadService: FileUploadService) {}
// Streaming target for direct uploads when storage has no presigned upload
// support (local driver, or S3 without presign enabled). The body is piped
// to the storage driver without ever being buffered in memory.
@Put('file-upload/:id')
@UseGuards(FileUploadTokenGuard, NoPermissionGuard)
async uploadFileById(
@Req() req: Request,
@Res() res: Response,
@Param('id') fileId: string,
) {
// oxlint-disable-next-line typescript/no-explicit-any
const workspaceId = (req as any)?.workspaceId;
await this.fileUploadService.receiveFileStream({
workspaceId,
fileId,
stream: req,
});
res.status(204).send();
}
}
@@ -0,0 +1,19 @@
import { Field, ObjectType } from '@nestjs/graphql';
import { UUIDScalarType } from 'src/engine/api/graphql/workspace-schema-builder/graphql-types/scalars';
@ObjectType('FileUploadTarget')
export class FileUploadTargetDTO {
@Field(() => UUIDScalarType)
fileId: string;
@Field()
uploadUrl: string;
// Content-Type header the client must send when uploading to uploadUrl
@Field()
contentType: string;
@Field(() => Date, { nullable: false })
expiresAt: Date;
}
@@ -0,0 +1,23 @@
import { type MessageDescriptor } from '@lingui/core';
import { CustomException } from 'src/utils/custom-exception';
export enum FileUploadExceptionCode {
BAD_REQUEST = 'BAD_REQUEST',
FILE_NOT_FOUND = 'FILE_NOT_FOUND',
FILE_NOT_UPLOADED = 'FILE_NOT_UPLOADED',
FILE_SIZE_MISMATCH = 'FILE_SIZE_MISMATCH',
FILE_TOO_LARGE = 'FILE_TOO_LARGE',
}
export class FileUploadException extends CustomException<FileUploadExceptionCode> {
constructor(
message: string,
code: FileUploadExceptionCode,
{ userFriendlyMessage }: { userFriendlyMessage: MessageDescriptor },
) {
super(message, code, {
userFriendlyMessage,
});
}
}
@@ -0,0 +1,40 @@
import { Module } from '@nestjs/common';
import { TypeOrmModule } from '@nestjs/typeorm';
import { ApplicationEntity } from 'src/engine/core-modules/application/application.entity';
import { ApplicationModule } from 'src/engine/core-modules/application/application.module';
import { FileStorageModule } from 'src/engine/core-modules/file-storage/file-storage.module';
import { FileEntity } from 'src/engine/core-modules/file/entities/file.entity';
import { FileUploadController } from 'src/engine/core-modules/file/file-upload/controllers/file-upload.controller';
import { FileUploadTokenGuard } from 'src/engine/core-modules/file/file-upload/guards/file-upload-token.guard';
import { FileUploadResolver } from 'src/engine/core-modules/file/file-upload/resolvers/file-upload.resolver';
import { FileUploadService } from 'src/engine/core-modules/file/file-upload/services/file-upload.service';
import { FileUrlModule } from 'src/engine/core-modules/file/file-url/file-url.module';
import { JwtModule } from 'src/engine/core-modules/jwt/jwt.module';
import { FieldMetadataEntity } from 'src/engine/metadata-modules/field-metadata/field-metadata.entity';
import { PermissionsModule } from 'src/engine/metadata-modules/permissions/permissions.module';
import { provideWorkspaceScopedRepository } from 'src/engine/twenty-orm/workspace-scoped-repository/provide-workspace-scoped-repository';
@Module({
imports: [
JwtModule,
TypeOrmModule.forFeature([
FileEntity,
ApplicationEntity,
FieldMetadataEntity,
]),
PermissionsModule,
FileStorageModule,
FileUrlModule,
ApplicationModule,
],
providers: [
FileUploadService,
FileUploadResolver,
FileUploadTokenGuard,
provideWorkspaceScopedRepository(FileEntity),
],
exports: [FileUploadService],
controllers: [FileUploadController],
})
export class FileUploadModule {}
@@ -0,0 +1,54 @@
import {
type ArgumentsHost,
Catch,
type ExceptionFilter,
} from '@nestjs/common';
import { type Response } from 'express';
import { HttpExceptionHandlerService } from 'src/engine/core-modules/exception-handler/http-exception-handler.service';
import {
FileUploadException,
FileUploadExceptionCode,
} from 'src/engine/core-modules/file/file-upload/file-upload.exception';
@Catch(FileUploadException)
export class FileUploadApiExceptionFilter implements ExceptionFilter {
constructor(
private readonly httpExceptionHandlerService: HttpExceptionHandlerService,
) {}
catch(exception: FileUploadException, host: ArgumentsHost) {
const ctx = host.switchToHttp();
const response = ctx.getResponse<Response>();
switch (exception.code) {
case FileUploadExceptionCode.FILE_NOT_FOUND:
return this.httpExceptionHandlerService.handleError(
exception,
response,
404,
);
case FileUploadExceptionCode.FILE_TOO_LARGE:
return this.httpExceptionHandlerService.handleError(
exception,
response,
413,
);
case FileUploadExceptionCode.BAD_REQUEST:
case FileUploadExceptionCode.FILE_NOT_UPLOADED:
case FileUploadExceptionCode.FILE_SIZE_MISMATCH:
return this.httpExceptionHandlerService.handleError(
exception,
response,
400,
);
default:
return this.httpExceptionHandlerService.handleError(
exception,
response,
500,
);
}
}
}
@@ -0,0 +1,42 @@
import { CanActivate, ExecutionContext, Injectable } from '@nestjs/common';
import { FileUploadTokenJwtPayload } from 'src/engine/core-modules/auth/types/file-upload-token-jwt-payload.type';
import { JwtTokenTypeEnum } from 'src/engine/core-modules/auth/types/jwt-token-type.enum';
import { JwtWrapperService } from 'src/engine/core-modules/jwt/services/jwt-wrapper.service';
@Injectable()
export class FileUploadTokenGuard implements CanActivate {
constructor(private readonly jwtWrapperService: JwtWrapperService) {}
async canActivate(context: ExecutionContext): Promise<boolean> {
const request = context.switchToHttp().getRequest();
const fileId = request.params.id;
const uploadToken = request.query.token;
if (!uploadToken) {
return false;
}
let payload: FileUploadTokenJwtPayload;
try {
payload = await this.jwtWrapperService.verifyJwtToken(uploadToken);
} catch {
return false;
}
// A FILE (download) token also carries workspaceId + fileId: reject
// anything that is not explicitly an upload token.
if (payload.type !== JwtTokenTypeEnum.FILE_UPLOAD) {
return false;
}
if (!payload.workspaceId || payload.fileId !== fileId) {
return false;
}
request.workspaceId = payload.workspaceId;
return true;
}
}
@@ -0,0 +1,68 @@
import { UseFilters, UseGuards, UsePipes } from '@nestjs/common';
import { Args, Mutation } from '@nestjs/graphql';
import { PermissionFlagType } from 'twenty-shared/constants';
import { FileFolder } from 'twenty-shared/types';
import { MetadataResolver } from 'src/engine/api/graphql/graphql-config/decorators/metadata-resolver.decorator';
import { FileWithSignedUrlDTO } from 'src/engine/core-modules/file/dtos/file-with-sign-url.dto';
import { FileUploadTargetDTO } from 'src/engine/core-modules/file/file-upload/dtos/file-upload-target.dto';
import { FileUploadService } from 'src/engine/core-modules/file/file-upload/services/file-upload.service';
import { PreventNestToAutoLogGraphqlErrorsFilter } from 'src/engine/core-modules/graphql/filters/prevent-nest-to-auto-log-graphql-errors.filter';
import { ResolverValidationPipe } from 'src/engine/core-modules/graphql/pipes/resolver-validation.pipe';
import { WorkspaceEntity } from 'src/engine/core-modules/workspace/workspace.entity';
import { AuthWorkspace } from 'src/engine/decorators/auth/auth-workspace.decorator';
import { SettingsPermissionGuard } from 'src/engine/guards/settings-permission.guard';
import { WorkspaceAuthGuard } from 'src/engine/guards/workspace-auth.guard';
@UseGuards(WorkspaceAuthGuard)
@UsePipes(ResolverValidationPipe)
@UseFilters(PreventNestToAutoLogGraphqlErrorsFilter)
@MetadataResolver()
export class FileUploadResolver {
constructor(private readonly fileUploadService: FileUploadService) {}
@Mutation(() => FileUploadTargetDTO)
@UseGuards(SettingsPermissionGuard(PermissionFlagType.UPLOAD_FILE))
async createFileUpload(
@AuthWorkspace()
{ id: workspaceId }: WorkspaceEntity,
@Args({ name: 'filename', type: () => String })
filename: string,
@Args({ name: 'size', type: () => Number })
size: number,
@Args({ name: 'fileFolder', type: () => FileFolder })
fileFolder: FileFolder,
@Args({ name: 'fieldMetadataId', type: () => String, nullable: true })
fieldMetadataId?: string,
@Args({
name: 'fieldMetadataUniversalIdentifier',
type: () => String,
nullable: true,
})
fieldMetadataUniversalIdentifier?: string,
): Promise<FileUploadTargetDTO> {
return await this.fileUploadService.createFileUpload({
workspaceId,
filename,
size,
fileFolder,
fieldMetadataId,
fieldMetadataUniversalIdentifier,
});
}
@Mutation(() => FileWithSignedUrlDTO)
@UseGuards(SettingsPermissionGuard(PermissionFlagType.UPLOAD_FILE))
async completeFileUpload(
@AuthWorkspace()
{ id: workspaceId }: WorkspaceEntity,
@Args({ name: 'fileId', type: () => String })
fileId: string,
): Promise<FileWithSignedUrlDTO> {
return await this.fileUploadService.completeFileUpload({
workspaceId,
fileId,
});
}
}
@@ -0,0 +1,309 @@
import { Test, type TestingModule } from '@nestjs/testing';
import { getRepositoryToken } from '@nestjs/typeorm';
import { FileFolder } from 'twenty-shared/types';
import { ApplicationEntity } from 'src/engine/core-modules/application/application.entity';
import { ApplicationService } from 'src/engine/core-modules/application/application.service';
import { FileStorageService } from 'src/engine/core-modules/file-storage/file-storage.service';
import { FileEntity } from 'src/engine/core-modules/file/entities/file.entity';
import {
FileUploadException,
FileUploadExceptionCode,
} from 'src/engine/core-modules/file/file-upload/file-upload.exception';
import { FileUploadService } from 'src/engine/core-modules/file/file-upload/services/file-upload.service';
import { FileUrlService } from 'src/engine/core-modules/file/file-url/file-url.service';
import { FILE_STATUS } from 'src/engine/core-modules/file/types/file-status.types';
import { JwtWrapperService } from 'src/engine/core-modules/jwt/services/jwt-wrapper.service';
import { TwentyConfigService } from 'src/engine/core-modules/twenty-config/twenty-config.service';
import { FieldMetadataEntity } from 'src/engine/metadata-modules/field-metadata/field-metadata.entity';
import { getWorkspaceScopedRepositoryToken } from 'src/engine/twenty-orm/workspace-scoped-repository/get-workspace-scoped-repository-token.util';
jest.mock('uuid', () => ({
v4: jest.fn(() => 'mocked-file-id'),
}));
describe('FileUploadService', () => {
let service: FileUploadService;
const fileStorageService = {
createPendingFile: jest.fn(),
getPresignedUploadUrl: jest.fn(),
getFileMetadata: jest.fn(),
writeFileStream: jest.fn(),
};
const fileUrlService = {
signFileByIdUrl: jest.fn().mockResolvedValue('https://signed-url'),
};
const jwtWrapperService = {
signAsyncOrThrow: jest.fn().mockResolvedValue('upload-token'),
};
const twentyConfigService = {
get: jest.fn((key: string) => {
if (key === 'STORAGE_S3_PRESIGNED_URL_EXPIRES_IN') {
return 900;
}
if (key === 'SERVER_URL') {
return 'https://server.tld';
}
return undefined;
}),
};
const applicationService = {
findWorkspaceTwentyStandardAndCustomApplicationOrThrow: jest
.fn()
.mockResolvedValue({
workspaceCustomFlatApplication: {
universalIdentifier: 'custom-app-uid',
},
}),
};
const applicationRepository = {
findOneOrFail: jest.fn().mockResolvedValue({
id: 'application-id',
universalIdentifier: 'application-uid',
}),
};
const fieldMetadataRepository = {
findOneOrFail: jest.fn().mockResolvedValue({
applicationId: 'application-id',
universalIdentifier: 'field-metadata-uid',
}),
};
const fileRepository = {
findOne: jest.fn(),
update: jest.fn(),
};
beforeEach(async () => {
jest.clearAllMocks();
const module: TestingModule = await Test.createTestingModule({
providers: [
FileUploadService,
{ provide: FileStorageService, useValue: fileStorageService },
{ provide: FileUrlService, useValue: fileUrlService },
{ provide: JwtWrapperService, useValue: jwtWrapperService },
{ provide: TwentyConfigService, useValue: twentyConfigService },
{ provide: ApplicationService, useValue: applicationService },
{
provide: getRepositoryToken(ApplicationEntity),
useValue: applicationRepository,
},
{
provide: getRepositoryToken(FieldMetadataEntity),
useValue: fieldMetadataRepository,
},
{
provide: getWorkspaceScopedRepositoryToken(FileEntity),
useValue: fileRepository,
},
],
}).compile();
service = module.get<FileUploadService>(FileUploadService);
});
describe('createFileUpload', () => {
it('should reject file folders without direct upload support', async () => {
await expect(
service.createFileUpload({
workspaceId: 'workspace-id',
filename: 'document.pdf',
size: 1024,
fileFolder: FileFolder.CorePicture,
}),
).rejects.toThrow(FileUploadException);
});
it('should reject an invalid declared size', async () => {
await expect(
service.createFileUpload({
workspaceId: 'workspace-id',
filename: 'document.pdf',
size: 0,
fileFolder: FileFolder.FilesField,
fieldMetadataId: 'field-metadata-id',
}),
).rejects.toThrow(FileUploadException);
});
it('should create a PENDING file and return the presigned url when storage supports it', async () => {
fileStorageService.getPresignedUploadUrl.mockResolvedValueOnce(
'https://bucket/presigned-put',
);
const result = await service.createFileUpload({
workspaceId: 'workspace-id',
filename: 'document.pdf',
size: 1024,
fileFolder: FileFolder.FilesField,
fieldMetadataId: 'field-metadata-id',
});
expect(fileStorageService.createPendingFile).toHaveBeenCalledWith(
expect.objectContaining({
fileId: 'mocked-file-id',
size: 1024,
mimeType: 'application/pdf',
resourcePath: 'field-metadata-uid/mocked-file-id.pdf',
settings: { isTemporaryFile: true, toDelete: false },
}),
);
expect(result.uploadUrl).toBe('https://bucket/presigned-put');
expect(result.contentType).toBe('application/pdf');
expect(result.fileId).toBe('mocked-file-id');
});
it('should fall back to the server streaming endpoint when presign is unavailable', async () => {
fileStorageService.getPresignedUploadUrl.mockResolvedValueOnce(null);
const result = await service.createFileUpload({
workspaceId: 'workspace-id',
filename: 'archive.zip',
size: 2048,
fileFolder: FileFolder.Workflow,
});
expect(jwtWrapperService.signAsyncOrThrow).toHaveBeenCalledWith(
expect.objectContaining({
workspaceId: 'workspace-id',
fileId: 'mocked-file-id',
}),
{ expiresIn: 900 },
);
expect(result.uploadUrl).toBe(
'https://server.tld/file-upload/mocked-file-id?token=upload-token',
);
expect(result.contentType).toBe('application/octet-stream');
});
});
describe('completeFileUpload', () => {
const pendingFile = {
id: 'file-id',
path: 'files-field/field-metadata-uid/file-id.pdf',
size: 1024,
applicationId: 'application-id',
mimeType: 'application/pdf',
status: FILE_STATUS.PENDING,
settings: { isTemporaryFile: true, toDelete: false },
createdAt: new Date(),
};
it('should throw when the file record does not exist', async () => {
fileRepository.findOne.mockResolvedValueOnce(null);
await expect(
service.completeFileUpload({
workspaceId: 'workspace-id',
fileId: 'file-id',
}),
).rejects.toThrow(FileUploadException);
});
it('should throw when the bytes are not in storage yet', async () => {
fileRepository.findOne.mockResolvedValueOnce(pendingFile);
fileStorageService.getFileMetadata.mockResolvedValueOnce(null);
await expect(
service.completeFileUpload({
workspaceId: 'workspace-id',
fileId: 'file-id',
}),
).rejects.toMatchObject({
code: FileUploadExceptionCode.FILE_NOT_UPLOADED,
});
expect(fileRepository.update).not.toHaveBeenCalled();
});
it('should throw when the stored size does not match the declared size', async () => {
fileRepository.findOne.mockResolvedValueOnce(pendingFile);
fileStorageService.getFileMetadata.mockResolvedValueOnce({ size: 999 });
await expect(
service.completeFileUpload({
workspaceId: 'workspace-id',
fileId: 'file-id',
}),
).rejects.toMatchObject({
code: FileUploadExceptionCode.FILE_SIZE_MISMATCH,
});
expect(fileRepository.update).not.toHaveBeenCalled();
});
it('should flip the file to UPLOADED when the stored size matches', async () => {
fileRepository.findOne.mockResolvedValueOnce(pendingFile);
fileStorageService.getFileMetadata.mockResolvedValueOnce({ size: 1024 });
const result = await service.completeFileUpload({
workspaceId: 'workspace-id',
fileId: 'file-id',
});
expect(fileRepository.update).toHaveBeenCalledWith(
'workspace-id',
{ id: 'file-id' },
{ status: FILE_STATUS.UPLOADED },
);
expect(result.url).toBe('https://signed-url');
});
it('should be idempotent when the file is already UPLOADED', async () => {
fileRepository.findOne.mockResolvedValueOnce({
...pendingFile,
status: FILE_STATUS.UPLOADED,
});
const result = await service.completeFileUpload({
workspaceId: 'workspace-id',
fileId: 'file-id',
});
expect(fileStorageService.getFileMetadata).not.toHaveBeenCalled();
expect(fileRepository.update).not.toHaveBeenCalled();
expect(result.url).toBe('https://signed-url');
});
it('should refuse confirming files that already left the upload flow', async () => {
fileRepository.findOne.mockResolvedValueOnce({
...pendingFile,
status: FILE_STATUS.UPLOADED,
settings: { isTemporaryFile: false, toDelete: false },
});
await expect(
service.completeFileUpload({
workspaceId: 'workspace-id',
fileId: 'file-id',
}),
).rejects.toMatchObject({
code: FileUploadExceptionCode.BAD_REQUEST,
});
});
it('should refuse files outside direct-upload folders', async () => {
fileRepository.findOne.mockResolvedValueOnce({
...pendingFile,
path: 'core-picture/file-id.png',
});
await expect(
service.completeFileUpload({
workspaceId: 'workspace-id',
fileId: 'file-id',
}),
).rejects.toMatchObject({
code: FileUploadExceptionCode.FILE_NOT_FOUND,
});
});
});
});
@@ -0,0 +1,511 @@
import { Injectable } from '@nestjs/common';
import { InjectRepository } from '@nestjs/typeorm';
import { type Readable, Transform } from 'stream';
import { pipeline } from 'stream/promises';
import { msg } from '@lingui/core/macro';
import { isNonEmptyString } from '@sniptt/guards';
import bytes from 'bytes';
import { lookup } from 'mrmime';
import { FileFolder } from 'twenty-shared/types';
import { isDefined } from 'twenty-shared/utils';
import { Repository } from 'typeorm';
import { v4 } from 'uuid';
import { settings } from 'src/engine/constants/settings';
import { ApplicationEntity } from 'src/engine/core-modules/application/application.entity';
import { ApplicationService } from 'src/engine/core-modules/application/application.service';
import { FileUploadTokenJwtPayload } from 'src/engine/core-modules/auth/types/file-upload-token-jwt-payload.type';
import { JwtTokenTypeEnum } from 'src/engine/core-modules/auth/types/jwt-token-type.enum';
import { FileStorageService } from 'src/engine/core-modules/file-storage/file-storage.service';
import { TWENTY_MIME_POLICY } from 'src/engine/core-modules/file/constants/twenty-mime-policy.constant';
import { FileWithSignedUrlDTO } from 'src/engine/core-modules/file/dtos/file-with-sign-url.dto';
import { FileEntity } from 'src/engine/core-modules/file/entities/file.entity';
import { FileUploadTargetDTO } from 'src/engine/core-modules/file/file-upload/dtos/file-upload-target.dto';
import {
FileUploadException,
FileUploadExceptionCode,
} from 'src/engine/core-modules/file/file-upload/file-upload.exception';
import { FileUrlService } from 'src/engine/core-modules/file/file-url/file-url.service';
import { FILE_STATUS } from 'src/engine/core-modules/file/types/file-status.types';
import { buildFileInfo } from 'src/engine/core-modules/file/utils/build-file-info.utils';
import { removeFileFolderFromFileEntityPath } from 'src/engine/core-modules/file/utils/remove-file-folder-from-file-entity-path.utils';
import { JwtWrapperService } from 'src/engine/core-modules/jwt/services/jwt-wrapper.service';
import { TwentyConfigService } from 'src/engine/core-modules/twenty-config/twenty-config.service';
import { FieldMetadataEntity } from 'src/engine/metadata-modules/field-metadata/field-metadata.entity';
import { InjectWorkspaceScopedRepository } from 'src/engine/twenty-orm/workspace-scoped-repository/inject-workspace-scoped-repository.decorator';
import { WorkspaceScopedRepository } from 'src/engine/twenty-orm/workspace-scoped-repository/workspace-scoped-repository';
export const DIRECT_UPLOAD_FILE_FOLDERS = [
FileFolder.FilesField,
FileFolder.Workflow,
] as const;
@Injectable()
export class FileUploadService {
constructor(
private readonly fileStorageService: FileStorageService,
private readonly fileUrlService: FileUrlService,
private readonly jwtWrapperService: JwtWrapperService,
private readonly twentyConfigService: TwentyConfigService,
private readonly applicationService: ApplicationService,
@InjectRepository(ApplicationEntity)
private readonly applicationRepository: Repository<ApplicationEntity>,
@InjectRepository(FieldMetadataEntity)
private readonly fieldMetadataRepository: Repository<FieldMetadataEntity>,
@InjectWorkspaceScopedRepository(FileEntity)
private readonly fileRepository: WorkspaceScopedRepository<FileEntity>,
) {}
async createFileUpload({
workspaceId,
filename,
size,
fileFolder,
fieldMetadataId,
fieldMetadataUniversalIdentifier,
}: {
workspaceId: string;
filename: string;
size: number;
fileFolder: FileFolder;
fieldMetadataId?: string;
fieldMetadataUniversalIdentifier?: string;
}): Promise<FileUploadTargetDTO> {
if (
!DIRECT_UPLOAD_FILE_FOLDERS.includes(
fileFolder as (typeof DIRECT_UPLOAD_FILE_FOLDERS)[number],
)
) {
throw new FileUploadException(
`Direct upload is not supported for file folder ${fileFolder}`,
FileUploadExceptionCode.BAD_REQUEST,
{
userFriendlyMessage: msg`Direct upload is not supported for this file type.`,
},
);
}
const maxFileSize = bytes(settings.storage.maxDirectUploadFileSize) ?? 0;
if (!Number.isInteger(size) || size <= 0 || size > maxFileSize) {
throw new FileUploadException(
`Invalid file size ${size} (max ${maxFileSize} bytes)`,
FileUploadExceptionCode.FILE_TOO_LARGE,
{
userFriendlyMessage: msg`The file is empty or exceeds the maximum allowed size.`,
},
);
}
const { ext } = buildFileInfo(filename);
// The file content cannot be sniffed before it reaches storage, so the
// mime type is derived from the extension only. Anything unknown is
// stored as octet-stream, and the serving path already forces
// Content-Disposition: attachment for non-inline-safe mime types.
const mimeType =
TWENTY_MIME_POLICY[ext] ??
(isNonEmptyString(ext) ? lookup(ext) : undefined) ??
'application/octet-stream';
const fileId = v4();
const name = `${fileId}${isNonEmptyString(ext) ? `.${ext}` : ''}`;
const { applicationUniversalIdentifier, resourcePath } =
await this.resolveUploadLocation({
workspaceId,
fileFolder,
name,
fieldMetadataId,
fieldMetadataUniversalIdentifier,
});
await this.fileStorageService.createPendingFile({
fileFolder,
applicationUniversalIdentifier,
workspaceId,
resourcePath,
fileId,
size,
mimeType,
settings: {
isTemporaryFile: true,
toDelete: false,
},
});
const expiresInSeconds = this.twentyConfigService.get(
'STORAGE_S3_PRESIGNED_URL_EXPIRES_IN',
);
const expiresAt = new Date(Date.now() + expiresInSeconds * 1000);
const presignedUploadUrl =
await this.fileStorageService.getPresignedUploadUrl({
fileFolder,
applicationUniversalIdentifier,
workspaceId,
resourcePath,
contentType: mimeType,
contentLength: size,
expiresInSeconds,
});
if (isDefined(presignedUploadUrl)) {
return {
fileId,
uploadUrl: presignedUploadUrl,
contentType: mimeType,
expiresAt,
};
}
// No presign support (local storage, or S3 without presign enabled):
// fall back to the token-authenticated streaming endpoint on the server.
const payload: FileUploadTokenJwtPayload = {
workspaceId,
fileId,
sub: workspaceId,
type: JwtTokenTypeEnum.FILE_UPLOAD,
};
const token = await this.jwtWrapperService.signAsyncOrThrow(payload, {
expiresIn: expiresInSeconds,
});
const serverUrl = this.twentyConfigService.get('SERVER_URL');
return {
fileId,
uploadUrl: `${serverUrl}/file-upload/${fileId}?token=${token}`,
// octet-stream keeps the request body away from the server's json/text
// body parsers; the real mime type is already on the file record.
contentType: 'application/octet-stream',
expiresAt,
};
}
// Streams the request body straight to the storage driver, bounded by the
// size declared at createFileUpload time. Memory usage stays constant no
// matter how large the file is.
async receiveFileStream({
workspaceId,
fileId,
stream,
}: {
workspaceId: string;
fileId: string;
stream: Readable;
}): Promise<void> {
const file = await this.findFileOrThrow({ workspaceId, fileId });
if (file.status !== FILE_STATUS.PENDING) {
throw new FileUploadException(
`File ${fileId} is not awaiting an upload`,
FileUploadExceptionCode.BAD_REQUEST,
{
userFriendlyMessage: msg`This file has already been uploaded.`,
},
);
}
const { application, fileFolder, resourcePath } =
await this.resolveFileLocation({ workspaceId, file });
const declaredSize = Number(file.size);
let receivedBytes = 0;
const sizeLimiter = new Transform({
transform: (chunk: Buffer, _encoding, callback) => {
receivedBytes += chunk.length;
if (receivedBytes > declaredSize) {
callback(
new FileUploadException(
`Upload exceeds declared size of ${declaredSize} bytes`,
FileUploadExceptionCode.FILE_TOO_LARGE,
{
userFriendlyMessage: msg`The uploaded file is larger than declared.`,
},
),
);
return;
}
callback(null, chunk);
},
});
try {
await Promise.all([
pipeline(stream, sizeLimiter),
this.fileStorageService.writeFileStream({
fileFolder,
applicationUniversalIdentifier: application.universalIdentifier,
workspaceId,
resourcePath,
stream: sizeLimiter,
mimeType: file.mimeType,
}),
]);
} catch (error) {
// The storage-side pipeline can lose the rejection race to the limiter:
// surface the size violation over the resulting stream teardown error.
if (receivedBytes > declaredSize) {
throw new FileUploadException(
`Upload exceeds declared size of ${declaredSize} bytes`,
FileUploadExceptionCode.FILE_TOO_LARGE,
{
userFriendlyMessage: msg`The uploaded file is larger than declared.`,
},
);
}
throw error;
}
if (receivedBytes !== declaredSize) {
// The short object stays in storage but the record stays PENDING, so it
// can never be served or attached: the client retries against the same
// upload url (overwriting it), and the pending-file cleanup cron
// (follow-up PR) reaps whatever is abandoned.
throw new FileUploadException(
`Uploaded ${receivedBytes} bytes but ${declaredSize} were declared`,
FileUploadExceptionCode.FILE_SIZE_MISMATCH,
{
userFriendlyMessage: msg`The uploaded file does not match the declared size. Please retry the upload.`,
},
);
}
}
// Verifies the bytes actually landed in storage with the declared size and
// flips the file to UPLOADED. Idempotent: confirming twice is a no-op.
async completeFileUpload({
workspaceId,
fileId,
}: {
workspaceId: string;
fileId: string;
}): Promise<FileWithSignedUrlDTO> {
const file = await this.findFileOrThrow({ workspaceId, fileId });
const [fileFolder] = file.path.split('/');
// Restrict to files created through createFileUpload so this mutation
// cannot be used to mint signed download urls for arbitrary files.
if (
!DIRECT_UPLOAD_FILE_FOLDERS.includes(
fileFolder as (typeof DIRECT_UPLOAD_FILE_FOLDERS)[number],
)
) {
throw new FileUploadException(
`File not found: ${fileId}`,
FileUploadExceptionCode.FILE_NOT_FOUND,
{
userFriendlyMessage: msg`File not found.`,
},
);
}
if (file.status === FILE_STATUS.UPLOADED) {
// Idempotent retry of a confirm that already succeeded. Only files not
// yet attached to a record qualify: this cannot be used to mint signed
// urls for files that went through the legacy flow and got attached.
if (!file.settings?.isTemporaryFile) {
throw new FileUploadException(
`File ${fileId} is not awaiting an upload confirmation`,
FileUploadExceptionCode.BAD_REQUEST,
{
userFriendlyMessage: msg`This file upload has already been finalized.`,
},
);
}
return this.toFileWithSignedUrl({
file,
fileFolder: fileFolder as FileFolder,
workspaceId,
});
}
const { application, resourcePath } = await this.resolveFileLocation({
workspaceId,
file,
});
const metadata = await this.fileStorageService.getFileMetadata({
fileFolder: fileFolder as FileFolder,
applicationUniversalIdentifier: application.universalIdentifier,
workspaceId,
resourcePath,
});
if (!isDefined(metadata)) {
throw new FileUploadException(
`File ${fileId} has not been uploaded to storage yet`,
FileUploadExceptionCode.FILE_NOT_UPLOADED,
{
userFriendlyMessage: msg`The file has not been uploaded yet. Please upload it before confirming.`,
},
);
}
if (metadata.size !== Number(file.size)) {
throw new FileUploadException(
`File ${fileId} has ${metadata.size} bytes in storage but ${file.size} were declared`,
FileUploadExceptionCode.FILE_SIZE_MISMATCH,
{
userFriendlyMessage: msg`The uploaded file does not match the declared size. Please retry the upload.`,
},
);
}
await this.fileRepository.update(
workspaceId,
{ id: fileId },
{ status: FILE_STATUS.UPLOADED },
);
return this.toFileWithSignedUrl({
file: { ...file, status: FILE_STATUS.UPLOADED },
fileFolder: fileFolder as FileFolder,
workspaceId,
});
}
private async resolveUploadLocation({
workspaceId,
fileFolder,
name,
fieldMetadataId,
fieldMetadataUniversalIdentifier,
}: {
workspaceId: string;
fileFolder: FileFolder;
name: string;
fieldMetadataId?: string;
fieldMetadataUniversalIdentifier?: string;
}): Promise<{
applicationUniversalIdentifier: string;
resourcePath: string;
}> {
if (fileFolder === FileFolder.FilesField) {
if (!fieldMetadataId && !fieldMetadataUniversalIdentifier) {
throw new FileUploadException(
'fieldMetadataId or fieldMetadataUniversalIdentifier must be provided',
FileUploadExceptionCode.BAD_REQUEST,
{
userFriendlyMessage: msg`fieldMetadataId or fieldMetadataUniversalIdentifier must be provided`,
},
);
}
const fieldMetadata = await this.fieldMetadataRepository.findOneOrFail({
select: ['applicationId', 'universalIdentifier'],
where: {
...(fieldMetadataId ? { id: fieldMetadataId } : {}),
...(fieldMetadataUniversalIdentifier
? { universalIdentifier: fieldMetadataUniversalIdentifier }
: {}),
workspaceId,
},
});
const application = await this.applicationRepository.findOneOrFail({
where: {
id: fieldMetadata.applicationId,
workspaceId,
},
});
return {
applicationUniversalIdentifier: application.universalIdentifier,
resourcePath: `${fieldMetadata.universalIdentifier}/${name}`,
};
}
const { workspaceCustomFlatApplication } =
await this.applicationService.findWorkspaceTwentyStandardAndCustomApplicationOrThrow(
{
workspaceId,
},
);
return {
applicationUniversalIdentifier:
workspaceCustomFlatApplication.universalIdentifier,
resourcePath: name,
};
}
private async findFileOrThrow({
workspaceId,
fileId,
}: {
workspaceId: string;
fileId: string;
}): Promise<FileEntity> {
const file = await this.fileRepository.findOne(workspaceId, {
where: { id: fileId },
});
if (!isDefined(file)) {
throw new FileUploadException(
`File not found: ${fileId}`,
FileUploadExceptionCode.FILE_NOT_FOUND,
{
userFriendlyMessage: msg`File not found.`,
},
);
}
return file;
}
private async resolveFileLocation({
workspaceId,
file,
}: {
workspaceId: string;
file: FileEntity;
}): Promise<{
application: ApplicationEntity;
fileFolder: FileFolder;
resourcePath: string;
}> {
const [fileFolder] = file.path.split('/');
const application = await this.applicationRepository.findOneOrFail({
where: {
id: file.applicationId,
workspaceId,
},
});
return {
application,
fileFolder: fileFolder as FileFolder,
resourcePath: removeFileFolderFromFileEntityPath(file.path),
};
}
private async toFileWithSignedUrl({
file,
fileFolder,
workspaceId,
}: {
file: FileEntity;
fileFolder: FileFolder;
workspaceId: string;
}): Promise<FileWithSignedUrlDTO> {
return {
...file,
url: await this.fileUrlService.signFileByIdUrl({
fileId: file.id,
workspaceId,
fileFolder,
}),
};
}
}
@@ -15,6 +15,7 @@ import { FileController } from './controllers/file.controller';
import { FileEntity } from './entities/file.entity';
import { FileCorePictureModule } from './file-core-picture/file-core-picture.module';
import { FileEmailAttachmentModule } from './file-email-attachment/file-email-attachment.module';
import { FileUploadModule } from './file-upload/file-upload.module';
import { FileUrlModule } from './file-url/file-url.module';
import { FileWorkflowModule } from './file-workflow/file-workflow.module';
import { FilesFieldModule } from './files-field/files-field.module';
@@ -33,6 +34,7 @@ import { FileService } from './services/file.service';
FileWorkflowModule,
FileAiChatModule,
FileEmailAttachmentModule,
FileUploadModule,
SecureHttpClientModule,
],
providers: [
@@ -50,6 +52,7 @@ import { FileService } from './services/file.service';
FileWorkflowModule,
FileAiChatModule,
FileEmailAttachmentModule,
FileUploadModule,
],
controllers: [FileController],
})
@@ -18,6 +18,7 @@ import {
IMMUTABLE_FILE_CACHE_CONTROL,
} from 'src/engine/core-modules/file/interfaces/file-folder.interface';
import { type FileResponse } from 'src/engine/core-modules/file/types/file-response.type';
import { FILE_STATUS } from 'src/engine/core-modules/file/types/file-status.types';
import { getContentDisposition } from 'src/engine/core-modules/file/utils/get-content-disposition.utils';
import { removeFileFolderFromFileEntityPath } from 'src/engine/core-modules/file/utils/remove-file-folder-from-file-entity-path.utils';
import { JwtWrapperService } from 'src/engine/core-modules/jwt/services/jwt-wrapper.service';
@@ -66,6 +67,7 @@ export class FileService {
where: {
path: `${fileFolder}/${filepath}`,
applicationId,
status: FILE_STATUS.UPLOADED,
},
});
@@ -94,6 +96,7 @@ export class FileService {
const file = await this.fileRepository.findOne(workspaceId, {
where: {
id: fileId,
status: FILE_STATUS.UPLOADED,
},
});
@@ -155,6 +158,7 @@ export class FileService {
where: {
id: params.fileId,
path: Like(`${params.fileFolder}/%`),
status: FILE_STATUS.UPLOADED,
},
});
@@ -251,6 +255,7 @@ export class FileService {
where: {
id: fileId,
path: Like(`${fileFolder}/%`),
status: FILE_STATUS.UPLOADED,
},
});
@@ -0,0 +1,8 @@
export const FILE_STATUS = {
// File record exists but bytes have not been confirmed in storage yet
// (direct upload initiated, waiting for the client to upload and confirm).
PENDING: 'PENDING',
UPLOADED: 'UPLOADED',
} as const;
export type FileStatus = (typeof FILE_STATUS)[keyof typeof FILE_STATUS];
@@ -565,7 +565,7 @@ export class ConfigVariables {
@ConfigVariablesMetadata({
group: ConfigVariablesGroup.STORAGE_CONFIG,
description:
'When enabled, file downloads are 302-redirected to S3 presigned URLs instead of being proxied through the server. Reduces server load and bandwidth.',
'When enabled, file downloads are 302-redirected to S3 presigned URLs and direct uploads go straight to S3 via presigned PUT URLs instead of being proxied through the server. Reduces server load and bandwidth. Requires a bucket CORS policy allowing PUT from the frontend origin.',
type: ConfigVariableType.BOOLEAN,
})
@ValidateIf((env) => env.STORAGE_TYPE === StorageDriverType.S_3)