feat(files): direct-to-storage upload endpoints with pending file lifecycle (#22449)
<img width="1484" height="404" alt="image" src="https://github.com/user-attachments/assets/b2d363bf-d9e1-49fb-9811-8cc98041aa79" /> ## Context Uploading large files currently OOMs the server: every upload resolver buffers the whole file in memory (`streamToBuffer`) before writing it to storage. This PR is the first of a series introducing direct client-to-storage uploads. It adds the server-side endpoints and driver support only — it is non-breaking and nothing consumes the new flow yet. Follow-up PRs will migrate the frontend upload paths, add a stale-pending-file cleanup cron, and cap the legacy buffered resolvers. ## What it does **New upload flow (initiate → PUT → confirm):** - `createFileUpload(filename, size, fileFolder, fieldMetadataId?)` validates the request (folder allowlist: `FilesField`/`Workflow`, max size, extension-derived mime type), creates the file record in a new `PENDING` status, and returns an upload target: - **S3 with presign enabled** → a presigned PUT URL with `Content-Type`/`Content-Length` pinned in the signature, so the client uploads straight to the bucket; - **local storage, or S3 without presign** → a token-authenticated streaming endpoint on the server (`PUT /file-upload/:id?token=…`, new `FILE_UPLOAD` JWT type) that pipes the request body to the storage driver with constant memory usage and a declared-size cap. - `completeFileUpload(fileId)` verifies the bytes actually landed in storage (HEAD + size match against the declared size) and flips the record to `UPLOADED`. Idempotent. **Pending lifecycle safety:** - New `status` column on `core.file` (`PENDING`/`UPLOADED`, default `UPLOADED` so all existing rows and the legacy upload path are unaffected) + fast instance command. - Files are refused by the serving endpoints and by FILES-field sync while `PENDING`. **Driver support (both drivers):** - `getPresignedUploadUrl` (S3: presigned PUT; local: `null` → server-endpoint fallback) - `writeFileStream` (local: `fs` pipeline with the existing symlink/containment hardening, partial-file cleanup on error; S3: `@aws-sdk/lib-storage` `Upload` for bounded-memory streaming) - `getFileMetadata` (HEAD/stat for confirm-time verification) ## Tests - `file-upload.service.spec.ts`: initiate validation (folder allowlist, size), presigned vs fallback target, confirm verification (missing object, size mismatch, happy path, idempotency) - `local.driver.spec.ts`: `writeFileStream` (content, symlink rejection, partial-file cleanup on stream error), `getFileMetadata` - `s3.driver.spec.ts`: `getPresignedUploadUrl` (disabled → null, PUT command with signed content-type/content-length) - `direct-file-upload.integration-spec.ts`: full end-to-end flow against the local driver (initiate → PUT → complete → download), plus error paths (complete without upload, oversized PUT → 413, invalid token → 403, unsupported folder, size above max) ## Notes for reviewers - The upload-size ceiling for direct uploads is `settings.storage.maxDirectUploadFileSize` (1GB), separate from the 10MB `maxFileSize` used for pictures. - Since content can't be sniffed before it reaches storage, the mime type is derived from the file extension (with the existing `TWENTY_MIME_POLICY` override) and unknown extensions fall back to `application/octet-stream`; the serving path already forces `Content-Disposition: attachment` for anything not on the inline-safe allowlist. - Self-hosters using S3 presign will need a bucket CORS policy allowing `PUT` from the frontend origin (config variable description updated). https://claude.ai/code/session_015UH8KWmsB9zdYaog8MFG1d --- _Generated by [Claude Code](https://claude.ai/code/session_015UH8KWmsB9zdYaog8MFG1d)_ <!-- This is an auto-generated description by cubic. --> <a href="https://cubic.dev/pr/twentyhq/twenty/pull/22449?utm_source=github" target="_blank" rel="noopener noreferrer" data-no-image-dialog="true"><picture><source media="(prefers-color-scheme: dark)" srcset="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"><source media="(prefers-color-scheme: light)" srcset="https://www.cubic.dev/buttons/review-in-cubic-light.svg"><img alt="Review in cubic" src="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"></picture></a> <!-- End of auto-generated description by cubic. -->
This commit is contained in:
+8
@@ -0,0 +1,8 @@
|
||||
import { type CommonPropertiesJwtPayload } from 'src/engine/core-modules/auth/types/common-properties-jwt-payload.type';
|
||||
import { JwtTokenTypeEnum } from 'src/engine/core-modules/auth/types/jwt-token-type.enum';
|
||||
|
||||
export type FileUploadTokenJwtPayload = CommonPropertiesJwtPayload & {
|
||||
type: JwtTokenTypeEnum.FILE_UPLOAD;
|
||||
workspaceId: string;
|
||||
fileId: string;
|
||||
};
|
||||
@@ -5,6 +5,7 @@ import { type ApplicationRefreshTokenJwtPayload } from 'src/engine/core-modules/
|
||||
import { type AppOAuthStateJwtPayload } from 'src/engine/core-modules/auth/types/app-oauth-state-jwt-payload.type';
|
||||
import { type ApprovedAccessDomainJwtPayload } from 'src/engine/core-modules/auth/types/approved-access-domain-jwt-payload.type';
|
||||
import { type FileTokenJwtPayload } from 'src/engine/core-modules/auth/types/file-token-jwt-payload.type';
|
||||
import { type FileUploadTokenJwtPayload } from 'src/engine/core-modules/auth/types/file-upload-token-jwt-payload.type';
|
||||
import { type FileTokenJwtPayloadLegacy } from 'src/engine/core-modules/auth/types/file-token-jwt-payload-legacy.type';
|
||||
import { type LoginTokenJwtPayload } from 'src/engine/core-modules/auth/types/login-token-jwt-payload.type';
|
||||
import { type PlaygroundTokenJwtPayload } from 'src/engine/core-modules/auth/types/playground-token-jwt-payload.type';
|
||||
@@ -23,6 +24,7 @@ export type JwtPayload =
|
||||
| RefreshTokenJwtPayload
|
||||
| FileTokenJwtPayload
|
||||
| FileTokenJwtPayloadLegacy
|
||||
| FileUploadTokenJwtPayload
|
||||
| AppOAuthStateJwtPayload
|
||||
| ApprovedAccessDomainJwtPayload
|
||||
| PlaygroundTokenJwtPayload;
|
||||
|
||||
@@ -4,6 +4,7 @@ export enum JwtTokenTypeEnum {
|
||||
WORKSPACE_AGNOSTIC = 'WORKSPACE_AGNOSTIC',
|
||||
LOGIN = 'LOGIN',
|
||||
FILE = 'FILE',
|
||||
FILE_UPLOAD = 'FILE_UPLOAD',
|
||||
API_KEY = 'API_KEY',
|
||||
REMOTE_SERVER = 'REMOTE_SERVER',
|
||||
KEY_ENCRYPTION_KEY = 'KEY_ENCRYPTION_KEY',
|
||||
|
||||
+114
-1
@@ -1,6 +1,15 @@
|
||||
import { mkdtemp, mkdir, readFile, rm, symlink, writeFile } from 'fs/promises';
|
||||
import {
|
||||
mkdtemp,
|
||||
mkdir,
|
||||
readFile,
|
||||
rm,
|
||||
stat,
|
||||
symlink,
|
||||
writeFile,
|
||||
} from 'fs/promises';
|
||||
import { tmpdir } from 'os';
|
||||
import path from 'path';
|
||||
import { Readable } from 'stream';
|
||||
|
||||
import { FileStorageExceptionCode } from 'src/engine/core-modules/file-storage/interfaces/file-storage-exception';
|
||||
|
||||
@@ -78,4 +87,108 @@ describe('LocalDriver security hardening', () => {
|
||||
code: FileStorageExceptionCode.ACCESS_DENIED,
|
||||
});
|
||||
});
|
||||
|
||||
describe('writeFileStream', () => {
|
||||
it('should write the streamed content to disk', async () => {
|
||||
const storagePath = await createTempDirectory('local-driver-storage-');
|
||||
const driver = new LocalDriver({ storagePath });
|
||||
|
||||
await driver.writeFileStream({
|
||||
filePath: 'workspace/app/streamed.txt',
|
||||
stream: Readable.from([
|
||||
Buffer.from('streamed-'),
|
||||
Buffer.from('content'),
|
||||
]),
|
||||
mimeType: 'text/plain',
|
||||
});
|
||||
|
||||
await expect(
|
||||
readFile(path.join(storagePath, 'workspace/app/streamed.txt'), 'utf8'),
|
||||
).resolves.toBe('streamed-content');
|
||||
});
|
||||
|
||||
it('should reject when target is a symlink', async () => {
|
||||
const storagePath = await createTempDirectory('local-driver-storage-');
|
||||
const outsidePath = await createTempDirectory('local-driver-outside-');
|
||||
const outsideFilePath = path.join(outsidePath, 'outside.txt');
|
||||
const symlinkFolderPath = path.join(storagePath, 'workspace', 'app');
|
||||
const symlinkFilePath = path.join(symlinkFolderPath, 'target.txt');
|
||||
|
||||
await mkdir(symlinkFolderPath, { recursive: true });
|
||||
await writeFile(outsideFilePath, 'outside');
|
||||
await symlink(outsideFilePath, symlinkFilePath);
|
||||
|
||||
const driver = new LocalDriver({ storagePath });
|
||||
|
||||
await expect(
|
||||
driver.writeFileStream({
|
||||
filePath: 'workspace/app/target.txt',
|
||||
stream: Readable.from([Buffer.from('new-content')]),
|
||||
mimeType: undefined,
|
||||
}),
|
||||
).rejects.toMatchObject({
|
||||
code: FileStorageExceptionCode.ACCESS_DENIED,
|
||||
});
|
||||
|
||||
await expect(readFile(outsideFilePath, 'utf8')).resolves.toBe('outside');
|
||||
});
|
||||
|
||||
it('should remove the partial file when the stream errors', async () => {
|
||||
const storagePath = await createTempDirectory('local-driver-storage-');
|
||||
const driver = new LocalDriver({ storagePath });
|
||||
|
||||
const failingStream = new Readable({
|
||||
read() {
|
||||
this.push(Buffer.from('partial'));
|
||||
this.destroy(new Error('stream interrupted'));
|
||||
},
|
||||
});
|
||||
|
||||
await expect(
|
||||
driver.writeFileStream({
|
||||
filePath: 'workspace/app/partial.txt',
|
||||
stream: failingStream,
|
||||
mimeType: undefined,
|
||||
}),
|
||||
).rejects.toThrow('stream interrupted');
|
||||
|
||||
await expect(
|
||||
stat(path.join(storagePath, 'workspace/app/partial.txt')),
|
||||
).rejects.toMatchObject({ code: 'ENOENT' });
|
||||
});
|
||||
});
|
||||
|
||||
describe('getFileMetadata', () => {
|
||||
it('should return the file size', async () => {
|
||||
const storagePath = await createTempDirectory('local-driver-storage-');
|
||||
const folderPath = path.join(storagePath, 'workspace', 'app');
|
||||
|
||||
await mkdir(folderPath, { recursive: true });
|
||||
await writeFile(path.join(folderPath, 'file.txt'), '12345');
|
||||
|
||||
const driver = new LocalDriver({ storagePath });
|
||||
|
||||
await expect(
|
||||
driver.getFileMetadata({ filePath: 'workspace/app/file.txt' }),
|
||||
).resolves.toEqual({ size: 5 });
|
||||
});
|
||||
|
||||
it('should return null when the file does not exist', async () => {
|
||||
const storagePath = await createTempDirectory('local-driver-storage-');
|
||||
const driver = new LocalDriver({ storagePath });
|
||||
|
||||
await expect(
|
||||
driver.getFileMetadata({ filePath: 'workspace/app/missing.txt' }),
|
||||
).resolves.toBeNull();
|
||||
});
|
||||
});
|
||||
|
||||
describe('getPresignedUploadUrl', () => {
|
||||
it('should return null so callers fall back to the server endpoint', async () => {
|
||||
const storagePath = await createTempDirectory('local-driver-storage-');
|
||||
const driver = new LocalDriver({ storagePath });
|
||||
|
||||
await expect(driver.getPresignedUploadUrl()).resolves.toBeNull();
|
||||
});
|
||||
});
|
||||
});
|
||||
|
||||
+64
-1
@@ -1,4 +1,4 @@
|
||||
import { GetObjectCommand } from '@aws-sdk/client-s3';
|
||||
import { GetObjectCommand, PutObjectCommand } from '@aws-sdk/client-s3';
|
||||
import { getSignedUrl } from '@aws-sdk/s3-request-presigner';
|
||||
|
||||
import { S3Driver } from 'src/engine/core-modules/file-storage/drivers/s3.driver';
|
||||
@@ -120,3 +120,66 @@ describe('S3Driver.getPresignedUrl', () => {
|
||||
);
|
||||
});
|
||||
});
|
||||
|
||||
describe('S3Driver.getPresignedUploadUrl', () => {
|
||||
afterEach(() => {
|
||||
jest.clearAllMocks();
|
||||
});
|
||||
|
||||
it('should return null when presigning is not enabled', async () => {
|
||||
const driver = new S3Driver({
|
||||
bucketName: 'test-bucket',
|
||||
region: 'us-east-1',
|
||||
});
|
||||
|
||||
const result = await driver.getPresignedUploadUrl({
|
||||
filePath: 'some/file.pdf',
|
||||
contentType: 'application/pdf',
|
||||
contentLength: 1024,
|
||||
});
|
||||
|
||||
expect(result).toBeNull();
|
||||
expect(getSignedUrl).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it('should presign a PUT with content-type and content-length in the signature', async () => {
|
||||
(getSignedUrl as jest.Mock).mockResolvedValue(
|
||||
'https://s3.us-east-1.amazonaws.com/test-bucket/some/file.pdf?X-Amz-Signature=abc',
|
||||
);
|
||||
|
||||
const driver = new S3Driver({
|
||||
bucketName: 'test-bucket',
|
||||
region: 'us-east-1',
|
||||
presignEnabled: true,
|
||||
});
|
||||
|
||||
const result = await driver.getPresignedUploadUrl({
|
||||
filePath: 'some/file.pdf',
|
||||
contentType: 'application/pdf',
|
||||
contentLength: 1024,
|
||||
expiresInSeconds: 900,
|
||||
});
|
||||
|
||||
expect(result).toBe(
|
||||
'https://s3.us-east-1.amazonaws.com/test-bucket/some/file.pdf?X-Amz-Signature=abc',
|
||||
);
|
||||
expect(getSignedUrl).toHaveBeenCalledWith(
|
||||
expect.anything(),
|
||||
expect.any(PutObjectCommand),
|
||||
{
|
||||
expiresIn: 900,
|
||||
signableHeaders: new Set(['content-type', 'content-length']),
|
||||
},
|
||||
);
|
||||
|
||||
const command = (getSignedUrl as jest.Mock).mock
|
||||
.calls[0][1] as PutObjectCommand;
|
||||
|
||||
expect(command.input).toMatchObject({
|
||||
Bucket: 'test-bucket',
|
||||
Key: 'some/file.pdf',
|
||||
ContentType: 'application/pdf',
|
||||
ContentLength: 1024,
|
||||
});
|
||||
});
|
||||
});
|
||||
|
||||
+87
@@ -8,6 +8,9 @@ import { ValidatedStorageDriver } from 'src/engine/core-modules/file-storage/dri
|
||||
const createMockDriver = (): jest.Mocked<StorageDriver> => ({
|
||||
readFile: jest.fn().mockResolvedValue(Readable.from([])),
|
||||
writeFile: jest.fn().mockResolvedValue(undefined),
|
||||
writeFileStream: jest.fn().mockResolvedValue(undefined),
|
||||
getFileMetadata: jest.fn().mockResolvedValue(null),
|
||||
getPresignedUploadUrl: jest.fn().mockResolvedValue(null),
|
||||
downloadFolder: jest.fn().mockResolvedValue(undefined),
|
||||
uploadFolder: jest.fn().mockResolvedValue(undefined),
|
||||
downloadFile: jest.fn().mockResolvedValue(undefined),
|
||||
@@ -138,6 +141,52 @@ describe('ValidatedStorageDriver', () => {
|
||||
responseContentDisposition: 'inline',
|
||||
});
|
||||
});
|
||||
|
||||
it('should delegate writeFileStream', async () => {
|
||||
const params = {
|
||||
filePath: 'folder/file.txt',
|
||||
stream: Readable.from([Buffer.from('data')]),
|
||||
mimeType: 'text/plain' as string | undefined,
|
||||
};
|
||||
|
||||
await driver.writeFileStream(params);
|
||||
|
||||
expect(mockDelegate.writeFileStream).toHaveBeenCalledWith(params);
|
||||
});
|
||||
|
||||
it('should delegate getFileMetadata', async () => {
|
||||
mockDelegate.getFileMetadata.mockResolvedValue({ size: 1024 });
|
||||
|
||||
const result = await driver.getFileMetadata({
|
||||
filePath: 'folder/file.txt',
|
||||
});
|
||||
|
||||
expect(result).toEqual({ size: 1024 });
|
||||
expect(mockDelegate.getFileMetadata).toHaveBeenCalledWith({
|
||||
filePath: 'folder/file.txt',
|
||||
});
|
||||
});
|
||||
|
||||
it('should delegate getPresignedUploadUrl', async () => {
|
||||
mockDelegate.getPresignedUploadUrl.mockResolvedValue(
|
||||
'https://s3.example.com/signed-put',
|
||||
);
|
||||
|
||||
const result = await driver.getPresignedUploadUrl({
|
||||
filePath: 'folder/file.txt',
|
||||
contentType: 'application/pdf',
|
||||
contentLength: 1024,
|
||||
expiresInSeconds: 900,
|
||||
});
|
||||
|
||||
expect(result).toBe('https://s3.example.com/signed-put');
|
||||
expect(mockDelegate.getPresignedUploadUrl).toHaveBeenCalledWith({
|
||||
filePath: 'folder/file.txt',
|
||||
contentType: 'application/pdf',
|
||||
contentLength: 1024,
|
||||
expiresInSeconds: 900,
|
||||
});
|
||||
});
|
||||
});
|
||||
|
||||
describe('rejects path traversal attempts', () => {
|
||||
@@ -165,6 +214,44 @@ describe('ValidatedStorageDriver', () => {
|
||||
expect(mockDelegate.writeFile).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it('should reject writeFileStream with traversal', async () => {
|
||||
await expect(
|
||||
driver.writeFileStream({
|
||||
filePath: '../../evil',
|
||||
stream: Readable.from([Buffer.from('x')]),
|
||||
mimeType: undefined,
|
||||
}),
|
||||
).rejects.toMatchObject({
|
||||
code: FileStorageExceptionCode.ACCESS_DENIED,
|
||||
});
|
||||
|
||||
expect(mockDelegate.writeFileStream).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it('should reject getFileMetadata with traversal', async () => {
|
||||
await expect(
|
||||
driver.getFileMetadata({ filePath: '../etc/passwd' }),
|
||||
).rejects.toMatchObject({
|
||||
code: FileStorageExceptionCode.ACCESS_DENIED,
|
||||
});
|
||||
|
||||
expect(mockDelegate.getFileMetadata).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it('should reject getPresignedUploadUrl with traversal', async () => {
|
||||
await expect(
|
||||
driver.getPresignedUploadUrl({
|
||||
filePath: '../etc/passwd',
|
||||
contentType: 'application/pdf',
|
||||
contentLength: 1024,
|
||||
}),
|
||||
).rejects.toMatchObject({
|
||||
code: FileStorageExceptionCode.ACCESS_DENIED,
|
||||
});
|
||||
|
||||
expect(mockDelegate.getPresignedUploadUrl).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it('should reject delete with traversal in folderPath', async () => {
|
||||
await expect(
|
||||
driver.delete({ folderPath: '../secret' }),
|
||||
|
||||
+17
@@ -8,6 +8,16 @@ export interface StorageDriver {
|
||||
mimeType: string | undefined;
|
||||
}): Promise<void>;
|
||||
|
||||
writeFileStream(params: {
|
||||
filePath: string;
|
||||
stream: Readable;
|
||||
mimeType: string | undefined;
|
||||
}): Promise<void>;
|
||||
|
||||
getFileMetadata(params: {
|
||||
filePath: string;
|
||||
}): Promise<{ size: number } | null>;
|
||||
|
||||
downloadFolder(params: {
|
||||
onStoragePath: string;
|
||||
localPath: string;
|
||||
@@ -42,4 +52,11 @@ export interface StorageDriver {
|
||||
responseContentDisposition?: string;
|
||||
responseCacheControl?: string;
|
||||
}): Promise<string | null>;
|
||||
|
||||
getPresignedUploadUrl(params: {
|
||||
filePath: string;
|
||||
contentType: string;
|
||||
contentLength: number;
|
||||
expiresInSeconds?: number;
|
||||
}): Promise<string | null>;
|
||||
}
|
||||
|
||||
+79
-9
@@ -1,7 +1,13 @@
|
||||
import { createReadStream, existsSync, realpathSync } from 'fs';
|
||||
import {
|
||||
createReadStream,
|
||||
createWriteStream,
|
||||
existsSync,
|
||||
realpathSync,
|
||||
} from 'fs';
|
||||
import * as fs from 'fs/promises';
|
||||
import path, { dirname, join } from 'path';
|
||||
import { type Readable } from 'stream';
|
||||
import { pipeline } from 'stream/promises';
|
||||
|
||||
import { type StorageDriver } from 'src/engine/core-modules/file-storage/drivers/interfaces/storage-driver.interface';
|
||||
import {
|
||||
@@ -64,18 +70,18 @@ export class LocalDriver implements StorageDriver {
|
||||
}
|
||||
}
|
||||
|
||||
async writeFile(params: {
|
||||
filePath: string;
|
||||
sourceFile: Buffer | Uint8Array | string;
|
||||
mimeType: string | undefined;
|
||||
}): Promise<void> {
|
||||
const filePath = path.resolve(this.options.storagePath, params.filePath);
|
||||
const folderPath = dirname(filePath);
|
||||
// Resolves the on-disk path for a write, creating the parent folder and
|
||||
// enforcing storage containment + symlink rejection.
|
||||
private async resolveWritableRealPathOrThrow(
|
||||
filePath: string,
|
||||
): Promise<string> {
|
||||
const resolvedPath = path.resolve(this.options.storagePath, filePath);
|
||||
const folderPath = dirname(resolvedPath);
|
||||
|
||||
await this.createFolder(folderPath);
|
||||
|
||||
const realFolderPath = realpathSync(folderPath);
|
||||
const realFilePath = path.join(realFolderPath, path.basename(filePath));
|
||||
const realFilePath = path.join(realFolderPath, path.basename(resolvedPath));
|
||||
|
||||
this.assertRealPathIsWithinStorage(realFilePath);
|
||||
|
||||
@@ -94,9 +100,67 @@ export class LocalDriver implements StorageDriver {
|
||||
}
|
||||
}
|
||||
|
||||
return realFilePath;
|
||||
}
|
||||
|
||||
async writeFile(params: {
|
||||
filePath: string;
|
||||
sourceFile: Buffer | Uint8Array | string;
|
||||
mimeType: string | undefined;
|
||||
}): Promise<void> {
|
||||
const realFilePath = await this.resolveWritableRealPathOrThrow(
|
||||
params.filePath,
|
||||
);
|
||||
|
||||
await fs.writeFile(realFilePath, params.sourceFile);
|
||||
}
|
||||
|
||||
async writeFileStream(params: {
|
||||
filePath: string;
|
||||
stream: Readable;
|
||||
mimeType: string | undefined;
|
||||
}): Promise<void> {
|
||||
const realFilePath = await this.resolveWritableRealPathOrThrow(
|
||||
params.filePath,
|
||||
);
|
||||
|
||||
try {
|
||||
await pipeline(params.stream, createWriteStream(realFilePath));
|
||||
} catch (error) {
|
||||
// Remove the partial file so a failed upload can be retried cleanly
|
||||
await fs.rm(realFilePath, { force: true });
|
||||
|
||||
throw error;
|
||||
}
|
||||
}
|
||||
|
||||
async getFileMetadata(params: {
|
||||
filePath: string;
|
||||
}): Promise<{ size: number } | null> {
|
||||
const joinedPath = join(this.options.storagePath, params.filePath);
|
||||
let filePath: string;
|
||||
|
||||
try {
|
||||
filePath = realpathSync(path.resolve(joinedPath));
|
||||
} catch {
|
||||
return null;
|
||||
}
|
||||
|
||||
this.assertRealPathIsWithinStorage(filePath);
|
||||
|
||||
try {
|
||||
const stats = await fs.stat(filePath);
|
||||
|
||||
return { size: stats.size };
|
||||
} catch (error) {
|
||||
if ((error as NodeJS.ErrnoException).code === 'ENOENT') {
|
||||
return null;
|
||||
}
|
||||
|
||||
throw error;
|
||||
}
|
||||
}
|
||||
|
||||
async downloadFile(params: {
|
||||
onStoragePath: string;
|
||||
localPath: string;
|
||||
@@ -304,6 +368,12 @@ export class LocalDriver implements StorageDriver {
|
||||
return null;
|
||||
}
|
||||
|
||||
// Local storage has no external endpoint to upload to: the caller falls
|
||||
// back to the server-side streaming upload endpoint.
|
||||
async getPresignedUploadUrl(): Promise<string | null> {
|
||||
return null;
|
||||
}
|
||||
|
||||
async checkFolderExists(params: { folderPath: string }): Promise<boolean> {
|
||||
const folderFullPath = path.resolve(
|
||||
this.options.storagePath,
|
||||
|
||||
@@ -20,6 +20,7 @@ import {
|
||||
S3,
|
||||
type S3ClientConfig,
|
||||
} from '@aws-sdk/client-s3';
|
||||
import { Upload } from '@aws-sdk/lib-storage';
|
||||
import { getSignedUrl } from '@aws-sdk/s3-request-presigner';
|
||||
import { isDefined } from 'twenty-shared/utils';
|
||||
|
||||
@@ -112,6 +113,47 @@ export class S3Driver implements StorageDriver {
|
||||
await this.s3Client.send(command);
|
||||
}
|
||||
|
||||
async writeFileStream(params: {
|
||||
filePath: string;
|
||||
stream: Readable;
|
||||
mimeType: string | undefined;
|
||||
}): Promise<void> {
|
||||
// Upload streams the body with bounded memory (multipart under the hood),
|
||||
// unlike PutObjectCommand which requires the whole payload upfront.
|
||||
const upload = new Upload({
|
||||
client: this.s3Client,
|
||||
params: {
|
||||
Bucket: this.bucketName,
|
||||
Key: params.filePath,
|
||||
Body: params.stream,
|
||||
ContentType: params.mimeType,
|
||||
},
|
||||
});
|
||||
|
||||
await upload.done();
|
||||
}
|
||||
|
||||
async getFileMetadata(params: {
|
||||
filePath: string;
|
||||
}): Promise<{ size: number } | null> {
|
||||
try {
|
||||
const head = await this.s3Client.send(
|
||||
new HeadObjectCommand({
|
||||
Bucket: this.bucketName,
|
||||
Key: params.filePath,
|
||||
}),
|
||||
);
|
||||
|
||||
return { size: head.ContentLength ?? 0 };
|
||||
} catch (error) {
|
||||
if (error instanceof NotFound) {
|
||||
return null;
|
||||
}
|
||||
|
||||
throw error;
|
||||
}
|
||||
}
|
||||
|
||||
private async createFolder(path: string) {
|
||||
return fs.mkdirSync(path, { recursive: true });
|
||||
}
|
||||
@@ -404,6 +446,31 @@ export class S3Driver implements StorageDriver {
|
||||
});
|
||||
}
|
||||
|
||||
async getPresignedUploadUrl(params: {
|
||||
filePath: string;
|
||||
contentType: string;
|
||||
contentLength: number;
|
||||
expiresInSeconds?: number;
|
||||
}): Promise<string | null> {
|
||||
if (!this.presignClient) {
|
||||
return null;
|
||||
}
|
||||
|
||||
const command = new PutObjectCommand({
|
||||
Bucket: this.bucketName,
|
||||
Key: params.filePath,
|
||||
ContentType: params.contentType,
|
||||
ContentLength: params.contentLength,
|
||||
});
|
||||
|
||||
// Content-Type and Content-Length are part of the signature so the client
|
||||
// cannot upload a payload of a different type or size than declared.
|
||||
return getSignedUrl(this.presignClient, command, {
|
||||
expiresIn: params.expiresInSeconds ?? 900,
|
||||
signableHeaders: new Set(['content-type', 'content-length']),
|
||||
});
|
||||
}
|
||||
|
||||
async checkBucketExists(args: HeadBucketCommandInput) {
|
||||
try {
|
||||
await this.s3Client.headBucket(args);
|
||||
|
||||
+29
@@ -23,6 +23,24 @@ export class ValidatedStorageDriver implements StorageDriver {
|
||||
return this.delegate.writeFile(params);
|
||||
}
|
||||
|
||||
async writeFileStream(params: {
|
||||
filePath: string;
|
||||
stream: Readable;
|
||||
mimeType: string | undefined;
|
||||
}): Promise<void> {
|
||||
assertStoragePathIsSafe(params.filePath);
|
||||
|
||||
return this.delegate.writeFileStream(params);
|
||||
}
|
||||
|
||||
async getFileMetadata(params: {
|
||||
filePath: string;
|
||||
}): Promise<{ size: number } | null> {
|
||||
assertStoragePathIsSafe(params.filePath);
|
||||
|
||||
return this.delegate.getFileMetadata(params);
|
||||
}
|
||||
|
||||
async downloadFolder(params: {
|
||||
onStoragePath: string;
|
||||
localPath: string;
|
||||
@@ -111,6 +129,17 @@ export class ValidatedStorageDriver implements StorageDriver {
|
||||
return this.delegate.getPresignedUrl(params);
|
||||
}
|
||||
|
||||
async getPresignedUploadUrl(params: {
|
||||
filePath: string;
|
||||
contentType: string;
|
||||
contentLength: number;
|
||||
expiresInSeconds?: number;
|
||||
}): Promise<string | null> {
|
||||
assertStoragePathIsSafe(params.filePath);
|
||||
|
||||
return this.delegate.getPresignedUploadUrl(params);
|
||||
}
|
||||
|
||||
async checkFileExists(params: { filePath: string }): Promise<boolean> {
|
||||
assertStoragePathIsSafe(params.filePath);
|
||||
|
||||
|
||||
@@ -19,6 +19,7 @@ import { validateFolderPath } from 'src/engine/core-modules/file-storage/utils/v
|
||||
import { validateStoragePathIsWithinWorkspaceOrThrow } from 'src/engine/core-modules/file-storage/utils/validate-storage-path-is-within-workspace-or-throw.util';
|
||||
import { FileEntity } from 'src/engine/core-modules/file/entities/file.entity';
|
||||
import { FileSettings } from 'src/engine/core-modules/file/types/file-settings.types';
|
||||
import { FILE_STATUS } from 'src/engine/core-modules/file/types/file-status.types';
|
||||
import { removeFileFolderFromFileEntityPath } from 'src/engine/core-modules/file/utils/remove-file-folder-from-file-entity-path.utils';
|
||||
import { InjectWorkspaceScopedRepository } from 'src/engine/twenty-orm/workspace-scoped-repository/inject-workspace-scoped-repository.decorator';
|
||||
import { WorkspaceScopedRepository } from 'src/engine/twenty-orm/workspace-scoped-repository/workspace-scoped-repository';
|
||||
@@ -187,6 +188,99 @@ export class FileStorageService {
|
||||
);
|
||||
}
|
||||
|
||||
// Creates the file record ahead of a direct client upload. The bytes are
|
||||
// not in storage yet: the record stays PENDING until the upload is
|
||||
// confirmed (completeFileUpload) or reaped by the cleanup cron.
|
||||
async createPendingFile({
|
||||
fileFolder,
|
||||
applicationUniversalIdentifier,
|
||||
workspaceId,
|
||||
resourcePath,
|
||||
fileId,
|
||||
size,
|
||||
mimeType,
|
||||
settings,
|
||||
}: ResourceIdentifier & {
|
||||
fileId: string;
|
||||
size: number;
|
||||
mimeType: string;
|
||||
settings: FileSettings;
|
||||
}): Promise<FileEntity> {
|
||||
const application = await this.applicationRepository.findOneOrFail({
|
||||
where: {
|
||||
universalIdentifier: applicationUniversalIdentifier,
|
||||
workspaceId,
|
||||
},
|
||||
});
|
||||
|
||||
const { filePath } = this.validateAndBuildFileStoragePathOrThrow({
|
||||
workspaceId,
|
||||
applicationUniversalIdentifier,
|
||||
fileFolder,
|
||||
resourcePath,
|
||||
});
|
||||
|
||||
return this.fileRepository.upsertAndReturnOne(
|
||||
workspaceId,
|
||||
{
|
||||
path: filePath,
|
||||
applicationId: application.id,
|
||||
id: fileId,
|
||||
mimeType,
|
||||
size,
|
||||
settings,
|
||||
status: FILE_STATUS.PENDING,
|
||||
},
|
||||
['path', 'workspaceId', 'applicationId'],
|
||||
);
|
||||
}
|
||||
|
||||
async writeFileStream(
|
||||
params: ResourceIdentifier & {
|
||||
stream: Readable;
|
||||
mimeType: string | undefined;
|
||||
},
|
||||
): Promise<void> {
|
||||
const driver = this.fileStorageDriverFactory.getCurrentDriver();
|
||||
const { onStorageFilePath } =
|
||||
this.validateAndBuildFileStoragePathOrThrow(params);
|
||||
|
||||
return driver.writeFileStream({
|
||||
filePath: onStorageFilePath,
|
||||
stream: params.stream,
|
||||
mimeType: params.mimeType,
|
||||
});
|
||||
}
|
||||
|
||||
async getFileMetadata(
|
||||
params: ResourceIdentifier,
|
||||
): Promise<{ size: number } | null> {
|
||||
const driver = this.fileStorageDriverFactory.getCurrentDriver();
|
||||
const { onStorageFilePath } =
|
||||
this.validateAndBuildFileStoragePathOrThrow(params);
|
||||
|
||||
return driver.getFileMetadata({ filePath: onStorageFilePath });
|
||||
}
|
||||
|
||||
async getPresignedUploadUrl(
|
||||
params: ResourceIdentifier & {
|
||||
contentType: string;
|
||||
contentLength: number;
|
||||
expiresInSeconds?: number;
|
||||
},
|
||||
): Promise<string | null> {
|
||||
const driver = this.fileStorageDriverFactory.getCurrentDriver();
|
||||
const { onStorageFilePath } =
|
||||
this.validateAndBuildFileStoragePathOrThrow(params);
|
||||
|
||||
return driver.getPresignedUploadUrl({
|
||||
filePath: onStorageFilePath,
|
||||
contentType: params.contentType,
|
||||
contentLength: params.contentLength,
|
||||
expiresInSeconds: params.expiresInSeconds,
|
||||
});
|
||||
}
|
||||
|
||||
async getPresignedUrl(
|
||||
params: ResourceIdentifier & {
|
||||
expiresInSeconds?: number;
|
||||
|
||||
@@ -12,12 +12,19 @@ import {
|
||||
UpdateDateColumn,
|
||||
} from 'typeorm';
|
||||
|
||||
import { ADD_STATUS_TO_FILE_UPGRADE_COMMAND_NAME } from 'src/database/commands/upgrade-version-command/2-19/add-status-to-file-upgrade-command-name.constant';
|
||||
import { ApplicationEntity } from 'src/engine/core-modules/application/application.entity';
|
||||
import { FileSettings } from 'src/engine/core-modules/file/types/file-settings.types';
|
||||
import {
|
||||
FILE_STATUS,
|
||||
FileStatus,
|
||||
} from 'src/engine/core-modules/file/types/file-status.types';
|
||||
import { WasIntroducedInUpgrade } from 'src/engine/core-modules/upgrade/decorators/was-introduced-in-upgrade.decorator';
|
||||
import { WorkspaceRelatedEntity } from 'src/engine/workspace-manager/types/workspace-related-entity';
|
||||
|
||||
@Entity('file')
|
||||
@Index('IDX_FILE_WORKSPACE_ID', ['workspaceId'])
|
||||
@Index('IDX_FILE_STATUS', ['status'])
|
||||
@Unique('IDX_APPLICATION_PATH_WORKSPACE_ID_APPLICATION_ID_UNIQUE', [
|
||||
'workspaceId',
|
||||
'applicationId',
|
||||
@@ -63,4 +70,15 @@ export class FileEntity extends WorkspaceRelatedEntity {
|
||||
default: 'application/octet-stream',
|
||||
})
|
||||
mimeType: string;
|
||||
|
||||
@WasIntroducedInUpgrade({
|
||||
upgradeCommandName: ADD_STATUS_TO_FILE_UPGRADE_COMMAND_NAME,
|
||||
})
|
||||
@Column({
|
||||
type: 'enum',
|
||||
enum: Object.values(FILE_STATUS),
|
||||
nullable: false,
|
||||
default: FILE_STATUS.UPLOADED,
|
||||
})
|
||||
status: FileStatus;
|
||||
}
|
||||
|
||||
+44
@@ -0,0 +1,44 @@
|
||||
import {
|
||||
Controller,
|
||||
Param,
|
||||
Put,
|
||||
Req,
|
||||
Res,
|
||||
UseFilters,
|
||||
UseGuards,
|
||||
} from '@nestjs/common';
|
||||
|
||||
import { Request, Response } from 'express';
|
||||
|
||||
import { FileUploadApiExceptionFilter } from 'src/engine/core-modules/file/file-upload/filters/file-upload-api-exception.filter';
|
||||
import { FileUploadTokenGuard } from 'src/engine/core-modules/file/file-upload/guards/file-upload-token.guard';
|
||||
import { FileUploadService } from 'src/engine/core-modules/file/file-upload/services/file-upload.service';
|
||||
import { NoPermissionGuard } from 'src/engine/guards/no-permission.guard';
|
||||
|
||||
@Controller()
|
||||
@UseFilters(FileUploadApiExceptionFilter)
|
||||
export class FileUploadController {
|
||||
constructor(private readonly fileUploadService: FileUploadService) {}
|
||||
|
||||
// Streaming target for direct uploads when storage has no presigned upload
|
||||
// support (local driver, or S3 without presign enabled). The body is piped
|
||||
// to the storage driver without ever being buffered in memory.
|
||||
@Put('file-upload/:id')
|
||||
@UseGuards(FileUploadTokenGuard, NoPermissionGuard)
|
||||
async uploadFileById(
|
||||
@Req() req: Request,
|
||||
@Res() res: Response,
|
||||
@Param('id') fileId: string,
|
||||
) {
|
||||
// oxlint-disable-next-line typescript/no-explicit-any
|
||||
const workspaceId = (req as any)?.workspaceId;
|
||||
|
||||
await this.fileUploadService.receiveFileStream({
|
||||
workspaceId,
|
||||
fileId,
|
||||
stream: req,
|
||||
});
|
||||
|
||||
res.status(204).send();
|
||||
}
|
||||
}
|
||||
+19
@@ -0,0 +1,19 @@
|
||||
import { Field, ObjectType } from '@nestjs/graphql';
|
||||
|
||||
import { UUIDScalarType } from 'src/engine/api/graphql/workspace-schema-builder/graphql-types/scalars';
|
||||
|
||||
@ObjectType('FileUploadTarget')
|
||||
export class FileUploadTargetDTO {
|
||||
@Field(() => UUIDScalarType)
|
||||
fileId: string;
|
||||
|
||||
@Field()
|
||||
uploadUrl: string;
|
||||
|
||||
// Content-Type header the client must send when uploading to uploadUrl
|
||||
@Field()
|
||||
contentType: string;
|
||||
|
||||
@Field(() => Date, { nullable: false })
|
||||
expiresAt: Date;
|
||||
}
|
||||
+23
@@ -0,0 +1,23 @@
|
||||
import { type MessageDescriptor } from '@lingui/core';
|
||||
|
||||
import { CustomException } from 'src/utils/custom-exception';
|
||||
|
||||
export enum FileUploadExceptionCode {
|
||||
BAD_REQUEST = 'BAD_REQUEST',
|
||||
FILE_NOT_FOUND = 'FILE_NOT_FOUND',
|
||||
FILE_NOT_UPLOADED = 'FILE_NOT_UPLOADED',
|
||||
FILE_SIZE_MISMATCH = 'FILE_SIZE_MISMATCH',
|
||||
FILE_TOO_LARGE = 'FILE_TOO_LARGE',
|
||||
}
|
||||
|
||||
export class FileUploadException extends CustomException<FileUploadExceptionCode> {
|
||||
constructor(
|
||||
message: string,
|
||||
code: FileUploadExceptionCode,
|
||||
{ userFriendlyMessage }: { userFriendlyMessage: MessageDescriptor },
|
||||
) {
|
||||
super(message, code, {
|
||||
userFriendlyMessage,
|
||||
});
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,40 @@
|
||||
import { Module } from '@nestjs/common';
|
||||
import { TypeOrmModule } from '@nestjs/typeorm';
|
||||
|
||||
import { ApplicationEntity } from 'src/engine/core-modules/application/application.entity';
|
||||
import { ApplicationModule } from 'src/engine/core-modules/application/application.module';
|
||||
import { FileStorageModule } from 'src/engine/core-modules/file-storage/file-storage.module';
|
||||
import { FileEntity } from 'src/engine/core-modules/file/entities/file.entity';
|
||||
import { FileUploadController } from 'src/engine/core-modules/file/file-upload/controllers/file-upload.controller';
|
||||
import { FileUploadTokenGuard } from 'src/engine/core-modules/file/file-upload/guards/file-upload-token.guard';
|
||||
import { FileUploadResolver } from 'src/engine/core-modules/file/file-upload/resolvers/file-upload.resolver';
|
||||
import { FileUploadService } from 'src/engine/core-modules/file/file-upload/services/file-upload.service';
|
||||
import { FileUrlModule } from 'src/engine/core-modules/file/file-url/file-url.module';
|
||||
import { JwtModule } from 'src/engine/core-modules/jwt/jwt.module';
|
||||
import { FieldMetadataEntity } from 'src/engine/metadata-modules/field-metadata/field-metadata.entity';
|
||||
import { PermissionsModule } from 'src/engine/metadata-modules/permissions/permissions.module';
|
||||
import { provideWorkspaceScopedRepository } from 'src/engine/twenty-orm/workspace-scoped-repository/provide-workspace-scoped-repository';
|
||||
|
||||
@Module({
|
||||
imports: [
|
||||
JwtModule,
|
||||
TypeOrmModule.forFeature([
|
||||
FileEntity,
|
||||
ApplicationEntity,
|
||||
FieldMetadataEntity,
|
||||
]),
|
||||
PermissionsModule,
|
||||
FileStorageModule,
|
||||
FileUrlModule,
|
||||
ApplicationModule,
|
||||
],
|
||||
providers: [
|
||||
FileUploadService,
|
||||
FileUploadResolver,
|
||||
FileUploadTokenGuard,
|
||||
provideWorkspaceScopedRepository(FileEntity),
|
||||
],
|
||||
exports: [FileUploadService],
|
||||
controllers: [FileUploadController],
|
||||
})
|
||||
export class FileUploadModule {}
|
||||
+54
@@ -0,0 +1,54 @@
|
||||
import {
|
||||
type ArgumentsHost,
|
||||
Catch,
|
||||
type ExceptionFilter,
|
||||
} from '@nestjs/common';
|
||||
|
||||
import { type Response } from 'express';
|
||||
|
||||
import { HttpExceptionHandlerService } from 'src/engine/core-modules/exception-handler/http-exception-handler.service';
|
||||
import {
|
||||
FileUploadException,
|
||||
FileUploadExceptionCode,
|
||||
} from 'src/engine/core-modules/file/file-upload/file-upload.exception';
|
||||
|
||||
@Catch(FileUploadException)
|
||||
export class FileUploadApiExceptionFilter implements ExceptionFilter {
|
||||
constructor(
|
||||
private readonly httpExceptionHandlerService: HttpExceptionHandlerService,
|
||||
) {}
|
||||
|
||||
catch(exception: FileUploadException, host: ArgumentsHost) {
|
||||
const ctx = host.switchToHttp();
|
||||
const response = ctx.getResponse<Response>();
|
||||
|
||||
switch (exception.code) {
|
||||
case FileUploadExceptionCode.FILE_NOT_FOUND:
|
||||
return this.httpExceptionHandlerService.handleError(
|
||||
exception,
|
||||
response,
|
||||
404,
|
||||
);
|
||||
case FileUploadExceptionCode.FILE_TOO_LARGE:
|
||||
return this.httpExceptionHandlerService.handleError(
|
||||
exception,
|
||||
response,
|
||||
413,
|
||||
);
|
||||
case FileUploadExceptionCode.BAD_REQUEST:
|
||||
case FileUploadExceptionCode.FILE_NOT_UPLOADED:
|
||||
case FileUploadExceptionCode.FILE_SIZE_MISMATCH:
|
||||
return this.httpExceptionHandlerService.handleError(
|
||||
exception,
|
||||
response,
|
||||
400,
|
||||
);
|
||||
default:
|
||||
return this.httpExceptionHandlerService.handleError(
|
||||
exception,
|
||||
response,
|
||||
500,
|
||||
);
|
||||
}
|
||||
}
|
||||
}
|
||||
+42
@@ -0,0 +1,42 @@
|
||||
import { CanActivate, ExecutionContext, Injectable } from '@nestjs/common';
|
||||
|
||||
import { FileUploadTokenJwtPayload } from 'src/engine/core-modules/auth/types/file-upload-token-jwt-payload.type';
|
||||
import { JwtTokenTypeEnum } from 'src/engine/core-modules/auth/types/jwt-token-type.enum';
|
||||
import { JwtWrapperService } from 'src/engine/core-modules/jwt/services/jwt-wrapper.service';
|
||||
|
||||
@Injectable()
|
||||
export class FileUploadTokenGuard implements CanActivate {
|
||||
constructor(private readonly jwtWrapperService: JwtWrapperService) {}
|
||||
|
||||
async canActivate(context: ExecutionContext): Promise<boolean> {
|
||||
const request = context.switchToHttp().getRequest();
|
||||
const fileId = request.params.id;
|
||||
const uploadToken = request.query.token;
|
||||
|
||||
if (!uploadToken) {
|
||||
return false;
|
||||
}
|
||||
|
||||
let payload: FileUploadTokenJwtPayload;
|
||||
|
||||
try {
|
||||
payload = await this.jwtWrapperService.verifyJwtToken(uploadToken);
|
||||
} catch {
|
||||
return false;
|
||||
}
|
||||
|
||||
// A FILE (download) token also carries workspaceId + fileId: reject
|
||||
// anything that is not explicitly an upload token.
|
||||
if (payload.type !== JwtTokenTypeEnum.FILE_UPLOAD) {
|
||||
return false;
|
||||
}
|
||||
|
||||
if (!payload.workspaceId || payload.fileId !== fileId) {
|
||||
return false;
|
||||
}
|
||||
|
||||
request.workspaceId = payload.workspaceId;
|
||||
|
||||
return true;
|
||||
}
|
||||
}
|
||||
+68
@@ -0,0 +1,68 @@
|
||||
import { UseFilters, UseGuards, UsePipes } from '@nestjs/common';
|
||||
import { Args, Mutation } from '@nestjs/graphql';
|
||||
|
||||
import { PermissionFlagType } from 'twenty-shared/constants';
|
||||
import { FileFolder } from 'twenty-shared/types';
|
||||
|
||||
import { MetadataResolver } from 'src/engine/api/graphql/graphql-config/decorators/metadata-resolver.decorator';
|
||||
import { FileWithSignedUrlDTO } from 'src/engine/core-modules/file/dtos/file-with-sign-url.dto';
|
||||
import { FileUploadTargetDTO } from 'src/engine/core-modules/file/file-upload/dtos/file-upload-target.dto';
|
||||
import { FileUploadService } from 'src/engine/core-modules/file/file-upload/services/file-upload.service';
|
||||
import { PreventNestToAutoLogGraphqlErrorsFilter } from 'src/engine/core-modules/graphql/filters/prevent-nest-to-auto-log-graphql-errors.filter';
|
||||
import { ResolverValidationPipe } from 'src/engine/core-modules/graphql/pipes/resolver-validation.pipe';
|
||||
import { WorkspaceEntity } from 'src/engine/core-modules/workspace/workspace.entity';
|
||||
import { AuthWorkspace } from 'src/engine/decorators/auth/auth-workspace.decorator';
|
||||
import { SettingsPermissionGuard } from 'src/engine/guards/settings-permission.guard';
|
||||
import { WorkspaceAuthGuard } from 'src/engine/guards/workspace-auth.guard';
|
||||
|
||||
@UseGuards(WorkspaceAuthGuard)
|
||||
@UsePipes(ResolverValidationPipe)
|
||||
@UseFilters(PreventNestToAutoLogGraphqlErrorsFilter)
|
||||
@MetadataResolver()
|
||||
export class FileUploadResolver {
|
||||
constructor(private readonly fileUploadService: FileUploadService) {}
|
||||
|
||||
@Mutation(() => FileUploadTargetDTO)
|
||||
@UseGuards(SettingsPermissionGuard(PermissionFlagType.UPLOAD_FILE))
|
||||
async createFileUpload(
|
||||
@AuthWorkspace()
|
||||
{ id: workspaceId }: WorkspaceEntity,
|
||||
@Args({ name: 'filename', type: () => String })
|
||||
filename: string,
|
||||
@Args({ name: 'size', type: () => Number })
|
||||
size: number,
|
||||
@Args({ name: 'fileFolder', type: () => FileFolder })
|
||||
fileFolder: FileFolder,
|
||||
@Args({ name: 'fieldMetadataId', type: () => String, nullable: true })
|
||||
fieldMetadataId?: string,
|
||||
@Args({
|
||||
name: 'fieldMetadataUniversalIdentifier',
|
||||
type: () => String,
|
||||
nullable: true,
|
||||
})
|
||||
fieldMetadataUniversalIdentifier?: string,
|
||||
): Promise<FileUploadTargetDTO> {
|
||||
return await this.fileUploadService.createFileUpload({
|
||||
workspaceId,
|
||||
filename,
|
||||
size,
|
||||
fileFolder,
|
||||
fieldMetadataId,
|
||||
fieldMetadataUniversalIdentifier,
|
||||
});
|
||||
}
|
||||
|
||||
@Mutation(() => FileWithSignedUrlDTO)
|
||||
@UseGuards(SettingsPermissionGuard(PermissionFlagType.UPLOAD_FILE))
|
||||
async completeFileUpload(
|
||||
@AuthWorkspace()
|
||||
{ id: workspaceId }: WorkspaceEntity,
|
||||
@Args({ name: 'fileId', type: () => String })
|
||||
fileId: string,
|
||||
): Promise<FileWithSignedUrlDTO> {
|
||||
return await this.fileUploadService.completeFileUpload({
|
||||
workspaceId,
|
||||
fileId,
|
||||
});
|
||||
}
|
||||
}
|
||||
+309
@@ -0,0 +1,309 @@
|
||||
import { Test, type TestingModule } from '@nestjs/testing';
|
||||
import { getRepositoryToken } from '@nestjs/typeorm';
|
||||
|
||||
import { FileFolder } from 'twenty-shared/types';
|
||||
|
||||
import { ApplicationEntity } from 'src/engine/core-modules/application/application.entity';
|
||||
import { ApplicationService } from 'src/engine/core-modules/application/application.service';
|
||||
import { FileStorageService } from 'src/engine/core-modules/file-storage/file-storage.service';
|
||||
import { FileEntity } from 'src/engine/core-modules/file/entities/file.entity';
|
||||
import {
|
||||
FileUploadException,
|
||||
FileUploadExceptionCode,
|
||||
} from 'src/engine/core-modules/file/file-upload/file-upload.exception';
|
||||
import { FileUploadService } from 'src/engine/core-modules/file/file-upload/services/file-upload.service';
|
||||
import { FileUrlService } from 'src/engine/core-modules/file/file-url/file-url.service';
|
||||
import { FILE_STATUS } from 'src/engine/core-modules/file/types/file-status.types';
|
||||
import { JwtWrapperService } from 'src/engine/core-modules/jwt/services/jwt-wrapper.service';
|
||||
import { TwentyConfigService } from 'src/engine/core-modules/twenty-config/twenty-config.service';
|
||||
import { FieldMetadataEntity } from 'src/engine/metadata-modules/field-metadata/field-metadata.entity';
|
||||
import { getWorkspaceScopedRepositoryToken } from 'src/engine/twenty-orm/workspace-scoped-repository/get-workspace-scoped-repository-token.util';
|
||||
|
||||
jest.mock('uuid', () => ({
|
||||
v4: jest.fn(() => 'mocked-file-id'),
|
||||
}));
|
||||
|
||||
describe('FileUploadService', () => {
|
||||
let service: FileUploadService;
|
||||
|
||||
const fileStorageService = {
|
||||
createPendingFile: jest.fn(),
|
||||
getPresignedUploadUrl: jest.fn(),
|
||||
getFileMetadata: jest.fn(),
|
||||
writeFileStream: jest.fn(),
|
||||
};
|
||||
|
||||
const fileUrlService = {
|
||||
signFileByIdUrl: jest.fn().mockResolvedValue('https://signed-url'),
|
||||
};
|
||||
|
||||
const jwtWrapperService = {
|
||||
signAsyncOrThrow: jest.fn().mockResolvedValue('upload-token'),
|
||||
};
|
||||
|
||||
const twentyConfigService = {
|
||||
get: jest.fn((key: string) => {
|
||||
if (key === 'STORAGE_S3_PRESIGNED_URL_EXPIRES_IN') {
|
||||
return 900;
|
||||
}
|
||||
if (key === 'SERVER_URL') {
|
||||
return 'https://server.tld';
|
||||
}
|
||||
|
||||
return undefined;
|
||||
}),
|
||||
};
|
||||
|
||||
const applicationService = {
|
||||
findWorkspaceTwentyStandardAndCustomApplicationOrThrow: jest
|
||||
.fn()
|
||||
.mockResolvedValue({
|
||||
workspaceCustomFlatApplication: {
|
||||
universalIdentifier: 'custom-app-uid',
|
||||
},
|
||||
}),
|
||||
};
|
||||
|
||||
const applicationRepository = {
|
||||
findOneOrFail: jest.fn().mockResolvedValue({
|
||||
id: 'application-id',
|
||||
universalIdentifier: 'application-uid',
|
||||
}),
|
||||
};
|
||||
|
||||
const fieldMetadataRepository = {
|
||||
findOneOrFail: jest.fn().mockResolvedValue({
|
||||
applicationId: 'application-id',
|
||||
universalIdentifier: 'field-metadata-uid',
|
||||
}),
|
||||
};
|
||||
|
||||
const fileRepository = {
|
||||
findOne: jest.fn(),
|
||||
update: jest.fn(),
|
||||
};
|
||||
|
||||
beforeEach(async () => {
|
||||
jest.clearAllMocks();
|
||||
|
||||
const module: TestingModule = await Test.createTestingModule({
|
||||
providers: [
|
||||
FileUploadService,
|
||||
{ provide: FileStorageService, useValue: fileStorageService },
|
||||
{ provide: FileUrlService, useValue: fileUrlService },
|
||||
{ provide: JwtWrapperService, useValue: jwtWrapperService },
|
||||
{ provide: TwentyConfigService, useValue: twentyConfigService },
|
||||
{ provide: ApplicationService, useValue: applicationService },
|
||||
{
|
||||
provide: getRepositoryToken(ApplicationEntity),
|
||||
useValue: applicationRepository,
|
||||
},
|
||||
{
|
||||
provide: getRepositoryToken(FieldMetadataEntity),
|
||||
useValue: fieldMetadataRepository,
|
||||
},
|
||||
{
|
||||
provide: getWorkspaceScopedRepositoryToken(FileEntity),
|
||||
useValue: fileRepository,
|
||||
},
|
||||
],
|
||||
}).compile();
|
||||
|
||||
service = module.get<FileUploadService>(FileUploadService);
|
||||
});
|
||||
|
||||
describe('createFileUpload', () => {
|
||||
it('should reject file folders without direct upload support', async () => {
|
||||
await expect(
|
||||
service.createFileUpload({
|
||||
workspaceId: 'workspace-id',
|
||||
filename: 'document.pdf',
|
||||
size: 1024,
|
||||
fileFolder: FileFolder.CorePicture,
|
||||
}),
|
||||
).rejects.toThrow(FileUploadException);
|
||||
});
|
||||
|
||||
it('should reject an invalid declared size', async () => {
|
||||
await expect(
|
||||
service.createFileUpload({
|
||||
workspaceId: 'workspace-id',
|
||||
filename: 'document.pdf',
|
||||
size: 0,
|
||||
fileFolder: FileFolder.FilesField,
|
||||
fieldMetadataId: 'field-metadata-id',
|
||||
}),
|
||||
).rejects.toThrow(FileUploadException);
|
||||
});
|
||||
|
||||
it('should create a PENDING file and return the presigned url when storage supports it', async () => {
|
||||
fileStorageService.getPresignedUploadUrl.mockResolvedValueOnce(
|
||||
'https://bucket/presigned-put',
|
||||
);
|
||||
|
||||
const result = await service.createFileUpload({
|
||||
workspaceId: 'workspace-id',
|
||||
filename: 'document.pdf',
|
||||
size: 1024,
|
||||
fileFolder: FileFolder.FilesField,
|
||||
fieldMetadataId: 'field-metadata-id',
|
||||
});
|
||||
|
||||
expect(fileStorageService.createPendingFile).toHaveBeenCalledWith(
|
||||
expect.objectContaining({
|
||||
fileId: 'mocked-file-id',
|
||||
size: 1024,
|
||||
mimeType: 'application/pdf',
|
||||
resourcePath: 'field-metadata-uid/mocked-file-id.pdf',
|
||||
settings: { isTemporaryFile: true, toDelete: false },
|
||||
}),
|
||||
);
|
||||
expect(result.uploadUrl).toBe('https://bucket/presigned-put');
|
||||
expect(result.contentType).toBe('application/pdf');
|
||||
expect(result.fileId).toBe('mocked-file-id');
|
||||
});
|
||||
|
||||
it('should fall back to the server streaming endpoint when presign is unavailable', async () => {
|
||||
fileStorageService.getPresignedUploadUrl.mockResolvedValueOnce(null);
|
||||
|
||||
const result = await service.createFileUpload({
|
||||
workspaceId: 'workspace-id',
|
||||
filename: 'archive.zip',
|
||||
size: 2048,
|
||||
fileFolder: FileFolder.Workflow,
|
||||
});
|
||||
|
||||
expect(jwtWrapperService.signAsyncOrThrow).toHaveBeenCalledWith(
|
||||
expect.objectContaining({
|
||||
workspaceId: 'workspace-id',
|
||||
fileId: 'mocked-file-id',
|
||||
}),
|
||||
{ expiresIn: 900 },
|
||||
);
|
||||
expect(result.uploadUrl).toBe(
|
||||
'https://server.tld/file-upload/mocked-file-id?token=upload-token',
|
||||
);
|
||||
expect(result.contentType).toBe('application/octet-stream');
|
||||
});
|
||||
});
|
||||
|
||||
describe('completeFileUpload', () => {
|
||||
const pendingFile = {
|
||||
id: 'file-id',
|
||||
path: 'files-field/field-metadata-uid/file-id.pdf',
|
||||
size: 1024,
|
||||
applicationId: 'application-id',
|
||||
mimeType: 'application/pdf',
|
||||
status: FILE_STATUS.PENDING,
|
||||
settings: { isTemporaryFile: true, toDelete: false },
|
||||
createdAt: new Date(),
|
||||
};
|
||||
|
||||
it('should throw when the file record does not exist', async () => {
|
||||
fileRepository.findOne.mockResolvedValueOnce(null);
|
||||
|
||||
await expect(
|
||||
service.completeFileUpload({
|
||||
workspaceId: 'workspace-id',
|
||||
fileId: 'file-id',
|
||||
}),
|
||||
).rejects.toThrow(FileUploadException);
|
||||
});
|
||||
|
||||
it('should throw when the bytes are not in storage yet', async () => {
|
||||
fileRepository.findOne.mockResolvedValueOnce(pendingFile);
|
||||
fileStorageService.getFileMetadata.mockResolvedValueOnce(null);
|
||||
|
||||
await expect(
|
||||
service.completeFileUpload({
|
||||
workspaceId: 'workspace-id',
|
||||
fileId: 'file-id',
|
||||
}),
|
||||
).rejects.toMatchObject({
|
||||
code: FileUploadExceptionCode.FILE_NOT_UPLOADED,
|
||||
});
|
||||
expect(fileRepository.update).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it('should throw when the stored size does not match the declared size', async () => {
|
||||
fileRepository.findOne.mockResolvedValueOnce(pendingFile);
|
||||
fileStorageService.getFileMetadata.mockResolvedValueOnce({ size: 999 });
|
||||
|
||||
await expect(
|
||||
service.completeFileUpload({
|
||||
workspaceId: 'workspace-id',
|
||||
fileId: 'file-id',
|
||||
}),
|
||||
).rejects.toMatchObject({
|
||||
code: FileUploadExceptionCode.FILE_SIZE_MISMATCH,
|
||||
});
|
||||
expect(fileRepository.update).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it('should flip the file to UPLOADED when the stored size matches', async () => {
|
||||
fileRepository.findOne.mockResolvedValueOnce(pendingFile);
|
||||
fileStorageService.getFileMetadata.mockResolvedValueOnce({ size: 1024 });
|
||||
|
||||
const result = await service.completeFileUpload({
|
||||
workspaceId: 'workspace-id',
|
||||
fileId: 'file-id',
|
||||
});
|
||||
|
||||
expect(fileRepository.update).toHaveBeenCalledWith(
|
||||
'workspace-id',
|
||||
{ id: 'file-id' },
|
||||
{ status: FILE_STATUS.UPLOADED },
|
||||
);
|
||||
expect(result.url).toBe('https://signed-url');
|
||||
});
|
||||
|
||||
it('should be idempotent when the file is already UPLOADED', async () => {
|
||||
fileRepository.findOne.mockResolvedValueOnce({
|
||||
...pendingFile,
|
||||
status: FILE_STATUS.UPLOADED,
|
||||
});
|
||||
|
||||
const result = await service.completeFileUpload({
|
||||
workspaceId: 'workspace-id',
|
||||
fileId: 'file-id',
|
||||
});
|
||||
|
||||
expect(fileStorageService.getFileMetadata).not.toHaveBeenCalled();
|
||||
expect(fileRepository.update).not.toHaveBeenCalled();
|
||||
expect(result.url).toBe('https://signed-url');
|
||||
});
|
||||
|
||||
it('should refuse confirming files that already left the upload flow', async () => {
|
||||
fileRepository.findOne.mockResolvedValueOnce({
|
||||
...pendingFile,
|
||||
status: FILE_STATUS.UPLOADED,
|
||||
settings: { isTemporaryFile: false, toDelete: false },
|
||||
});
|
||||
|
||||
await expect(
|
||||
service.completeFileUpload({
|
||||
workspaceId: 'workspace-id',
|
||||
fileId: 'file-id',
|
||||
}),
|
||||
).rejects.toMatchObject({
|
||||
code: FileUploadExceptionCode.BAD_REQUEST,
|
||||
});
|
||||
});
|
||||
|
||||
it('should refuse files outside direct-upload folders', async () => {
|
||||
fileRepository.findOne.mockResolvedValueOnce({
|
||||
...pendingFile,
|
||||
path: 'core-picture/file-id.png',
|
||||
});
|
||||
|
||||
await expect(
|
||||
service.completeFileUpload({
|
||||
workspaceId: 'workspace-id',
|
||||
fileId: 'file-id',
|
||||
}),
|
||||
).rejects.toMatchObject({
|
||||
code: FileUploadExceptionCode.FILE_NOT_FOUND,
|
||||
});
|
||||
});
|
||||
});
|
||||
});
|
||||
+511
@@ -0,0 +1,511 @@
|
||||
import { Injectable } from '@nestjs/common';
|
||||
import { InjectRepository } from '@nestjs/typeorm';
|
||||
|
||||
import { type Readable, Transform } from 'stream';
|
||||
import { pipeline } from 'stream/promises';
|
||||
|
||||
import { msg } from '@lingui/core/macro';
|
||||
import { isNonEmptyString } from '@sniptt/guards';
|
||||
import bytes from 'bytes';
|
||||
import { lookup } from 'mrmime';
|
||||
import { FileFolder } from 'twenty-shared/types';
|
||||
import { isDefined } from 'twenty-shared/utils';
|
||||
import { Repository } from 'typeorm';
|
||||
import { v4 } from 'uuid';
|
||||
|
||||
import { settings } from 'src/engine/constants/settings';
|
||||
import { ApplicationEntity } from 'src/engine/core-modules/application/application.entity';
|
||||
import { ApplicationService } from 'src/engine/core-modules/application/application.service';
|
||||
import { FileUploadTokenJwtPayload } from 'src/engine/core-modules/auth/types/file-upload-token-jwt-payload.type';
|
||||
import { JwtTokenTypeEnum } from 'src/engine/core-modules/auth/types/jwt-token-type.enum';
|
||||
import { FileStorageService } from 'src/engine/core-modules/file-storage/file-storage.service';
|
||||
import { TWENTY_MIME_POLICY } from 'src/engine/core-modules/file/constants/twenty-mime-policy.constant';
|
||||
import { FileWithSignedUrlDTO } from 'src/engine/core-modules/file/dtos/file-with-sign-url.dto';
|
||||
import { FileEntity } from 'src/engine/core-modules/file/entities/file.entity';
|
||||
import { FileUploadTargetDTO } from 'src/engine/core-modules/file/file-upload/dtos/file-upload-target.dto';
|
||||
import {
|
||||
FileUploadException,
|
||||
FileUploadExceptionCode,
|
||||
} from 'src/engine/core-modules/file/file-upload/file-upload.exception';
|
||||
import { FileUrlService } from 'src/engine/core-modules/file/file-url/file-url.service';
|
||||
import { FILE_STATUS } from 'src/engine/core-modules/file/types/file-status.types';
|
||||
import { buildFileInfo } from 'src/engine/core-modules/file/utils/build-file-info.utils';
|
||||
import { removeFileFolderFromFileEntityPath } from 'src/engine/core-modules/file/utils/remove-file-folder-from-file-entity-path.utils';
|
||||
import { JwtWrapperService } from 'src/engine/core-modules/jwt/services/jwt-wrapper.service';
|
||||
import { TwentyConfigService } from 'src/engine/core-modules/twenty-config/twenty-config.service';
|
||||
import { FieldMetadataEntity } from 'src/engine/metadata-modules/field-metadata/field-metadata.entity';
|
||||
import { InjectWorkspaceScopedRepository } from 'src/engine/twenty-orm/workspace-scoped-repository/inject-workspace-scoped-repository.decorator';
|
||||
import { WorkspaceScopedRepository } from 'src/engine/twenty-orm/workspace-scoped-repository/workspace-scoped-repository';
|
||||
|
||||
export const DIRECT_UPLOAD_FILE_FOLDERS = [
|
||||
FileFolder.FilesField,
|
||||
FileFolder.Workflow,
|
||||
] as const;
|
||||
|
||||
@Injectable()
|
||||
export class FileUploadService {
|
||||
constructor(
|
||||
private readonly fileStorageService: FileStorageService,
|
||||
private readonly fileUrlService: FileUrlService,
|
||||
private readonly jwtWrapperService: JwtWrapperService,
|
||||
private readonly twentyConfigService: TwentyConfigService,
|
||||
private readonly applicationService: ApplicationService,
|
||||
@InjectRepository(ApplicationEntity)
|
||||
private readonly applicationRepository: Repository<ApplicationEntity>,
|
||||
@InjectRepository(FieldMetadataEntity)
|
||||
private readonly fieldMetadataRepository: Repository<FieldMetadataEntity>,
|
||||
@InjectWorkspaceScopedRepository(FileEntity)
|
||||
private readonly fileRepository: WorkspaceScopedRepository<FileEntity>,
|
||||
) {}
|
||||
|
||||
async createFileUpload({
|
||||
workspaceId,
|
||||
filename,
|
||||
size,
|
||||
fileFolder,
|
||||
fieldMetadataId,
|
||||
fieldMetadataUniversalIdentifier,
|
||||
}: {
|
||||
workspaceId: string;
|
||||
filename: string;
|
||||
size: number;
|
||||
fileFolder: FileFolder;
|
||||
fieldMetadataId?: string;
|
||||
fieldMetadataUniversalIdentifier?: string;
|
||||
}): Promise<FileUploadTargetDTO> {
|
||||
if (
|
||||
!DIRECT_UPLOAD_FILE_FOLDERS.includes(
|
||||
fileFolder as (typeof DIRECT_UPLOAD_FILE_FOLDERS)[number],
|
||||
)
|
||||
) {
|
||||
throw new FileUploadException(
|
||||
`Direct upload is not supported for file folder ${fileFolder}`,
|
||||
FileUploadExceptionCode.BAD_REQUEST,
|
||||
{
|
||||
userFriendlyMessage: msg`Direct upload is not supported for this file type.`,
|
||||
},
|
||||
);
|
||||
}
|
||||
|
||||
const maxFileSize = bytes(settings.storage.maxDirectUploadFileSize) ?? 0;
|
||||
|
||||
if (!Number.isInteger(size) || size <= 0 || size > maxFileSize) {
|
||||
throw new FileUploadException(
|
||||
`Invalid file size ${size} (max ${maxFileSize} bytes)`,
|
||||
FileUploadExceptionCode.FILE_TOO_LARGE,
|
||||
{
|
||||
userFriendlyMessage: msg`The file is empty or exceeds the maximum allowed size.`,
|
||||
},
|
||||
);
|
||||
}
|
||||
|
||||
const { ext } = buildFileInfo(filename);
|
||||
// The file content cannot be sniffed before it reaches storage, so the
|
||||
// mime type is derived from the extension only. Anything unknown is
|
||||
// stored as octet-stream, and the serving path already forces
|
||||
// Content-Disposition: attachment for non-inline-safe mime types.
|
||||
const mimeType =
|
||||
TWENTY_MIME_POLICY[ext] ??
|
||||
(isNonEmptyString(ext) ? lookup(ext) : undefined) ??
|
||||
'application/octet-stream';
|
||||
|
||||
const fileId = v4();
|
||||
const name = `${fileId}${isNonEmptyString(ext) ? `.${ext}` : ''}`;
|
||||
|
||||
const { applicationUniversalIdentifier, resourcePath } =
|
||||
await this.resolveUploadLocation({
|
||||
workspaceId,
|
||||
fileFolder,
|
||||
name,
|
||||
fieldMetadataId,
|
||||
fieldMetadataUniversalIdentifier,
|
||||
});
|
||||
|
||||
await this.fileStorageService.createPendingFile({
|
||||
fileFolder,
|
||||
applicationUniversalIdentifier,
|
||||
workspaceId,
|
||||
resourcePath,
|
||||
fileId,
|
||||
size,
|
||||
mimeType,
|
||||
settings: {
|
||||
isTemporaryFile: true,
|
||||
toDelete: false,
|
||||
},
|
||||
});
|
||||
|
||||
const expiresInSeconds = this.twentyConfigService.get(
|
||||
'STORAGE_S3_PRESIGNED_URL_EXPIRES_IN',
|
||||
);
|
||||
const expiresAt = new Date(Date.now() + expiresInSeconds * 1000);
|
||||
|
||||
const presignedUploadUrl =
|
||||
await this.fileStorageService.getPresignedUploadUrl({
|
||||
fileFolder,
|
||||
applicationUniversalIdentifier,
|
||||
workspaceId,
|
||||
resourcePath,
|
||||
contentType: mimeType,
|
||||
contentLength: size,
|
||||
expiresInSeconds,
|
||||
});
|
||||
|
||||
if (isDefined(presignedUploadUrl)) {
|
||||
return {
|
||||
fileId,
|
||||
uploadUrl: presignedUploadUrl,
|
||||
contentType: mimeType,
|
||||
expiresAt,
|
||||
};
|
||||
}
|
||||
|
||||
// No presign support (local storage, or S3 without presign enabled):
|
||||
// fall back to the token-authenticated streaming endpoint on the server.
|
||||
const payload: FileUploadTokenJwtPayload = {
|
||||
workspaceId,
|
||||
fileId,
|
||||
sub: workspaceId,
|
||||
type: JwtTokenTypeEnum.FILE_UPLOAD,
|
||||
};
|
||||
|
||||
const token = await this.jwtWrapperService.signAsyncOrThrow(payload, {
|
||||
expiresIn: expiresInSeconds,
|
||||
});
|
||||
|
||||
const serverUrl = this.twentyConfigService.get('SERVER_URL');
|
||||
|
||||
return {
|
||||
fileId,
|
||||
uploadUrl: `${serverUrl}/file-upload/${fileId}?token=${token}`,
|
||||
// octet-stream keeps the request body away from the server's json/text
|
||||
// body parsers; the real mime type is already on the file record.
|
||||
contentType: 'application/octet-stream',
|
||||
expiresAt,
|
||||
};
|
||||
}
|
||||
|
||||
// Streams the request body straight to the storage driver, bounded by the
|
||||
// size declared at createFileUpload time. Memory usage stays constant no
|
||||
// matter how large the file is.
|
||||
async receiveFileStream({
|
||||
workspaceId,
|
||||
fileId,
|
||||
stream,
|
||||
}: {
|
||||
workspaceId: string;
|
||||
fileId: string;
|
||||
stream: Readable;
|
||||
}): Promise<void> {
|
||||
const file = await this.findFileOrThrow({ workspaceId, fileId });
|
||||
|
||||
if (file.status !== FILE_STATUS.PENDING) {
|
||||
throw new FileUploadException(
|
||||
`File ${fileId} is not awaiting an upload`,
|
||||
FileUploadExceptionCode.BAD_REQUEST,
|
||||
{
|
||||
userFriendlyMessage: msg`This file has already been uploaded.`,
|
||||
},
|
||||
);
|
||||
}
|
||||
|
||||
const { application, fileFolder, resourcePath } =
|
||||
await this.resolveFileLocation({ workspaceId, file });
|
||||
|
||||
const declaredSize = Number(file.size);
|
||||
let receivedBytes = 0;
|
||||
|
||||
const sizeLimiter = new Transform({
|
||||
transform: (chunk: Buffer, _encoding, callback) => {
|
||||
receivedBytes += chunk.length;
|
||||
|
||||
if (receivedBytes > declaredSize) {
|
||||
callback(
|
||||
new FileUploadException(
|
||||
`Upload exceeds declared size of ${declaredSize} bytes`,
|
||||
FileUploadExceptionCode.FILE_TOO_LARGE,
|
||||
{
|
||||
userFriendlyMessage: msg`The uploaded file is larger than declared.`,
|
||||
},
|
||||
),
|
||||
);
|
||||
|
||||
return;
|
||||
}
|
||||
|
||||
callback(null, chunk);
|
||||
},
|
||||
});
|
||||
|
||||
try {
|
||||
await Promise.all([
|
||||
pipeline(stream, sizeLimiter),
|
||||
this.fileStorageService.writeFileStream({
|
||||
fileFolder,
|
||||
applicationUniversalIdentifier: application.universalIdentifier,
|
||||
workspaceId,
|
||||
resourcePath,
|
||||
stream: sizeLimiter,
|
||||
mimeType: file.mimeType,
|
||||
}),
|
||||
]);
|
||||
} catch (error) {
|
||||
// The storage-side pipeline can lose the rejection race to the limiter:
|
||||
// surface the size violation over the resulting stream teardown error.
|
||||
if (receivedBytes > declaredSize) {
|
||||
throw new FileUploadException(
|
||||
`Upload exceeds declared size of ${declaredSize} bytes`,
|
||||
FileUploadExceptionCode.FILE_TOO_LARGE,
|
||||
{
|
||||
userFriendlyMessage: msg`The uploaded file is larger than declared.`,
|
||||
},
|
||||
);
|
||||
}
|
||||
|
||||
throw error;
|
||||
}
|
||||
|
||||
if (receivedBytes !== declaredSize) {
|
||||
// The short object stays in storage but the record stays PENDING, so it
|
||||
// can never be served or attached: the client retries against the same
|
||||
// upload url (overwriting it), and the pending-file cleanup cron
|
||||
// (follow-up PR) reaps whatever is abandoned.
|
||||
throw new FileUploadException(
|
||||
`Uploaded ${receivedBytes} bytes but ${declaredSize} were declared`,
|
||||
FileUploadExceptionCode.FILE_SIZE_MISMATCH,
|
||||
{
|
||||
userFriendlyMessage: msg`The uploaded file does not match the declared size. Please retry the upload.`,
|
||||
},
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
// Verifies the bytes actually landed in storage with the declared size and
|
||||
// flips the file to UPLOADED. Idempotent: confirming twice is a no-op.
|
||||
async completeFileUpload({
|
||||
workspaceId,
|
||||
fileId,
|
||||
}: {
|
||||
workspaceId: string;
|
||||
fileId: string;
|
||||
}): Promise<FileWithSignedUrlDTO> {
|
||||
const file = await this.findFileOrThrow({ workspaceId, fileId });
|
||||
const [fileFolder] = file.path.split('/');
|
||||
|
||||
// Restrict to files created through createFileUpload so this mutation
|
||||
// cannot be used to mint signed download urls for arbitrary files.
|
||||
if (
|
||||
!DIRECT_UPLOAD_FILE_FOLDERS.includes(
|
||||
fileFolder as (typeof DIRECT_UPLOAD_FILE_FOLDERS)[number],
|
||||
)
|
||||
) {
|
||||
throw new FileUploadException(
|
||||
`File not found: ${fileId}`,
|
||||
FileUploadExceptionCode.FILE_NOT_FOUND,
|
||||
{
|
||||
userFriendlyMessage: msg`File not found.`,
|
||||
},
|
||||
);
|
||||
}
|
||||
|
||||
if (file.status === FILE_STATUS.UPLOADED) {
|
||||
// Idempotent retry of a confirm that already succeeded. Only files not
|
||||
// yet attached to a record qualify: this cannot be used to mint signed
|
||||
// urls for files that went through the legacy flow and got attached.
|
||||
if (!file.settings?.isTemporaryFile) {
|
||||
throw new FileUploadException(
|
||||
`File ${fileId} is not awaiting an upload confirmation`,
|
||||
FileUploadExceptionCode.BAD_REQUEST,
|
||||
{
|
||||
userFriendlyMessage: msg`This file upload has already been finalized.`,
|
||||
},
|
||||
);
|
||||
}
|
||||
|
||||
return this.toFileWithSignedUrl({
|
||||
file,
|
||||
fileFolder: fileFolder as FileFolder,
|
||||
workspaceId,
|
||||
});
|
||||
}
|
||||
|
||||
const { application, resourcePath } = await this.resolveFileLocation({
|
||||
workspaceId,
|
||||
file,
|
||||
});
|
||||
|
||||
const metadata = await this.fileStorageService.getFileMetadata({
|
||||
fileFolder: fileFolder as FileFolder,
|
||||
applicationUniversalIdentifier: application.universalIdentifier,
|
||||
workspaceId,
|
||||
resourcePath,
|
||||
});
|
||||
|
||||
if (!isDefined(metadata)) {
|
||||
throw new FileUploadException(
|
||||
`File ${fileId} has not been uploaded to storage yet`,
|
||||
FileUploadExceptionCode.FILE_NOT_UPLOADED,
|
||||
{
|
||||
userFriendlyMessage: msg`The file has not been uploaded yet. Please upload it before confirming.`,
|
||||
},
|
||||
);
|
||||
}
|
||||
|
||||
if (metadata.size !== Number(file.size)) {
|
||||
throw new FileUploadException(
|
||||
`File ${fileId} has ${metadata.size} bytes in storage but ${file.size} were declared`,
|
||||
FileUploadExceptionCode.FILE_SIZE_MISMATCH,
|
||||
{
|
||||
userFriendlyMessage: msg`The uploaded file does not match the declared size. Please retry the upload.`,
|
||||
},
|
||||
);
|
||||
}
|
||||
|
||||
await this.fileRepository.update(
|
||||
workspaceId,
|
||||
{ id: fileId },
|
||||
{ status: FILE_STATUS.UPLOADED },
|
||||
);
|
||||
|
||||
return this.toFileWithSignedUrl({
|
||||
file: { ...file, status: FILE_STATUS.UPLOADED },
|
||||
fileFolder: fileFolder as FileFolder,
|
||||
workspaceId,
|
||||
});
|
||||
}
|
||||
|
||||
private async resolveUploadLocation({
|
||||
workspaceId,
|
||||
fileFolder,
|
||||
name,
|
||||
fieldMetadataId,
|
||||
fieldMetadataUniversalIdentifier,
|
||||
}: {
|
||||
workspaceId: string;
|
||||
fileFolder: FileFolder;
|
||||
name: string;
|
||||
fieldMetadataId?: string;
|
||||
fieldMetadataUniversalIdentifier?: string;
|
||||
}): Promise<{
|
||||
applicationUniversalIdentifier: string;
|
||||
resourcePath: string;
|
||||
}> {
|
||||
if (fileFolder === FileFolder.FilesField) {
|
||||
if (!fieldMetadataId && !fieldMetadataUniversalIdentifier) {
|
||||
throw new FileUploadException(
|
||||
'fieldMetadataId or fieldMetadataUniversalIdentifier must be provided',
|
||||
FileUploadExceptionCode.BAD_REQUEST,
|
||||
{
|
||||
userFriendlyMessage: msg`fieldMetadataId or fieldMetadataUniversalIdentifier must be provided`,
|
||||
},
|
||||
);
|
||||
}
|
||||
|
||||
const fieldMetadata = await this.fieldMetadataRepository.findOneOrFail({
|
||||
select: ['applicationId', 'universalIdentifier'],
|
||||
where: {
|
||||
...(fieldMetadataId ? { id: fieldMetadataId } : {}),
|
||||
...(fieldMetadataUniversalIdentifier
|
||||
? { universalIdentifier: fieldMetadataUniversalIdentifier }
|
||||
: {}),
|
||||
workspaceId,
|
||||
},
|
||||
});
|
||||
|
||||
const application = await this.applicationRepository.findOneOrFail({
|
||||
where: {
|
||||
id: fieldMetadata.applicationId,
|
||||
workspaceId,
|
||||
},
|
||||
});
|
||||
|
||||
return {
|
||||
applicationUniversalIdentifier: application.universalIdentifier,
|
||||
resourcePath: `${fieldMetadata.universalIdentifier}/${name}`,
|
||||
};
|
||||
}
|
||||
|
||||
const { workspaceCustomFlatApplication } =
|
||||
await this.applicationService.findWorkspaceTwentyStandardAndCustomApplicationOrThrow(
|
||||
{
|
||||
workspaceId,
|
||||
},
|
||||
);
|
||||
|
||||
return {
|
||||
applicationUniversalIdentifier:
|
||||
workspaceCustomFlatApplication.universalIdentifier,
|
||||
resourcePath: name,
|
||||
};
|
||||
}
|
||||
|
||||
private async findFileOrThrow({
|
||||
workspaceId,
|
||||
fileId,
|
||||
}: {
|
||||
workspaceId: string;
|
||||
fileId: string;
|
||||
}): Promise<FileEntity> {
|
||||
const file = await this.fileRepository.findOne(workspaceId, {
|
||||
where: { id: fileId },
|
||||
});
|
||||
|
||||
if (!isDefined(file)) {
|
||||
throw new FileUploadException(
|
||||
`File not found: ${fileId}`,
|
||||
FileUploadExceptionCode.FILE_NOT_FOUND,
|
||||
{
|
||||
userFriendlyMessage: msg`File not found.`,
|
||||
},
|
||||
);
|
||||
}
|
||||
|
||||
return file;
|
||||
}
|
||||
|
||||
private async resolveFileLocation({
|
||||
workspaceId,
|
||||
file,
|
||||
}: {
|
||||
workspaceId: string;
|
||||
file: FileEntity;
|
||||
}): Promise<{
|
||||
application: ApplicationEntity;
|
||||
fileFolder: FileFolder;
|
||||
resourcePath: string;
|
||||
}> {
|
||||
const [fileFolder] = file.path.split('/');
|
||||
|
||||
const application = await this.applicationRepository.findOneOrFail({
|
||||
where: {
|
||||
id: file.applicationId,
|
||||
workspaceId,
|
||||
},
|
||||
});
|
||||
|
||||
return {
|
||||
application,
|
||||
fileFolder: fileFolder as FileFolder,
|
||||
resourcePath: removeFileFolderFromFileEntityPath(file.path),
|
||||
};
|
||||
}
|
||||
|
||||
private async toFileWithSignedUrl({
|
||||
file,
|
||||
fileFolder,
|
||||
workspaceId,
|
||||
}: {
|
||||
file: FileEntity;
|
||||
fileFolder: FileFolder;
|
||||
workspaceId: string;
|
||||
}): Promise<FileWithSignedUrlDTO> {
|
||||
return {
|
||||
...file,
|
||||
url: await this.fileUrlService.signFileByIdUrl({
|
||||
fileId: file.id,
|
||||
workspaceId,
|
||||
fileFolder,
|
||||
}),
|
||||
};
|
||||
}
|
||||
}
|
||||
@@ -15,6 +15,7 @@ import { FileController } from './controllers/file.controller';
|
||||
import { FileEntity } from './entities/file.entity';
|
||||
import { FileCorePictureModule } from './file-core-picture/file-core-picture.module';
|
||||
import { FileEmailAttachmentModule } from './file-email-attachment/file-email-attachment.module';
|
||||
import { FileUploadModule } from './file-upload/file-upload.module';
|
||||
import { FileUrlModule } from './file-url/file-url.module';
|
||||
import { FileWorkflowModule } from './file-workflow/file-workflow.module';
|
||||
import { FilesFieldModule } from './files-field/files-field.module';
|
||||
@@ -33,6 +34,7 @@ import { FileService } from './services/file.service';
|
||||
FileWorkflowModule,
|
||||
FileAiChatModule,
|
||||
FileEmailAttachmentModule,
|
||||
FileUploadModule,
|
||||
SecureHttpClientModule,
|
||||
],
|
||||
providers: [
|
||||
@@ -50,6 +52,7 @@ import { FileService } from './services/file.service';
|
||||
FileWorkflowModule,
|
||||
FileAiChatModule,
|
||||
FileEmailAttachmentModule,
|
||||
FileUploadModule,
|
||||
],
|
||||
controllers: [FileController],
|
||||
})
|
||||
|
||||
@@ -18,6 +18,7 @@ import {
|
||||
IMMUTABLE_FILE_CACHE_CONTROL,
|
||||
} from 'src/engine/core-modules/file/interfaces/file-folder.interface';
|
||||
import { type FileResponse } from 'src/engine/core-modules/file/types/file-response.type';
|
||||
import { FILE_STATUS } from 'src/engine/core-modules/file/types/file-status.types';
|
||||
import { getContentDisposition } from 'src/engine/core-modules/file/utils/get-content-disposition.utils';
|
||||
import { removeFileFolderFromFileEntityPath } from 'src/engine/core-modules/file/utils/remove-file-folder-from-file-entity-path.utils';
|
||||
import { JwtWrapperService } from 'src/engine/core-modules/jwt/services/jwt-wrapper.service';
|
||||
@@ -66,6 +67,7 @@ export class FileService {
|
||||
where: {
|
||||
path: `${fileFolder}/${filepath}`,
|
||||
applicationId,
|
||||
status: FILE_STATUS.UPLOADED,
|
||||
},
|
||||
});
|
||||
|
||||
@@ -94,6 +96,7 @@ export class FileService {
|
||||
const file = await this.fileRepository.findOne(workspaceId, {
|
||||
where: {
|
||||
id: fileId,
|
||||
status: FILE_STATUS.UPLOADED,
|
||||
},
|
||||
});
|
||||
|
||||
@@ -155,6 +158,7 @@ export class FileService {
|
||||
where: {
|
||||
id: params.fileId,
|
||||
path: Like(`${params.fileFolder}/%`),
|
||||
status: FILE_STATUS.UPLOADED,
|
||||
},
|
||||
});
|
||||
|
||||
@@ -251,6 +255,7 @@ export class FileService {
|
||||
where: {
|
||||
id: fileId,
|
||||
path: Like(`${fileFolder}/%`),
|
||||
status: FILE_STATUS.UPLOADED,
|
||||
},
|
||||
});
|
||||
|
||||
|
||||
@@ -0,0 +1,8 @@
|
||||
export const FILE_STATUS = {
|
||||
// File record exists but bytes have not been confirmed in storage yet
|
||||
// (direct upload initiated, waiting for the client to upload and confirm).
|
||||
PENDING: 'PENDING',
|
||||
UPLOADED: 'UPLOADED',
|
||||
} as const;
|
||||
|
||||
export type FileStatus = (typeof FILE_STATUS)[keyof typeof FILE_STATUS];
|
||||
@@ -565,7 +565,7 @@ export class ConfigVariables {
|
||||
@ConfigVariablesMetadata({
|
||||
group: ConfigVariablesGroup.STORAGE_CONFIG,
|
||||
description:
|
||||
'When enabled, file downloads are 302-redirected to S3 presigned URLs instead of being proxied through the server. Reduces server load and bandwidth.',
|
||||
'When enabled, file downloads are 302-redirected to S3 presigned URLs and direct uploads go straight to S3 via presigned PUT URLs instead of being proxied through the server. Reduces server load and bandwidth. Requires a bucket CORS policy allowing PUT from the frontend origin.',
|
||||
type: ConfigVariableType.BOOLEAN,
|
||||
})
|
||||
@ValidateIf((env) => env.STORAGE_TYPE === StorageDriverType.S_3)
|
||||
|
||||
Reference in New Issue
Block a user