feat(files): direct-to-storage upload endpoints with pending file lifecycle (#22449)
<img width="1484" height="404" alt="image" src="https://github.com/user-attachments/assets/b2d363bf-d9e1-49fb-9811-8cc98041aa79" /> ## Context Uploading large files currently OOMs the server: every upload resolver buffers the whole file in memory (`streamToBuffer`) before writing it to storage. This PR is the first of a series introducing direct client-to-storage uploads. It adds the server-side endpoints and driver support only — it is non-breaking and nothing consumes the new flow yet. Follow-up PRs will migrate the frontend upload paths, add a stale-pending-file cleanup cron, and cap the legacy buffered resolvers. ## What it does **New upload flow (initiate → PUT → confirm):** - `createFileUpload(filename, size, fileFolder, fieldMetadataId?)` validates the request (folder allowlist: `FilesField`/`Workflow`, max size, extension-derived mime type), creates the file record in a new `PENDING` status, and returns an upload target: - **S3 with presign enabled** → a presigned PUT URL with `Content-Type`/`Content-Length` pinned in the signature, so the client uploads straight to the bucket; - **local storage, or S3 without presign** → a token-authenticated streaming endpoint on the server (`PUT /file-upload/:id?token=…`, new `FILE_UPLOAD` JWT type) that pipes the request body to the storage driver with constant memory usage and a declared-size cap. - `completeFileUpload(fileId)` verifies the bytes actually landed in storage (HEAD + size match against the declared size) and flips the record to `UPLOADED`. Idempotent. **Pending lifecycle safety:** - New `status` column on `core.file` (`PENDING`/`UPLOADED`, default `UPLOADED` so all existing rows and the legacy upload path are unaffected) + fast instance command. - Files are refused by the serving endpoints and by FILES-field sync while `PENDING`. **Driver support (both drivers):** - `getPresignedUploadUrl` (S3: presigned PUT; local: `null` → server-endpoint fallback) - `writeFileStream` (local: `fs` pipeline with the existing symlink/containment hardening, partial-file cleanup on error; S3: `@aws-sdk/lib-storage` `Upload` for bounded-memory streaming) - `getFileMetadata` (HEAD/stat for confirm-time verification) ## Tests - `file-upload.service.spec.ts`: initiate validation (folder allowlist, size), presigned vs fallback target, confirm verification (missing object, size mismatch, happy path, idempotency) - `local.driver.spec.ts`: `writeFileStream` (content, symlink rejection, partial-file cleanup on stream error), `getFileMetadata` - `s3.driver.spec.ts`: `getPresignedUploadUrl` (disabled → null, PUT command with signed content-type/content-length) - `direct-file-upload.integration-spec.ts`: full end-to-end flow against the local driver (initiate → PUT → complete → download), plus error paths (complete without upload, oversized PUT → 413, invalid token → 403, unsupported folder, size above max) ## Notes for reviewers - The upload-size ceiling for direct uploads is `settings.storage.maxDirectUploadFileSize` (1GB), separate from the 10MB `maxFileSize` used for pictures. - Since content can't be sniffed before it reaches storage, the mime type is derived from the file extension (with the existing `TWENTY_MIME_POLICY` override) and unknown extensions fall back to `application/octet-stream`; the serving path already forces `Content-Disposition: attachment` for anything not on the inline-safe allowlist. - Self-hosters using S3 presign will need a bucket CORS policy allowing `PUT` from the frontend origin (config variable description updated). https://claude.ai/code/session_015UH8KWmsB9zdYaog8MFG1d --- _Generated by [Claude Code](https://claude.ai/code/session_015UH8KWmsB9zdYaog8MFG1d)_ <!-- This is an auto-generated description by cubic. --> <a href="https://cubic.dev/pr/twentyhq/twenty/pull/22449?utm_source=github" target="_blank" rel="noopener noreferrer" data-no-image-dialog="true"><picture><source media="(prefers-color-scheme: dark)" srcset="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"><source media="(prefers-color-scheme: light)" srcset="https://www.cubic.dev/buttons/review-in-cubic-light.svg"><img alt="Review in cubic" src="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"></picture></a> <!-- End of auto-generated description by cubic. -->
This commit is contained in:
martmull
@@ -1771,6 +1771,23 @@ __metadata:
|
||||
languageName: node
|
||||
linkType: hard
|
||||
|
||||
"@aws-sdk/lib-storage@npm:3.1001.0":
|
||||
version: 3.1001.0
|
||||
resolution: "@aws-sdk/lib-storage@npm:3.1001.0"
|
||||
dependencies:
|
||||
"@smithy/abort-controller": "npm:^4.2.10"
|
||||
"@smithy/middleware-endpoint": "npm:^4.4.21"
|
||||
"@smithy/smithy-client": "npm:^4.12.1"
|
||||
buffer: "npm:5.6.0"
|
||||
events: "npm:3.3.0"
|
||||
stream-browserify: "npm:3.0.0"
|
||||
tslib: "npm:^2.6.2"
|
||||
peerDependencies:
|
||||
"@aws-sdk/client-s3": ^3.1001.0
|
||||
checksum: 10c0/9ead85eb70c2be7ecf1b28a2e40926da170908e5fe0ac1e9d034a86c2c2b39424f4ac0c2a3560aed4469449e89be8353bab3b663a29df20fb57bcb19d7f57fe4
|
||||
languageName: node
|
||||
linkType: hard
|
||||
|
||||
"@aws-sdk/middleware-bucket-endpoint@npm:^3.972.3":
|
||||
version: 3.972.13
|
||||
resolution: "@aws-sdk/middleware-bucket-endpoint@npm:3.972.13"
|
||||
@@ -27204,7 +27221,7 @@ __metadata:
|
||||
languageName: node
|
||||
linkType: hard
|
||||
|
||||
"base64-js@npm:1.5.1, base64-js@npm:^1.1.2, base64-js@npm:^1.3.0, base64-js@npm:^1.3.1, base64-js@npm:^1.5.1":
|
||||
"base64-js@npm:1.5.1, base64-js@npm:^1.0.2, base64-js@npm:^1.1.2, base64-js@npm:^1.3.0, base64-js@npm:^1.3.1, base64-js@npm:^1.5.1":
|
||||
version: 1.5.1
|
||||
resolution: "base64-js@npm:1.5.1"
|
||||
checksum: 10c0/f23823513b63173a001030fae4f2dabe283b99a9d324ade3ad3d148e218134676f1ee8568c877cd79ec1c53158dcf2d2ba527a97c606618928ba99dd930102bf
|
||||
@@ -27738,6 +27755,16 @@ __metadata:
|
||||
languageName: node
|
||||
linkType: hard
|
||||
|
||||
"buffer@npm:5.6.0":
|
||||
version: 5.6.0
|
||||
resolution: "buffer@npm:5.6.0"
|
||||
dependencies:
|
||||
base64-js: "npm:^1.0.2"
|
||||
ieee754: "npm:^1.1.4"
|
||||
checksum: 10c0/07037a0278b07fbc779920f1ba1b473933ffb4a2e2f7b387c55daf6ac64a05b58c27da9e85730a4046e8f97a49f8acd9f7bf89605c0a4dfda88ebfb7e08bfe4a
|
||||
languageName: node
|
||||
linkType: hard
|
||||
|
||||
"buffer@npm:5.7.1, buffer@npm:^5.2.1, buffer@npm:^5.5.0":
|
||||
version: 5.7.1
|
||||
resolution: "buffer@npm:5.7.1"
|
||||
@@ -32447,7 +32474,7 @@ __metadata:
|
||||
languageName: node
|
||||
linkType: hard
|
||||
|
||||
"events@npm:^3.2.0, events@npm:^3.3.0":
|
||||
"events@npm:3.3.0, events@npm:^3.2.0, events@npm:^3.3.0":
|
||||
version: 3.3.0
|
||||
resolution: "events@npm:3.3.0"
|
||||
checksum: 10c0/d6b6f2adbccbcda74ddbab52ed07db727ef52e31a61ed26db9feb7dc62af7fc8e060defa65e5f8af9449b86b52cc1a1f6a79f2eafcf4e62add2b7a1fa4a432f6
|
||||
@@ -36187,7 +36214,7 @@ __metadata:
|
||||
languageName: node
|
||||
linkType: hard
|
||||
|
||||
"ieee754@npm:1.2.1, ieee754@npm:^1.1.13, ieee754@npm:^1.2.1":
|
||||
"ieee754@npm:1.2.1, ieee754@npm:^1.1.13, ieee754@npm:^1.1.4, ieee754@npm:^1.2.1":
|
||||
version: 1.2.1
|
||||
resolution: "ieee754@npm:1.2.1"
|
||||
checksum: 10c0/b0782ef5e0935b9f12883a2e2aa37baa75da6e66ce6515c168697b42160807d9330de9a32ec1ed73149aea02e0d822e572bca6f1e22bdcbd2149e13b050b17bb
|
||||
@@ -47486,7 +47513,7 @@ __metadata:
|
||||
languageName: node
|
||||
linkType: hard
|
||||
|
||||
"readable-stream@npm:3, readable-stream@npm:3.6.2, readable-stream@npm:^3.0.2, readable-stream@npm:^3.0.6, readable-stream@npm:^3.1.1, readable-stream@npm:^3.4.0, readable-stream@npm:^3.6.0":
|
||||
"readable-stream@npm:3, readable-stream@npm:3.6.2, readable-stream@npm:^3.0.2, readable-stream@npm:^3.0.6, readable-stream@npm:^3.1.1, readable-stream@npm:^3.4.0, readable-stream@npm:^3.5.0, readable-stream@npm:^3.6.0":
|
||||
version: 3.6.2
|
||||
resolution: "readable-stream@npm:3.6.2"
|
||||
dependencies:
|
||||
@@ -50763,6 +50790,16 @@ __metadata:
|
||||
languageName: node
|
||||
linkType: hard
|
||||
|
||||
"stream-browserify@npm:3.0.0":
|
||||
version: 3.0.0
|
||||
resolution: "stream-browserify@npm:3.0.0"
|
||||
dependencies:
|
||||
inherits: "npm:~2.0.4"
|
||||
readable-stream: "npm:^3.5.0"
|
||||
checksum: 10c0/ec3b975a4e0aa4b3dc5e70ffae3fc8fd29ac725353a14e72f213dff477b00330140ad014b163a8cbb9922dfe90803f81a5ea2b269e1bbfd8bd71511b88f889ad
|
||||
languageName: node
|
||||
linkType: hard
|
||||
|
||||
"stream-buffers@npm:~2.2.0":
|
||||
version: 2.2.0
|
||||
resolution: "stream-buffers@npm:2.2.0"
|
||||
@@ -53068,6 +53105,7 @@ __metadata:
|
||||
"@aws-sdk/client-sesv2": "npm:3.1001.0"
|
||||
"@aws-sdk/client-sts": "npm:3.1001.0"
|
||||
"@aws-sdk/credential-providers": "npm:3.1001.0"
|
||||
"@aws-sdk/lib-storage": "npm:3.1001.0"
|
||||
"@aws-sdk/s3-request-presigner": "npm:3.1001.0"
|
||||
"@azure/msal-node": "npm:^5.2.3"
|
||||
"@blocknote/server-util": "npm:^0.51.4"
|
||||
|
||||
Reference in New Issue
Block a user