cdeebb1a185e4d53186464fc5ffe576dfd3b6fd9
2 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 martmull
|
1a85b88d38 |
feat(files): direct-to-storage upload endpoints with pending file lifecycle (#22449)
<img width="1484" height="404" alt="image" src="https://github.com/user-attachments/assets/b2d363bf-d9e1-49fb-9811-8cc98041aa79" /> ## Context Uploading large files currently OOMs the server: every upload resolver buffers the whole file in memory (`streamToBuffer`) before writing it to storage. This PR is the first of a series introducing direct client-to-storage uploads. It adds the server-side endpoints and driver support only — it is non-breaking and nothing consumes the new flow yet. Follow-up PRs will migrate the frontend upload paths, add a stale-pending-file cleanup cron, and cap the legacy buffered resolvers. ## What it does **New upload flow (initiate → PUT → confirm):** - `createFileUpload(filename, size, fileFolder, fieldMetadataId?)` validates the request (folder allowlist: `FilesField`/`Workflow`, max size, extension-derived mime type), creates the file record in a new `PENDING` status, and returns an upload target: - **S3 with presign enabled** → a presigned PUT URL with `Content-Type`/`Content-Length` pinned in the signature, so the client uploads straight to the bucket; - **local storage, or S3 without presign** → a token-authenticated streaming endpoint on the server (`PUT /file-upload/:id?token=…`, new `FILE_UPLOAD` JWT type) that pipes the request body to the storage driver with constant memory usage and a declared-size cap. - `completeFileUpload(fileId)` verifies the bytes actually landed in storage (HEAD + size match against the declared size) and flips the record to `UPLOADED`. Idempotent. **Pending lifecycle safety:** - New `status` column on `core.file` (`PENDING`/`UPLOADED`, default `UPLOADED` so all existing rows and the legacy upload path are unaffected) + fast instance command. - Files are refused by the serving endpoints and by FILES-field sync while `PENDING`. **Driver support (both drivers):** - `getPresignedUploadUrl` (S3: presigned PUT; local: `null` → server-endpoint fallback) - `writeFileStream` (local: `fs` pipeline with the existing symlink/containment hardening, partial-file cleanup on error; S3: `@aws-sdk/lib-storage` `Upload` for bounded-memory streaming) - `getFileMetadata` (HEAD/stat for confirm-time verification) ## Tests - `file-upload.service.spec.ts`: initiate validation (folder allowlist, size), presigned vs fallback target, confirm verification (missing object, size mismatch, happy path, idempotency) - `local.driver.spec.ts`: `writeFileStream` (content, symlink rejection, partial-file cleanup on stream error), `getFileMetadata` - `s3.driver.spec.ts`: `getPresignedUploadUrl` (disabled → null, PUT command with signed content-type/content-length) - `direct-file-upload.integration-spec.ts`: full end-to-end flow against the local driver (initiate → PUT → complete → download), plus error paths (complete without upload, oversized PUT → 413, invalid token → 403, unsupported folder, size above max) ## Notes for reviewers - The upload-size ceiling for direct uploads is `settings.storage.maxDirectUploadFileSize` (1GB), separate from the 10MB `maxFileSize` used for pictures. - Since content can't be sniffed before it reaches storage, the mime type is derived from the file extension (with the existing `TWENTY_MIME_POLICY` override) and unknown extensions fall back to `application/octet-stream`; the serving path already forces `Content-Disposition: attachment` for anything not on the inline-safe allowlist. - Self-hosters using S3 presign will need a bucket CORS policy allowing `PUT` from the frontend origin (config variable description updated). https://claude.ai/code/session_015UH8KWmsB9zdYaog8MFG1d --- _Generated by [Claude Code](https://claude.ai/code/session_015UH8KWmsB9zdYaog8MFG1d)_ <!-- This is an auto-generated description by cubic. --> <a href="https://cubic.dev/pr/twentyhq/twenty/pull/22449?utm_source=github" target="_blank" rel="noopener noreferrer" data-no-image-dialog="true"><picture><source media="(prefers-color-scheme: dark)" srcset="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"><source media="(prefers-color-scheme: light)" srcset="https://www.cubic.dev/buttons/review-in-cubic-light.svg"><img alt="Review in cubic" src="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"></picture></a> <!-- End of auto-generated description by cubic. --> |
||
|
Charles Bochet
|
9d57bc39e5 |
Migrate from ESLint to OxLint (#18443)
## Summary Fully replaces ESLint with OxLint across the entire monorepo: - **Replaced all ESLint configs** (`eslint.config.mjs`) with OxLint configs (`.oxlintrc.json`) for every package: `twenty-front`, `twenty-server`, `twenty-emails`, `twenty-ui`, `twenty-shared`, `twenty-sdk`, `twenty-zapier`, `twenty-docs`, `twenty-website`, `twenty-apps/*`, `create-twenty-app` - **Migrated custom lint rules** from ESLint plugin format to OxLint JS plugin system (`@oxlint/plugins`), including `styled-components-prefixed-with-styled`, `no-hardcoded-colors`, `sort-css-properties-alphabetically`, `graphql-resolvers-should-be-guarded`, `rest-api-methods-should-be-guarded`, `max-consts-per-file`, and Jotai-related rules - **Migrated custom rule tests** from ESLint `RuleTester` + Jest to `oxlint/plugins-dev` `RuleTester` + Vitest - **Removed all ESLint dependencies** from `package.json` files and regenerated lockfiles - **Updated Nx targets** (`lint`, `lint:diff-with-main`, `fmt`) in `nx.json` and per-project `project.json` to use `oxlint` commands with proper `dependsOn` for plugin builds - **Updated CI workflows** (`.github/workflows/ci-*.yaml`) — no more ESLint executor - **Updated IDE setup**: replaced `dbaeumer.vscode-eslint` with `oxc.oxc-vscode` extension, configured `source.fixAll.oxc` and format-on-save with Prettier - **Replaced all `eslint-disable` comments** with `oxlint-disable` equivalents across the codebase - **Updated docs** (`twenty-docs`) to reference OxLint instead of ESLint - **Renamed** `twenty-eslint-rules` package to `twenty-oxlint-rules` ### Temporarily disabled rules (tracked in `OXLINT_MIGRATION_TODO.md`) | Rule | Package | Violations | Auto-fixable | |------|---------|-----------|-------------| | `twenty/sort-css-properties-alphabetically` | twenty-front | 578 | Yes | | `typescript/consistent-type-imports` | twenty-server | 3814 | Yes | | `twenty/max-consts-per-file` | twenty-server | 94 | No | ### Dropped plugins (no OxLint equivalent) `eslint-plugin-project-structure`, `lingui/*`, `@stylistic/*`, `import/order`, `prefer-arrow/prefer-arrow-functions`, `eslint-plugin-mdx`, `@next/eslint-plugin-next`, `eslint-plugin-storybook`, `eslint-plugin-react-refresh`. Partial coverage for `jsx-a11y` and `unused-imports`. ### Additional fixes (pre-existing issues exposed by merge) - Fixed `EmailThreadPreview.tsx` broken import from main rename (`useOpenEmailThreadInSidePanel`) - Restored truthiness guard in `getActivityTargetObjectRecords.ts` - Fixed `AgentTurnResolver` return types to match entity (virtual `fileMediaType`/`fileUrl` are resolved via `@ResolveField()`) ## Test plan - [x] `npx nx lint twenty-front` passes - [x] `npx nx lint twenty-server` passes - [x] `npx nx lint twenty-docs` passes - [x] Custom oxlint rules validated with Vitest: `npx nx test twenty-oxlint-rules` - [x] `npx nx typecheck twenty-front` passes - [x] `npx nx typecheck twenty-server` passes - [x] CI workflows trigger correctly with `dependsOn: ["twenty-oxlint-rules:build"]` - [x] IDE linting works with `oxc.oxc-vscode` extension |
