feat(files): reap stale pending direct-upload files via hourly cron (#22531)
## Context Follow-up to #22449 (direct-to-storage upload endpoints). That PR introduced the `PENDING` → `UPLOADED` file lifecycle: `createFileUpload` inserts a file record in `PENDING`, the client uploads the bytes directly to storage, then `completeFileUpload` flips it to `UPLOADED`. A client that initiates an upload but never confirms — a crash, a closed tab, an expired presigned URL — leaves a `PENDING` file record and a possibly-partial storage object behind forever. This PR reaps them. ## What this does Adds an hourly cron that hard-deletes `PENDING` files older than 24h together with their storage objects, in bounded batches. - **`PendingFileCleanupService`** — finds `PENDING` files with `createdAt` older than `PENDING_FILE_MAX_AGE_MS` (24h), capped at `PENDING_FILE_CLEANUP_BATCH_SIZE` (200) per run, and deletes each via `FileStorageService.deleteByFileId` (which tolerates a missing object). A failure on one file is logged and skipped so the rest of the batch still gets cleaned. - **`PendingFileCleanupCronJob`** — `@Processor(cronQueue)` job that runs the service and reports exceptions. - **`PendingFileCleanupCronCommand`** — registers the job on the hourly pattern (`0 * * * *`). - Wired into `FileUploadModule` (providers + export) and registered in `cron:register:all`. ### Why 24h The reaper threshold sits well past the presigned URL expiry, so a `PENDING` file only becomes reapable long after any legitimate in-flight upload could still complete — the cleanup can never race a real upload. A file that was never confirmed is referenced by nothing; the client recovery path is simply re-uploading under a fresh `fileId`, so we never promote to `UPLOADED`. ## Tests `pending-file-cleanup.service.spec.ts` covers: the query shape (status + age threshold + batch cap), deleting each stale file and returning the count, continuing past a per-file deletion failure, and the empty-batch no-op. ## Scope Server-only, non-breaking, no user-facing change. Part of the incremental direct-upload rollout being split into small PRs. https://claude.ai/code/session_015UH8KWmsB9zdYaog8MFG1d --- _Generated by [Claude Code](https://claude.ai/code/session_015UH8KWmsB9zdYaog8MFG1d)_ <!-- This is an auto-generated description by cubic. --> <a href="https://cubic.dev/pr/twentyhq/twenty/pull/22531?utm_source=github" target="_blank" rel="noopener noreferrer" data-no-image-dialog="true"><picture><source media="(prefers-color-scheme: dark)" srcset="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"><source media="(prefers-color-scheme: light)" srcset="https://www.cubic.dev/buttons/review-in-cubic-light.svg"><img alt="Review in cubic" src="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"></picture></a> <!-- End of auto-generated description by cubic. -->
This commit is contained in:
@@ -9,6 +9,7 @@ import { ApplicationVersionCheckCronCommand } from 'src/engine/core-modules/appl
|
||||
import { BillingReminderCronCommand } from 'src/engine/core-modules/billing/reminders/crons/commands/billing-reminder.cron.command';
|
||||
import { EnterpriseKeyValidationCronCommand } from 'src/engine/core-modules/enterprise/cron/command/enterprise-key-validation.cron.command';
|
||||
import { EventLogCleanupCronCommand } from 'src/engine/core-modules/event-logs/cleanup/commands/event-log-cleanup.cron.command';
|
||||
import { PendingFileCleanupCronCommand } from 'src/engine/core-modules/file/file-upload/crons/commands/pending-file-cleanup.cron.command';
|
||||
import { RotateSigningKeysCronCommand } from 'src/engine/core-modules/jwt/crons/commands/rotate-signing-keys.cron.command';
|
||||
import { CronTriggerCronCommand } from 'src/engine/core-modules/logic-function/logic-function-trigger/triggers/cron/cron-trigger.cron.command';
|
||||
import { CheckPublicDomainsValidRecordsCronCommand } from 'src/engine/core-modules/public-domain/crons/commands/check-public-domains-valid-records.cron.command';
|
||||
@@ -65,6 +66,7 @@ export class CronRegisterAllCommand extends CommandRunner {
|
||||
private readonly marketplaceCatalogSyncCronCommand: MarketplaceCatalogSyncCronCommand,
|
||||
private readonly applicationVersionCheckCronCommand: ApplicationVersionCheckCronCommand,
|
||||
private readonly staleRegistrationCleanupCronCommand: StaleRegistrationCleanupCronCommand,
|
||||
private readonly pendingFileCleanupCronCommand: PendingFileCleanupCronCommand,
|
||||
private readonly billingReminderCronCommand: BillingReminderCronCommand,
|
||||
private readonly twentyConfigService: TwentyConfigService,
|
||||
) {
|
||||
@@ -183,6 +185,10 @@ export class CronRegisterAllCommand extends CommandRunner {
|
||||
name: 'StaleRegistrationCleanup',
|
||||
command: this.staleRegistrationCleanupCronCommand,
|
||||
},
|
||||
{
|
||||
name: 'PendingFileCleanup',
|
||||
command: this.pendingFileCleanupCronCommand,
|
||||
},
|
||||
{
|
||||
name: 'BillingReminder',
|
||||
command: this.billingReminderCronCommand,
|
||||
|
||||
Reference in New Issue
Block a user