feat(server): support server-scoped files via nullable workspaceId on file table (#22587)

Part of the app settings architecture cleanup
(twentyhq/core-team-issues#2456) — PR 1 of the server-level documents
plan, reworked after the revert of #22560 (#22579). Same capability,
different shape: **no new entity** — server-level documents live in the
existing `file` table with a nullable `workspaceId`.

## Problem

All file storage is workspace-scoped (`FileEntity.workspaceId NOT NULL`,
`{workspaceId}/{app}/…` storage keys). Server-level data like
application-registration manifests and tarballs for ownerless catalog
registrations has no first-class home, forcing raw-driver bypasses
(`DefaultAiCatalogService`, prototype #22556).

## Changes (core storage layer only — no HTTP serving, no GraphQL
exposure)

**`FileEntity` gains server scope** (mirrors `KeyValuePairEntity`, which
already supports both instance-level and per-workspace rows):
- `workspaceId` uuid becomes **nullable** — NULL means server-scoped;
the entity no longer extends `WorkspaceRelatedEntity` and declares its
columns directly
- `applicationRegistrationId` nullable FK (`onDelete: CASCADE`) —
registration-owned documents follow their registration
- ownership checks: `workspaceId IS NOT NULL OR
applicationRegistrationId IS NOT NULL` and `workspaceId IS NULL OR
applicationRegistrationId IS NULL` — every row has exactly one owner
- `IDX_FILE_APPLICATION_REGISTRATION_ID_PATH_UNIQUE` UNIQUE
(`applicationRegistrationId`, `path`) — mirrors the workspace
unique-constraint pattern; workspace rows are exempt via their NULL
`applicationRegistrationId`

**New `ServerFileStorageService`** (`file-storage/services/`, exported
from the global `FileStorageModule`; `FileStorageService` moved
alongside it):
- storage keys
`server/{fileFolder}/{applicationRegistrationId}/{resourcePath}` — the
registration segment is injected by the service itself, so paths cannot
collide across registrations; scope-validation util mirroring
`validateStoragePathIsWithinWorkspaceOrThrow`; new `ServerFileFolder`
enum in twenty-shared
- `writeServerFile` (upsert on (`applicationRegistrationId`, `path`) +
driver write; throws on failure), `readServerFile`/`readServerFileById`
(missing row or bytes surfaces `FILE_NOT_FOUND`),
`checkServerFileExists`, `deleteServerFile`/`deleteByServerFileId`
(bytes best-effort, row authoritative),
`deleteByApplicationRegistrationId`
- rows are accessed through a plain repository pinned to `workspaceId:
IsNull()` on every query; workspace-file code paths still go through
`WorkspaceScopedRepository`, which never sees NULL rows

**Null-safety ripples** (workspaceId is now `string | null`):
- `WorkspaceScopedEntity` bound widened to `workspaceId: string | null`
(the wrapper always filters with a concrete id)
- `list-and-delete-orphaned-workspace-entities` now skips `workspaceId
IS NULL` rows — previously `NOT EXISTS` would have flagged server rows
as orphans and deleted them
- `PendingFileCleanupService` sweeps only `workspaceId IS NOT NULL`
rows; `application-package-fetcher` pins its tarball lookup to workspace
rows (tarball migration to server scope is a follow-up PR)

**Migration**: `allow-server-scoped-file` ships as a **2-20 fast
instance command** (2.20.0 is current since #22639; re-slotted from 2-19
per review). Command runs are tracked by name, so instances that already
executed the 2-20 `standardOverrides` drop command still pick this one
up. Its realistic timestamp sorts before that drop command's fabricated
`1825000000000`, which the
`ci:allow-upgrade-command-timestamp-exception` label covers.

## Next PRs in the plan

- PR 2: HTTP serving + token type for server files
- PR 3: application-registration manifests stored as versioned server
files (rework of draft #22556)
- PR 4 (optional): registration tarballs migrate to server scope

## Verification

- New spec `server-file-storage.service.spec.ts` (traversal table,
upsert conflict semantics, row-before-bytes reads, best-effort byte
deletion, registration cascade) + scope-validation util spec; affected
suites all green
- Typecheck (server + shared), `lint:diff-with-main`, full `oxfmt
--check src/` on both packages clean
- Fresh `database:reset` on the re-slotted branch: the 2-20 command
executes, generator then reports **no schema drift**; both ownership
checks and the composite unique verified live (dual-owner insert and
duplicate registration+path both rejected)
This commit is contained in:
martmull
2026-07-08 11:56:24 +02:00
committed by GitHub
parent 2f5e9d47ef
commit b733a79821
54 changed files with 987 additions and 53 deletions
@@ -14,6 +14,8 @@ import {
} from 'typeorm';
import { ADD_STATUS_TO_FILE_UPGRADE_COMMAND_NAME } from 'src/database/commands/upgrade-version-command/2-19/add-status-to-file-upgrade-command-name.constant';
import { ALLOW_SERVER_SCOPED_FILE_UPGRADE_COMMAND_NAME } from 'src/database/commands/upgrade-version-command/2-20/allow-server-scoped-file-upgrade-command-name.constant';
import { type ApplicationRegistrationEntity } from 'src/engine/core-modules/application/application-registration/application-registration.entity';
import { ApplicationEntity } from 'src/engine/core-modules/application/application.entity';
import { FileSettings } from 'src/engine/core-modules/file/types/file-settings.types';
import {
@@ -21,24 +23,47 @@ import {
FileStatus,
} from 'src/engine/core-modules/file/types/file-status.types';
import { WasIntroducedInUpgrade } from 'src/engine/core-modules/upgrade/decorators/was-introduced-in-upgrade.decorator';
import { WorkspaceRelatedEntity } from 'src/engine/workspace-manager/types/workspace-related-entity';
import { type WorkspaceEntity } from 'src/engine/core-modules/workspace/workspace.entity';
@Entity('file')
@Check(
'CHK_FILE_PENDING_MIME_OCTET_STREAM',
`"status" != 'PENDING' OR "mimeType" = 'application/octet-stream'`,
)
@Check(
'CHK_FILE_WORKSPACE_ID_OR_APPLICATION_REGISTRATION_ID',
`"workspaceId" IS NOT NULL OR "applicationRegistrationId" IS NOT NULL`,
)
@Check(
'CHK_FILE_WORKSPACE_ID_XOR_APPLICATION_REGISTRATION_ID',
`"workspaceId" IS NULL OR "applicationRegistrationId" IS NULL`,
)
@Index('IDX_FILE_WORKSPACE_ID', ['workspaceId'])
@Index('IDX_FILE_STATUS', ['status'])
@Index('IDX_FILE_APPLICATION_REGISTRATION_ID', ['applicationRegistrationId'])
@Unique('IDX_APPLICATION_PATH_WORKSPACE_ID_APPLICATION_ID_UNIQUE', [
'workspaceId',
'applicationId',
'path',
])
export class FileEntity extends WorkspaceRelatedEntity {
@Unique('IDX_FILE_APPLICATION_REGISTRATION_ID_PATH_UNIQUE', [
'applicationRegistrationId',
'path',
])
export class FileEntity {
@PrimaryGeneratedColumn('uuid')
id: string;
@Column({ nullable: true, type: 'uuid' })
workspaceId: string | null;
@ManyToOne('WorkspaceEntity', {
onDelete: 'CASCADE',
nullable: true,
})
@JoinColumn({ name: 'workspaceId' })
workspace: Relation<WorkspaceEntity> | null;
@Column({ nullable: true, type: 'uuid' })
applicationId: string;
@@ -48,6 +73,19 @@ export class FileEntity extends WorkspaceRelatedEntity {
@JoinColumn({ name: 'applicationId' })
application: Relation<ApplicationEntity>;
@WasIntroducedInUpgrade({
upgradeCommandName: ALLOW_SERVER_SCOPED_FILE_UPGRADE_COMMAND_NAME,
})
@Column({ nullable: true, type: 'uuid' })
applicationRegistrationId: string | null;
@ManyToOne('ApplicationRegistrationEntity', {
onDelete: 'CASCADE',
nullable: true,
})
@JoinColumn({ name: 'applicationRegistrationId' })
applicationRegistration: Relation<ApplicationRegistrationEntity> | null;
@Column({ nullable: false })
path: string;
@@ -5,7 +5,7 @@ import { FileFolder } from 'twenty-shared/types';
import { v4 } from 'uuid';
import { ApplicationService } from 'src/engine/core-modules/application/application.service';
import { FileStorageService } from 'src/engine/core-modules/file-storage/file-storage.service';
import { FileStorageService } from 'src/engine/core-modules/file-storage/services/file-storage.service';
import { FileWithSignedUrlDTO } from 'src/engine/core-modules/file/dtos/file-with-sign-url.dto';
import { FileUrlService } from 'src/engine/core-modules/file/file-url/file-url.service';
import { extractFileInfoOrThrow } from 'src/engine/core-modules/file/utils/extract-file-info-or-throw.utils';
@@ -20,7 +20,7 @@ import {
AuthException,
AuthExceptionCode,
} from 'src/engine/core-modules/auth/auth.exception';
import { FileStorageService } from 'src/engine/core-modules/file-storage/file-storage.service';
import { FileStorageService } from 'src/engine/core-modules/file-storage/services/file-storage.service';
import { FileWithSignedUrlDTO } from 'src/engine/core-modules/file/dtos/file-with-sign-url.dto';
import { FileEntity } from 'src/engine/core-modules/file/entities/file.entity';
import { FileUrlService } from 'src/engine/core-modules/file/file-url/file-url.service';
@@ -5,7 +5,7 @@ import { FileFolder } from 'twenty-shared/types';
import { v4 } from 'uuid';
import { ApplicationService } from 'src/engine/core-modules/application/application.service';
import { FileStorageService } from 'src/engine/core-modules/file-storage/file-storage.service';
import { FileStorageService } from 'src/engine/core-modules/file-storage/services/file-storage.service';
import { FileWithSignedUrlDTO } from 'src/engine/core-modules/file/dtos/file-with-sign-url.dto';
import { FileUrlService } from 'src/engine/core-modules/file/file-url/file-url.service';
import { extractFileInfoOrThrow } from 'src/engine/core-modules/file/utils/extract-file-info-or-throw.utils';
@@ -4,7 +4,7 @@ import { getRepositoryToken } from '@nestjs/typeorm';
import { FileFolder } from 'twenty-shared/types';
import { ApplicationEntity } from 'src/engine/core-modules/application/application.entity';
import { FileStorageService } from 'src/engine/core-modules/file-storage/file-storage.service';
import { FileStorageService } from 'src/engine/core-modules/file-storage/services/file-storage.service';
import { FileEntity } from 'src/engine/core-modules/file/entities/file.entity';
import {
PENDING_FILE_CLEANUP_BATCH_SIZE,
@@ -7,7 +7,7 @@ import { FileFolder } from 'twenty-shared/types';
import { ApplicationEntity } from 'src/engine/core-modules/application/application.entity';
import { ApplicationService } from 'src/engine/core-modules/application/application.service';
import { FileStorageService } from 'src/engine/core-modules/file-storage/file-storage.service';
import { FileStorageService } from 'src/engine/core-modules/file-storage/services/file-storage.service';
import { FileEntity } from 'src/engine/core-modules/file/entities/file.entity';
import {
FileUploadException,
@@ -17,7 +17,7 @@ import { ApplicationEntity } from 'src/engine/core-modules/application/applicati
import { ApplicationService } from 'src/engine/core-modules/application/application.service';
import { FileUploadTokenJwtPayload } from 'src/engine/core-modules/auth/types/file-upload-token-jwt-payload.type';
import { JwtTokenTypeEnum } from 'src/engine/core-modules/auth/types/jwt-token-type.enum';
import { FileStorageService } from 'src/engine/core-modules/file-storage/file-storage.service';
import { FileStorageService } from 'src/engine/core-modules/file-storage/services/file-storage.service';
import { FileWithSignedUrlDTO } from 'src/engine/core-modules/file/dtos/file-with-sign-url.dto';
import { FileEntity } from 'src/engine/core-modules/file/entities/file.entity';
import { FILE_CONTENT_SNIFF_BYTE_COUNT } from 'src/engine/core-modules/file/file-upload/constants/file-content-sniff.constant';
@@ -3,10 +3,10 @@ import { InjectRepository } from '@nestjs/typeorm';
import { FileFolder } from 'twenty-shared/types';
import { isDefined } from 'twenty-shared/utils';
import { LessThan, Repository } from 'typeorm';
import { IsNull, LessThan, Not, Repository } from 'typeorm';
import { ApplicationEntity } from 'src/engine/core-modules/application/application.entity';
import { FileStorageService } from 'src/engine/core-modules/file-storage/file-storage.service';
import { FileStorageService } from 'src/engine/core-modules/file-storage/services/file-storage.service';
import { FileEntity } from 'src/engine/core-modules/file/entities/file.entity';
import {
PENDING_FILE_CLEANUP_BATCH_SIZE,
@@ -40,6 +40,7 @@ export class PendingFileCleanupService {
where: {
status: FILE_STATUS.PENDING,
createdAt: LessThan(staleThreshold),
workspaceId: Not(IsNull()),
},
take: PENDING_FILE_CLEANUP_BATCH_SIZE,
});
@@ -78,6 +79,10 @@ export class PendingFileCleanupService {
// partial, possibly absent) storage object. A failure here leaks bytes but
// never data, so it is logged rather than retried.
private async deleteStorageObject(file: FileEntity): Promise<void> {
if (!isDefined(file.workspaceId)) {
return;
}
const [fileFolder] = file.path.split('/');
const application = await this.applicationRepository.findOne({
@@ -5,7 +5,7 @@ import { FileFolder } from 'twenty-shared/types';
import { v4 } from 'uuid';
import { ApplicationService } from 'src/engine/core-modules/application/application.service';
import { FileStorageService } from 'src/engine/core-modules/file-storage/file-storage.service';
import { FileStorageService } from 'src/engine/core-modules/file-storage/services/file-storage.service';
import { FileWithSignedUrlDTO } from 'src/engine/core-modules/file/dtos/file-with-sign-url.dto';
import { FileUrlService } from 'src/engine/core-modules/file/file-url/file-url.service';
import { extractFileInfoOrThrow } from 'src/engine/core-modules/file/utils/extract-file-info-or-throw.utils';
@@ -8,7 +8,7 @@ import { Repository } from 'typeorm';
import { v4 } from 'uuid';
import { ApplicationEntity } from 'src/engine/core-modules/application/application.entity';
import { FileStorageService } from 'src/engine/core-modules/file-storage/file-storage.service';
import { FileStorageService } from 'src/engine/core-modules/file-storage/services/file-storage.service';
import { FileWithSignedUrlDTO } from 'src/engine/core-modules/file/dtos/file-with-sign-url.dto';
import { FileUrlService } from 'src/engine/core-modules/file/file-url/file-url.service';
import {
@@ -1,6 +1,6 @@
import { FileFolder } from 'twenty-shared/types';
import { FileStorageService } from 'src/engine/core-modules/file-storage/file-storage.service';
import { FileStorageService } from 'src/engine/core-modules/file-storage/services/file-storage.service';
import { Process } from 'src/engine/core-modules/message-queue/decorators/process.decorator';
import { Processor } from 'src/engine/core-modules/message-queue/decorators/processor.decorator';
import { MessageQueue } from 'src/engine/core-modules/message-queue/message-queue.constants';
@@ -2,7 +2,7 @@ import { Test, type TestingModule } from '@nestjs/testing';
import { getRepositoryToken } from '@nestjs/typeorm';
import { ApplicationEntity } from 'src/engine/core-modules/application/application.entity';
import { FileStorageService } from 'src/engine/core-modules/file-storage/file-storage.service';
import { FileStorageService } from 'src/engine/core-modules/file-storage/services/file-storage.service';
import { FileEntity } from 'src/engine/core-modules/file/entities/file.entity';
import { JwtWrapperService } from 'src/engine/core-modules/jwt/services/jwt-wrapper.service';
import { TwentyConfigService } from 'src/engine/core-modules/twenty-config/twenty-config.service';
@@ -7,7 +7,7 @@ import { FileFolder } from 'twenty-shared/types';
import { Like, Repository } from 'typeorm';
import { ApplicationEntity } from 'src/engine/core-modules/application/application.entity';
import { FileStorageService } from 'src/engine/core-modules/file-storage/file-storage.service';
import { FileStorageService } from 'src/engine/core-modules/file-storage/services/file-storage.service';
import {
FileStorageException,
FileStorageExceptionCode,