feat(server): support server-scoped files via nullable workspaceId on file table (#22587)
Part of the app settings architecture cleanup (twentyhq/core-team-issues#2456) — PR 1 of the server-level documents plan, reworked after the revert of #22560 (#22579). Same capability, different shape: **no new entity** — server-level documents live in the existing `file` table with a nullable `workspaceId`. ## Problem All file storage is workspace-scoped (`FileEntity.workspaceId NOT NULL`, `{workspaceId}/{app}/…` storage keys). Server-level data like application-registration manifests and tarballs for ownerless catalog registrations has no first-class home, forcing raw-driver bypasses (`DefaultAiCatalogService`, prototype #22556). ## Changes (core storage layer only — no HTTP serving, no GraphQL exposure) **`FileEntity` gains server scope** (mirrors `KeyValuePairEntity`, which already supports both instance-level and per-workspace rows): - `workspaceId` uuid becomes **nullable** — NULL means server-scoped; the entity no longer extends `WorkspaceRelatedEntity` and declares its columns directly - `applicationRegistrationId` nullable FK (`onDelete: CASCADE`) — registration-owned documents follow their registration - ownership checks: `workspaceId IS NOT NULL OR applicationRegistrationId IS NOT NULL` and `workspaceId IS NULL OR applicationRegistrationId IS NULL` — every row has exactly one owner - `IDX_FILE_APPLICATION_REGISTRATION_ID_PATH_UNIQUE` UNIQUE (`applicationRegistrationId`, `path`) — mirrors the workspace unique-constraint pattern; workspace rows are exempt via their NULL `applicationRegistrationId` **New `ServerFileStorageService`** (`file-storage/services/`, exported from the global `FileStorageModule`; `FileStorageService` moved alongside it): - storage keys `server/{fileFolder}/{applicationRegistrationId}/{resourcePath}` — the registration segment is injected by the service itself, so paths cannot collide across registrations; scope-validation util mirroring `validateStoragePathIsWithinWorkspaceOrThrow`; new `ServerFileFolder` enum in twenty-shared - `writeServerFile` (upsert on (`applicationRegistrationId`, `path`) + driver write; throws on failure), `readServerFile`/`readServerFileById` (missing row or bytes surfaces `FILE_NOT_FOUND`), `checkServerFileExists`, `deleteServerFile`/`deleteByServerFileId` (bytes best-effort, row authoritative), `deleteByApplicationRegistrationId` - rows are accessed through a plain repository pinned to `workspaceId: IsNull()` on every query; workspace-file code paths still go through `WorkspaceScopedRepository`, which never sees NULL rows **Null-safety ripples** (workspaceId is now `string | null`): - `WorkspaceScopedEntity` bound widened to `workspaceId: string | null` (the wrapper always filters with a concrete id) - `list-and-delete-orphaned-workspace-entities` now skips `workspaceId IS NULL` rows — previously `NOT EXISTS` would have flagged server rows as orphans and deleted them - `PendingFileCleanupService` sweeps only `workspaceId IS NOT NULL` rows; `application-package-fetcher` pins its tarball lookup to workspace rows (tarball migration to server scope is a follow-up PR) **Migration**: `allow-server-scoped-file` ships as a **2-20 fast instance command** (2.20.0 is current since #22639; re-slotted from 2-19 per review). Command runs are tracked by name, so instances that already executed the 2-20 `standardOverrides` drop command still pick this one up. Its realistic timestamp sorts before that drop command's fabricated `1825000000000`, which the `ci:allow-upgrade-command-timestamp-exception` label covers. ## Next PRs in the plan - PR 2: HTTP serving + token type for server files - PR 3: application-registration manifests stored as versioned server files (rework of draft #22556) - PR 4 (optional): registration tarballs migrate to server scope ## Verification - New spec `server-file-storage.service.spec.ts` (traversal table, upsert conflict semantics, row-before-bytes reads, best-effort byte deletion, registration cascade) + scope-validation util spec; affected suites all green - Typecheck (server + shared), `lint:diff-with-main`, full `oxfmt --check src/` on both packages clean - Fresh `database:reset` on the re-slotted branch: the 2-20 command executes, generator then reports **no schema drift**; both ownership checks and the composite unique verified live (dual-owner insert and duplicate registration+path both rejected)
This commit is contained in:
@@ -14,6 +14,8 @@ import {
|
||||
} from 'typeorm';
|
||||
|
||||
import { ADD_STATUS_TO_FILE_UPGRADE_COMMAND_NAME } from 'src/database/commands/upgrade-version-command/2-19/add-status-to-file-upgrade-command-name.constant';
|
||||
import { ALLOW_SERVER_SCOPED_FILE_UPGRADE_COMMAND_NAME } from 'src/database/commands/upgrade-version-command/2-20/allow-server-scoped-file-upgrade-command-name.constant';
|
||||
import { type ApplicationRegistrationEntity } from 'src/engine/core-modules/application/application-registration/application-registration.entity';
|
||||
import { ApplicationEntity } from 'src/engine/core-modules/application/application.entity';
|
||||
import { FileSettings } from 'src/engine/core-modules/file/types/file-settings.types';
|
||||
import {
|
||||
@@ -21,24 +23,47 @@ import {
|
||||
FileStatus,
|
||||
} from 'src/engine/core-modules/file/types/file-status.types';
|
||||
import { WasIntroducedInUpgrade } from 'src/engine/core-modules/upgrade/decorators/was-introduced-in-upgrade.decorator';
|
||||
import { WorkspaceRelatedEntity } from 'src/engine/workspace-manager/types/workspace-related-entity';
|
||||
import { type WorkspaceEntity } from 'src/engine/core-modules/workspace/workspace.entity';
|
||||
|
||||
@Entity('file')
|
||||
@Check(
|
||||
'CHK_FILE_PENDING_MIME_OCTET_STREAM',
|
||||
`"status" != 'PENDING' OR "mimeType" = 'application/octet-stream'`,
|
||||
)
|
||||
@Check(
|
||||
'CHK_FILE_WORKSPACE_ID_OR_APPLICATION_REGISTRATION_ID',
|
||||
`"workspaceId" IS NOT NULL OR "applicationRegistrationId" IS NOT NULL`,
|
||||
)
|
||||
@Check(
|
||||
'CHK_FILE_WORKSPACE_ID_XOR_APPLICATION_REGISTRATION_ID',
|
||||
`"workspaceId" IS NULL OR "applicationRegistrationId" IS NULL`,
|
||||
)
|
||||
@Index('IDX_FILE_WORKSPACE_ID', ['workspaceId'])
|
||||
@Index('IDX_FILE_STATUS', ['status'])
|
||||
@Index('IDX_FILE_APPLICATION_REGISTRATION_ID', ['applicationRegistrationId'])
|
||||
@Unique('IDX_APPLICATION_PATH_WORKSPACE_ID_APPLICATION_ID_UNIQUE', [
|
||||
'workspaceId',
|
||||
'applicationId',
|
||||
'path',
|
||||
])
|
||||
export class FileEntity extends WorkspaceRelatedEntity {
|
||||
@Unique('IDX_FILE_APPLICATION_REGISTRATION_ID_PATH_UNIQUE', [
|
||||
'applicationRegistrationId',
|
||||
'path',
|
||||
])
|
||||
export class FileEntity {
|
||||
@PrimaryGeneratedColumn('uuid')
|
||||
id: string;
|
||||
|
||||
@Column({ nullable: true, type: 'uuid' })
|
||||
workspaceId: string | null;
|
||||
|
||||
@ManyToOne('WorkspaceEntity', {
|
||||
onDelete: 'CASCADE',
|
||||
nullable: true,
|
||||
})
|
||||
@JoinColumn({ name: 'workspaceId' })
|
||||
workspace: Relation<WorkspaceEntity> | null;
|
||||
|
||||
@Column({ nullable: true, type: 'uuid' })
|
||||
applicationId: string;
|
||||
|
||||
@@ -48,6 +73,19 @@ export class FileEntity extends WorkspaceRelatedEntity {
|
||||
@JoinColumn({ name: 'applicationId' })
|
||||
application: Relation<ApplicationEntity>;
|
||||
|
||||
@WasIntroducedInUpgrade({
|
||||
upgradeCommandName: ALLOW_SERVER_SCOPED_FILE_UPGRADE_COMMAND_NAME,
|
||||
})
|
||||
@Column({ nullable: true, type: 'uuid' })
|
||||
applicationRegistrationId: string | null;
|
||||
|
||||
@ManyToOne('ApplicationRegistrationEntity', {
|
||||
onDelete: 'CASCADE',
|
||||
nullable: true,
|
||||
})
|
||||
@JoinColumn({ name: 'applicationRegistrationId' })
|
||||
applicationRegistration: Relation<ApplicationRegistrationEntity> | null;
|
||||
|
||||
@Column({ nullable: false })
|
||||
path: string;
|
||||
|
||||
|
||||
+1
-1
@@ -5,7 +5,7 @@ import { FileFolder } from 'twenty-shared/types';
|
||||
import { v4 } from 'uuid';
|
||||
|
||||
import { ApplicationService } from 'src/engine/core-modules/application/application.service';
|
||||
import { FileStorageService } from 'src/engine/core-modules/file-storage/file-storage.service';
|
||||
import { FileStorageService } from 'src/engine/core-modules/file-storage/services/file-storage.service';
|
||||
import { FileWithSignedUrlDTO } from 'src/engine/core-modules/file/dtos/file-with-sign-url.dto';
|
||||
import { FileUrlService } from 'src/engine/core-modules/file/file-url/file-url.service';
|
||||
import { extractFileInfoOrThrow } from 'src/engine/core-modules/file/utils/extract-file-info-or-throw.utils';
|
||||
|
||||
+1
-1
@@ -20,7 +20,7 @@ import {
|
||||
AuthException,
|
||||
AuthExceptionCode,
|
||||
} from 'src/engine/core-modules/auth/auth.exception';
|
||||
import { FileStorageService } from 'src/engine/core-modules/file-storage/file-storage.service';
|
||||
import { FileStorageService } from 'src/engine/core-modules/file-storage/services/file-storage.service';
|
||||
import { FileWithSignedUrlDTO } from 'src/engine/core-modules/file/dtos/file-with-sign-url.dto';
|
||||
import { FileEntity } from 'src/engine/core-modules/file/entities/file.entity';
|
||||
import { FileUrlService } from 'src/engine/core-modules/file/file-url/file-url.service';
|
||||
|
||||
+1
-1
@@ -5,7 +5,7 @@ import { FileFolder } from 'twenty-shared/types';
|
||||
import { v4 } from 'uuid';
|
||||
|
||||
import { ApplicationService } from 'src/engine/core-modules/application/application.service';
|
||||
import { FileStorageService } from 'src/engine/core-modules/file-storage/file-storage.service';
|
||||
import { FileStorageService } from 'src/engine/core-modules/file-storage/services/file-storage.service';
|
||||
import { FileWithSignedUrlDTO } from 'src/engine/core-modules/file/dtos/file-with-sign-url.dto';
|
||||
import { FileUrlService } from 'src/engine/core-modules/file/file-url/file-url.service';
|
||||
import { extractFileInfoOrThrow } from 'src/engine/core-modules/file/utils/extract-file-info-or-throw.utils';
|
||||
|
||||
+1
-1
@@ -4,7 +4,7 @@ import { getRepositoryToken } from '@nestjs/typeorm';
|
||||
import { FileFolder } from 'twenty-shared/types';
|
||||
|
||||
import { ApplicationEntity } from 'src/engine/core-modules/application/application.entity';
|
||||
import { FileStorageService } from 'src/engine/core-modules/file-storage/file-storage.service';
|
||||
import { FileStorageService } from 'src/engine/core-modules/file-storage/services/file-storage.service';
|
||||
import { FileEntity } from 'src/engine/core-modules/file/entities/file.entity';
|
||||
import {
|
||||
PENDING_FILE_CLEANUP_BATCH_SIZE,
|
||||
|
||||
+1
-1
@@ -7,7 +7,7 @@ import { FileFolder } from 'twenty-shared/types';
|
||||
|
||||
import { ApplicationEntity } from 'src/engine/core-modules/application/application.entity';
|
||||
import { ApplicationService } from 'src/engine/core-modules/application/application.service';
|
||||
import { FileStorageService } from 'src/engine/core-modules/file-storage/file-storage.service';
|
||||
import { FileStorageService } from 'src/engine/core-modules/file-storage/services/file-storage.service';
|
||||
import { FileEntity } from 'src/engine/core-modules/file/entities/file.entity';
|
||||
import {
|
||||
FileUploadException,
|
||||
|
||||
+1
-1
@@ -17,7 +17,7 @@ import { ApplicationEntity } from 'src/engine/core-modules/application/applicati
|
||||
import { ApplicationService } from 'src/engine/core-modules/application/application.service';
|
||||
import { FileUploadTokenJwtPayload } from 'src/engine/core-modules/auth/types/file-upload-token-jwt-payload.type';
|
||||
import { JwtTokenTypeEnum } from 'src/engine/core-modules/auth/types/jwt-token-type.enum';
|
||||
import { FileStorageService } from 'src/engine/core-modules/file-storage/file-storage.service';
|
||||
import { FileStorageService } from 'src/engine/core-modules/file-storage/services/file-storage.service';
|
||||
import { FileWithSignedUrlDTO } from 'src/engine/core-modules/file/dtos/file-with-sign-url.dto';
|
||||
import { FileEntity } from 'src/engine/core-modules/file/entities/file.entity';
|
||||
import { FILE_CONTENT_SNIFF_BYTE_COUNT } from 'src/engine/core-modules/file/file-upload/constants/file-content-sniff.constant';
|
||||
|
||||
+7
-2
@@ -3,10 +3,10 @@ import { InjectRepository } from '@nestjs/typeorm';
|
||||
|
||||
import { FileFolder } from 'twenty-shared/types';
|
||||
import { isDefined } from 'twenty-shared/utils';
|
||||
import { LessThan, Repository } from 'typeorm';
|
||||
import { IsNull, LessThan, Not, Repository } from 'typeorm';
|
||||
|
||||
import { ApplicationEntity } from 'src/engine/core-modules/application/application.entity';
|
||||
import { FileStorageService } from 'src/engine/core-modules/file-storage/file-storage.service';
|
||||
import { FileStorageService } from 'src/engine/core-modules/file-storage/services/file-storage.service';
|
||||
import { FileEntity } from 'src/engine/core-modules/file/entities/file.entity';
|
||||
import {
|
||||
PENDING_FILE_CLEANUP_BATCH_SIZE,
|
||||
@@ -40,6 +40,7 @@ export class PendingFileCleanupService {
|
||||
where: {
|
||||
status: FILE_STATUS.PENDING,
|
||||
createdAt: LessThan(staleThreshold),
|
||||
workspaceId: Not(IsNull()),
|
||||
},
|
||||
take: PENDING_FILE_CLEANUP_BATCH_SIZE,
|
||||
});
|
||||
@@ -78,6 +79,10 @@ export class PendingFileCleanupService {
|
||||
// partial, possibly absent) storage object. A failure here leaks bytes but
|
||||
// never data, so it is logged rather than retried.
|
||||
private async deleteStorageObject(file: FileEntity): Promise<void> {
|
||||
if (!isDefined(file.workspaceId)) {
|
||||
return;
|
||||
}
|
||||
|
||||
const [fileFolder] = file.path.split('/');
|
||||
|
||||
const application = await this.applicationRepository.findOne({
|
||||
|
||||
+1
-1
@@ -5,7 +5,7 @@ import { FileFolder } from 'twenty-shared/types';
|
||||
import { v4 } from 'uuid';
|
||||
|
||||
import { ApplicationService } from 'src/engine/core-modules/application/application.service';
|
||||
import { FileStorageService } from 'src/engine/core-modules/file-storage/file-storage.service';
|
||||
import { FileStorageService } from 'src/engine/core-modules/file-storage/services/file-storage.service';
|
||||
import { FileWithSignedUrlDTO } from 'src/engine/core-modules/file/dtos/file-with-sign-url.dto';
|
||||
import { FileUrlService } from 'src/engine/core-modules/file/file-url/file-url.service';
|
||||
import { extractFileInfoOrThrow } from 'src/engine/core-modules/file/utils/extract-file-info-or-throw.utils';
|
||||
|
||||
+1
-1
@@ -8,7 +8,7 @@ import { Repository } from 'typeorm';
|
||||
import { v4 } from 'uuid';
|
||||
|
||||
import { ApplicationEntity } from 'src/engine/core-modules/application/application.entity';
|
||||
import { FileStorageService } from 'src/engine/core-modules/file-storage/file-storage.service';
|
||||
import { FileStorageService } from 'src/engine/core-modules/file-storage/services/file-storage.service';
|
||||
import { FileWithSignedUrlDTO } from 'src/engine/core-modules/file/dtos/file-with-sign-url.dto';
|
||||
import { FileUrlService } from 'src/engine/core-modules/file/file-url/file-url.service';
|
||||
import {
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
import { FileFolder } from 'twenty-shared/types';
|
||||
|
||||
import { FileStorageService } from 'src/engine/core-modules/file-storage/file-storage.service';
|
||||
import { FileStorageService } from 'src/engine/core-modules/file-storage/services/file-storage.service';
|
||||
import { Process } from 'src/engine/core-modules/message-queue/decorators/process.decorator';
|
||||
import { Processor } from 'src/engine/core-modules/message-queue/decorators/processor.decorator';
|
||||
import { MessageQueue } from 'src/engine/core-modules/message-queue/message-queue.constants';
|
||||
|
||||
@@ -2,7 +2,7 @@ import { Test, type TestingModule } from '@nestjs/testing';
|
||||
import { getRepositoryToken } from '@nestjs/typeorm';
|
||||
|
||||
import { ApplicationEntity } from 'src/engine/core-modules/application/application.entity';
|
||||
import { FileStorageService } from 'src/engine/core-modules/file-storage/file-storage.service';
|
||||
import { FileStorageService } from 'src/engine/core-modules/file-storage/services/file-storage.service';
|
||||
import { FileEntity } from 'src/engine/core-modules/file/entities/file.entity';
|
||||
import { JwtWrapperService } from 'src/engine/core-modules/jwt/services/jwt-wrapper.service';
|
||||
import { TwentyConfigService } from 'src/engine/core-modules/twenty-config/twenty-config.service';
|
||||
|
||||
@@ -7,7 +7,7 @@ import { FileFolder } from 'twenty-shared/types';
|
||||
import { Like, Repository } from 'typeorm';
|
||||
|
||||
import { ApplicationEntity } from 'src/engine/core-modules/application/application.entity';
|
||||
import { FileStorageService } from 'src/engine/core-modules/file-storage/file-storage.service';
|
||||
import { FileStorageService } from 'src/engine/core-modules/file-storage/services/file-storage.service';
|
||||
import {
|
||||
FileStorageException,
|
||||
FileStorageExceptionCode,
|
||||
|
||||
Reference in New Issue
Block a user