feat(server): support server-scoped files via nullable workspaceId on file table (#22587)
Part of the app settings architecture cleanup (twentyhq/core-team-issues#2456) — PR 1 of the server-level documents plan, reworked after the revert of #22560 (#22579). Same capability, different shape: **no new entity** — server-level documents live in the existing `file` table with a nullable `workspaceId`. ## Problem All file storage is workspace-scoped (`FileEntity.workspaceId NOT NULL`, `{workspaceId}/{app}/…` storage keys). Server-level data like application-registration manifests and tarballs for ownerless catalog registrations has no first-class home, forcing raw-driver bypasses (`DefaultAiCatalogService`, prototype #22556). ## Changes (core storage layer only — no HTTP serving, no GraphQL exposure) **`FileEntity` gains server scope** (mirrors `KeyValuePairEntity`, which already supports both instance-level and per-workspace rows): - `workspaceId` uuid becomes **nullable** — NULL means server-scoped; the entity no longer extends `WorkspaceRelatedEntity` and declares its columns directly - `applicationRegistrationId` nullable FK (`onDelete: CASCADE`) — registration-owned documents follow their registration - ownership checks: `workspaceId IS NOT NULL OR applicationRegistrationId IS NOT NULL` and `workspaceId IS NULL OR applicationRegistrationId IS NULL` — every row has exactly one owner - `IDX_FILE_APPLICATION_REGISTRATION_ID_PATH_UNIQUE` UNIQUE (`applicationRegistrationId`, `path`) — mirrors the workspace unique-constraint pattern; workspace rows are exempt via their NULL `applicationRegistrationId` **New `ServerFileStorageService`** (`file-storage/services/`, exported from the global `FileStorageModule`; `FileStorageService` moved alongside it): - storage keys `server/{fileFolder}/{applicationRegistrationId}/{resourcePath}` — the registration segment is injected by the service itself, so paths cannot collide across registrations; scope-validation util mirroring `validateStoragePathIsWithinWorkspaceOrThrow`; new `ServerFileFolder` enum in twenty-shared - `writeServerFile` (upsert on (`applicationRegistrationId`, `path`) + driver write; throws on failure), `readServerFile`/`readServerFileById` (missing row or bytes surfaces `FILE_NOT_FOUND`), `checkServerFileExists`, `deleteServerFile`/`deleteByServerFileId` (bytes best-effort, row authoritative), `deleteByApplicationRegistrationId` - rows are accessed through a plain repository pinned to `workspaceId: IsNull()` on every query; workspace-file code paths still go through `WorkspaceScopedRepository`, which never sees NULL rows **Null-safety ripples** (workspaceId is now `string | null`): - `WorkspaceScopedEntity` bound widened to `workspaceId: string | null` (the wrapper always filters with a concrete id) - `list-and-delete-orphaned-workspace-entities` now skips `workspaceId IS NULL` rows — previously `NOT EXISTS` would have flagged server rows as orphans and deleted them - `PendingFileCleanupService` sweeps only `workspaceId IS NOT NULL` rows; `application-package-fetcher` pins its tarball lookup to workspace rows (tarball migration to server scope is a follow-up PR) **Migration**: `allow-server-scoped-file` ships as a **2-20 fast instance command** (2.20.0 is current since #22639; re-slotted from 2-19 per review). Command runs are tracked by name, so instances that already executed the 2-20 `standardOverrides` drop command still pick this one up. Its realistic timestamp sorts before that drop command's fabricated `1825000000000`, which the `ci:allow-upgrade-command-timestamp-exception` label covers. ## Next PRs in the plan - PR 2: HTTP serving + token type for server files - PR 3: application-registration manifests stored as versioned server files (rework of draft #22556) - PR 4 (optional): registration tarballs migrate to server scope ## Verification - New spec `server-file-storage.service.spec.ts` (traversal table, upsert conflict semantics, row-before-bytes reads, best-effort byte deletion, registration cascade) + scope-validation util spec; affected suites all green - Typecheck (server + shared), `lint:diff-with-main`, full `oxfmt --check src/` on both packages clean - Fresh `database:reset` on the re-slotted branch: the 2-20 command executes, generator then reports **no schema drift**; both ownership checks and the composite unique verified live (dual-owner insert and duplicate registration+path both rejected)
This commit is contained in:
+1
-1
@@ -31,7 +31,7 @@ import {
|
||||
} from 'src/engine/core-modules/application/application.exception';
|
||||
import { ApplicationService } from 'src/engine/core-modules/application/application.service';
|
||||
import { CacheLockService } from 'src/engine/core-modules/cache-lock/cache-lock.service';
|
||||
import { FileStorageService } from 'src/engine/core-modules/file-storage/file-storage.service';
|
||||
import { FileStorageService } from 'src/engine/core-modules/file-storage/services/file-storage.service';
|
||||
import { validateFilePath } from 'src/engine/core-modules/file-storage/utils/validate-file-path.util';
|
||||
import { FileDTO } from 'src/engine/core-modules/file/dtos/file.dto';
|
||||
import { ResolverValidationPipe } from 'src/engine/core-modules/graphql/pipes/resolver-validation.pipe';
|
||||
|
||||
+1
-1
@@ -27,7 +27,7 @@ import {
|
||||
} from 'src/engine/core-modules/application/application-package/application-version-validation.service';
|
||||
import { ApplicationSyncService } from 'src/engine/core-modules/application/application-manifest/application-sync.service';
|
||||
import { CacheLockService } from 'src/engine/core-modules/cache-lock/cache-lock.service';
|
||||
import { FileStorageService } from 'src/engine/core-modules/file-storage/file-storage.service';
|
||||
import { FileStorageService } from 'src/engine/core-modules/file-storage/services/file-storage.service';
|
||||
import {
|
||||
LogicFunctionTriggerJob,
|
||||
type LogicFunctionTriggerJobData,
|
||||
|
||||
+1
-1
@@ -18,7 +18,7 @@ import {
|
||||
} from 'src/engine/core-modules/application/application.exception';
|
||||
import { ApplicationService } from 'src/engine/core-modules/application/application.service';
|
||||
import { type FlatApplication } from 'src/engine/core-modules/application/types/flat-application.type';
|
||||
import { FileStorageService } from 'src/engine/core-modules/file-storage/file-storage.service';
|
||||
import { FileStorageService } from 'src/engine/core-modules/file-storage/services/file-storage.service';
|
||||
import { LOGIC_FUNCTION_DRIVER_FACTORY_TOKEN } from 'src/engine/core-modules/logic-function/logic-function-drivers/constants/logic-function-driver-factory.token';
|
||||
import { type LogicFunctionDriverFactory } from 'src/engine/core-modules/logic-function/logic-function-drivers/logic-function-driver.factory';
|
||||
import { createEmptyAllFlatEntityMaps } from 'src/engine/metadata-modules/flat-entity/constant/create-empty-all-flat-entity-maps.constant';
|
||||
|
||||
+13
-3
@@ -10,7 +10,7 @@ import { type Manifest } from 'twenty-shared/application';
|
||||
import { FileFolder } from 'twenty-shared/types';
|
||||
import { isDefined } from 'twenty-shared/utils';
|
||||
import { type PackageJson } from 'type-fest';
|
||||
import { Repository } from 'typeorm';
|
||||
import { IsNull, Not, Repository } from 'typeorm';
|
||||
import { v4 } from 'uuid';
|
||||
|
||||
import { ApplicationRegistrationEntity } from 'src/engine/core-modules/application/application-registration/application-registration.entity';
|
||||
@@ -24,7 +24,7 @@ import { assertValidNpmPackageName } from 'src/engine/core-modules/application/a
|
||||
import { extractTarballSecurely } from 'src/engine/core-modules/application/application-package/utils/extract-tarball-securely.util';
|
||||
import { readJsonFileOrThrow } from 'src/engine/core-modules/application/application-package/utils/read-json-file.util';
|
||||
import { resolvePackageContentDir } from 'src/engine/core-modules/application/application-package/utils/tarball-utils';
|
||||
import { FileStorageService } from 'src/engine/core-modules/file-storage/file-storage.service';
|
||||
import { FileStorageService } from 'src/engine/core-modules/file-storage/services/file-storage.service';
|
||||
import { FileEntity } from 'src/engine/core-modules/file/entities/file.entity';
|
||||
import { removeFileFolderFromFileEntityPath } from 'src/engine/core-modules/file/utils/remove-file-folder-from-file-entity-path.utils';
|
||||
import { SecureHttpClientService } from 'src/engine/core-modules/secure-http-client/secure-http-client.service';
|
||||
@@ -178,13 +178,23 @@ export class ApplicationPackageFetcherService implements OnModuleInit {
|
||||
|
||||
try {
|
||||
const file = await this.fileRepository.findOneOrFail({
|
||||
where: { id: appRegistration.tarballFileId },
|
||||
where: {
|
||||
id: appRegistration.tarballFileId,
|
||||
workspaceId: Not(IsNull()),
|
||||
},
|
||||
});
|
||||
|
||||
const application = await this.applicationRepository.findOneOrFail({
|
||||
where: { id: file.applicationId },
|
||||
});
|
||||
|
||||
if (!isDefined(file.workspaceId)) {
|
||||
throw new ApplicationException(
|
||||
`Tarball file ${file.id} for app registration ${appRegistration.id} has no workspaceId`,
|
||||
ApplicationExceptionCode.TARBALL_EXTRACTION_FAILED,
|
||||
);
|
||||
}
|
||||
|
||||
const tarballStream = await this.fileStorageService.readFile({
|
||||
workspaceId: file.workspaceId,
|
||||
applicationUniversalIdentifier: application.universalIdentifier,
|
||||
|
||||
+1
-1
@@ -28,7 +28,7 @@ import {
|
||||
import { ApplicationRegistrationSourceType } from 'src/engine/core-modules/application/application-registration/enums/application-registration-source-type.enum';
|
||||
import { fromManifestApplicationToDisplayFields } from 'src/engine/core-modules/application/application-registration/utils/from-manifest-application-to-display-fields.util';
|
||||
import { ApplicationService } from 'src/engine/core-modules/application/application.service';
|
||||
import { FileStorageService } from 'src/engine/core-modules/file-storage/file-storage.service';
|
||||
import { FileStorageService } from 'src/engine/core-modules/file-storage/services/file-storage.service';
|
||||
import type { ApplicationManifest } from 'twenty-shared/application';
|
||||
|
||||
@Injectable()
|
||||
|
||||
@@ -18,7 +18,7 @@ import {
|
||||
} from 'src/engine/core-modules/application/application.exception';
|
||||
import { WORKSPACE_CUSTOM_APPLICATION_NAME } from 'src/engine/core-modules/application/constants/workspace-custom-application.constant';
|
||||
import { type FlatApplication } from 'src/engine/core-modules/application/types/flat-application.type';
|
||||
import { FileStorageService } from 'src/engine/core-modules/file-storage/file-storage.service';
|
||||
import { FileStorageService } from 'src/engine/core-modules/file-storage/services/file-storage.service';
|
||||
import { WorkspaceEntity } from 'src/engine/core-modules/workspace/workspace.entity';
|
||||
import { AgentEntity } from 'src/engine/metadata-modules/ai/ai-agent/entities/agent.entity';
|
||||
import { CommandMenuItemEntity } from 'src/engine/metadata-modules/command-menu-item/entities/command-menu-item.entity';
|
||||
|
||||
+1
-1
@@ -6,7 +6,7 @@ import { ApplicationService } from 'src/engine/core-modules/application/applicat
|
||||
import { DpaAgreementEntity } from 'src/engine/core-modules/dpa/entities/dpa-agreement.entity';
|
||||
import { DpaRegionService } from 'src/engine/core-modules/dpa/services/dpa-region.service';
|
||||
import { DpaService } from 'src/engine/core-modules/dpa/services/dpa.service';
|
||||
import { FileStorageService } from 'src/engine/core-modules/file-storage/file-storage.service';
|
||||
import { FileStorageService } from 'src/engine/core-modules/file-storage/services/file-storage.service';
|
||||
import { FileUrlService } from 'src/engine/core-modules/file/file-url/file-url.service';
|
||||
import { TwentyConfigService } from 'src/engine/core-modules/twenty-config/twenty-config.service';
|
||||
|
||||
|
||||
@@ -18,7 +18,7 @@ import { type ResolvedDpa } from 'src/engine/core-modules/dpa/types/dpa.types';
|
||||
import { buildDpaAgreementRecord } from 'src/engine/core-modules/dpa/utils/build-dpa-agreement-record.util';
|
||||
import { resolveDpa } from 'src/engine/core-modules/dpa/utils/resolve-dpa.util';
|
||||
import { renderDpaToPdfBuffer } from 'src/engine/core-modules/dpa/pdf/render-dpa-to-pdf.util';
|
||||
import { FileStorageService } from 'src/engine/core-modules/file-storage/file-storage.service';
|
||||
import { FileStorageService } from 'src/engine/core-modules/file-storage/services/file-storage.service';
|
||||
import { FileUrlService } from 'src/engine/core-modules/file/file-url/file-url.service';
|
||||
import { TwentyConfigService } from 'src/engine/core-modules/twenty-config/twenty-config.service';
|
||||
import { type WorkspaceEntity } from 'src/engine/core-modules/workspace/workspace.entity';
|
||||
|
||||
+1
@@ -0,0 +1 @@
|
||||
export const SERVER_FILE_STORAGE_PREFIX = 'server';
|
||||
@@ -5,7 +5,8 @@ import { TypeOrmModule } from '@nestjs/typeorm';
|
||||
import { ApplicationEntity } from 'src/engine/core-modules/application/application.entity';
|
||||
import { FileStorageExceptionFilter } from 'src/engine/core-modules/file-storage/file-storage-exception-filter';
|
||||
import { FileStorageDriverFactory } from 'src/engine/core-modules/file-storage/file-storage-driver.factory';
|
||||
import { FileStorageService } from 'src/engine/core-modules/file-storage/file-storage.service';
|
||||
import { FileStorageService } from 'src/engine/core-modules/file-storage/services/file-storage.service';
|
||||
import { ServerFileStorageService } from 'src/engine/core-modules/file-storage/services/server-file-storage.service';
|
||||
import { FileEntity } from 'src/engine/core-modules/file/entities/file.entity';
|
||||
import { TwentyConfigModule } from 'src/engine/core-modules/twenty-config/twenty-config.module';
|
||||
import { provideWorkspaceScopedRepository } from 'src/engine/twenty-orm/workspace-scoped-repository/provide-workspace-scoped-repository';
|
||||
@@ -21,13 +22,18 @@ export class FileStorageModule {
|
||||
providers: [
|
||||
FileStorageDriverFactory,
|
||||
FileStorageService,
|
||||
ServerFileStorageService,
|
||||
provideWorkspaceScopedRepository(FileEntity),
|
||||
{
|
||||
provide: APP_FILTER,
|
||||
useClass: FileStorageExceptionFilter,
|
||||
},
|
||||
],
|
||||
exports: [FileStorageDriverFactory, FileStorageService],
|
||||
exports: [
|
||||
FileStorageDriverFactory,
|
||||
FileStorageService,
|
||||
ServerFileStorageService,
|
||||
],
|
||||
};
|
||||
}
|
||||
}
|
||||
|
||||
+1
-1
@@ -9,7 +9,7 @@ import {
|
||||
|
||||
import { ApplicationEntity } from 'src/engine/core-modules/application/application.entity';
|
||||
import { FileStorageDriverFactory } from 'src/engine/core-modules/file-storage/file-storage-driver.factory';
|
||||
import { FileStorageService } from 'src/engine/core-modules/file-storage/file-storage.service';
|
||||
import { FileStorageService } from 'src/engine/core-modules/file-storage/services/file-storage.service';
|
||||
import { FileStorageExceptionCode } from 'src/engine/core-modules/file-storage/interfaces/file-storage-exception';
|
||||
import { FileEntity } from 'src/engine/core-modules/file/entities/file.entity';
|
||||
import { getWorkspaceScopedRepositoryToken } from 'src/engine/twenty-orm/workspace-scoped-repository/get-workspace-scoped-repository-token.util';
|
||||
+444
@@ -0,0 +1,444 @@
|
||||
import { Test, type TestingModule } from '@nestjs/testing';
|
||||
import { getRepositoryToken } from '@nestjs/typeorm';
|
||||
|
||||
import { Readable } from 'stream';
|
||||
|
||||
import { ServerFileFolder } from 'twenty-shared/types';
|
||||
import { IsNull } from 'typeorm';
|
||||
|
||||
import { FileStorageDriverFactory } from 'src/engine/core-modules/file-storage/file-storage-driver.factory';
|
||||
import { ServerFileStorageService } from 'src/engine/core-modules/file-storage/services/server-file-storage.service';
|
||||
import {
|
||||
FileStorageException,
|
||||
FileStorageExceptionCode,
|
||||
} from 'src/engine/core-modules/file-storage/interfaces/file-storage-exception';
|
||||
import { FileEntity } from 'src/engine/core-modules/file/entities/file.entity';
|
||||
|
||||
describe('ServerFileStorageService', () => {
|
||||
let service: ServerFileStorageService;
|
||||
|
||||
const mockFileStorageDriverFactory = {
|
||||
getCurrentDriver: jest.fn(),
|
||||
};
|
||||
|
||||
const mockServerFileRepository = {
|
||||
upsert: jest.fn(),
|
||||
findOneBy: jest.fn(),
|
||||
findOneByOrFail: jest.fn(),
|
||||
findBy: jest.fn(),
|
||||
delete: jest.fn(),
|
||||
};
|
||||
|
||||
const mockDriver = {
|
||||
writeFile: jest.fn(),
|
||||
readFile: jest.fn(),
|
||||
delete: jest.fn(),
|
||||
checkFileExists: jest.fn(),
|
||||
};
|
||||
|
||||
beforeEach(async () => {
|
||||
const module: TestingModule = await Test.createTestingModule({
|
||||
providers: [
|
||||
ServerFileStorageService,
|
||||
{
|
||||
provide: FileStorageDriverFactory,
|
||||
useValue: mockFileStorageDriverFactory,
|
||||
},
|
||||
{
|
||||
provide: getRepositoryToken(FileEntity),
|
||||
useValue: mockServerFileRepository,
|
||||
},
|
||||
],
|
||||
}).compile();
|
||||
|
||||
service = module.get<ServerFileStorageService>(ServerFileStorageService);
|
||||
|
||||
jest.clearAllMocks();
|
||||
|
||||
mockFileStorageDriverFactory.getCurrentDriver.mockReturnValue(mockDriver);
|
||||
});
|
||||
|
||||
describe.each([
|
||||
[
|
||||
'readServerFile',
|
||||
(resourcePath: string) =>
|
||||
service.readServerFile({
|
||||
fileFolder: ServerFileFolder.ApplicationRegistration,
|
||||
applicationRegistrationId: 'registration-id',
|
||||
resourcePath,
|
||||
}),
|
||||
],
|
||||
[
|
||||
'checkServerFileExists',
|
||||
(resourcePath: string) =>
|
||||
service.checkServerFileExists({
|
||||
fileFolder: ServerFileFolder.ApplicationRegistration,
|
||||
applicationRegistrationId: 'registration-id',
|
||||
resourcePath,
|
||||
}),
|
||||
],
|
||||
[
|
||||
'deleteServerFile',
|
||||
(resourcePath: string) =>
|
||||
service.deleteServerFile({
|
||||
fileFolder: ServerFileFolder.ApplicationRegistration,
|
||||
applicationRegistrationId: 'registration-id',
|
||||
resourcePath,
|
||||
}),
|
||||
],
|
||||
] as const)('%s traversal protection', (_methodName, invoke) => {
|
||||
it.each(['../workspace-id/stolen.json', 'a/../../escape.json'])(
|
||||
'should reject traversal resource path %s without touching storage',
|
||||
async (resourcePath) => {
|
||||
await expect(
|
||||
(async () => {
|
||||
await invoke(resourcePath);
|
||||
})(),
|
||||
).rejects.toThrow(
|
||||
expect.objectContaining({
|
||||
code: FileStorageExceptionCode.ACCESS_DENIED,
|
||||
}),
|
||||
);
|
||||
|
||||
expect(mockDriver.readFile).not.toHaveBeenCalled();
|
||||
expect(mockDriver.checkFileExists).not.toHaveBeenCalled();
|
||||
expect(mockDriver.delete).not.toHaveBeenCalled();
|
||||
},
|
||||
);
|
||||
});
|
||||
|
||||
describe('writeServerFile', () => {
|
||||
it('should write bytes with the server prefix and upsert the row on path conflict', async () => {
|
||||
const serverFile = {
|
||||
id: 'server-file-id',
|
||||
path: 'application-registration/registration-id/manifests/manifest.json',
|
||||
} as FileEntity;
|
||||
|
||||
mockServerFileRepository.findOneByOrFail.mockResolvedValue(serverFile);
|
||||
|
||||
const result = await service.writeServerFile({
|
||||
fileFolder: ServerFileFolder.ApplicationRegistration,
|
||||
applicationRegistrationId: 'registration-id',
|
||||
resourcePath: 'manifests/manifest.json',
|
||||
contents: '{"name":"my-app"}',
|
||||
mimeType: 'application/json',
|
||||
});
|
||||
|
||||
expect(mockDriver.writeFile).toHaveBeenCalledWith({
|
||||
filePath:
|
||||
'server/application-registration/registration-id/manifests/manifest.json',
|
||||
mimeType: 'application/json',
|
||||
sourceFile: '{"name":"my-app"}',
|
||||
});
|
||||
expect(mockServerFileRepository.upsert).toHaveBeenCalledWith(
|
||||
{
|
||||
path: 'application-registration/registration-id/manifests/manifest.json',
|
||||
workspaceId: null,
|
||||
size: Buffer.byteLength('{"name":"my-app"}'),
|
||||
mimeType: 'application/json',
|
||||
applicationRegistrationId: 'registration-id',
|
||||
},
|
||||
{
|
||||
conflictPaths: ['applicationRegistrationId', 'path'],
|
||||
},
|
||||
);
|
||||
expect(mockServerFileRepository.findOneByOrFail).toHaveBeenCalledWith({
|
||||
applicationRegistrationId: 'registration-id',
|
||||
path: 'application-registration/registration-id/manifests/manifest.json',
|
||||
workspaceId: IsNull(),
|
||||
});
|
||||
expect(result).toEqual(serverFile);
|
||||
});
|
||||
|
||||
it('should reject a traversal resource path without touching storage', async () => {
|
||||
await expect(
|
||||
service.writeServerFile({
|
||||
fileFolder: ServerFileFolder.ApplicationRegistration,
|
||||
applicationRegistrationId: 'registration-id',
|
||||
resourcePath: '../workspace-id/stolen.json',
|
||||
contents: '{}',
|
||||
mimeType: 'application/json',
|
||||
}),
|
||||
).rejects.toThrow(
|
||||
expect.objectContaining({
|
||||
code: FileStorageExceptionCode.ACCESS_DENIED,
|
||||
}),
|
||||
);
|
||||
|
||||
expect(mockDriver.writeFile).not.toHaveBeenCalled();
|
||||
expect(mockServerFileRepository.upsert).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it('should propagate driver write failures without upserting the row', async () => {
|
||||
mockDriver.writeFile.mockRejectedValueOnce(new Error('Write failed'));
|
||||
|
||||
await expect(
|
||||
service.writeServerFile({
|
||||
fileFolder: ServerFileFolder.ApplicationRegistration,
|
||||
applicationRegistrationId: 'registration-id',
|
||||
resourcePath: 'manifest.json',
|
||||
contents: '{}',
|
||||
mimeType: 'application/json',
|
||||
}),
|
||||
).rejects.toThrow('Write failed');
|
||||
|
||||
expect(mockServerFileRepository.upsert).not.toHaveBeenCalled();
|
||||
});
|
||||
});
|
||||
|
||||
describe('readServerFile', () => {
|
||||
it('should read from the server-prefixed storage path', async () => {
|
||||
const stream = Readable.from(['{}']);
|
||||
|
||||
mockServerFileRepository.findOneBy.mockResolvedValue({
|
||||
id: 'server-file-id',
|
||||
path: 'application-registration/registration-id/manifests/manifest.json',
|
||||
} as FileEntity);
|
||||
mockDriver.readFile.mockResolvedValue(stream);
|
||||
|
||||
const result = await service.readServerFile({
|
||||
fileFolder: ServerFileFolder.ApplicationRegistration,
|
||||
applicationRegistrationId: 'registration-id',
|
||||
resourcePath: 'manifests/manifest.json',
|
||||
});
|
||||
|
||||
expect(mockServerFileRepository.findOneBy).toHaveBeenCalledWith({
|
||||
path: 'application-registration/registration-id/manifests/manifest.json',
|
||||
workspaceId: IsNull(),
|
||||
});
|
||||
expect(mockDriver.readFile).toHaveBeenCalledWith({
|
||||
filePath:
|
||||
'server/application-registration/registration-id/manifests/manifest.json',
|
||||
});
|
||||
expect(result).toBe(stream);
|
||||
});
|
||||
|
||||
it('should propagate the missing-file exception from the driver', async () => {
|
||||
mockServerFileRepository.findOneBy.mockResolvedValue({
|
||||
id: 'server-file-id',
|
||||
path: 'application-registration/registration-id/missing.json',
|
||||
} as FileEntity);
|
||||
mockDriver.readFile.mockRejectedValueOnce(
|
||||
new FileStorageException(
|
||||
'File not found',
|
||||
FileStorageExceptionCode.FILE_NOT_FOUND,
|
||||
),
|
||||
);
|
||||
|
||||
await expect(
|
||||
service.readServerFile({
|
||||
fileFolder: ServerFileFolder.ApplicationRegistration,
|
||||
applicationRegistrationId: 'registration-id',
|
||||
resourcePath: 'missing.json',
|
||||
}),
|
||||
).rejects.toThrow(
|
||||
expect.objectContaining({
|
||||
code: FileStorageExceptionCode.FILE_NOT_FOUND,
|
||||
}),
|
||||
);
|
||||
});
|
||||
|
||||
it('should throw FILE_NOT_FOUND without reading bytes when the row is missing', async () => {
|
||||
mockServerFileRepository.findOneBy.mockResolvedValue(null);
|
||||
|
||||
await expect(
|
||||
service.readServerFile({
|
||||
fileFolder: ServerFileFolder.ApplicationRegistration,
|
||||
applicationRegistrationId: 'registration-id',
|
||||
resourcePath: 'deleted.json',
|
||||
}),
|
||||
).rejects.toThrow(
|
||||
expect.objectContaining({
|
||||
code: FileStorageExceptionCode.FILE_NOT_FOUND,
|
||||
}),
|
||||
);
|
||||
|
||||
expect(mockDriver.readFile).not.toHaveBeenCalled();
|
||||
});
|
||||
});
|
||||
|
||||
describe('readServerFileById', () => {
|
||||
it('should read the bytes of the row storage path', async () => {
|
||||
const stream = Readable.from(['{}']);
|
||||
|
||||
mockServerFileRepository.findOneBy.mockResolvedValue({
|
||||
id: 'server-file-id',
|
||||
path: 'application-registration/registration-id/manifest.json',
|
||||
} as FileEntity);
|
||||
mockDriver.readFile.mockResolvedValue(stream);
|
||||
|
||||
const result = await service.readServerFileById('server-file-id');
|
||||
|
||||
expect(mockServerFileRepository.findOneBy).toHaveBeenCalledWith({
|
||||
id: 'server-file-id',
|
||||
workspaceId: IsNull(),
|
||||
});
|
||||
expect(mockDriver.readFile).toHaveBeenCalledWith({
|
||||
filePath:
|
||||
'server/application-registration/registration-id/manifest.json',
|
||||
});
|
||||
expect(result).toBe(stream);
|
||||
});
|
||||
|
||||
it('should throw a missing-file exception when the row does not exist', async () => {
|
||||
mockServerFileRepository.findOneBy.mockResolvedValue(null);
|
||||
|
||||
await expect(service.readServerFileById('unknown-id')).rejects.toThrow(
|
||||
expect.objectContaining({
|
||||
code: FileStorageExceptionCode.FILE_NOT_FOUND,
|
||||
}),
|
||||
);
|
||||
|
||||
expect(mockDriver.readFile).not.toHaveBeenCalled();
|
||||
});
|
||||
});
|
||||
|
||||
describe('checkServerFileExists', () => {
|
||||
it('should check existence on the server-prefixed storage path', async () => {
|
||||
mockDriver.checkFileExists.mockResolvedValue(true);
|
||||
|
||||
const result = await service.checkServerFileExists({
|
||||
fileFolder: ServerFileFolder.ApplicationRegistration,
|
||||
applicationRegistrationId: 'registration-id',
|
||||
resourcePath: 'manifest.json',
|
||||
});
|
||||
|
||||
expect(mockDriver.checkFileExists).toHaveBeenCalledWith({
|
||||
filePath:
|
||||
'server/application-registration/registration-id/manifest.json',
|
||||
});
|
||||
expect(result).toBe(true);
|
||||
});
|
||||
});
|
||||
|
||||
describe('deleteServerFile', () => {
|
||||
it('should delete the bytes and the row', async () => {
|
||||
await service.deleteServerFile({
|
||||
fileFolder: ServerFileFolder.ApplicationRegistration,
|
||||
applicationRegistrationId: 'registration-id',
|
||||
resourcePath: 'manifests/manifest.json',
|
||||
});
|
||||
|
||||
expect(mockDriver.delete).toHaveBeenCalledWith({
|
||||
folderPath: 'server/application-registration/registration-id/manifests',
|
||||
filename: 'manifest.json',
|
||||
});
|
||||
expect(mockServerFileRepository.delete).toHaveBeenCalledWith({
|
||||
path: 'application-registration/registration-id/manifests/manifest.json',
|
||||
workspaceId: IsNull(),
|
||||
});
|
||||
});
|
||||
|
||||
it('should still delete the row when the bytes deletion fails', async () => {
|
||||
mockDriver.delete.mockRejectedValueOnce(new Error('Delete failed'));
|
||||
|
||||
await service.deleteServerFile({
|
||||
fileFolder: ServerFileFolder.ApplicationRegistration,
|
||||
applicationRegistrationId: 'registration-id',
|
||||
resourcePath: 'manifest.json',
|
||||
});
|
||||
|
||||
expect(mockServerFileRepository.delete).toHaveBeenCalledWith({
|
||||
path: 'application-registration/registration-id/manifest.json',
|
||||
workspaceId: IsNull(),
|
||||
});
|
||||
});
|
||||
|
||||
it('should propagate row deletion failures', async () => {
|
||||
mockServerFileRepository.delete.mockRejectedValueOnce(
|
||||
new Error('Row deletion failed'),
|
||||
);
|
||||
|
||||
await expect(
|
||||
service.deleteServerFile({
|
||||
fileFolder: ServerFileFolder.ApplicationRegistration,
|
||||
applicationRegistrationId: 'registration-id',
|
||||
resourcePath: 'manifest.json',
|
||||
}),
|
||||
).rejects.toThrow('Row deletion failed');
|
||||
});
|
||||
});
|
||||
|
||||
describe('deleteByServerFileId', () => {
|
||||
it('should delete the bytes and the row of the given id', async () => {
|
||||
mockServerFileRepository.findOneBy.mockResolvedValue({
|
||||
id: 'server-file-id',
|
||||
path: 'application-registration/registration-id/manifest.json',
|
||||
} as FileEntity);
|
||||
|
||||
await service.deleteByServerFileId('server-file-id');
|
||||
|
||||
expect(mockDriver.delete).toHaveBeenCalledWith({
|
||||
folderPath: 'server/application-registration/registration-id',
|
||||
filename: 'manifest.json',
|
||||
});
|
||||
expect(mockServerFileRepository.delete).toHaveBeenCalledWith({
|
||||
id: 'server-file-id',
|
||||
workspaceId: IsNull(),
|
||||
});
|
||||
});
|
||||
|
||||
it('should throw a missing-file exception when the row does not exist', async () => {
|
||||
mockServerFileRepository.findOneBy.mockResolvedValue(null);
|
||||
|
||||
await expect(service.deleteByServerFileId('unknown-id')).rejects.toThrow(
|
||||
expect.objectContaining({
|
||||
code: FileStorageExceptionCode.FILE_NOT_FOUND,
|
||||
}),
|
||||
);
|
||||
|
||||
expect(mockServerFileRepository.delete).not.toHaveBeenCalled();
|
||||
});
|
||||
});
|
||||
|
||||
describe('deleteByApplicationRegistrationId', () => {
|
||||
it('should delete the bytes of every file then the rows', async () => {
|
||||
mockServerFileRepository.findBy.mockResolvedValue([
|
||||
{
|
||||
id: 'file-1',
|
||||
path: 'application-registration/registration-id/manifest.json',
|
||||
},
|
||||
{
|
||||
id: 'file-2',
|
||||
path: 'application-registration/registration-id/nested/settings.json',
|
||||
},
|
||||
] as FileEntity[]);
|
||||
|
||||
await service.deleteByApplicationRegistrationId('registration-id');
|
||||
|
||||
expect(mockServerFileRepository.findBy).toHaveBeenCalledWith({
|
||||
applicationRegistrationId: 'registration-id',
|
||||
workspaceId: IsNull(),
|
||||
});
|
||||
expect(mockDriver.delete).toHaveBeenCalledWith({
|
||||
folderPath: 'server/application-registration/registration-id',
|
||||
filename: 'manifest.json',
|
||||
});
|
||||
expect(mockDriver.delete).toHaveBeenCalledWith({
|
||||
folderPath: 'server/application-registration/registration-id/nested',
|
||||
filename: 'settings.json',
|
||||
});
|
||||
expect(mockServerFileRepository.delete).toHaveBeenCalledWith({
|
||||
applicationRegistrationId: 'registration-id',
|
||||
workspaceId: IsNull(),
|
||||
});
|
||||
});
|
||||
|
||||
it('should still delete the rows when a bytes deletion fails', async () => {
|
||||
mockServerFileRepository.findBy.mockResolvedValue([
|
||||
{
|
||||
id: 'file-1',
|
||||
path: 'application-registration/registration-id/manifest.json',
|
||||
},
|
||||
] as FileEntity[]);
|
||||
mockDriver.delete.mockRejectedValueOnce(new Error('Delete failed'));
|
||||
|
||||
await service.deleteByApplicationRegistrationId('registration-id');
|
||||
|
||||
expect(mockServerFileRepository.delete).toHaveBeenCalledWith({
|
||||
applicationRegistrationId: 'registration-id',
|
||||
workspaceId: IsNull(),
|
||||
});
|
||||
});
|
||||
});
|
||||
});
|
||||
+1
-1
@@ -429,7 +429,7 @@ export class FileStorageService {
|
||||
});
|
||||
|
||||
const application = await this.applicationRepository.findOneOrFail({
|
||||
where: { id: file.applicationId, workspaceId: file.workspaceId },
|
||||
where: { id: file.applicationId, workspaceId },
|
||||
});
|
||||
|
||||
await this.deleteFile({
|
||||
+267
@@ -0,0 +1,267 @@
|
||||
import { Injectable, Logger } from '@nestjs/common';
|
||||
import { InjectRepository } from '@nestjs/typeorm';
|
||||
|
||||
import { basename, dirname, join } from 'path';
|
||||
import { type Readable } from 'stream';
|
||||
|
||||
import { type ServerFileFolder } from 'twenty-shared/types';
|
||||
import { isDefined } from 'twenty-shared/utils';
|
||||
import { IsNull, Repository } from 'typeorm';
|
||||
|
||||
import { SERVER_FILE_STORAGE_PREFIX } from 'src/engine/core-modules/file-storage/constants/server-file-storage-prefix.constant';
|
||||
import { FileStorageDriverFactory } from 'src/engine/core-modules/file-storage/file-storage-driver.factory';
|
||||
import {
|
||||
FileStorageException,
|
||||
FileStorageExceptionCode,
|
||||
} from 'src/engine/core-modules/file-storage/interfaces/file-storage-exception';
|
||||
import { validateFilePath } from 'src/engine/core-modules/file-storage/utils/validate-file-path.util';
|
||||
import { validateStoragePathIsWithinServerScopeOrThrow } from 'src/engine/core-modules/file-storage/utils/validate-storage-path-is-within-server-scope-or-throw.util';
|
||||
import { FileEntity } from 'src/engine/core-modules/file/entities/file.entity';
|
||||
|
||||
export type ServerResourceIdentifier = {
|
||||
fileFolder: ServerFileFolder;
|
||||
applicationRegistrationId: string;
|
||||
resourcePath: string;
|
||||
};
|
||||
|
||||
@Injectable()
|
||||
export class ServerFileStorageService {
|
||||
private readonly logger = new Logger(ServerFileStorageService.name);
|
||||
|
||||
constructor(
|
||||
private readonly fileStorageDriverFactory: FileStorageDriverFactory,
|
||||
// eslint-disable-next-line twenty/prefer-workspace-scoped-repository -- server-scoped rows (workspaceId IS NULL) are unreachable through the scoped wrapper; every query below pins workspaceId to IsNull()
|
||||
@InjectRepository(FileEntity)
|
||||
private readonly serverFileRepository: Repository<FileEntity>,
|
||||
) {}
|
||||
|
||||
private validateAndBuildServerFileStoragePathOrThrow({
|
||||
fileFolder,
|
||||
applicationRegistrationId,
|
||||
resourcePath,
|
||||
}: ServerResourceIdentifier): {
|
||||
onStorageFilePath: string;
|
||||
filePath: string;
|
||||
} {
|
||||
const validationResult = validateFilePath({ resourcePath, fileFolder });
|
||||
|
||||
if (!validationResult.isValid) {
|
||||
throw new FileStorageException(
|
||||
validationResult.error,
|
||||
FileStorageExceptionCode.ACCESS_DENIED,
|
||||
);
|
||||
}
|
||||
|
||||
const filePath = join(
|
||||
fileFolder,
|
||||
applicationRegistrationId,
|
||||
resourcePath,
|
||||
).replace(/\/+/g, '/');
|
||||
|
||||
const onStorageFilePath = join(
|
||||
SERVER_FILE_STORAGE_PREFIX,
|
||||
filePath,
|
||||
).replace(/\/+/g, '/');
|
||||
|
||||
validateStoragePathIsWithinServerScopeOrThrow({
|
||||
onStoragePath: onStorageFilePath,
|
||||
fileFolder,
|
||||
});
|
||||
|
||||
return { onStorageFilePath, filePath };
|
||||
}
|
||||
|
||||
async writeServerFile({
|
||||
fileFolder,
|
||||
applicationRegistrationId,
|
||||
resourcePath,
|
||||
contents,
|
||||
mimeType,
|
||||
}: ServerResourceIdentifier & {
|
||||
contents: Buffer | string;
|
||||
mimeType: string;
|
||||
}): Promise<FileEntity> {
|
||||
const driver = this.fileStorageDriverFactory.getCurrentDriver();
|
||||
|
||||
const { onStorageFilePath, filePath } =
|
||||
this.validateAndBuildServerFileStoragePathOrThrow({
|
||||
fileFolder,
|
||||
applicationRegistrationId,
|
||||
resourcePath,
|
||||
});
|
||||
|
||||
await driver.writeFile({
|
||||
filePath: onStorageFilePath,
|
||||
mimeType,
|
||||
sourceFile: contents,
|
||||
});
|
||||
|
||||
await this.serverFileRepository.upsert(
|
||||
{
|
||||
path: filePath,
|
||||
workspaceId: null,
|
||||
size:
|
||||
typeof contents === 'string'
|
||||
? Buffer.byteLength(contents)
|
||||
: contents.length,
|
||||
mimeType,
|
||||
applicationRegistrationId,
|
||||
},
|
||||
{
|
||||
conflictPaths: ['applicationRegistrationId', 'path'],
|
||||
},
|
||||
);
|
||||
|
||||
return this.serverFileRepository.findOneByOrFail({
|
||||
applicationRegistrationId,
|
||||
path: filePath,
|
||||
workspaceId: IsNull(),
|
||||
});
|
||||
}
|
||||
|
||||
async readServerFile({
|
||||
fileFolder,
|
||||
applicationRegistrationId,
|
||||
resourcePath,
|
||||
}: ServerResourceIdentifier): Promise<Readable> {
|
||||
const driver = this.fileStorageDriverFactory.getCurrentDriver();
|
||||
|
||||
const { onStorageFilePath, filePath } =
|
||||
this.validateAndBuildServerFileStoragePathOrThrow({
|
||||
fileFolder,
|
||||
applicationRegistrationId,
|
||||
resourcePath,
|
||||
});
|
||||
|
||||
const serverFile = await this.serverFileRepository.findOneBy({
|
||||
path: filePath,
|
||||
workspaceId: IsNull(),
|
||||
});
|
||||
|
||||
if (!isDefined(serverFile)) {
|
||||
throw new FileStorageException(
|
||||
`Server file ${filePath} not found`,
|
||||
FileStorageExceptionCode.FILE_NOT_FOUND,
|
||||
);
|
||||
}
|
||||
|
||||
return driver.readFile({ filePath: onStorageFilePath });
|
||||
}
|
||||
|
||||
async readServerFileById(id: string): Promise<Readable> {
|
||||
const serverFile = await this.findServerFileByIdOrThrow(id);
|
||||
|
||||
const driver = this.fileStorageDriverFactory.getCurrentDriver();
|
||||
|
||||
return driver.readFile({
|
||||
filePath: this.buildServerOnStorageFilePath(serverFile),
|
||||
});
|
||||
}
|
||||
|
||||
checkServerFileExists({
|
||||
fileFolder,
|
||||
applicationRegistrationId,
|
||||
resourcePath,
|
||||
}: ServerResourceIdentifier): Promise<boolean> {
|
||||
const driver = this.fileStorageDriverFactory.getCurrentDriver();
|
||||
|
||||
const { onStorageFilePath } =
|
||||
this.validateAndBuildServerFileStoragePathOrThrow({
|
||||
fileFolder,
|
||||
applicationRegistrationId,
|
||||
resourcePath,
|
||||
});
|
||||
|
||||
return driver.checkFileExists({ filePath: onStorageFilePath });
|
||||
}
|
||||
|
||||
async deleteServerFile({
|
||||
fileFolder,
|
||||
applicationRegistrationId,
|
||||
resourcePath,
|
||||
}: ServerResourceIdentifier): Promise<void> {
|
||||
const { onStorageFilePath, filePath } =
|
||||
this.validateAndBuildServerFileStoragePathOrThrow({
|
||||
fileFolder,
|
||||
applicationRegistrationId,
|
||||
resourcePath,
|
||||
});
|
||||
|
||||
await this.deleteServerFileBytesBestEffort(onStorageFilePath);
|
||||
|
||||
await this.serverFileRepository.delete({
|
||||
path: filePath,
|
||||
workspaceId: IsNull(),
|
||||
});
|
||||
}
|
||||
|
||||
async deleteByServerFileId(id: string): Promise<void> {
|
||||
const serverFile = await this.findServerFileByIdOrThrow(id);
|
||||
|
||||
await this.deleteServerFileBytesBestEffort(
|
||||
this.buildServerOnStorageFilePath(serverFile),
|
||||
);
|
||||
|
||||
await this.serverFileRepository.delete({
|
||||
id,
|
||||
workspaceId: IsNull(),
|
||||
});
|
||||
}
|
||||
|
||||
async deleteByApplicationRegistrationId(
|
||||
applicationRegistrationId: string,
|
||||
): Promise<void> {
|
||||
const serverFiles = await this.serverFileRepository.findBy({
|
||||
applicationRegistrationId,
|
||||
workspaceId: IsNull(),
|
||||
});
|
||||
|
||||
for (const serverFile of serverFiles) {
|
||||
await this.deleteServerFileBytesBestEffort(
|
||||
this.buildServerOnStorageFilePath(serverFile),
|
||||
);
|
||||
}
|
||||
|
||||
await this.serverFileRepository.delete({
|
||||
applicationRegistrationId,
|
||||
workspaceId: IsNull(),
|
||||
});
|
||||
}
|
||||
|
||||
private async findServerFileByIdOrThrow(id: string): Promise<FileEntity> {
|
||||
const serverFile = await this.serverFileRepository.findOneBy({
|
||||
id,
|
||||
workspaceId: IsNull(),
|
||||
});
|
||||
|
||||
if (!isDefined(serverFile)) {
|
||||
throw new FileStorageException(
|
||||
`Server file ${id} not found`,
|
||||
FileStorageExceptionCode.FILE_NOT_FOUND,
|
||||
);
|
||||
}
|
||||
|
||||
return serverFile;
|
||||
}
|
||||
|
||||
private buildServerOnStorageFilePath(serverFile: FileEntity): string {
|
||||
return join(SERVER_FILE_STORAGE_PREFIX, serverFile.path);
|
||||
}
|
||||
|
||||
private async deleteServerFileBytesBestEffort(
|
||||
onStorageFilePath: string,
|
||||
): Promise<void> {
|
||||
const driver = this.fileStorageDriverFactory.getCurrentDriver();
|
||||
|
||||
try {
|
||||
await driver.delete({
|
||||
folderPath: dirname(onStorageFilePath),
|
||||
filename: basename(onStorageFilePath),
|
||||
});
|
||||
} catch (error) {
|
||||
this.logger.warn(
|
||||
`Failed to delete server file bytes at ${onStorageFilePath}: ${error}`,
|
||||
);
|
||||
}
|
||||
}
|
||||
}
|
||||
+78
@@ -0,0 +1,78 @@
|
||||
import { ServerFileFolder } from 'twenty-shared/types';
|
||||
import { FileStorageExceptionCode } from 'src/engine/core-modules/file-storage/interfaces/file-storage-exception';
|
||||
|
||||
import { validateStoragePathIsWithinServerScopeOrThrow } from 'src/engine/core-modules/file-storage/utils/validate-storage-path-is-within-server-scope-or-throw.util';
|
||||
|
||||
const primitives = {
|
||||
fileFolder: ServerFileFolder.ApplicationRegistration,
|
||||
} as const;
|
||||
|
||||
describe('validateStoragePathIsWithinServerScopeOrThrow', () => {
|
||||
it.each([
|
||||
{
|
||||
title: 'nested path within prefix',
|
||||
onStoragePath: 'server/application-registration/manifests/manifest.json',
|
||||
},
|
||||
{
|
||||
title: 'file directly under prefix',
|
||||
onStoragePath: 'server/application-registration/manifest.json',
|
||||
},
|
||||
])('should accept valid path: $title', ({ onStoragePath }) => {
|
||||
expect(() =>
|
||||
validateStoragePathIsWithinServerScopeOrThrow({
|
||||
onStoragePath,
|
||||
...primitives,
|
||||
}),
|
||||
).not.toThrow();
|
||||
});
|
||||
|
||||
it.each([
|
||||
{
|
||||
title: 'workspace-like prefix instead of server prefix',
|
||||
onStoragePath: 'workspace-id/app-uid/source/file.json',
|
||||
},
|
||||
{
|
||||
title: 'different file folder',
|
||||
onStoragePath: 'server/other-folder/file.json',
|
||||
},
|
||||
{
|
||||
title: 'prefix without trailing file',
|
||||
onStoragePath: 'server/application-registration',
|
||||
},
|
||||
{
|
||||
title: 'partial prefix match (malicious suffix)',
|
||||
onStoragePath: 'server/application-registrationMalicious/file.json',
|
||||
},
|
||||
{
|
||||
title: 'traversal out of the server prefix',
|
||||
onStoragePath:
|
||||
'server/application-registration/../../workspace-id/file.json',
|
||||
},
|
||||
{
|
||||
title: 'traversal segments kept after normalization',
|
||||
onStoragePath: 'server/application-registration/../../../etc/passwd',
|
||||
},
|
||||
{
|
||||
title: 'absolute path',
|
||||
onStoragePath: '/server/application-registration/file.json',
|
||||
},
|
||||
{
|
||||
title: 'null byte in path',
|
||||
onStoragePath: 'server/application-registration/file\0.json',
|
||||
},
|
||||
])(
|
||||
'should reject path that escapes server scope: $title',
|
||||
({ onStoragePath }) => {
|
||||
expect(() =>
|
||||
validateStoragePathIsWithinServerScopeOrThrow({
|
||||
onStoragePath,
|
||||
...primitives,
|
||||
}),
|
||||
).toThrow(
|
||||
expect.objectContaining({
|
||||
code: FileStorageExceptionCode.ACCESS_DENIED,
|
||||
}),
|
||||
);
|
||||
},
|
||||
);
|
||||
});
|
||||
+2
-2
@@ -1,5 +1,5 @@
|
||||
import { t } from '@lingui/core/macro';
|
||||
import { type FileFolder } from 'twenty-shared/types';
|
||||
import { type FileFolder, type ServerFileFolder } from 'twenty-shared/types';
|
||||
|
||||
import { ALLOWED_EXTENSIONS_BY_APPLICATION_FILE_FOLDER } from 'src/engine/core-modules/file-storage/constants/allowed-extensions-by-application-file-folder.constant';
|
||||
import { type ResourcePathValidationResult } from 'src/engine/core-modules/file-storage/types/resource-path-validation-result.type';
|
||||
@@ -10,7 +10,7 @@ export const validateFileExtension = ({
|
||||
fileFolder,
|
||||
}: {
|
||||
resourcePath: string;
|
||||
fileFolder: FileFolder;
|
||||
fileFolder: FileFolder | ServerFileFolder;
|
||||
}): ResourcePathValidationResult => {
|
||||
const allowedExtensions =
|
||||
ALLOWED_EXTENSIONS_BY_APPLICATION_FILE_FOLDER[
|
||||
|
||||
+2
-2
@@ -1,5 +1,5 @@
|
||||
import { t } from '@lingui/core/macro';
|
||||
import { type FileFolder } from 'twenty-shared/types';
|
||||
import { type FileFolder, type ServerFileFolder } from 'twenty-shared/types';
|
||||
|
||||
import { type ResourcePathValidationResult } from 'src/engine/core-modules/file-storage/types/resource-path-validation-result.type';
|
||||
import { validateFileExtension } from 'src/engine/core-modules/file-storage/utils/validate-file-extension.util';
|
||||
@@ -11,7 +11,7 @@ export const validateFilePath = ({
|
||||
fileFolder,
|
||||
}: {
|
||||
resourcePath: string;
|
||||
fileFolder: FileFolder;
|
||||
fileFolder: FileFolder | ServerFileFolder;
|
||||
}): ResourcePathValidationResult => {
|
||||
const safePathResult = validateSafeRelativePath({ resourcePath });
|
||||
|
||||
|
||||
+32
@@ -0,0 +1,32 @@
|
||||
import { join, normalize } from 'path';
|
||||
|
||||
import { type ServerFileFolder } from 'twenty-shared/types';
|
||||
|
||||
import {
|
||||
FileStorageException,
|
||||
FileStorageExceptionCode,
|
||||
} from 'src/engine/core-modules/file-storage/interfaces/file-storage-exception';
|
||||
import { assertStoragePathIsSafe } from 'src/engine/core-modules/file-storage/utils/assert-storage-path-is-safe.util';
|
||||
import { SERVER_FILE_STORAGE_PREFIX } from 'src/engine/core-modules/file-storage/constants/server-file-storage-prefix.constant';
|
||||
|
||||
export const validateStoragePathIsWithinServerScopeOrThrow = ({
|
||||
onStoragePath,
|
||||
fileFolder,
|
||||
}: {
|
||||
onStoragePath: string;
|
||||
fileFolder: ServerFileFolder;
|
||||
}): void => {
|
||||
assertStoragePathIsSafe(onStoragePath);
|
||||
|
||||
const expectedPrefix = join(SERVER_FILE_STORAGE_PREFIX, fileFolder);
|
||||
|
||||
const normalizedPath = normalize(onStoragePath);
|
||||
const normalizedPrefix = normalize(expectedPrefix + '/');
|
||||
|
||||
if (!normalizedPath.startsWith(normalizedPrefix)) {
|
||||
throw new FileStorageException(
|
||||
'Invalid storage path: resolved path escapes the server scope',
|
||||
FileStorageExceptionCode.ACCESS_DENIED,
|
||||
);
|
||||
}
|
||||
};
|
||||
@@ -14,6 +14,8 @@ import {
|
||||
} from 'typeorm';
|
||||
|
||||
import { ADD_STATUS_TO_FILE_UPGRADE_COMMAND_NAME } from 'src/database/commands/upgrade-version-command/2-19/add-status-to-file-upgrade-command-name.constant';
|
||||
import { ALLOW_SERVER_SCOPED_FILE_UPGRADE_COMMAND_NAME } from 'src/database/commands/upgrade-version-command/2-20/allow-server-scoped-file-upgrade-command-name.constant';
|
||||
import { type ApplicationRegistrationEntity } from 'src/engine/core-modules/application/application-registration/application-registration.entity';
|
||||
import { ApplicationEntity } from 'src/engine/core-modules/application/application.entity';
|
||||
import { FileSettings } from 'src/engine/core-modules/file/types/file-settings.types';
|
||||
import {
|
||||
@@ -21,24 +23,47 @@ import {
|
||||
FileStatus,
|
||||
} from 'src/engine/core-modules/file/types/file-status.types';
|
||||
import { WasIntroducedInUpgrade } from 'src/engine/core-modules/upgrade/decorators/was-introduced-in-upgrade.decorator';
|
||||
import { WorkspaceRelatedEntity } from 'src/engine/workspace-manager/types/workspace-related-entity';
|
||||
import { type WorkspaceEntity } from 'src/engine/core-modules/workspace/workspace.entity';
|
||||
|
||||
@Entity('file')
|
||||
@Check(
|
||||
'CHK_FILE_PENDING_MIME_OCTET_STREAM',
|
||||
`"status" != 'PENDING' OR "mimeType" = 'application/octet-stream'`,
|
||||
)
|
||||
@Check(
|
||||
'CHK_FILE_WORKSPACE_ID_OR_APPLICATION_REGISTRATION_ID',
|
||||
`"workspaceId" IS NOT NULL OR "applicationRegistrationId" IS NOT NULL`,
|
||||
)
|
||||
@Check(
|
||||
'CHK_FILE_WORKSPACE_ID_XOR_APPLICATION_REGISTRATION_ID',
|
||||
`"workspaceId" IS NULL OR "applicationRegistrationId" IS NULL`,
|
||||
)
|
||||
@Index('IDX_FILE_WORKSPACE_ID', ['workspaceId'])
|
||||
@Index('IDX_FILE_STATUS', ['status'])
|
||||
@Index('IDX_FILE_APPLICATION_REGISTRATION_ID', ['applicationRegistrationId'])
|
||||
@Unique('IDX_APPLICATION_PATH_WORKSPACE_ID_APPLICATION_ID_UNIQUE', [
|
||||
'workspaceId',
|
||||
'applicationId',
|
||||
'path',
|
||||
])
|
||||
export class FileEntity extends WorkspaceRelatedEntity {
|
||||
@Unique('IDX_FILE_APPLICATION_REGISTRATION_ID_PATH_UNIQUE', [
|
||||
'applicationRegistrationId',
|
||||
'path',
|
||||
])
|
||||
export class FileEntity {
|
||||
@PrimaryGeneratedColumn('uuid')
|
||||
id: string;
|
||||
|
||||
@Column({ nullable: true, type: 'uuid' })
|
||||
workspaceId: string | null;
|
||||
|
||||
@ManyToOne('WorkspaceEntity', {
|
||||
onDelete: 'CASCADE',
|
||||
nullable: true,
|
||||
})
|
||||
@JoinColumn({ name: 'workspaceId' })
|
||||
workspace: Relation<WorkspaceEntity> | null;
|
||||
|
||||
@Column({ nullable: true, type: 'uuid' })
|
||||
applicationId: string;
|
||||
|
||||
@@ -48,6 +73,19 @@ export class FileEntity extends WorkspaceRelatedEntity {
|
||||
@JoinColumn({ name: 'applicationId' })
|
||||
application: Relation<ApplicationEntity>;
|
||||
|
||||
@WasIntroducedInUpgrade({
|
||||
upgradeCommandName: ALLOW_SERVER_SCOPED_FILE_UPGRADE_COMMAND_NAME,
|
||||
})
|
||||
@Column({ nullable: true, type: 'uuid' })
|
||||
applicationRegistrationId: string | null;
|
||||
|
||||
@ManyToOne('ApplicationRegistrationEntity', {
|
||||
onDelete: 'CASCADE',
|
||||
nullable: true,
|
||||
})
|
||||
@JoinColumn({ name: 'applicationRegistrationId' })
|
||||
applicationRegistration: Relation<ApplicationRegistrationEntity> | null;
|
||||
|
||||
@Column({ nullable: false })
|
||||
path: string;
|
||||
|
||||
|
||||
+1
-1
@@ -5,7 +5,7 @@ import { FileFolder } from 'twenty-shared/types';
|
||||
import { v4 } from 'uuid';
|
||||
|
||||
import { ApplicationService } from 'src/engine/core-modules/application/application.service';
|
||||
import { FileStorageService } from 'src/engine/core-modules/file-storage/file-storage.service';
|
||||
import { FileStorageService } from 'src/engine/core-modules/file-storage/services/file-storage.service';
|
||||
import { FileWithSignedUrlDTO } from 'src/engine/core-modules/file/dtos/file-with-sign-url.dto';
|
||||
import { FileUrlService } from 'src/engine/core-modules/file/file-url/file-url.service';
|
||||
import { extractFileInfoOrThrow } from 'src/engine/core-modules/file/utils/extract-file-info-or-throw.utils';
|
||||
|
||||
+1
-1
@@ -20,7 +20,7 @@ import {
|
||||
AuthException,
|
||||
AuthExceptionCode,
|
||||
} from 'src/engine/core-modules/auth/auth.exception';
|
||||
import { FileStorageService } from 'src/engine/core-modules/file-storage/file-storage.service';
|
||||
import { FileStorageService } from 'src/engine/core-modules/file-storage/services/file-storage.service';
|
||||
import { FileWithSignedUrlDTO } from 'src/engine/core-modules/file/dtos/file-with-sign-url.dto';
|
||||
import { FileEntity } from 'src/engine/core-modules/file/entities/file.entity';
|
||||
import { FileUrlService } from 'src/engine/core-modules/file/file-url/file-url.service';
|
||||
|
||||
+1
-1
@@ -5,7 +5,7 @@ import { FileFolder } from 'twenty-shared/types';
|
||||
import { v4 } from 'uuid';
|
||||
|
||||
import { ApplicationService } from 'src/engine/core-modules/application/application.service';
|
||||
import { FileStorageService } from 'src/engine/core-modules/file-storage/file-storage.service';
|
||||
import { FileStorageService } from 'src/engine/core-modules/file-storage/services/file-storage.service';
|
||||
import { FileWithSignedUrlDTO } from 'src/engine/core-modules/file/dtos/file-with-sign-url.dto';
|
||||
import { FileUrlService } from 'src/engine/core-modules/file/file-url/file-url.service';
|
||||
import { extractFileInfoOrThrow } from 'src/engine/core-modules/file/utils/extract-file-info-or-throw.utils';
|
||||
|
||||
+1
-1
@@ -4,7 +4,7 @@ import { getRepositoryToken } from '@nestjs/typeorm';
|
||||
import { FileFolder } from 'twenty-shared/types';
|
||||
|
||||
import { ApplicationEntity } from 'src/engine/core-modules/application/application.entity';
|
||||
import { FileStorageService } from 'src/engine/core-modules/file-storage/file-storage.service';
|
||||
import { FileStorageService } from 'src/engine/core-modules/file-storage/services/file-storage.service';
|
||||
import { FileEntity } from 'src/engine/core-modules/file/entities/file.entity';
|
||||
import {
|
||||
PENDING_FILE_CLEANUP_BATCH_SIZE,
|
||||
|
||||
+1
-1
@@ -7,7 +7,7 @@ import { FileFolder } from 'twenty-shared/types';
|
||||
|
||||
import { ApplicationEntity } from 'src/engine/core-modules/application/application.entity';
|
||||
import { ApplicationService } from 'src/engine/core-modules/application/application.service';
|
||||
import { FileStorageService } from 'src/engine/core-modules/file-storage/file-storage.service';
|
||||
import { FileStorageService } from 'src/engine/core-modules/file-storage/services/file-storage.service';
|
||||
import { FileEntity } from 'src/engine/core-modules/file/entities/file.entity';
|
||||
import {
|
||||
FileUploadException,
|
||||
|
||||
+1
-1
@@ -17,7 +17,7 @@ import { ApplicationEntity } from 'src/engine/core-modules/application/applicati
|
||||
import { ApplicationService } from 'src/engine/core-modules/application/application.service';
|
||||
import { FileUploadTokenJwtPayload } from 'src/engine/core-modules/auth/types/file-upload-token-jwt-payload.type';
|
||||
import { JwtTokenTypeEnum } from 'src/engine/core-modules/auth/types/jwt-token-type.enum';
|
||||
import { FileStorageService } from 'src/engine/core-modules/file-storage/file-storage.service';
|
||||
import { FileStorageService } from 'src/engine/core-modules/file-storage/services/file-storage.service';
|
||||
import { FileWithSignedUrlDTO } from 'src/engine/core-modules/file/dtos/file-with-sign-url.dto';
|
||||
import { FileEntity } from 'src/engine/core-modules/file/entities/file.entity';
|
||||
import { FILE_CONTENT_SNIFF_BYTE_COUNT } from 'src/engine/core-modules/file/file-upload/constants/file-content-sniff.constant';
|
||||
|
||||
+7
-2
@@ -3,10 +3,10 @@ import { InjectRepository } from '@nestjs/typeorm';
|
||||
|
||||
import { FileFolder } from 'twenty-shared/types';
|
||||
import { isDefined } from 'twenty-shared/utils';
|
||||
import { LessThan, Repository } from 'typeorm';
|
||||
import { IsNull, LessThan, Not, Repository } from 'typeorm';
|
||||
|
||||
import { ApplicationEntity } from 'src/engine/core-modules/application/application.entity';
|
||||
import { FileStorageService } from 'src/engine/core-modules/file-storage/file-storage.service';
|
||||
import { FileStorageService } from 'src/engine/core-modules/file-storage/services/file-storage.service';
|
||||
import { FileEntity } from 'src/engine/core-modules/file/entities/file.entity';
|
||||
import {
|
||||
PENDING_FILE_CLEANUP_BATCH_SIZE,
|
||||
@@ -40,6 +40,7 @@ export class PendingFileCleanupService {
|
||||
where: {
|
||||
status: FILE_STATUS.PENDING,
|
||||
createdAt: LessThan(staleThreshold),
|
||||
workspaceId: Not(IsNull()),
|
||||
},
|
||||
take: PENDING_FILE_CLEANUP_BATCH_SIZE,
|
||||
});
|
||||
@@ -78,6 +79,10 @@ export class PendingFileCleanupService {
|
||||
// partial, possibly absent) storage object. A failure here leaks bytes but
|
||||
// never data, so it is logged rather than retried.
|
||||
private async deleteStorageObject(file: FileEntity): Promise<void> {
|
||||
if (!isDefined(file.workspaceId)) {
|
||||
return;
|
||||
}
|
||||
|
||||
const [fileFolder] = file.path.split('/');
|
||||
|
||||
const application = await this.applicationRepository.findOne({
|
||||
|
||||
+1
-1
@@ -5,7 +5,7 @@ import { FileFolder } from 'twenty-shared/types';
|
||||
import { v4 } from 'uuid';
|
||||
|
||||
import { ApplicationService } from 'src/engine/core-modules/application/application.service';
|
||||
import { FileStorageService } from 'src/engine/core-modules/file-storage/file-storage.service';
|
||||
import { FileStorageService } from 'src/engine/core-modules/file-storage/services/file-storage.service';
|
||||
import { FileWithSignedUrlDTO } from 'src/engine/core-modules/file/dtos/file-with-sign-url.dto';
|
||||
import { FileUrlService } from 'src/engine/core-modules/file/file-url/file-url.service';
|
||||
import { extractFileInfoOrThrow } from 'src/engine/core-modules/file/utils/extract-file-info-or-throw.utils';
|
||||
|
||||
+1
-1
@@ -8,7 +8,7 @@ import { Repository } from 'typeorm';
|
||||
import { v4 } from 'uuid';
|
||||
|
||||
import { ApplicationEntity } from 'src/engine/core-modules/application/application.entity';
|
||||
import { FileStorageService } from 'src/engine/core-modules/file-storage/file-storage.service';
|
||||
import { FileStorageService } from 'src/engine/core-modules/file-storage/services/file-storage.service';
|
||||
import { FileWithSignedUrlDTO } from 'src/engine/core-modules/file/dtos/file-with-sign-url.dto';
|
||||
import { FileUrlService } from 'src/engine/core-modules/file/file-url/file-url.service';
|
||||
import {
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
import { FileFolder } from 'twenty-shared/types';
|
||||
|
||||
import { FileStorageService } from 'src/engine/core-modules/file-storage/file-storage.service';
|
||||
import { FileStorageService } from 'src/engine/core-modules/file-storage/services/file-storage.service';
|
||||
import { Process } from 'src/engine/core-modules/message-queue/decorators/process.decorator';
|
||||
import { Processor } from 'src/engine/core-modules/message-queue/decorators/processor.decorator';
|
||||
import { MessageQueue } from 'src/engine/core-modules/message-queue/message-queue.constants';
|
||||
|
||||
@@ -2,7 +2,7 @@ import { Test, type TestingModule } from '@nestjs/testing';
|
||||
import { getRepositoryToken } from '@nestjs/typeorm';
|
||||
|
||||
import { ApplicationEntity } from 'src/engine/core-modules/application/application.entity';
|
||||
import { FileStorageService } from 'src/engine/core-modules/file-storage/file-storage.service';
|
||||
import { FileStorageService } from 'src/engine/core-modules/file-storage/services/file-storage.service';
|
||||
import { FileEntity } from 'src/engine/core-modules/file/entities/file.entity';
|
||||
import { JwtWrapperService } from 'src/engine/core-modules/jwt/services/jwt-wrapper.service';
|
||||
import { TwentyConfigService } from 'src/engine/core-modules/twenty-config/twenty-config.service';
|
||||
|
||||
@@ -7,7 +7,7 @@ import { FileFolder } from 'twenty-shared/types';
|
||||
import { Like, Repository } from 'typeorm';
|
||||
|
||||
import { ApplicationEntity } from 'src/engine/core-modules/application/application.entity';
|
||||
import { FileStorageService } from 'src/engine/core-modules/file-storage/file-storage.service';
|
||||
import { FileStorageService } from 'src/engine/core-modules/file-storage/services/file-storage.service';
|
||||
import {
|
||||
FileStorageException,
|
||||
FileStorageExceptionCode,
|
||||
|
||||
+1
-1
@@ -9,7 +9,7 @@ import { isDefined } from 'twenty-shared/utils';
|
||||
|
||||
import { FileStorageExceptionCode } from 'src/engine/core-modules/file-storage/interfaces/file-storage-exception';
|
||||
|
||||
import { FileStorageService } from 'src/engine/core-modules/file-storage/file-storage.service';
|
||||
import { FileStorageService } from 'src/engine/core-modules/file-storage/services/file-storage.service';
|
||||
import {
|
||||
getLogicFunctionSeedProjectFiles,
|
||||
LogicFunctionSeedProjectFile,
|
||||
|
||||
+1
-1
@@ -6,7 +6,7 @@ import { Repository } from 'typeorm';
|
||||
import { WorkspaceSchemaFactory } from 'src/engine/api/graphql/workspace-schema.factory';
|
||||
import { ApplicationEntity } from 'src/engine/core-modules/application/application.entity';
|
||||
import { ApplicationService } from 'src/engine/core-modules/application/application.service';
|
||||
import { FileStorageService } from 'src/engine/core-modules/file-storage/file-storage.service';
|
||||
import { FileStorageService } from 'src/engine/core-modules/file-storage/services/file-storage.service';
|
||||
import { getQueueToken } from 'src/engine/core-modules/message-queue/utils/get-queue-token.util';
|
||||
import { MessageQueue } from 'src/engine/core-modules/message-queue/message-queue.constants';
|
||||
import { MessageQueueService } from 'src/engine/core-modules/message-queue/services/message-queue.service';
|
||||
|
||||
+1
-1
@@ -7,7 +7,7 @@ import { FileFolder } from 'twenty-shared/types';
|
||||
import { Repository } from 'typeorm';
|
||||
|
||||
import { ApplicationEntity } from 'src/engine/core-modules/application/application.entity';
|
||||
import { FileStorageService } from 'src/engine/core-modules/file-storage/file-storage.service';
|
||||
import { FileStorageService } from 'src/engine/core-modules/file-storage/services/file-storage.service';
|
||||
import {
|
||||
FileStorageException,
|
||||
FileStorageExceptionCode,
|
||||
|
||||
+1
-1
@@ -12,7 +12,7 @@ import { Repository } from 'typeorm';
|
||||
import { WorkspaceSchemaFactory } from 'src/engine/api/graphql/workspace-schema.factory';
|
||||
import { ApplicationEntity } from 'src/engine/core-modules/application/application.entity';
|
||||
import { ApplicationService } from 'src/engine/core-modules/application/application.service';
|
||||
import { FileStorageService } from 'src/engine/core-modules/file-storage/file-storage.service';
|
||||
import { FileStorageService } from 'src/engine/core-modules/file-storage/services/file-storage.service';
|
||||
import { createZipFile } from 'src/engine/core-modules/logic-function/logic-function-drivers/utils/create-zip-file';
|
||||
import { TemporaryDirManager } from 'src/engine/core-modules/logic-function/logic-function-drivers/utils/temporary-dir-manager';
|
||||
import { InjectMessageQueue } from 'src/engine/core-modules/message-queue/decorators/message-queue.decorator';
|
||||
|
||||
+7
-4
@@ -1,13 +1,16 @@
|
||||
import { FileFolder } from 'twenty-shared/types';
|
||||
|
||||
import { type ApplicationService } from 'src/engine/core-modules/application/application.service';
|
||||
import { type FileStorageService } from 'src/engine/core-modules/file-storage/file-storage.service';
|
||||
import { type FileStorageService } from 'src/engine/core-modules/file-storage/services/file-storage.service';
|
||||
import { ToolOutputSpillService } from 'src/engine/core-modules/tool/services/tool-output-spill.service';
|
||||
import { type ToolOutput } from 'src/engine/core-modules/tool/types/tool-output.type';
|
||||
|
||||
jest.mock('src/engine/core-modules/file-storage/file-storage.service', () => ({
|
||||
FileStorageService: class {},
|
||||
}));
|
||||
jest.mock(
|
||||
'src/engine/core-modules/file-storage/services/file-storage.service',
|
||||
() => ({
|
||||
FileStorageService: class {},
|
||||
}),
|
||||
);
|
||||
jest.mock('src/engine/core-modules/application/application.service', () => ({
|
||||
ApplicationService: class {},
|
||||
}));
|
||||
|
||||
+1
-1
@@ -6,7 +6,7 @@ import { v4 } from 'uuid';
|
||||
|
||||
import { isObject } from 'class-validator';
|
||||
import { ApplicationService } from 'src/engine/core-modules/application/application.service';
|
||||
import { FileStorageService } from 'src/engine/core-modules/file-storage/file-storage.service';
|
||||
import { FileStorageService } from 'src/engine/core-modules/file-storage/services/file-storage.service';
|
||||
import { MAX_INLINE_TOOL_OUTPUT_BYTES } from 'src/engine/core-modules/tool/tools/output-navigation-tool/constants/max-inline-tool-output-bytes.constant';
|
||||
import { OUTPUT_NAVIGATION_TOOL_NAMES } from 'src/engine/core-modules/tool/tools/output-navigation-tool/constants/output-navigation-tool-names.constant';
|
||||
import { type ToolOutput } from 'src/engine/core-modules/tool/types/tool-output.type';
|
||||
|
||||
+1
-1
@@ -19,7 +19,7 @@ import { ApplicationService } from 'src/engine/core-modules/application/applicat
|
||||
import { type AccessTokenJwtPayload } from 'src/engine/core-modules/auth/types/access-token-jwt-payload.type';
|
||||
import { JwtTokenTypeEnum } from 'src/engine/core-modules/auth/types/jwt-token-type.enum';
|
||||
import { CodeInterpreterService } from 'src/engine/core-modules/code-interpreter/code-interpreter.service';
|
||||
import { FileStorageService } from 'src/engine/core-modules/file-storage/file-storage.service';
|
||||
import { FileStorageService } from 'src/engine/core-modules/file-storage/services/file-storage.service';
|
||||
import { FileUrlService } from 'src/engine/core-modules/file/file-url/file-url.service';
|
||||
import { FileService } from 'src/engine/core-modules/file/services/file.service';
|
||||
import { JwtWrapperService } from 'src/engine/core-modules/jwt/services/jwt-wrapper.service';
|
||||
|
||||
+1
-1
@@ -11,7 +11,7 @@ import { LoginTokenService } from 'src/engine/core-modules/auth/token/services/l
|
||||
import { type AuthContextUser } from 'src/engine/core-modules/auth/types/auth-context.type';
|
||||
import { WorkspaceDomainsService } from 'src/engine/core-modules/domain/workspace-domains/services/workspace-domains.service';
|
||||
import { FeatureFlagService } from 'src/engine/core-modules/feature-flag/services/feature-flag.service';
|
||||
import { FileStorageService } from 'src/engine/core-modules/file-storage/file-storage.service';
|
||||
import { FileStorageService } from 'src/engine/core-modules/file-storage/services/file-storage.service';
|
||||
import { FileCorePictureService } from 'src/engine/core-modules/file/file-core-picture/services/file-core-picture.service';
|
||||
import { FileUrlService } from 'src/engine/core-modules/file/file-url/file-url.service';
|
||||
import { OnboardingService } from 'src/engine/core-modules/onboarding/onboarding.service';
|
||||
|
||||
Reference in New Issue
Block a user