Resolve application registration logo and gallery image urls at query time (#22827)

## Context

Application registration logo and gallery image urls were baked into the
stored manifest and display columns at write time, with each source flow
doing it differently: npm catalog sync baked CDN urls, local dev sync
baked `public-assets` urls, and tarball uploads left raw manifest paths
that never displayed in the UI. The entity also carried a `logoUrl`
getter computed field.

This moves url generation to query time, the same way the workspace logo
works.

## What changed

**Read side**
- `ApplicationRegistrationAssetUrlService` builds display urls when
queried: stored files are served by fileId, absolute urls pass through
untouched, and not-yet-rehosted npm assets fall back to the registry CDN
from `sourcePackage@latestAvailableVersion`.
- The `logoUrl` getter on `ApplicationRegistrationEntity` is replaced by
`logoUrl` and `galleryImages` `@ResolveField`s on the metadata resolver,
the admin panel resolver, and a new resolver for
`ApplicationRegistrationSummary` (used by
`Application.applicationRegistration`).
- The marketplace detail/card DTOs and the public OAuth authorize DTO
(`findApplicationRegistrationByClientId`) go through the same url
builder.
- New public route `GET /file/application-registration/:id` streams
registration server files (these are instance-global marketplace assets,
also shown on the public OAuth authorize page).
`ServerFileStorageService.readServerFileById` now returns the mime type
alongside the stream.

**Write side**
- New `logoFileId` column on `applicationRegistration` (2.21 fast
instance command, constraint names match TypeORM naming), complementing
the fileIds already stored in the `galleryImages` jsonb.
- `ApplicationRegistrationAssetService` copies the manifest logo and
gallery images into instance-global server file storage, so all three
sources behave the same:
- **TARBALL**: from the uploaded package (previously only gallery images
were stored, never the logo).
- **LOCAL**: dev sync reads the already-uploaded public assets from
workspace storage (the CLI uploads files before syncing).
- **NPM**: catalog sync downloads the assets from the registry CDN.
Downloads are skipped when the package version is unchanged and the
files are already stored; failed or pending downloads fall back to CDN
urls at query time.
- Write-time url rewriting is removed
(`ManifestAssetUrlResolverService`, `resolveManifestAssetUrls`);
manifests now keep raw asset paths. Existing rows with baked absolute
urls keep working through the absolute-url passthrough, so no backfill
is needed.
- `updateFromManifest` and `upsertFromCatalog` preserve stored gallery
fileIds for unchanged paths, so installs and the hourly catalog sync no
longer clobber them.

## How it was verified

Against a local Postgres/Redis with the server running:
- Fresh database init runs the new instance command; column and FK/UQ
constraint names match TypeORM's generated names, and the CI
pending-migration check produces no diff.
- `findManyApplicationRegistrations { logoUrl galleryImages }` returns
fileId-served urls for a TARBALL registration (absolute urls passed
through), and null/[] for a LOCAL registration without assets.
- Ran `marketplace:catalog-sync` against the real npm registry: 14
packages synced, logos and gallery images rehosted from unpkg with
fileIds set; a second run re-downloaded nothing (version-unchanged
skip); `findMarketplaceAppDetail` for `twenty-linear` returns
fileId-served urls for the logo and all four gallery images.
- `GET /file/application-registration/:id` serves stored files with the
right content type (png and svg verified), 404s on unknown ids, and the
token-guarded generic `/file/:folder/:id` route still returns 403
without a token.
- Unit tests for the url builder and the assets-stored check; server
unit test suites for the application module pass; typecheck and lint
clean.
This commit is contained in:
martmull
2026-07-13 17:09:34 +02:00
committed by GitHub
parent 2b0b62235e
commit 94192a2164
40 changed files with 1174 additions and 449 deletions
@@ -1,11 +1,16 @@
import { Context, Parent, ResolveField } from '@nestjs/graphql';
import { AdminResolver } from 'src/engine/api/graphql/graphql-config/decorators/admin-resolver.decorator';
import { ApplicationRegistrationAssetUrlService } from 'src/engine/core-modules/application/application-registration/application-registration-asset-url.service';
import { ApplicationRegistrationEntity } from 'src/engine/core-modules/application/application-registration/application-registration.entity';
import { type IDataloaders } from 'src/engine/dataloaders/dataloader.interface';
@AdminResolver(() => ApplicationRegistrationEntity)
export class AdminPanelApplicationRegistrationResolver {
constructor(
private readonly applicationRegistrationAssetUrlService: ApplicationRegistrationAssetUrlService,
) {}
@ResolveField(() => Boolean)
async isConfigured(
@Parent() registration: ApplicationRegistrationEntity,
@@ -15,4 +20,22 @@ export class AdminPanelApplicationRegistrationResolver {
applicationRegistrationId: registration.id,
});
}
@ResolveField(() => String, { nullable: true })
logoUrl(
@Parent() registration: ApplicationRegistrationEntity,
): string | null {
return this.applicationRegistrationAssetUrlService.buildLogoUrl(
registration,
);
}
@ResolveField(() => [String])
galleryImagesUrls(
@Parent() registration: ApplicationRegistrationEntity,
): string[] {
return this.applicationRegistrationAssetUrlService.buildGalleryImageUrls(
registration,
);
}
}
@@ -7,10 +7,10 @@ import { type ApplicationInput } from 'src/engine/core-modules/application/appli
import { type DevelopmentApplicationDTO } from 'src/engine/core-modules/application/application-development/dtos/development-application.dto';
import { type WorkspaceMigrationDTO } from 'src/engine/core-modules/application/application-development/dtos/workspace-migration.dto';
import { ApplicationSyncService } from 'src/engine/core-modules/application/application-manifest/application-sync.service';
import { resolveManifestAssetUrls } from 'src/engine/core-modules/application/application-marketplace/utils/resolve-manifest-asset-urls.util';
import { ApplicationVersionValidationService } from 'src/engine/core-modules/application/application-package/application-version-validation.service';
import { VERSION_REASON_TO_APPLICATION_EXCEPTION_CODE } from 'src/engine/core-modules/application/application-package/constants/version-reason-to-exception-code.constant';
import { ApplicationRegistrationVariableService } from 'src/engine/core-modules/application/application-registration-variable/application-registration-variable.service';
import { ApplicationRegistrationAssetService } from 'src/engine/core-modules/application/application-registration/application-registration-asset.service';
import { ApplicationRegistrationService } from 'src/engine/core-modules/application/application-registration/application-registration.service';
import { ApplicationRegistrationSourceType } from 'src/engine/core-modules/application/application-registration/enums/application-registration-source-type.enum';
import {
@@ -24,7 +24,7 @@ import { validateFilePath } from 'src/engine/core-modules/file-storage/utils/val
import { type FileDTO } from 'src/engine/core-modules/file/dtos/file.dto';
import { SdkClientGenerationService } from 'src/engine/core-modules/sdk-client/sdk-client-generation.service';
import { ThrottlerService } from 'src/engine/core-modules/throttler/throttler.service';
import { TwentyConfigService } from 'src/engine/core-modules/twenty-config/twenty-config.service';
import { streamToBuffer } from 'src/utils/stream-to-buffer';
const APP_DEV_RATE_LIMIT_MAX = 30;
const APP_DEV_RATE_LIMIT_WINDOW_MS = 30_000;
@@ -46,10 +46,10 @@ export class ApplicationDevelopmentService {
private readonly applicationSyncService: ApplicationSyncService,
private readonly applicationRegistrationService: ApplicationRegistrationService,
private readonly applicationRegistrationVariableService: ApplicationRegistrationVariableService,
private readonly applicationRegistrationAssetService: ApplicationRegistrationAssetService,
private readonly applicationVersionValidationService: ApplicationVersionValidationService,
private readonly fileStorageService: FileStorageService,
private readonly sdkClientGenerationService: SdkClientGenerationService,
private readonly twentyConfigService: TwentyConfigService,
private readonly throttlerService: ThrottlerService,
private readonly cacheLockService: CacheLockService,
) {}
@@ -253,7 +253,6 @@ export class ApplicationDevelopmentService {
applicationRegistrationId,
manifest,
workspaceId,
application.id,
);
return {
@@ -297,7 +296,6 @@ export class ApplicationDevelopmentService {
applicationRegistrationId: string,
manifest: ApplicationInput['manifest'],
workspaceId: string,
applicationId: string,
): Promise<void> {
const registration =
await this.applicationRegistrationService.findOneByIdGlobal(
@@ -316,20 +314,27 @@ export class ApplicationDevelopmentService {
return;
}
const serverUrl = this.twentyConfigService.get('SERVER_URL');
const manifestWithResolvedUrls = resolveManifestAssetUrls(
manifest,
(filePath) =>
`${serverUrl}/public-assets/${workspaceId}/${applicationId}/${filePath}`,
);
await this.applicationRegistrationService.updateFromManifest({
applicationRegistrationId,
manifest: manifestWithResolvedUrls,
manifest,
sourceType: ApplicationRegistrationSourceType.LOCAL,
});
// Public assets are uploaded to workspace storage before the sync, so the
// logo and gallery images can be copied into the registration's
// instance-global server files here.
await this.applicationRegistrationAssetService.storeRegistrationAssets({
applicationRegistrationId,
manifestApplication: manifest.application,
readAsset: (path) =>
this.readPublicAssetFromWorkspaceStorage({
workspaceId,
applicationUniversalIdentifier:
manifest.application.universalIdentifier,
path,
}),
});
if (manifest.application.serverVariables) {
await this.applicationRegistrationVariableService.syncVariableSchemas(
applicationRegistrationId,
@@ -337,4 +342,27 @@ export class ApplicationDevelopmentService {
);
}
}
private async readPublicAssetFromWorkspaceStorage({
workspaceId,
applicationUniversalIdentifier,
path,
}: {
workspaceId: string;
applicationUniversalIdentifier: string;
path: string;
}): Promise<Buffer | null> {
try {
const stream = await this.fileStorageService.readFile({
workspaceId,
applicationUniversalIdentifier,
fileFolder: FileFolder.PublicAsset,
resourcePath: path,
});
return await streamToBuffer(stream);
} catch {
return null;
}
}
}
@@ -18,7 +18,6 @@ import { isImageFilePath } from 'src/engine/core-modules/application/application
import { ApplicationRegistrationEntity } from 'src/engine/core-modules/application/application-registration/application-registration.entity';
import { ApplicationRegistrationService } from 'src/engine/core-modules/application/application-registration/application-registration.service';
import { ApplicationRegistrationSourceType } from 'src/engine/core-modules/application/application-registration/enums/application-registration-source-type.enum';
import { ManifestAssetUrlResolverService } from 'src/engine/core-modules/application/application-registration/manifest-asset-url-resolver.service';
import { ApplicationEntity } from 'src/engine/core-modules/application/application.entity';
import { ApplicationService } from 'src/engine/core-modules/application/application.service';
import { ApplicationPackageFetcherService } from 'src/engine/core-modules/application/application-package/application-package-fetcher.service';
@@ -57,7 +56,6 @@ export class ApplicationInstallService {
@InjectMessageQueue(MessageQueue.logicFunctionQueue)
private readonly messageQueueService: MessageQueueService,
private readonly workspaceCacheService: WorkspaceCacheService,
private readonly manifestAssetUrlResolverService: ManifestAssetUrlResolverService,
) {}
async installApplication(params: {
@@ -306,12 +304,7 @@ export class ApplicationInstallService {
await this.applicationRegistrationService.updateFromManifest({
applicationRegistrationId: appRegistration.id,
manifest: this.manifestAssetUrlResolverService.resolveFromRegistration({
sourceType: appRegistration.sourceType,
sourcePackage: appRegistration.sourcePackage,
manifest,
version: installedVersion,
}),
manifest,
latestAvailableVersion: installedVersion,
preventVersionDowngrade: true,
});
@@ -1,9 +1,12 @@
import { Injectable, Logger } from '@nestjs/common';
import { isDefined } from 'twenty-shared/utils';
import { MarketplaceService } from 'src/engine/core-modules/application/application-marketplace/marketplace.service';
import { ApplicationRegistrationAssetService } from 'src/engine/core-modules/application/application-registration/application-registration-asset.service';
import { ApplicationRegistrationService } from 'src/engine/core-modules/application/application-registration/application-registration.service';
import { ApplicationRegistrationSourceType } from 'src/engine/core-modules/application/application-registration/enums/application-registration-source-type.enum';
import { MarketplaceService } from 'src/engine/core-modules/application/application-marketplace/marketplace.service';
import { ManifestAssetUrlResolverService } from 'src/engine/core-modules/application/application-registration/manifest-asset-url-resolver.service';
import { areRegistrationAssetsStored } from 'src/engine/core-modules/application/application-registration/utils/are-registration-assets-stored.util';
@Injectable()
export class MarketplaceCatalogSyncService {
@@ -11,8 +14,8 @@ export class MarketplaceCatalogSyncService {
constructor(
private readonly applicationRegistrationService: ApplicationRegistrationService,
private readonly applicationRegistrationAssetService: ApplicationRegistrationAssetService,
private readonly marketplaceService: MarketplaceService,
private readonly manifestAssetUrlResolverService: ManifestAssetUrlResolverService,
) {}
async syncCatalog(): Promise<void> {
@@ -43,12 +46,11 @@ export class MarketplaceCatalogSyncService {
const universalIdentifier =
fetchedManifest.application.universalIdentifier;
const manifestWithResolvedUrls =
this.manifestAssetUrlResolverService.resolveFromRegistrySource({
manifest: fetchedManifest,
packageName: pkg.name,
version: pkg.version,
});
const previousVersion = (
await this.applicationRegistrationService.findOneByUniversalIdentifier(
universalIdentifier,
)
)?.latestAvailableVersion;
await this.applicationRegistrationService.upsertFromCatalog({
universalIdentifier,
@@ -56,8 +58,42 @@ export class MarketplaceCatalogSyncService {
sourceType: ApplicationRegistrationSourceType.NPM,
sourcePackage: pkg.name,
latestAvailableVersion: pkg.version ?? null,
manifest: manifestWithResolvedUrls,
manifest: fetchedManifest,
});
const registration =
await this.applicationRegistrationService.findOneByUniversalIdentifier(
universalIdentifier,
);
if (!isDefined(registration)) {
continue;
}
// Rehost the logo and gallery images from the registry CDN so display
// urls are served from fileIds like every other source. Skipped when
// the version is unchanged and the files are already stored; the
// query-time url builder falls back to CDN urls until they are.
if (
previousVersion !== pkg.version ||
!areRegistrationAssetsStored(
registration,
fetchedManifest.application,
)
) {
await this.applicationRegistrationAssetService.storeRegistrationAssets(
{
applicationRegistrationId: registration.id,
manifestApplication: fetchedManifest.application,
readAsset: (path) =>
this.marketplaceService.fetchAssetFromRegistryCdn(
pkg.name,
pkg.version,
path,
),
},
);
}
} catch (error) {
this.logger.error(
`Failed to sync registry app "${pkg.name}": ${error instanceof Error ? error.message : String(error)}`,
@@ -8,18 +8,19 @@ import { MARKETPLACE_CATALOG_CACHE_ENTITY_ID } from 'src/engine/core-modules/app
import { MarketplaceAppDTO } from 'src/engine/core-modules/application/application-marketplace/dtos/marketplace-app.dto';
import { MarketplaceAppDetailDTO } from 'src/engine/core-modules/application/application-marketplace/dtos/marketplace-app-detail.dto';
import { MarketplaceAppRoleDTO } from 'src/engine/core-modules/application/application-marketplace/dtos/marketplace-app-role.dto';
import { ApplicationRegistrationAssetUrlService } from 'src/engine/core-modules/application/application-registration/application-registration-asset-url.service';
import { type ApplicationRegistrationEntity } from 'src/engine/core-modules/application/application-registration/application-registration.entity';
import {
ApplicationRegistrationException,
ApplicationRegistrationExceptionCode,
} from 'src/engine/core-modules/application/application-registration/application-registration.exception';
import { toGalleryImagePaths } from 'src/engine/core-modules/application/application-registration/utils/to-gallery-image-paths.util';
import { ApplicationRegistrationService } from 'src/engine/core-modules/application/application-registration/application-registration.service';
@Injectable()
export class MarketplaceQueryService {
constructor(
private readonly applicationRegistrationService: ApplicationRegistrationService,
private readonly applicationRegistrationAssetUrlService: ApplicationRegistrationAssetUrlService,
private readonly coreEntityCacheService: CoreEntityCacheService,
) {}
@@ -82,13 +83,10 @@ export class MarketplaceQueryService {
private toMarketplaceAppDetailDTO(
registration: ApplicationRegistrationEntity,
): MarketplaceAppDetailDTO {
// TODO: simplify in a follow-up PR to read galleryImages only, once the
// deprecated screenshots column and manifest fallback are backfilled away.
const galleryImagePaths = isNonEmptyArray(registration.galleryImages)
? registration.galleryImages.map((galleryImage) => galleryImage.path)
: isNonEmptyArray(registration.screenshots)
? registration.screenshots
: toGalleryImagePaths(registration.manifest?.application);
const galleryImageUrls =
this.applicationRegistrationAssetUrlService.buildGalleryImageUrls(
registration,
);
return {
id: registration.id,
@@ -111,7 +109,10 @@ export class MarketplaceQueryService {
registration.category ??
registration.manifest?.application?.category ??
undefined,
logo: registration.logoUrl ?? undefined,
logo:
this.applicationRegistrationAssetUrlService.buildLogoUrl(
registration,
) ?? undefined,
websiteUrl:
registration.websiteUrl ??
registration.manifest?.application?.websiteUrl ??
@@ -132,8 +133,8 @@ export class MarketplaceQueryService {
registration.issueReportUrl ??
registration.manifest?.application?.issueReportUrl ??
undefined,
screenshots: galleryImagePaths,
galleryImages: galleryImagePaths,
screenshots: galleryImageUrls,
galleryImages: galleryImageUrls,
defaultRoleUniversalIdentifier:
registration.manifest?.application?.defaultRoleUniversalIdentifier,
roles: registration.manifest?.roles?.map((role) =>
@@ -7,6 +7,8 @@ import { z } from 'zod';
import { buildRegistryCdnUrl } from 'src/engine/core-modules/application/application-marketplace/utils/build-registry-cdn-url.util';
import { TwentyConfigService } from 'src/engine/core-modules/twenty-config/twenty-config.service';
const MAX_REGISTRY_ASSET_SIZE_BYTES = 100 * 1024 * 1024; // 100Mb
export type RegistryPackageInfo = {
name: string;
version: string;
@@ -73,6 +75,37 @@ export class MarketplaceService {
}
}
async fetchAssetFromRegistryCdn(
packageName: string,
version: string,
filePath: string,
): Promise<Buffer | null> {
const cdnBaseUrl = this.twentyConfigService.get('APP_REGISTRY_CDN_URL');
const url = buildRegistryCdnUrl({
cdnBaseUrl,
packageName,
version,
filePath,
});
try {
const { data } = await axios.get<ArrayBuffer>(url, {
headers: { 'User-Agent': 'Twenty-Marketplace' },
timeout: 10_000,
responseType: 'arraybuffer',
maxContentLength: MAX_REGISTRY_ASSET_SIZE_BYTES,
});
return Buffer.from(data);
} catch {
this.logger.debug(
`Could not fetch asset "${filePath}" from CDN for ${packageName}@${version}`,
);
return null;
}
}
async fetchAppsFromRegistry(): Promise<RegistryPackageInfo[]> {
const registryUrl = this.twentyConfigService.get('APP_REGISTRY_URL');
@@ -1,190 +0,0 @@
import { type Manifest } from 'twenty-shared/application';
import { resolveManifestAssetUrls } from 'src/engine/core-modules/application/application-marketplace/utils/resolve-manifest-asset-urls.util';
const buildMinimalManifest = (
overrides: Partial<Manifest['application']> = {},
): Manifest => ({
application: {
universalIdentifier: 'app-1',
defaultRoleUniversalIdentifier: 'role-1',
displayName: 'Test App',
description: 'A test app',
packageJsonChecksum: null,
yarnLockChecksum: null,
...overrides,
},
objects: [],
fields: [],
logicFunctions: [],
frontComponents: [],
permissionFlags: [],
roles: [],
skills: [],
agents: [],
publicAssets: [],
views: [],
viewFields: [],
navigationMenuItems: [],
pageLayouts: [],
pageLayoutTabs: [],
commandMenuItems: [],
});
describe('resolveManifestAssetUrls', () => {
const urlBuilder = (filePath: string) =>
`https://cdn.example.com/pkg/${filePath}`;
it('should resolve a relative logoUrl', () => {
const manifest = buildMinimalManifest({ logoUrl: 'logo.png' });
const result = resolveManifestAssetUrls(manifest, urlBuilder);
expect(result.application.logoUrl).toBe(
'https://cdn.example.com/pkg/logo.png',
);
});
it('should not modify an absolute logoUrl', () => {
const manifest = buildMinimalManifest({
logoUrl: 'https://example.com/logo.png',
});
const result = resolveManifestAssetUrls(manifest, urlBuilder);
expect(result.application.logoUrl).toBe('https://example.com/logo.png');
});
it('should not modify an http logoUrl', () => {
const manifest = buildMinimalManifest({
logoUrl: 'http://example.com/logo.png',
});
const result = resolveManifestAssetUrls(manifest, urlBuilder);
expect(result.application.logoUrl).toBe('http://example.com/logo.png');
});
it('should leave logoUrl undefined when not set', () => {
const manifest = buildMinimalManifest();
const result = resolveManifestAssetUrls(manifest, urlBuilder);
expect(result.application.logoUrl).toBeUndefined();
});
it('should resolve relative screenshot paths', () => {
const manifest = buildMinimalManifest({
screenshots: ['screen1.png', 'screen2.png'],
});
const result = resolveManifestAssetUrls(manifest, urlBuilder);
expect(result.application.screenshots).toEqual([
'https://cdn.example.com/pkg/screen1.png',
'https://cdn.example.com/pkg/screen2.png',
]);
});
it('should not modify absolute screenshot URLs', () => {
const manifest = buildMinimalManifest({
screenshots: [
'https://example.com/screen1.png',
'https://example.com/screen2.png',
],
});
const result = resolveManifestAssetUrls(manifest, urlBuilder);
expect(result.application.screenshots).toEqual([
'https://example.com/screen1.png',
'https://example.com/screen2.png',
]);
});
it('should handle a mix of relative and absolute screenshot URLs', () => {
const manifest = buildMinimalManifest({
screenshots: ['relative.png', 'https://example.com/absolute.png'],
});
const result = resolveManifestAssetUrls(manifest, urlBuilder);
expect(result.application.screenshots).toEqual([
'https://cdn.example.com/pkg/relative.png',
'https://example.com/absolute.png',
]);
});
it('should handle empty screenshots array', () => {
const manifest = buildMinimalManifest({ screenshots: [] });
const result = resolveManifestAssetUrls(manifest, urlBuilder);
expect(result.application.screenshots).toEqual([]);
});
it('should handle undefined screenshots', () => {
const manifest = buildMinimalManifest();
const result = resolveManifestAssetUrls(manifest, urlBuilder);
expect(result.application.screenshots).toEqual([]);
});
it('should resolve a relative logo', () => {
const manifest = buildMinimalManifest({ logo: 'logo.png' });
const result = resolveManifestAssetUrls(manifest, urlBuilder);
expect(result.application.logo).toBe(
'https://cdn.example.com/pkg/logo.png',
);
});
it('should not modify an absolute logo', () => {
const manifest = buildMinimalManifest({
logo: 'https://example.com/logo.png',
});
const result = resolveManifestAssetUrls(manifest, urlBuilder);
expect(result.application.logo).toBe('https://example.com/logo.png');
});
it('should resolve galleryImages paths and leave absolute ones untouched', () => {
const manifest = buildMinimalManifest({
galleryImages: ['a.png', 'https://example.com/b.png'],
});
const result = resolveManifestAssetUrls(manifest, urlBuilder);
expect(result.application.galleryImages).toEqual([
'https://cdn.example.com/pkg/a.png',
'https://example.com/b.png',
]);
});
it('should handle undefined galleryImages', () => {
const manifest = buildMinimalManifest();
const result = resolveManifestAssetUrls(manifest, urlBuilder);
expect(result.application.galleryImages).toEqual([]);
});
it('should preserve all other manifest properties', () => {
const manifest = buildMinimalManifest({
logoUrl: 'logo.png',
author: 'Test Author',
websiteUrl: 'https://test.com',
});
manifest.objects = [{ universalIdentifier: 'obj-1' } as never];
const result = resolveManifestAssetUrls(manifest, urlBuilder);
expect(result.application.author).toBe('Test Author');
expect(result.application.websiteUrl).toBe('https://test.com');
expect(result.application.displayName).toBe('Test App');
expect(result.objects).toEqual([{ universalIdentifier: 'obj-1' }]);
});
});
@@ -1,27 +0,0 @@
import { type Manifest } from 'twenty-shared/application';
const isAbsoluteUrl = (url: string): boolean =>
url.startsWith('http://') || url.startsWith('https://');
export const resolveManifestAssetUrls = (
manifest: Manifest,
urlBuilder: (filePath: string) => string,
): Manifest => {
const resolveUrl = (url: string): string =>
isAbsoluteUrl(url) ? url : urlBuilder(url);
return {
...manifest,
application: {
...manifest.application,
logoUrl: manifest.application.logoUrl
? resolveUrl(manifest.application.logoUrl)
: undefined,
logo: manifest.application.logo
? resolveUrl(manifest.application.logo)
: undefined,
screenshots: (manifest.application.screenshots ?? []).map(resolveUrl),
galleryImages: (manifest.application.galleryImages ?? []).map(resolveUrl),
},
};
};
@@ -0,0 +1,149 @@
import { Test, type TestingModule } from '@nestjs/testing';
import { type Manifest } from 'twenty-shared/application';
import { ApplicationRegistrationAssetUrlService } from 'src/engine/core-modules/application/application-registration/application-registration-asset-url.service';
import { ApplicationRegistrationSourceType } from 'src/engine/core-modules/application/application-registration/enums/application-registration-source-type.enum';
import { TwentyConfigService } from 'src/engine/core-modules/twenty-config/twenty-config.service';
const CONFIG_VALUES: Record<string, string> = {
SERVER_URL: 'https://api.twenty.com',
APP_REGISTRY_CDN_URL: 'https://cdn.registry.com',
};
const baseRegistration = {
sourceType: ApplicationRegistrationSourceType.TARBALL,
sourcePackage: null,
latestAvailableVersion: '1.0.0',
logo: null,
logoFileId: null,
galleryImages: null,
screenshots: [],
manifest: null,
};
describe('ApplicationRegistrationAssetUrlService', () => {
let service: ApplicationRegistrationAssetUrlService;
beforeEach(async () => {
const module: TestingModule = await Test.createTestingModule({
providers: [
ApplicationRegistrationAssetUrlService,
{
provide: TwentyConfigService,
useValue: {
get: jest.fn((key: string) => CONFIG_VALUES[key]),
},
},
],
}).compile();
service = module.get(ApplicationRegistrationAssetUrlService);
});
describe('buildLogoUrl', () => {
it('should build a server file url when a logoFileId is set', () => {
const logoUrl = service.buildLogoUrl({
...baseRegistration,
logo: 'public/logo.png',
logoFileId: 'file-id',
});
expect(logoUrl).toBe(
'https://api.twenty.com/file/server/application-registration/file-id',
);
});
it('should pass through absolute logo urls', () => {
const logoUrl = service.buildLogoUrl({
...baseRegistration,
logo: 'https://example.com/logo.png',
});
expect(logoUrl).toBe('https://example.com/logo.png');
});
it('should build a registry cdn url for npm registrations', () => {
const logoUrl = service.buildLogoUrl({
...baseRegistration,
sourceType: ApplicationRegistrationSourceType.NPM,
sourcePackage: '@twenty/my-app',
latestAvailableVersion: '2.3.4',
logo: 'public/logo.png',
});
expect(logoUrl).toBe(
'https://cdn.registry.com/@twenty/my-app@2.3.4/public/logo.png',
);
});
it('should return null for a relative path without a stored file on non-npm registrations', () => {
const logoUrl = service.buildLogoUrl({
...baseRegistration,
logo: 'public/logo.png',
});
expect(logoUrl).toBeNull();
});
it('should return null when there is no logo at all', () => {
const logoUrl = service.buildLogoUrl(baseRegistration);
expect(logoUrl).toBeNull();
});
});
describe('buildGalleryImageUrls', () => {
it('should resolve stored gallery images by fileId and keep absolute urls', () => {
const urls = service.buildGalleryImageUrls({
...baseRegistration,
galleryImages: [
{ path: 'public/one.png', fileId: 'file-1' },
{ path: 'https://example.com/two.png', fileId: null },
{ path: 'public/missing.png', fileId: null },
],
});
expect(urls).toEqual([
'https://api.twenty.com/file/server/application-registration/file-1',
'https://example.com/two.png',
]);
});
it('should resolve npm gallery images from the registry cdn', () => {
const urls = service.buildGalleryImageUrls({
...baseRegistration,
sourceType: ApplicationRegistrationSourceType.NPM,
sourcePackage: '@twenty/my-app',
latestAvailableVersion: '2.3.4',
galleryImages: [{ path: 'public/one.png', fileId: null }],
});
expect(urls).toEqual([
'https://cdn.registry.com/@twenty/my-app@2.3.4/public/one.png',
]);
});
it('should fall back to the deprecated screenshots column', () => {
const urls = service.buildGalleryImageUrls({
...baseRegistration,
screenshots: ['https://example.com/screenshot.png'],
});
expect(urls).toEqual(['https://example.com/screenshot.png']);
});
it('should fall back to manifest gallery images when nothing is backfilled', () => {
const urls = service.buildGalleryImageUrls({
...baseRegistration,
manifest: {
application: {
galleryImages: ['https://example.com/from-manifest.png'],
},
} as unknown as Manifest,
});
expect(urls).toEqual(['https://example.com/from-manifest.png']);
});
});
});
@@ -0,0 +1,109 @@
import { Injectable } from '@nestjs/common';
import { ServerFileFolder } from 'twenty-shared/types';
import { isDefined, isNonEmptyArray } from 'twenty-shared/utils';
import { buildRegistryCdnUrl } from 'src/engine/core-modules/application/application-marketplace/utils/build-registry-cdn-url.util';
import { SERVER_FILE_STORAGE_PREFIX } from 'src/engine/core-modules/file-storage/constants/server-file-storage-prefix.constant';
import { type ApplicationRegistrationEntity } from 'src/engine/core-modules/application/application-registration/application-registration.entity';
import { ApplicationRegistrationSourceType } from 'src/engine/core-modules/application/application-registration/enums/application-registration-source-type.enum';
import { toGalleryImagePaths } from 'src/engine/core-modules/application/application-registration/utils/to-gallery-image-paths.util';
import { isAbsoluteUrl } from 'src/engine/core-modules/application/application-registration/utils/is-absolute-url.util';
import { TwentyConfigService } from 'src/engine/core-modules/twenty-config/twenty-config.service';
type AssetSourceFields = Pick<
ApplicationRegistrationEntity,
'sourceType' | 'sourcePackage' | 'latestAvailableVersion'
>;
// Builds display URLs for application registration assets at query time.
// Assets stored as instance-global server files (TARBALL, LOCAL) are served by
// fileId; NPM assets are served straight from the registry CDN; absolute URLs
// pass through untouched.
@Injectable()
export class ApplicationRegistrationAssetUrlService {
constructor(private readonly twentyConfigService: TwentyConfigService) {}
buildLogoUrl(
registration: AssetSourceFields &
Pick<ApplicationRegistrationEntity, 'logo' | 'logoFileId'>,
): string | null {
return this.resolveAssetUrl({
fileId: registration.logoFileId,
path: registration.logo,
registration,
});
}
buildGalleryImageUrls(
registration: AssetSourceFields &
Pick<
ApplicationRegistrationEntity,
'galleryImages' | 'screenshots' | 'manifest'
>,
): string[] {
// TODO: read galleryImages only, once the deprecated screenshots column
// and manifest fallback are backfilled away.
const galleryImages = isNonEmptyArray(registration.galleryImages)
? registration.galleryImages
: this.toGalleryImageFallbackEntries(registration);
return galleryImages
.map(({ path, fileId }) =>
this.resolveAssetUrl({ fileId, path, registration }),
)
.filter(isDefined);
}
private toGalleryImageFallbackEntries(
registration: Pick<
ApplicationRegistrationEntity,
'screenshots' | 'manifest'
>,
): { path: string; fileId: null }[] {
const paths = isNonEmptyArray(registration.screenshots)
? registration.screenshots
: toGalleryImagePaths(registration.manifest?.application);
return paths.map((path) => ({ path, fileId: null }));
}
private resolveAssetUrl({
fileId,
path,
registration,
}: {
fileId: string | null;
path: string | null;
registration: AssetSourceFields;
}): string | null {
if (isDefined(fileId)) {
const serverUrl = this.twentyConfigService.get('SERVER_URL');
return `${serverUrl}/file/${SERVER_FILE_STORAGE_PREFIX}/${ServerFileFolder.ApplicationRegistration}/${fileId}`;
}
if (!isDefined(path) || path.length === 0) {
return null;
}
if (isAbsoluteUrl(path)) {
return path;
}
if (
registration.sourceType === ApplicationRegistrationSourceType.NPM &&
isDefined(registration.sourcePackage) &&
isDefined(registration.latestAvailableVersion)
) {
return buildRegistryCdnUrl({
cdnBaseUrl: this.twentyConfigService.get('APP_REGISTRY_CDN_URL'),
packageName: registration.sourcePackage,
version: registration.latestAvailableVersion,
filePath: path,
});
}
return null;
}
}
@@ -0,0 +1,182 @@
import { Injectable, Logger } from '@nestjs/common';
import { InjectRepository } from '@nestjs/typeorm';
import { ServerFileFolder } from 'twenty-shared/types';
import { isDefined } from 'twenty-shared/utils';
import { Repository } from 'typeorm';
import { type QueryDeepPartialEntity } from 'typeorm/query-builder/QueryPartialEntity';
import { ApplicationRegistrationEntity } from 'src/engine/core-modules/application/application-registration/application-registration.entity';
import { type ApplicationRegistrationGalleryImage } from 'src/engine/core-modules/application/application-registration/types/application-registration-gallery-image.type';
import { isStorableAssetPath } from 'src/engine/core-modules/application/application-registration/utils/is-storable-asset-path.util';
import { toGalleryImagePaths } from 'src/engine/core-modules/application/application-registration/utils/to-gallery-image-paths.util';
import { ServerFileStorageService } from 'src/engine/core-modules/file-storage/services/server-file-storage.service';
import { prepareFileForStorageOrThrow } from 'src/engine/core-modules/file-storage/utils/prepare-file-for-storage-or-throw.util';
import type { ApplicationManifest } from 'twenty-shared/application';
export type ReadRegistrationAsset = (path: string) => Promise<Buffer | null>;
// Copies the manifest logo and gallery images into instance-global server file
// storage so their display URLs can be built at query time from fileIds,
// regardless of how the registration was created (LOCAL, TARBALL, NPM).
@Injectable()
export class ApplicationRegistrationAssetService {
private readonly logger = new Logger(
ApplicationRegistrationAssetService.name,
);
constructor(
@InjectRepository(ApplicationRegistrationEntity)
private readonly applicationRegistrationRepository: Repository<ApplicationRegistrationEntity>,
private readonly serverFileStorageService: ServerFileStorageService,
) {}
async storeRegistrationAssets({
applicationRegistrationId,
manifestApplication,
readAsset,
}: {
applicationRegistrationId: string;
manifestApplication: ApplicationManifest | undefined;
readAsset: ReadRegistrationAsset;
}): Promise<void> {
const existing = await this.applicationRegistrationRepository.findOneOrFail(
{
select: ['id', 'logo', 'logoFileId', 'galleryImages'],
where: { id: applicationRegistrationId },
},
);
const logoFileId = await this.storeLogoFile({
applicationRegistrationId,
manifestApplication,
readAsset,
});
const galleryImages = await this.storeGalleryImageFiles({
applicationRegistrationId,
manifestApplication,
readAsset,
});
// A transient read/download failure must not clobber a working asset:
// keep the previously stored fileId when the path did not change.
const logoPath = manifestApplication?.logo ?? manifestApplication?.logoUrl;
const existingFileIdByPath = new Map(
(existing.galleryImages ?? []).map(({ path, fileId }) => [path, fileId]),
);
await this.applicationRegistrationRepository.update(
applicationRegistrationId,
{
logoFileId:
logoFileId ??
(isDefined(logoPath) &&
isStorableAssetPath(logoPath) &&
existing.logo === logoPath
? existing.logoFileId
: null),
galleryImages: galleryImages.map((galleryImage) => ({
...galleryImage,
fileId:
galleryImage.fileId ??
existingFileIdByPath.get(galleryImage.path) ??
null,
})),
} as QueryDeepPartialEntity<ApplicationRegistrationEntity>,
);
}
private async storeLogoFile({
applicationRegistrationId,
manifestApplication,
readAsset,
}: {
applicationRegistrationId: string;
manifestApplication: ApplicationManifest | undefined;
readAsset: ReadRegistrationAsset;
}): Promise<string | null> {
const logoPath = manifestApplication?.logo ?? manifestApplication?.logoUrl;
if (!isDefined(logoPath)) {
return null;
}
return this.storeAssetFile({
applicationRegistrationId,
path: logoPath,
readAsset,
});
}
private async storeGalleryImageFiles({
applicationRegistrationId,
manifestApplication,
readAsset,
}: {
applicationRegistrationId: string;
manifestApplication: ApplicationManifest | undefined;
readAsset: ReadRegistrationAsset;
}): Promise<ApplicationRegistrationGalleryImage[]> {
const galleryImages: ApplicationRegistrationGalleryImage[] = [];
for (const path of toGalleryImagePaths(manifestApplication)) {
const fileId = await this.storeAssetFile({
applicationRegistrationId,
path,
readAsset,
});
// Entries without a fileId (absolute URLs, missing files) are kept so
// the query-time URL resolution can still fall back on the raw path.
galleryImages.push({ path, fileId });
}
return galleryImages;
}
private async storeAssetFile({
applicationRegistrationId,
path,
readAsset,
}: {
applicationRegistrationId: string;
path: string;
readAsset: ReadRegistrationAsset;
}): Promise<string | null> {
if (!isStorableAssetPath(path)) {
return null;
}
try {
const contents = await readAsset(path);
if (!isDefined(contents)) {
return null;
}
const { sourceFile, mimeType } = await prepareFileForStorageOrThrow({
sourceFile: contents,
resourcePath: path,
});
const savedFile = await this.serverFileStorageService.writeServerFile({
fileFolder: ServerFileFolder.ApplicationRegistration,
applicationRegistrationId,
resourcePath: path,
contents: Buffer.isBuffer(sourceFile)
? sourceFile
: Buffer.from(sourceFile),
mimeType,
});
return savedFile.id;
} catch (error) {
this.logger.warn(
`Failed to store asset "${path}" for registration ${applicationRegistrationId}: ${error.message}`,
);
return null;
}
}
}
@@ -0,0 +1,24 @@
import { Parent, ResolveField } from '@nestjs/graphql';
import { MetadataResolver } from 'src/engine/api/graphql/graphql-config/decorators/metadata-resolver.decorator';
import { ApplicationRegistrationAssetUrlService } from 'src/engine/core-modules/application/application-registration/application-registration-asset-url.service';
import { ApplicationRegistrationEntity } from 'src/engine/core-modules/application/application-registration/application-registration.entity';
import { ApplicationRegistrationSummaryDTO } from 'src/engine/core-modules/application/application-registration/dtos/application-registration-summary.dto';
// The summary is resolved from ApplicationRegistrationEntity instances loaded
// through the Application.applicationRegistration relation.
@MetadataResolver(() => ApplicationRegistrationSummaryDTO)
export class ApplicationRegistrationSummaryResolver {
constructor(
private readonly applicationRegistrationAssetUrlService: ApplicationRegistrationAssetUrlService,
) {}
@ResolveField(() => String, { nullable: true })
logoUrl(
@Parent() registration: ApplicationRegistrationEntity,
): string | null {
return this.applicationRegistrationAssetUrlService.buildLogoUrl(
registration,
);
}
}
@@ -150,6 +150,17 @@ export class ApplicationRegistrationEntity {
})
logo: string | null;
@Column({ nullable: true, type: 'uuid' })
@WasIntroducedInUpgrade({
upgradeCommandName:
'2.21.0_AddLogoFileIdToApplicationRegistrationFastInstanceCommand_1783945979243',
})
logoFileId: string | null;
@OneToOne(() => FileEntity, { onDelete: 'SET NULL', nullable: true })
@JoinColumn({ name: 'logoFileId' })
logoFile: Relation<FileEntity> | null;
@Column({ nullable: true, type: 'text' })
@WasIntroducedInUpgrade({
upgradeCommandName:
@@ -213,16 +224,6 @@ export class ApplicationRegistrationEntity {
})
screenshots: string[];
@Field(() => String, { nullable: true })
get logoUrl(): string | null {
return (
this.logo ??
this.manifest?.application?.logo ??
this.manifest?.application?.logoUrl ??
null
);
}
@OneToMany(
() => ApplicationRegistrationVariableEntity,
(variable) => variable.applicationRegistration,
@@ -2,12 +2,14 @@ import { Module } from '@nestjs/common';
import { TypeOrmModule } from '@nestjs/typeorm';
import { CoreEntityCacheModule } from 'src/engine/core-entity-cache/core-entity-cache.module';
import { ApplicationRegistrationAssetUrlService } from 'src/engine/core-modules/application/application-registration/application-registration-asset-url.service';
import { ApplicationRegistrationAssetService } from 'src/engine/core-modules/application/application-registration/application-registration-asset.service';
import { ApplicationRegistrationEntity } from 'src/engine/core-modules/application/application-registration/application-registration.entity';
import { ApplicationRegistrationResolver } from 'src/engine/core-modules/application/application-registration/application-registration.resolver';
import { ApplicationRegistrationService } from 'src/engine/core-modules/application/application-registration/application-registration.service';
import { ApplicationRegistrationSummaryResolver } from 'src/engine/core-modules/application/application-registration/application-registration-summary.resolver';
import { ApplicationRegistrationVariableModule } from 'src/engine/core-modules/application/application-registration-variable/application-registration-variable.module';
import { ApplicationTarballService } from 'src/engine/core-modules/application/application-registration/application-tarball.service';
import { ManifestAssetUrlResolverService } from 'src/engine/core-modules/application/application-registration/manifest-asset-url-resolver.service';
import { ApplicationPackageModule } from 'src/engine/core-modules/application/application-package/application-package.module';
import { ApplicationEntity } from 'src/engine/core-modules/application/application.entity';
import { ApplicationModule } from 'src/engine/core-modules/application/application.module';
@@ -42,13 +44,16 @@ import { WorkspaceCacheStorageModule } from 'src/engine/workspace-cache-storage/
providers: [
ApplicationRegistrationService,
ApplicationRegistrationResolver,
ApplicationRegistrationSummaryResolver,
ApplicationTarballService,
ManifestAssetUrlResolverService,
ApplicationRegistrationAssetService,
ApplicationRegistrationAssetUrlService,
],
exports: [
ApplicationRegistrationService,
ApplicationRegistrationVariableModule,
ManifestAssetUrlResolverService,
ApplicationRegistrationAssetService,
ApplicationRegistrationAssetUrlService,
],
})
export class ApplicationRegistrationModule {}
@@ -23,6 +23,7 @@ import { ApplicationRegistrationVariableService } from 'src/engine/core-modules/
import { CreateApplicationRegistrationVariableInput } from 'src/engine/core-modules/application/application-registration-variable/dtos/create-application-registration-variable.input';
import { UpdateApplicationRegistrationVariableInput } from 'src/engine/core-modules/application/application-registration-variable/dtos/update-application-registration-variable.input';
import { ApplicationRegistrationExceptionFilter } from 'src/engine/core-modules/application/application-registration/application-registration-exception-filter';
import { ApplicationRegistrationAssetUrlService } from 'src/engine/core-modules/application/application-registration/application-registration-asset-url.service';
import { ApplicationRegistrationEntity } from 'src/engine/core-modules/application/application-registration/application-registration.entity';
import { ApplicationRegistrationService } from 'src/engine/core-modules/application/application-registration/application-registration.service';
import { ApplicationTarballService } from 'src/engine/core-modules/application/application-registration/application-tarball.service';
@@ -67,6 +68,7 @@ export class ApplicationRegistrationResolver {
private readonly applicationRegistrationService: ApplicationRegistrationService,
private readonly applicationRegistrationVariableService: ApplicationRegistrationVariableService,
private readonly applicationTarballService: ApplicationTarballService,
private readonly applicationRegistrationAssetUrlService: ApplicationRegistrationAssetUrlService,
private readonly fileUrlService: FileUrlService,
private readonly twentyConfigService: TwentyConfigService,
) {}
@@ -356,4 +358,22 @@ export class ApplicationRegistrationResolver {
applicationRegistrationId: registration.id,
});
}
@ResolveField(() => String, { nullable: true })
logoUrl(
@Parent() registration: ApplicationRegistrationEntity,
): string | null {
return this.applicationRegistrationAssetUrlService.buildLogoUrl(
registration,
);
}
@ResolveField(() => [String])
galleryImagesUrls(
@Parent() registration: ApplicationRegistrationEntity,
): string[] {
return this.applicationRegistrationAssetUrlService.buildGalleryImageUrls(
registration,
);
}
}
@@ -11,7 +11,9 @@ import { v4 } from 'uuid';
import { ALL_OAUTH_SCOPES } from 'src/engine/core-modules/application/application-oauth/constants/oauth-scopes';
import { shouldRefreshApplicationRegistrationOnInstall } from 'src/engine/core-modules/application/application-install/utils/should-refresh-application-registration-on-install.util';
import { ApplicationRegistrationAssetUrlService } from 'src/engine/core-modules/application/application-registration/application-registration-asset-url.service';
import { CacheLockService } from 'src/engine/core-modules/cache-lock/cache-lock.service';
import { ServerFileStorageService } from 'src/engine/core-modules/file-storage/services/server-file-storage.service';
import { ApplicationRegistrationEntity } from 'src/engine/core-modules/application/application-registration/application-registration.entity';
import { TWENTY_CLI_APPLICATION_REGISTRATION } from 'src/engine/workspace-manager/twenty-standard-application/constants/twenty-cli-application-registration.constant';
import {
@@ -59,6 +61,7 @@ const APPLICATION_REGISTRATION_WITHOUT_MANIFEST_SELECT: (keyof ApplicationRegist
'isVetted',
'isPreInstalled',
'logo',
'logoFileId',
'description',
'author',
'category',
@@ -97,6 +100,8 @@ export class ApplicationRegistrationService {
@InjectRepository(WorkspaceEntity)
private readonly workspaceRepository: Repository<WorkspaceEntity>,
private readonly applicationRegistrationVariableService: ApplicationRegistrationVariableService,
private readonly applicationRegistrationAssetUrlService: ApplicationRegistrationAssetUrlService,
private readonly serverFileStorageService: ServerFileStorageService,
private readonly cacheLockService: CacheLockService,
private readonly coreEntityCacheService: CoreEntityCacheService,
) {}
@@ -225,7 +230,17 @@ export class ApplicationRegistrationService {
): Promise<PublicApplicationRegistrationDTO | null> {
const registration = await this.applicationRegistrationRepository.findOne({
where: { oAuthClientId: clientId },
select: ['id', 'name', 'logo', 'websiteUrl', 'oAuthScopes'],
select: [
'id',
'name',
'logo',
'logoFileId',
'sourceType',
'sourcePackage',
'latestAvailableVersion',
'websiteUrl',
'oAuthScopes',
],
});
if (!registration) {
@@ -235,7 +250,8 @@ export class ApplicationRegistrationService {
return {
id: registration.id,
name: registration.name,
logoUrl: registration.logo,
logoUrl:
this.applicationRegistrationAssetUrlService.buildLogoUrl(registration),
websiteUrl: registration.websiteUrl,
oAuthScopes: registration.oAuthScopes,
};
@@ -389,11 +405,28 @@ export class ApplicationRegistrationService {
return;
}
const displayFields = fromManifestApplicationToDisplayFields(
manifest.application,
);
// Gallery image files are stored by the source-specific flows (tarball
// upload, dev sync); keep their fileIds for paths that did not change.
const existingFileIdByPath = new Map(
(existing.galleryImages ?? []).map(({ path, fileId }) => [
path,
fileId,
]),
);
await this.applicationRegistrationRepository.save({
...existing,
name: manifest.application.displayName,
manifest,
...fromManifestApplicationToDisplayFields(manifest.application),
...displayFields,
galleryImages: displayFields.galleryImages.map((galleryImage) => ({
...galleryImage,
fileId: existingFileIdByPath.get(galleryImage.path) ?? null,
})),
...(sourceType !== undefined && { sourceType }),
...(latestAvailableVersion !== undefined && {
latestAvailableVersion,
@@ -408,6 +441,17 @@ export class ApplicationRegistrationService {
await this.findOneById(id, ownerWorkspaceId);
await this.applicationRegistrationRepository.softDelete(id);
// Stored assets (logo, gallery images) are gone for good; deleting the
// file rows also nulls logoFileId through its ON DELETE SET NULL fk.
try {
await this.serverFileStorageService.deleteByApplicationRegistrationId(id);
} catch (error) {
this.logger.error(
`Failed to delete server files for registration ${id}`,
error,
);
}
await this.invalidateMarketplaceAppsCache();
return true;
@@ -464,6 +508,17 @@ export class ApplicationRegistrationService {
const isVetted = vettedIdentifiers.has(params.universalIdentifier);
if (isDefined(existing)) {
const displayFields = fromManifestApplicationToDisplayFields(
params.manifest?.application,
);
const existingFileIdByPath = new Map(
(existing.galleryImages ?? []).map(({ path, fileId }) => [
path,
fileId,
]),
);
await this.applicationRegistrationRepository.save({
...existing,
name: params.name,
@@ -472,7 +527,11 @@ export class ApplicationRegistrationService {
latestAvailableVersion: params.latestAvailableVersion,
isVetted,
manifest: params.manifest,
...fromManifestApplicationToDisplayFields(params.manifest?.application),
...displayFields,
galleryImages: displayFields.galleryImages.map((galleryImage) => ({
...galleryImage,
fileId: existingFileIdByPath.get(galleryImage.path) ?? null,
})),
});
} else {
const registration = this.applicationRegistrationRepository.create({
@@ -552,9 +611,12 @@ export class ApplicationRegistrationService {
'id',
'universalIdentifier',
'name',
'sourceType',
'sourcePackage',
'latestAvailableVersion',
'isVetted',
'logo',
'logoFileId',
'description',
'author',
'category',
@@ -574,7 +636,8 @@ export class ApplicationRegistrationService {
description: registration.description,
author: registration.author,
category: registration.category,
logoUrl: registration.logo,
logoUrl:
this.applicationRegistrationAssetUrlService.buildLogoUrl(registration),
}));
}
@@ -6,7 +6,7 @@ import { tmpdir } from 'os';
import { isAbsolute, join, relative, resolve } from 'path';
import semver from 'semver';
import { FileFolder, ServerFileFolder } from 'twenty-shared/types';
import { FileFolder } from 'twenty-shared/types';
import { isDefined } from 'twenty-shared/utils';
import { Repository } from 'typeorm';
import { type QueryDeepPartialEntity } from 'typeorm/query-builder/QueryPartialEntity';
@@ -18,20 +18,16 @@ import { extractTarballSecurely } from 'src/engine/core-modules/application/appl
import { readJsonFile } from 'src/engine/core-modules/application/application-package/utils/read-json-file.util';
import { resolvePackageContentDir } from 'src/engine/core-modules/application/application-package/utils/tarball-utils';
import { ApplicationRegistrationVariableService } from 'src/engine/core-modules/application/application-registration-variable/application-registration-variable.service';
import { ApplicationRegistrationAssetService } from 'src/engine/core-modules/application/application-registration/application-registration-asset.service';
import { ApplicationRegistrationEntity } from 'src/engine/core-modules/application/application-registration/application-registration.entity';
import {
ApplicationRegistrationException,
ApplicationRegistrationExceptionCode,
} from 'src/engine/core-modules/application/application-registration/application-registration.exception';
import { ApplicationRegistrationSourceType } from 'src/engine/core-modules/application/application-registration/enums/application-registration-source-type.enum';
import { type ApplicationRegistrationGalleryImage } from 'src/engine/core-modules/application/application-registration/types/application-registration-gallery-image.type';
import { fromManifestApplicationToDisplayFields } from 'src/engine/core-modules/application/application-registration/utils/from-manifest-application-to-display-fields.util';
import { isImageFilePath } from 'src/engine/core-modules/application/application-registration/utils/is-image-file-path.util';
import { toGalleryImagePaths } from 'src/engine/core-modules/application/application-registration/utils/to-gallery-image-paths.util';
import { ApplicationService } from 'src/engine/core-modules/application/application.service';
import { FileStorageService } from 'src/engine/core-modules/file-storage/services/file-storage.service';
import { ServerFileStorageService } from 'src/engine/core-modules/file-storage/services/server-file-storage.service';
import { prepareFileForStorageOrThrow } from 'src/engine/core-modules/file-storage/utils/prepare-file-for-storage-or-throw.util';
import type { ApplicationManifest } from 'twenty-shared/application';
@Injectable()
@@ -42,7 +38,7 @@ export class ApplicationTarballService {
@InjectRepository(ApplicationRegistrationEntity)
private readonly appRegistrationRepository: Repository<ApplicationRegistrationEntity>,
private readonly fileStorageService: FileStorageService,
private readonly serverFileStorageService: ServerFileStorageService,
private readonly applicationRegistrationAssetService: ApplicationRegistrationAssetService,
private readonly applicationService: ApplicationService,
private readonly applicationRegistrationVariableService: ApplicationRegistrationVariableService,
private readonly applicationVersionValidationService: ApplicationVersionValidationService,
@@ -208,10 +204,10 @@ export class ApplicationTarballService {
ownerWorkspaceId: params.ownerWorkspaceId,
} as QueryDeepPartialEntity<ApplicationRegistrationEntity>);
await this.storeGalleryImageFiles({
await this.applicationRegistrationAssetService.storeRegistrationAssets({
applicationRegistrationId: appRegistration.id,
manifestApplication: manifest.application,
contentDir,
readAsset: (path) => this.readAssetFromContentDir(contentDir, path),
});
if (manifest.application?.serverVariables) {
@@ -233,51 +229,10 @@ export class ApplicationTarballService {
}
}
private async storeGalleryImageFiles({
applicationRegistrationId,
manifestApplication,
contentDir,
}: {
applicationRegistrationId: string;
manifestApplication: ApplicationManifest | undefined;
contentDir: string;
}): Promise<void> {
const galleryImages: ApplicationRegistrationGalleryImage[] = [];
for (const path of toGalleryImagePaths(manifestApplication)) {
const fileId = await this.storeGalleryImageFile({
applicationRegistrationId,
contentDir,
path,
});
if (isDefined(fileId)) {
galleryImages.push({ path, fileId });
}
}
await this.appRegistrationRepository.update(applicationRegistrationId, {
galleryImages,
} as QueryDeepPartialEntity<ApplicationRegistrationEntity>);
}
private async storeGalleryImageFile({
applicationRegistrationId,
contentDir,
path,
}: {
applicationRegistrationId: string;
contentDir: string;
path: string;
}): Promise<string | null> {
if (
path.startsWith('http://') ||
path.startsWith('https://') ||
!isImageFilePath(path)
) {
return null;
}
private async readAssetFromContentDir(
contentDir: string,
path: string,
): Promise<Buffer | null> {
const absolutePath = resolve(contentDir, path);
const relativeToContentDir = relative(contentDir, absolutePath);
@@ -287,37 +242,12 @@ export class ApplicationTarballService {
isAbsolute(relativeToContentDir)
) {
this.logger.warn(
`Gallery image "${path}" escapes the package directory; skipping`,
`Asset "${path}" escapes the package directory; skipping`,
);
return null;
}
try {
const contents = await fs.readFile(absolutePath);
const { sourceFile, mimeType } = await prepareFileForStorageOrThrow({
sourceFile: contents,
resourcePath: path,
});
const savedFile = await this.serverFileStorageService.writeServerFile({
fileFolder: ServerFileFolder.ApplicationRegistration,
applicationRegistrationId,
resourcePath: path,
contents: Buffer.isBuffer(sourceFile)
? sourceFile
: Buffer.from(sourceFile),
mimeType,
});
return savedFile.id;
} catch (error) {
this.logger.warn(
`Failed to store gallery image "${path}" for registration ${applicationRegistrationId}: ${error.message}`,
);
return null;
}
return fs.readFile(absolutePath);
}
}
@@ -1,55 +0,0 @@
import { Injectable } from '@nestjs/common';
import { type Manifest } from 'twenty-shared/application';
import { isDefined } from 'twenty-shared/utils';
import { buildRegistryCdnUrl } from 'src/engine/core-modules/application/application-marketplace/utils/build-registry-cdn-url.util';
import { resolveManifestAssetUrls } from 'src/engine/core-modules/application/application-marketplace/utils/resolve-manifest-asset-urls.util';
import { ApplicationRegistrationSourceType } from 'src/engine/core-modules/application/application-registration/enums/application-registration-source-type.enum';
import { TwentyConfigService } from 'src/engine/core-modules/twenty-config/twenty-config.service';
@Injectable()
export class ManifestAssetUrlResolverService {
constructor(private readonly twentyConfigService: TwentyConfigService) {}
resolveFromRegistrySource({
manifest,
packageName,
version,
}: {
manifest: Manifest;
packageName: string;
version: string;
}): Manifest {
const cdnBaseUrl = this.twentyConfigService.get('APP_REGISTRY_CDN_URL');
return resolveManifestAssetUrls(manifest, (filePath) =>
buildRegistryCdnUrl({ cdnBaseUrl, packageName, version, filePath }),
);
}
resolveFromRegistration({
sourceType,
sourcePackage,
manifest,
version,
}: {
sourceType: ApplicationRegistrationSourceType;
sourcePackage: string | null;
manifest: Manifest;
version: string;
}): Manifest {
if (
sourceType !== ApplicationRegistrationSourceType.NPM ||
!isDefined(sourcePackage)
) {
return manifest;
}
return this.resolveFromRegistrySource({
manifest,
packageName: sourcePackage,
version,
});
}
}
@@ -0,0 +1,82 @@
import { areRegistrationAssetsStored } from 'src/engine/core-modules/application/application-registration/utils/are-registration-assets-stored.util';
import type { ApplicationManifest } from 'twenty-shared/application';
const manifestApplication = (
overrides: Partial<ApplicationManifest>,
): ApplicationManifest =>
({
universalIdentifier: 'universal-identifier',
displayName: 'App',
...overrides,
}) as ApplicationManifest;
describe('areRegistrationAssetsStored', () => {
it('should be true when the manifest declares no storable assets', () => {
expect(
areRegistrationAssetsStored(
{ logoFileId: null, galleryImages: null },
manifestApplication({
logo: 'https://example.com/logo.png',
galleryImages: ['https://example.com/shot.png'],
}),
),
).toBe(true);
});
it('should be false when a relative logo has no stored file', () => {
expect(
areRegistrationAssetsStored(
{ logoFileId: null, galleryImages: [] },
manifestApplication({ logo: 'public/logo.png' }),
),
).toBe(false);
});
it('should be false when a relative gallery image has no stored file', () => {
expect(
areRegistrationAssetsStored(
{
logoFileId: 'logo-file-id',
galleryImages: [{ path: 'public/shot.png', fileId: null }],
},
manifestApplication({
logo: 'public/logo.png',
galleryImages: ['public/shot.png'],
}),
),
).toBe(false);
});
it('should be true when every storable asset has a stored file', () => {
expect(
areRegistrationAssetsStored(
{
logoFileId: 'logo-file-id',
galleryImages: [
{ path: 'public/shot.png', fileId: 'shot-file-id' },
{ path: 'https://example.com/external.png', fileId: null },
],
},
manifestApplication({
logo: 'public/logo.png',
galleryImages: [
'public/shot.png',
'https://example.com/external.png',
],
}),
),
).toBe(true);
});
it('should be false when the manifest declares a gallery image the registration has never stored', () => {
expect(
areRegistrationAssetsStored(
{ logoFileId: 'logo-file-id', galleryImages: [] },
manifestApplication({
logo: 'public/logo.png',
galleryImages: ['public/new-shot.png'],
}),
),
).toBe(false);
});
});
@@ -0,0 +1,37 @@
import { isDefined } from 'twenty-shared/utils';
import { type ApplicationRegistrationEntity } from 'src/engine/core-modules/application/application-registration/application-registration.entity';
import { isStorableAssetPath } from 'src/engine/core-modules/application/application-registration/utils/is-storable-asset-path.util';
import { toGalleryImagePaths } from 'src/engine/core-modules/application/application-registration/utils/to-gallery-image-paths.util';
import type { ApplicationManifest } from 'twenty-shared/application';
// True when every storable asset declared by the manifest already has a file
// stored for the registration, so a sync can skip re-downloading them.
export const areRegistrationAssetsStored = (
registration: Pick<
ApplicationRegistrationEntity,
'logoFileId' | 'galleryImages'
>,
manifestApplication: ApplicationManifest | undefined,
): boolean => {
const logoPath = manifestApplication?.logo ?? manifestApplication?.logoUrl;
if (
isDefined(logoPath) &&
isStorableAssetPath(logoPath) &&
!isDefined(registration.logoFileId)
) {
return false;
}
const storedFileIdByPath = new Map(
(registration.galleryImages ?? []).map(({ path, fileId }) => [
path,
fileId,
]),
);
return toGalleryImagePaths(manifestApplication)
.filter(isStorableAssetPath)
.every((path) => isDefined(storedFileIdByPath.get(path)));
};
@@ -0,0 +1,2 @@
export const isAbsoluteUrl = (url: string): boolean =>
/^https?:\/\//i.test(url);
@@ -0,0 +1,7 @@
import { isAbsoluteUrl } from 'src/engine/core-modules/application/application-registration/utils/is-absolute-url.util';
import { isImageFilePath } from 'src/engine/core-modules/application/application-registration/utils/is-image-file-path.util';
// Only relative image paths are copied into server file storage; absolute
// urls are served as-is at query time.
export const isStorableAssetPath = (path: string): boolean =>
!isAbsoluteUrl(path) && isImageFilePath(path);
@@ -264,10 +264,14 @@ describe('ServerFileStorageService', () => {
mockServerFileRepository.findOneBy.mockResolvedValue({
id: 'server-file-id',
path: 'application-registration/registration-id/manifest.json',
mimeType: 'application/json',
} as FileEntity);
mockDriver.readFile.mockResolvedValue(stream);
const result = await service.readServerFileById('server-file-id');
const result = await service.readServerFileById(
'server-file-id',
ServerFileFolder.ApplicationRegistration,
);
expect(mockServerFileRepository.findOneBy).toHaveBeenCalledWith({
id: 'server-file-id',
@@ -277,13 +281,18 @@ describe('ServerFileStorageService', () => {
filePath:
'server/application-registration/registration-id/manifest.json',
});
expect(result).toBe(stream);
expect(result).toEqual({ stream, mimeType: 'application/json' });
});
it('should throw a missing-file exception when the row does not exist', async () => {
mockServerFileRepository.findOneBy.mockResolvedValue(null);
await expect(service.readServerFileById('unknown-id')).rejects.toThrow(
await expect(
service.readServerFileById(
'unknown-id',
ServerFileFolder.ApplicationRegistration,
),
).rejects.toThrow(
expect.objectContaining({
code: FileStorageExceptionCode.FILE_NOT_FOUND,
}),
@@ -148,14 +148,26 @@ export class ServerFileStorageService {
return driver.readFile({ filePath: onStorageFilePath });
}
async readServerFileById(id: string): Promise<Readable> {
async readServerFileById(
id: string,
fileFolder: ServerFileFolder,
): Promise<{ stream: Readable; mimeType: string }> {
const serverFile = await this.findServerFileByIdOrThrow(id);
if (!serverFile.path.startsWith(`${fileFolder}/`)) {
throw new FileStorageException(
`Server file ${id} not found`,
FileStorageExceptionCode.FILE_NOT_FOUND,
);
}
const driver = this.fileStorageDriverFactory.getCurrentDriver();
return driver.readFile({
const stream = await driver.readFile({
filePath: this.buildServerOnStorageFilePath(serverFile),
});
return { stream, mimeType: serverFile.mimeType };
}
checkServerFileExists({
@@ -4,12 +4,17 @@ import { Test, type TestingModule } from '@nestjs/testing';
import { Readable } from 'stream';
import { pipeline } from 'node:stream/promises';
import { FileFolder } from 'twenty-shared/types';
import { FileFolder, ServerFileFolder } from 'twenty-shared/types';
jest.mock('node:stream/promises', () => ({
pipeline: jest.fn(),
}));
import {
FileStorageException,
FileStorageExceptionCode,
} from 'src/engine/core-modules/file-storage/interfaces/file-storage-exception';
import { ServerFileStorageService } from 'src/engine/core-modules/file-storage/services/server-file-storage.service';
import {
FileException,
FileExceptionCode,
@@ -45,6 +50,7 @@ const mockPipeline = jest.mocked(pipeline);
describe('FileController', () => {
let controller: FileController;
let fileService: FileService;
let serverFileStorageService: ServerFileStorageService;
const mock_FileByIdGuard: CanActivate = { canActivate: jest.fn(() => true) };
const mock_PublicEndpointGuard: CanActivate = {
canActivate: jest.fn(() => true),
@@ -65,6 +71,12 @@ describe('FileController', () => {
getFilePresignedUrlOrStreamById: jest.fn(),
},
},
{
provide: ServerFileStorageService,
useValue: {
readServerFileById: jest.fn(),
},
},
],
})
.overrideGuard(FileByIdGuard)
@@ -79,6 +91,9 @@ describe('FileController', () => {
controller = module.get<FileController>(FileController);
fileService = module.get<FileService>(FileService);
serverFileStorageService = module.get<ServerFileStorageService>(
ServerFileStorageService,
);
// Default to a resolved pipeline so happy-path tests don't have to wire it up.
mockPipeline.mockResolvedValue(undefined);
@@ -307,6 +322,108 @@ describe('FileController', () => {
});
});
describe('getApplicationRegistrationFileById', () => {
it('should stream the file with public cache headers', async () => {
const mockStream = createMockStream();
jest
.spyOn(serverFileStorageService, 'readServerFileById')
.mockResolvedValue({
stream: mockStream,
mimeType: 'image/png',
});
const mockResponse = createMockResponse() as any;
await controller.getApplicationRegistrationFileById(
mockResponse,
'file-123',
);
expect(serverFileStorageService.readServerFileById).toHaveBeenCalledWith(
'file-123',
ServerFileFolder.ApplicationRegistration,
);
expect(mockResponse.setHeader).toHaveBeenCalledWith(
'Content-Type',
'image/png',
);
expect(mockResponse.setHeader).toHaveBeenCalledWith(
'Cache-Control',
'public, max-age=3600',
);
expect(mockPipeline).toHaveBeenCalledWith(mockStream, mockResponse);
});
it('should throw FILE_NOT_FOUND when the file does not exist', async () => {
jest
.spyOn(serverFileStorageService, 'readServerFileById')
.mockRejectedValue(
new FileStorageException(
'Server file unknown-id not found',
FileStorageExceptionCode.FILE_NOT_FOUND,
),
);
const mockResponse = createMockResponse() as any;
await expect(
controller.getApplicationRegistrationFileById(
mockResponse,
'unknown-id',
),
).rejects.toThrow(
new FileException('File not found', FileExceptionCode.FILE_NOT_FOUND),
);
expect(mockPipeline).not.toHaveBeenCalled();
});
it('should throw INTERNAL_SERVER_ERROR when the stream errors before headers are sent', async () => {
jest
.spyOn(serverFileStorageService, 'readServerFileById')
.mockResolvedValue({
stream: createMockStream(),
mimeType: 'image/png',
});
mockPipeline.mockRejectedValue(new Error('source backend exploded'));
const mockResponse = createMockResponse({ headersSent: false }) as any;
await expect(
controller.getApplicationRegistrationFileById(mockResponse, 'file-123'),
).rejects.toThrow(
new FileException(
'Error streaming file from storage',
FileExceptionCode.INTERNAL_SERVER_ERROR,
),
);
expect(mockResponse.destroy).not.toHaveBeenCalled();
});
it('should destroy the response without throwing when the stream errors after headers are sent', async () => {
jest
.spyOn(serverFileStorageService, 'readServerFileById')
.mockResolvedValue({
stream: createMockStream(),
mimeType: 'image/png',
});
mockPipeline.mockRejectedValue(new Error('socket reset mid-flight'));
const mockResponse = createMockResponse({ headersSent: true }) as any;
await controller.getApplicationRegistrationFileById(
mockResponse,
'file-123',
);
expect(mockResponse.destroy).toHaveBeenCalledTimes(1);
});
});
describe('getPublicAssets', () => {
it('should 302 redirect when presigned URL is available', async () => {
jest
@@ -11,15 +11,23 @@ import {
import { pipeline } from 'node:stream/promises';
import { join } from 'path';
import { type Readable } from 'stream';
import { Request, Response } from 'express';
import { FileFolder } from 'twenty-shared/types';
import { FileFolder, ServerFileFolder } from 'twenty-shared/types';
import { SERVER_FILE_STORAGE_PREFIX } from 'src/engine/core-modules/file-storage/constants/server-file-storage-prefix.constant';
import {
FileStorageException,
FileStorageExceptionCode,
} from 'src/engine/core-modules/file-storage/interfaces/file-storage-exception';
import { ServerFileStorageService } from 'src/engine/core-modules/file-storage/services/server-file-storage.service';
import { validateFilePath } from 'src/engine/core-modules/file-storage/utils/validate-file-path.util';
import {
FileException,
FileExceptionCode,
} from 'src/engine/core-modules/file/file.exception';
import { PUBLIC_ASSET_CACHE_CONTROL } from 'src/engine/core-modules/file/interfaces/file-folder.interface';
import { FileApiExceptionFilter } from 'src/engine/core-modules/file/filters/file-api-exception.filter';
import {
FileByIdGuard,
@@ -35,7 +43,68 @@ import { PublicEndpointGuard } from 'src/engine/guards/public-endpoint.guard';
export class FileController {
private readonly logger = new Logger(FileController.name);
constructor(private readonly fileService: FileService) {}
constructor(
private readonly fileService: FileService,
private readonly serverFileStorageService: ServerFileStorageService,
) {}
// Serves application registration assets (logo, gallery images). These are
// instance-global marketplace resources, also displayed on the public OAuth
// authorize page, hence no auth token. The /server/ segment separates
// instance-global server files from the workspace-scoped /file/:folder/:id.
@Get(`file/${SERVER_FILE_STORAGE_PREFIX}/application-registration/:id`)
@UseGuards(PublicEndpointGuard, NoPermissionGuard)
async getApplicationRegistrationFileById(
@Res() res: Response,
@Param('id') fileId: string,
) {
let fileResponse: { stream: Readable; mimeType: string };
try {
fileResponse = await this.serverFileStorageService.readServerFileById(
fileId,
ServerFileFolder.ApplicationRegistration,
);
} catch (error) {
if (
error instanceof FileStorageException &&
error.code === FileStorageExceptionCode.FILE_NOT_FOUND
) {
throw new FileException(
'File not found',
FileExceptionCode.FILE_NOT_FOUND,
);
}
this.logger.error('readServerFileById failed unexpectedly', { error });
throw new FileException(
'Error retrieving file',
FileExceptionCode.INTERNAL_SERVER_ERROR,
);
}
setFileResponseHeaders(res, fileResponse.mimeType);
res.setHeader('Cache-Control', PUBLIC_ASSET_CACHE_CONTROL);
try {
await pipeline(fileResponse.stream, res);
} catch (error) {
this.logger.error(
'Application registration file stream failed mid-transfer',
{ error },
);
if (!res.headersSent) {
throw new FileException(
'Error streaming file from storage',
FileExceptionCode.INTERNAL_SERVER_ERROR,
);
}
res.destroy();
}
}
@Get('public-assets/:workspaceId/:applicationId/*path')
@UseGuards(PublicEndpointGuard, NoPermissionGuard)