diff --git a/npm-app-distribution-plan.md b/npm-app-distribution-plan.md deleted file mode 100644 index 246a466df7..0000000000 --- a/npm-app-distribution-plan.md +++ /dev/null @@ -1,283 +0,0 @@ -# npm-Based App Distribution for Twenty - -*Technical Design Document -- February 2026* - -## Overview - -Add npm registry support for distributing Twenty apps (public and private), with per-AppRegistration registry overrides, direct tarball upload as escape hatch, and version upgrade detection. - -## Assumptions - -- The marketplace install flow (currently a TODO in the resolver and frontend) will be implemented separately. This plan provides the infrastructure that install flow will call into. -- The existing `app:dev` flow (individual file uploads via CLI) remains unchanged. -- The existing GitHub-based marketplace discovery remains as a curated fallback. - -## Architecture - -``` - Developer - / | \ - npm publish | twenty app:push - / | \ - npmjs.com Private Reg Server REST Upload - | | | - v v v - [Discovery Layer] [Direct Upload] - npm search API | - GitHub curated list | - | | - v | - MarketplaceService | - (sourcePackage in DTO) | - | | - v v - AppPackageResolverService <--------+ - .npmrc generation - yarn add / tarball extract - | - v - ApplicationSyncService (existing) - WorkspaceMigrationRunnerService - | - v - AppRegistration + Application entities -``` - -## Phase 1: Entity and Config Changes - -### 1a. Extend AppRegistrationEntity - -Add four columns to `application-registration.entity.ts`: - -- **`sourcePackage`** (text, nullable) -- npm package name, e.g. `"twenty-app-fireflies"` or `"@myorg/twenty-app-crm"`. Null for tarball-only or OAuth-only apps. -- **`tarballFileId`** (uuid, nullable) -- FK to a FileEntity storing a directly-uploaded `.tar.gz`. Null when the app comes from npm. -- **`registryUrl`** (text, nullable) -- per-registration npm registry override. Null means "use the server default `APP_REGISTRY_URL`." This is how a single server can pull public apps from npmjs.com while pulling `@mycompany/*` apps from GitHub Packages. -- **`latestAvailableVersion`** (text, nullable) -- cached latest version from the registry, updated periodically. Compared against `Application.version` to surface upgrade availability. - -**Source resolution priority:** - -1. `sourcePackage` is set → resolve from npm via `yarn add` -2. `tarballFileId` is set → extract from file storage -3. Neither → OAuth-only app, no server-side code - -### 1b. Extend ApplicationEntity.sourceType - -Widen the `sourceType` union from `'local'` to `'local' | 'npm' | 'tarball'`: - -- `'local'` -- existing behavior (CLI `app:dev`, individual file uploads, workspace-custom) -- `'npm'` -- installed from an npm registry via `yarn add` -- `'tarball'` -- installed from a directly-uploaded tarball - -This lets the system distinguish how an app was installed, which matters for upgrade logic (npm apps can check the registry for newer versions; tarball apps cannot). - -### 1c. Add server-wide config variables - -Add a new `APP_REGISTRY_CONFIG` group to ConfigVariablesGroup: - -- **`APP_REGISTRY_URL`** (string, default `https://registry.npmjs.org`) -- default npm registry URL -- **`APP_REGISTRY_TOKEN`** (string, optional, sensitive) -- auth token for the default registry - -### 1d. Generate migration - -TypeORM migration adding `sourcePackage`, `tarballFileId`, `registryUrl`, `latestAvailableVersion` to `core.applicationRegistration`. - -## Phase 2: App Package Resolver Service - -### 2a. Create AppPackageResolverService - -New service with core method: - -``` -resolvePackage(appRegistration, options?: { targetVersion? }) → ResolvedPackage | null -``` - -Returns `{ manifestPath, packageJsonPath, filesDir }` or null for OAuth-only apps. - -**Resolution logic:** - -``` -if sourcePackage: - 1. Determine registry: appRegistration.registryUrl ?? APP_REGISTRY_URL - 2. Determine auth token for the resolved registry - 3. Generate temporary .npmrc in an isolated working directory - 4. Run: yarn add @ - 5. Read manifest from node_modules//.twenty/output/manifest.json - 6. Return paths - -if tarballFileId: - 1. Download tarball from FileStorageService - 2. Extract to temporary directory - 3. Read manifest from extracted files - 4. Return paths - -else: - return null (OAuth-only) -``` - -### 2b. Isolated working directories - -Each resolution runs in a temporary directory under `{os.tmpdir()}/twenty-app-resolver/{uuid}/`. This avoids contaminating the server's own `node_modules` and isolates apps from each other. Cleaned up after files are copied to storage. - -### 2c. .npmrc generation - -For scoped packages (`@scope/twenty-app-*`): - -``` -@scope:registry=https://npm.pkg.github.com -//npm.pkg.github.com/:_authToken=TOKEN -``` - -For unscoped packages with a non-default registry: - -``` -registry=https://my-verdaccio.internal:4873 -//my-verdaccio.internal:4873/:_authToken=TOKEN -``` - -### 2d. Post-resolution file transfer - -After resolving, copies files into the app's storage path using the existing FileStorageService layout: - -``` -{workspaceId}/{applicationUniversalIdentifier}/ - built-logic-function/... - built-front-component/... - dependencies/package.json - dependencies/yarn.lock - public-asset/... - source/... -``` - -This reuses the same FileFolder enum paths that `app:dev` uses, so downstream ApplicationSyncService works unchanged. - -## Phase 3: Marketplace Discovery via npm - -### 3a. Update MarketplaceService - -Add npm-based discovery alongside the existing GitHub path: - -- Query npm search API: `GET {registryUrl}/-/v1/search?text=keywords:twenty-app&size=250` -- Map each result to MarketplaceAppDTO using package.json metadata - -**Merge strategy:** - -1. Fetch from npm search API (apps with `keywords: ["twenty-app"]`) -2. Fetch from GitHub (existing curated list) -3. Merge by `universalIdentifier` -- GitHub entries override npm entries (allowing curation) -4. Cache merged result with existing 1-hour TTL - -### 3b. Add sourcePackage to MarketplaceAppDTO - -The DTO needs a `sourcePackage: string | null` field so the install flow knows which npm package to resolve. For npm-discovered apps, this is the package name. For GitHub-only apps, this is null. - -## Phase 4: Version Upgrade Support - -### 4a. Create AppUpgradeService - -**Periodic version check (npm-sourced apps only):** - -- Fetches `{registryUrl}/{sourcePackage}/latest` from the npm registry -- Stores result in `AppRegistration.latestAvailableVersion` -- Frontend compares against `Application.version` to show "Update available" - -**Upgrade trigger:** - -1. Resolve the new version via AppPackageResolverService -2. Sync via existing ApplicationSyncService (triggers workspace migration for schema changes) -3. Update Application.version - -**Rollback strategy:** If sync fails (e.g., migration validation error), re-resolve the previous version and re-sync. This is possible because npm retains all published versions. - -### 4b. Version check scheduling - -Lightweight cron or check-on-access pattern. Iterates over AppRegistrations where `sourcePackage IS NOT NULL` and calls `checkForUpdates()`. Frequency: once per hour, matching the existing marketplace cache TTL. - -## Phase 5: SDK CLI Commands - -### 5a. Finalize `twenty app:build` - -Ensure `.twenty/output/` is npm-publishable. The build step generates a `package.json` in the output directory: - -```json -{ - "name": "twenty-app-fireflies", - "version": "1.2.0", - "keywords": ["twenty-app"], - "twenty": { - "universalIdentifier": "a4df0c0f-c65e-44e5-8436-24814182d4ac" - }, - "files": ["manifest.json", "built-logic-function", "built-front-component", "public-asset"] -} -``` - -The developer then publishes with standard `npm publish` -- no custom command needed. - -### 5b. `twenty app:pack` (new command) - -``` -twenty app:pack [appPath] -``` - -- Runs `app:build` if `.twenty/output/` doesn't exist or is stale -- Uses existing TarballService to create `{name}-{version}.tar.gz` -- Outputs the file path for manual distribution - -### 5c. `twenty app:push` (new command) - -``` -twenty app:push [appPath] --server --token -``` - -- Runs `app:pack` to produce the tarball -- Reads universalIdentifier from manifest to find or create the AppRegistration -- Uploads via `POST /api/app-registrations/upload-tarball` -- Reports success with the registration ID -- Reuses `twenty auth:login` credentials if `--server` is not specified - -## Phase 6: Server Tarball Upload Endpoint - -### 6a. REST controller - -``` -POST /api/app-registrations/upload-tarball -Content-Type: multipart/form-data -Body: tarball file + optional universalIdentifier -``` - -**Validation:** - -- Max file size: 50MB -- Must be a valid `.tar.gz` -- Extracted contents must contain `manifest.json` with a valid `universalIdentifier` -- The `universalIdentifier` must not conflict with an existing registration owned by a different user - -**Flow:** - -1. Extract tarball to temp directory -2. Validate manifest structure -3. Find or create AppRegistration by universalIdentifier -4. Store tarball in FileStorageService under `FileFolder.AppTarball` -5. Set `tarballFileId` on the AppRegistration -6. Return the AppRegistration entity - -### 6b. Add FileFolder.AppTarball - -New enum value `AppTarball = 'app-tarball'` in FileFolder. - -## Key Design Decisions - -| Decision | Rationale | -|---|---| -| Per-AppRegistration registry override | `registryUrl` on the entity allows mixing registries. Public apps from npmjs.com, private from GitHub Packages/Verdaccio. Server-wide `APP_REGISTRY_URL` is the fallback. | -| npm publish is standard | No custom publish infra. Free versioning, README, `npm audit`, download stats, proven auth model. | -| Tarball as escape hatch | Air-gapped environments, CI pipelines, one-off installs. Cannot auto-upgrade. | -| sourceType distinction | `'npm' \| 'tarball' \| 'local'` lets the system know which upgrade path is available. Only npm apps can check for newer versions. | -| Backward compatible | `app:dev` flow unchanged. GitHub marketplace unchanged. All new fields nullable. | -| Upgrade rollback | Re-resolve previous version from npm on failure. Safe because npm never deletes published versions. | - -## Edge Cases - -- **npm unreachable**: Timeout after 30s, throw clear error. App remains at current installed version. -- **Package name conflicts**: The `universalIdentifier` in the `twenty` field of `package.json` is the source of truth, not the npm package name. Two packages with the same universalIdentifier conflict at the AppRegistration level (unique index). -- **Scoped vs unscoped packages**: Both work. Scoped packages naturally route to a private registry via `.npmrc` scope mapping. -- **Multiple workspaces, same server**: AppRegistration is server-level (core schema). Application is workspace-level. One AppRegistration can be installed in multiple workspaces at different versions. diff --git a/npm-app-distribution-plan.pdf b/npm-app-distribution-plan.pdf deleted file mode 100644 index e37caf5d4b..0000000000 Binary files a/npm-app-distribution-plan.pdf and /dev/null differ