baa84bb2e01816cd5b7ed4d137094880bc15889e
1160 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 martmull
|
baa84bb2e0 |
Add auto-upgrade-in-apps (#23001)
We need to auto upgrade application, lets add this column in application entity, and add an admin button to autoupgrade all applications to latest app registrration version manually <img width="1131" height="372" alt="image" src="https://github.com/user-attachments/assets/4e755abc-38ad-4895-a2e8-d55ee1948ac2" /> <img width="906" height="533" alt="image" src="https://github.com/user-attachments/assets/ce002057-0341-4581-bf9a-66ac2bd84a9b" /> |
||
|
Paul Rastoin
|
6b55a6b51c |
Fix duplicate searchFieldMetadata inserts in the 2.16 backfill upgrade command (#23060)
## Context
A self-hosted instance upgrading from 2.0.3 to v2.22.0 got stuck with
one workspace failing at `2.16.0_BackfillSearchFieldMetadataCommand`:
```
[QueryFailedError] duplicate key value violates unique constraint "IDX_SEARCH_FIELD_METADATA_OBJECT_FIELD_UNIQUE"
Detail: Key ("objectMetadataId", "fieldMetadataId")=(...) already exists.
```
The failure happened on a retry after a previous partial run, and
reproduced even though the command already recomputes
`flatSearchFieldMetadataMaps` before deriving the create-set (#22884).
## Root cause
The idempotency dedupe compares `(objectMetadataId, fieldMetadataId)`
pairs across two differently-fresh caches:
- The **existing rows** side comes from `flatSearchFieldMetadataMaps`,
which is recomputed from the database (real current ids).
- The **candidate** side resolves ids through `flatObjectMetadataMaps` /
`flatFieldMetadataMaps`, which are **not** invalidated. During a
cross-version upgrade these can be stale, since the migration runner
only invalidates the cache keys a migration touched.
When a stale map resolves a candidate to an outdated id, the dedupe key
doesn't match the existing row and the row is re-emitted. The migration
runner then re-resolves the universal identifiers against fresh maps at
execution time and inserts with the real current ids — exactly the pair
already committed by the earlier partial run (each per-application
migration commits independently) — tripping the unique constraint and
failing the upgrade.
## Fix
Two independent layers, either of which would have prevented the
failure:
1. **Consistent snapshot for the build phase**: the command now
invalidates and recomputes all three maps the dedupe depends on
(`flatObjectMetadataMaps`, `flatFieldMetadataMaps`,
`flatSearchFieldMetadataMaps`), so candidate resolution, existing-row
keys, and the runner all see the same database state.
2. **Id-churn-proof dedupe**: every row this command creates carries a
deterministic universal identifier (`getSearchFieldUniversalIdentifier`,
derived from application + field universal identifiers, no database ids
involved) and `(workspaceId, universalIdentifier)` is unique. The build
util now also skips any candidate whose deterministic universal
identifier already exists, catching leftovers from a previous partial
run even if objects/fields were recreated under new ids in between.
Deliberately **not** done: `ON CONFLICT DO NOTHING` in the create action
handler — it is shared by all runtime `searchFieldMetadata` creation,
and swallowing a conflict would leave the flat-entity cache holding an
entity id that differs from the row actually in the database.
## Test
Added a regression test reproducing the failure shape: an existing row
with the same deterministic universal identifier but stale metadata ids
must not be re-emitted by the backfill.
Note: `ReconcileSearchFieldMetadataCommand` (2.20) has the same
stale-cache exposure; hardening it is left to a follow-up.
<!-- This is an auto-generated description by cubic. -->
<a
href="https://cubic.dev/pr/twentyhq/twenty/pull/23060?utm_source=github"
target="_blank" rel="noopener noreferrer"
data-no-image-dialog="true"><picture><source
media="(prefers-color-scheme: dark)"
srcset="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"><source
media="(prefers-color-scheme: light)"
srcset="https://www.cubic.dev/buttons/review-in-cubic-light.svg"><img
alt="Review in cubic"
src="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"></picture></a>
<!-- End of auto-generated description by cubic. -->
|
||
|
Thomas Trompette
|
87729a2822 |
feat(workflow): backfill + dual-write for core workflow entity (#22776)
Workflow-side soft-ref sync — the `coreWorkflowId` mirror of the merged version side (#22821 / #22940 / #22944 / #22961). Rebased onto current main; supersedes the original shared-UUID version of this PR. ## What Gives `core.workflow` a per-workspace copy of each workflow (`name`, `lastPublishedVersionId`), soft-reffed from the workspace record via `coreWorkflowId`, so app-shipped workflows have a core home. Does not touch reads/dispatch (that's Phase B). - **Sync service** (`WorkflowCoreSyncService`): core rows get their own id (`uuidv5(workspaceId:recordId)`), the workspace record links via `coreWorkflowId` (written back after the upsert), and the write-back is **guarded** on the `coreWorkflowId` field being present (skips with a warning otherwise — mirrors #22940). Injected repo renamed `coreWorkflowRepository`. - **Dual-write listener** on the `workflow` object: CREATED/UPDATED/RESTORED upsert, DELETED/DESTROYED delete by `coreWorkflowId`. Always-on; failures routed to Sentry so they never break the user write. - **2-20 backfill** (`backfill-workflow-to-core`): reads via the provided `RunOnWorkspaceArgs.dataSource`, upserts each workspace workflow into core. - **2-22 provisioning** (mirrors #22944/#22961): - `add-workflow-core-soft-ref-field`: adds the `coreWorkflowId` system field on existing workspaces (flat-entity legacy migration). - `backfill-workflow-core-links`: full rebuild — per workspace, in one raw-SQL transaction, wipes all `core.workflow` rows, inserts a fresh own-id row per workflow, and re-links every record. (No trigger-map cache to invalidate on `core.workflow`.) Simpler than the version side: `core.workflow` has no one-active-per-workflow index and no trigger-map cache, and it was never backfilled in prod, so there are no legacy shared-id rows. ## Test Fresh `database:reset` + full sequence (2-20 backfill → 2-22 add-field → 2-22 rebuild link): 4/4 workspace records linked via `coreWorkflowId` (id != coreWorkflowId), links resolve, **0 dangling**, no duplicate core rows, names populated. Typecheck + lint + oxfmt clean. |
||
|
neo773
|
c04714b9e0 |
fix(messaging): register and harden webhook subscription renewal cron (#23006)
The renewal cron was never wired into cron:register:all, so Gmail/Calendar/Graph watches were never renewed and went dark ~7 days after connect (the max watch lifetime all three providers allow). Register it, fan renewals out as per-channel queue jobs scoped to active workspaces, retry FAILED channels, and recreate Google Calendar watches before stopping the old one. <!-- This is an auto-generated description by cubic. --> <a href="https://cubic.dev/pr/twentyhq/twenty/pull/23006?utm_source=github" target="_blank" rel="noopener noreferrer" data-no-image-dialog="true"><picture><source media="(prefers-color-scheme: dark)" srcset="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"><source media="(prefers-color-scheme: light)" srcset="https://www.cubic.dev/buttons/review-in-cubic-light.svg"><img alt="Review in cubic" src="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"></picture></a> <!-- End of auto-generated description by cubic. --> |
||
|
Charles Bochet
|
2e671342f5 |
[2/2] Write CREATED at activation, clean stale onboarding workspaces (#22915)
## Context Follow-up to #22904 (merged), which introduced the `CREATED` activation status (schema provisioned, onboarding incomplete — no billing subscription), its enum migration, and the read path. This PR turns the status on and closes the zombie-workspace leak (~60–110 subscription-less ACTIVE workspaces per day since v2 onboarding, 935+ total). ✅ **Deploy gating satisfied**: #22904's slow enum migration shipped with the release deployed to prod on 2026-07-17, so writing `CREATED` is now safe. Rebased on main (clean, no conflicts) — main's #22943/#22955 guarded-transition rework already handles `CREATED` correctly: the webhook suspend switch only suspends `ACTIVE` workspaces, and `reactivateWorkspace` promotes `CREATED`→`ACTIVE`. ## What this PR does 1. **Write path** — `activateWorkspace` sets `CREATED` instead of `ACTIVE` when the workspace has no billing subscription. `hasWorkspaceAnySubscription` returns true when billing is disabled, so self-hosted workspaces keep going straight to `ACTIVE` — no behavior change outside cloud. 2. **Cleanup** — `CREATED` joins `PENDING_CREATION`/`ONGOING_CREATION` in the existing onboarding cleaning flow (cron + `workspace:clean:onboarding` with `--dry-run`): workspaces older than the same seven-day threshold are soft-deleted, then hard-deleted on a later run. **No suspension step and no emails** — an abandoned onboarding is treated as never having completed, exactly like a workspace stuck in creation. A workspace that subscribes before cleanup exits the flow (`CREATED`→`ACTIVE` synchronously via checkout). 3. **Backfill** — slow instance command moving `ACTIVE` workspaces with no `billingSubscription` row, created since v2 onboarding shipped (2026-07-01), to `CREATED`. Gated on `IS_BILLING_ENABLED` so self-hosted instances are untouched. 4. **Resolves #22904's text-cast TODO on the billing activation update** — this PR only deploys after the enum migration, so the `CREATED`→`ACTIVE` promotion is a plain status-scoped update again. The upgrade-path filters (`activationStatusIn`) keep the `::text` cast: upgrade tooling has to run against databases coming from pre-2.22 versions, so its TODO now points at the real removal trigger (dropping pre-2.22 upgrade support). ## Ops note before deploying The backfilled zombies are all older than seven days, so the first cron run after the backfill **soft-deletes them and the next run destroys them (schema and data), with no user-facing communication**. The backfill also catches any post-July-1 cloud workspace that is ACTIVE without a subscription — including intentionally comped/demo/internal ones if any were created since then (verified locally: the seeded demo workspaces matched). **Run the backfill's SELECT as a dry-run against prod and review the list before deploying.** ## CI note ~~`cross-version-upgrade` (and its `ci-server-status-check` aggregate) is red due to a pre-existing regression on main — `Field metadata "coreWorkflowVersionId" is missing in object metadata workflowVersion` on the seed workspaces.~~ Resolved: the rebase picks up main's #22944/#22961 which fixed that regression. ## Verification Server-side (billing-enabled local instance, Stripe test mode) and through the full onboarding UI in both billing modes: - **Billing enabled, UI**: signup → workspace creation → **`activationStatus: CREATED`** in DB mid-onboarding → profile/invite steps work on the CREATED workspace → plan-required page → no-card trial → app loads, workspace **`ACTIVE`** with a `trialing` subscription (exercises #22904's synchronous promotion). - **Billing disabled, UI**: signup → workspace creation → **`ACTIVE` directly**, no plan step anywhere, app loads — self-hosted behavior unchanged. - **Cleanup**: a `CREATED` workspace backdated 8 days is listed by `workspace:clean:onboarding --dry-run`; the real run soft-deletes it silently (no suspension, no email) and the next run hard-deletes it (workspace row and schema gone). - **Backfill**: synthetic `ACTIVE` no-sub workspaces — created 2026-07-05 flips to `CREATED`, created 2026-06-15 stays `ACTIVE`, subscribed workspaces stay `ACTIVE`; billing-disabled short-circuit returns without touching anything. - Lint + typecheck green. |
||
|
Thomas Trompette
|
f8b7ecf680 |
fix(workflow): rebuild core workflowVersion rows in the 2-22 backfill (#22961)
## Problem
Syncing workflowVersion to core fails with `duplicate key value violates
unique constraint "IDX_WORKFLOW_VERSION_ONE_ACTIVE_PER_WORKFLOW"`. The
sync's `INSERT ... ON CONFLICT ("id")` only dedupes the primary key — it
can't dedupe `IDX_WORKFLOW_VERSION_ONE_ACTIVE_PER_WORKFLOW`
(`(workspaceId, workflowId) WHERE status='ACTIVE'`). When a leftover
ACTIVE core row exists for a `(workspaceId, workflowId)` with a stale id
(accumulated across the sync's earlier id schemes), inserting the new
active row collides, and the per-id purge misses it.
## Fix
Rewrite the 2-22 `backfill-workflow-version-core-links` command as a
**full rebuild**. Per workspace, in one raw-SQL transaction (core and
workspace schemas are the same database):
1. `DELETE` all `core.workflowVersion` for the workspace — clears every
stale/leftover row.
2. Insert a fresh own-id core row for **every** workspace version.
3. `UPDATE` `coreWorkflowVersionId` on **every** workspace record to its
new core id.
Because it wipes first and re-links all records, leftover ACTIVE rows
can't collide and no record is left pointing at a deleted core row — so
the dual-write's `linked → update` path stays correct afterward.
## Test (local)
Fresh reset, and a reproduced dirty state (leftover ACTIVE core row with
a stale id + a stale link + an unlinked record):
- rebuild runs with no `ONE_ACTIVE` / duplicate-key error,
- leftover wiped, stale link replaced, every record re-linked to a fresh
own-id row,
- 0 dangling/unlinked, no duplicate core rows, exactly one ACTIVE core
version per workflow.
Typecheck + lint + oxfmt clean.
## Note
The 2-20 `backfill-workflow-version-to-core` can still log per-workspace
conflicts on already-dirty instances, but they're non-fatal (the
iterator continues) and this rebuild corrects the end state. The
dual-write (`upsertToCore`) is unchanged and works on the clean data
this produces.
|
||
|
Thomas Trompette
|
988f8ff900 |
feat(workflow): provision coreWorkflowVersionId on existing workspaces and link them (#22944)
Stacked on the write-back guard hotfix (#22940). Completes the version soft-ref for workspaces that predate the `coreWorkflowVersionId` field. ## Why New standard fields aren't auto-synced to existing workspaces; they're only built at workspace creation or added by an explicit upgrade command. Deployed 2.20/2.21 instances also carry **legacy core rows with `id = record.id`** (the pre-soft-ref shared-UUID model, from the already-run #22663 backfill). The original backfill has already run there and won't re-run (tracking is by command name), so the migration to soft-ref has to be **new appended commands**. ## What (two appended 2-22 workspace commands) 1. `add-workflow-version-core-soft-ref-field` (`1784193206000`): adds the `coreWorkflowVersionId` system field to existing workspaces missing it (flat-entity migration, `add-message-campaign-stat-fields` pattern). Idempotent, dry-run aware, skips workspaces without the `workflowVersion` object. 2. `backfill-workflow-version-core-links` (`1784193207000`): re-runs the sync (`upsertToCore`). For each version, `upsertToCore` **purges the legacy shared-id core row** (`id === record id`) then upserts a deterministic own-id row and writes the link back onto the workspace record. Targeted per-id delete — it does not wipe unrelated core rows. Ordering: both run after the original 2-20 backfill. On instances that run 2-20 fresh (e.g. 2.19 → 2.22) that backfill creates core rows and — via the hotfix guard — skips the write-back until the field exists; command 1 provisions the field; command 2 purges + relinks. On already-migrated 2.21 instances the 2-20 backfill won't re-run, so command 2 is what clears their shared-id rows. The same per-id purge in `upsertToCore` also covers the dual-write path: between the 2.22 deploy and command 2 running, an edited version would otherwise collide with its shared-id row on the one-active-per-workflow index. Scope: version side only. The workflow-side equivalent (`coreWorkflowId`) ships with the workflow-side sync PR; `core.workflow` was never backfilled in prod, so it has no legacy shared-id rows. ## Test - Unit: guard skips write-back when the field is absent; runs it when present. - Happy path (fresh reset): original backfill → add-field no-op → link → 4/4 linked, own ids, 0 duplicates. - Deployed migration (simulated 2.21: shared-id ACTIVE core rows + field removed): add-field re-provisions → link purges the shared-id rows and rebuilds → records linked to own-id rows, 0 duplicates, exactly one ACTIVE core version per workflow (index intact), no collision. - Idempotent re-run: still 4 core rows, links resolve. - Typecheck + lint + oxfmt clean. |
||
|
Charles Bochet
|
8e03921372 |
Add CREATED workspace activation status (read path + enum migration) (#22904)
## Context Since v2 onboarding (#22303), workspaces are activated **before** the billing plan step (now the last onboarding step). Users abandoning at the plan step leave ACTIVE workspaces with a Stripe customer but no subscription (~60–110/day on cloud, 935+ so far), and no cleanup mechanism ever touches them: billing webhooks never fire (no subscription), the suspended-workspaces cron only handles SUSPENDED, the onboarding cron only handles PENDING_CREATION/ONGOING_CREATION. Target lifecycle (across two PRs): `PENDING_CREATION → ONGOING_CREATION → CREATED → ACTIVE → SUSPENDED → deleted`. **`CREATED`** = the workspace schema is provisioned but onboarding is not complete — no billing subscription yet. It is **not** considered active: | Concern | CREATED behavior | |---|---| | Sign-in / invited teammates joining | allowed (invite-team step precedes the plan step) | | Member + metadata loading (app shell) | allowed (user must finish onboarding) | | Permissions | real permission checks (no PENDING-style bypass) | | Version upgrades / workspace migrations | **included** (schema must not drift) | | Messaging/calendar/workflow/etc. crons | **excluded** — no background processing until a plan is chosen | | PLAN_REQUIRED onboarding lock | unchanged (still derived from subscription existence) | ## What this PR does (read path only) The enum addition ships as a **slow** instance command, which can run after deploy — so nothing in this PR ever **writes** `CREATED`. The write path (setting it at activation, the cleanup sweep, the backfill of the existing zombie cohort) is a follow-up PR that ships once this migration has run everywhere. - **twenty-shared**: `CREATED` enum value; `PROVISIONED_WORKSPACE_ACTIVATION_STATUSES` + `isWorkspaceProvisioned` ("schema exists": CREATED | ACTIVE | SUSPENDED), replacing `isWorkspaceActiveOrSuspended` — all call sites (server member loading, access-token workspace-member lookup, front metadata-store gates) meant "has schema/members". - **Slow instance command** (2.22.0): swaps `core.workspace_activationStatus_enum` using the rename→recreate→alter-column idiom. The CHECK constraints on `core.workspace` embed casts to the enum type and would break the swap — the command captures them from `pg_constraint`, drops them, swaps the type, and restores them. - **Pre-migration-safe queries**: Postgres rejects `IN ('CREATED', ...)` when the enum value does not exist yet — even for reads, and the instance-command runner itself queries provisioned workspaces before migrating (a fresh database could never initialize). All provisioned-status filters go through a new `activationStatusIn` util comparing on `"activationStatus"::text`, valid before and after the migration. - **Upgrade path**: workspace iterator, command runner, upgrade-status and workspace-version services iterate CREATED workspaces. Since they now cover more than ACTIVE/SUSPENDED, the stale names were renamed to `ProvisionedWorkspaceCommandRunner`, `hasProvisionedWorkspaces`, `getProvisionedWorkspaceIds`, `loadProvisionedWorkspaces` (the mechanical import rename in old version-command dirs is why this PR carries the `ci:allow-previous-version-upgrade-mutation` label). - **Sign-in**: `throwIfWorkspaceIsNotReadyForSignInUp` accepts CREATED so invited members can join during onboarding (join authorization itself is unchanged — enforced upstream in `checkAccessForSignIn`); `activateWorkspace` idempotent-retry accepts CREATED as a terminal state. - **Transitions out of CREATED** (only write ACTIVE — safe to ship now, dead until the write path lands): the Stripe webhook reactivation branch also promotes CREATED, and `syncSubscriptionToDatabase` promotes synchronously; both gated on `WORKSPACE_ACTIVATING_SUBSCRIPTION_STATUSES` (Active/Trialing — extracted from `shouldReactivateWorkspace`, behavior-preserving) so an `incomplete` subscription created by the payment-intent flow before payment never promotes the workspace. - Deliberately untouched: all background crons, permission guards, JWT strategy, PLAN_REQUIRED logic, admin panel (renders the raw status string). ## Follow-up PR (after this migration has run) 1. `activateWorkspace` sets `hasWorkspaceAnySubscription ? ACTIVE : CREATED` (billing disabled → always ACTIVE, self-hosted unchanged). 2. Cleanup: suspend CREATED workspaces older than N days (config var), handing them to the existing suspended pipeline (warn → soft-delete → destroy). 3. Backfill: cloud-only slow command moving ACTIVE workspaces with no billingSubscription row (created since Jul 1) to CREATED. ## Verification - Migration exercised against a real database via the command class: up → down → up; `enum_range` and `pg_get_constraintdef` checked after each step (constraints restored against the new type, `DEFAULT 'INACTIVE'` preserved). - Pre-migration safety exercised for real: with the migration rolled back (enum without CREATED), `run-instance-commands` — the exact fresh-database CI path that failed before the `::text` fix — completes cleanly. - End-to-end with a workspace manually set to CREATED and the branch server+front running: sign-in issues tokens, `currentUser` loads workspaceMember(s), the full app loads with no console errors; GraphQL returns `activationStatus: CREATED`. - Workspace creation ran end-to-end locally in **both billing modes** on this branch: - billing disabled: signup → workspace creation → ACTIVE immediately → onboarding completes with no plan step → app loads (unchanged behavior); - billing enabled (Stripe test mode): signup creates the Stripe customer eagerly → activation ends ACTIVE → subscription-less workspace is pinned to the plan-required page → no-card trial checkout creates a `trialing` subscription via `createDirectSubscription`/`syncSubscriptionToDatabase` → app loads. - `twenty-shared` unit tests, server specs on touched services, `lint:diff-with-main` and `typecheck` for shared/server/front all green; full CI green. |
||
|
Weiko
|
25bd2897a3 |
Add weekly layout to record calendar (#22819)
## Summary - Add a week layout to record calendar views and persist the selected layout. - Render `DATE` calendars as an all-day week and `DATE_TIME` calendars as an hourly week. - Add an optional end date field across calendar configuration, metadata, persistence, and complete-view upserts. - Use configured end values for ranged and multi-day events, with a one-hour fallback when a `DATE_TIME` end is absent or invalid. - Keep calendar cards consistent with the existing compact view, including checkbox selection and whole-card record opening. - Gate the weekly layout and end-date behavior behind the public Labs `IS_CALENDAR_WEEK_VIEW_ENABLED` workspace feature flag. ## Week interactions - Show overlapping timed events side by side and cap the visible records at two per day. - Display start and end times on timed cards, enforce a readable 30-minute minimum height, and keep today’s text contrast stronger. - Drag timed events between days and times with 30-minute snapping while preserving their duration, including zero-duration events. - Show a create button when hovering a 30-minute slot; keyboard users can focus a day, move the slot with the arrow keys, and reach the same contextual action. - Initialize new records with the selected slot time and a compatible writable end value one hour later. - Show the workspace time zone and current-time indicator in timed weeks; date-only weeks keep the all-day section without an hourly grid. ## Configuration and data loading - Only allow end fields that match the start field type, and prevent selecting the same field for both boundaries. - Load records whose ranges overlap the visible period so month and week layouts display the same relevant records. - Resolve and persist calendar end fields when updating existing views through `upsert_complete_view`. - Fall back to Month and ignore the configured end field while the flag is disabled, without overwriting either persisted setting, so re-enabling restores the previous configuration. - Expose the flag in Labs and keep it default-off for workspaces without a stored value; enable it in the development seeder. <img width="1285" height="808" alt="Screenshot 2026-07-15 at 15 50 17" src="https://github.com/user-attachments/assets/b7e3f7f1-ca77-492f-8cce-cca186ebca0b" /> |
||
|
Abdul Rahman
|
f4ff234db8 |
feat: make record avatar/icon resolution data-driven via a configurable image identifier field (#22644)
## Summary Today the avatar/icon shown for a record is hardcoded per object — Company pulls a favicon from its domain link, Person uses `avatarUrl`, etc. This PR replaces that hardcoding with a generic, data-driven abstraction based on a configurable **image identifier field** on each object's metadata (mirroring the existing **label identifier** concept). An object's image identifier can point to: - a **`FILES`** field → the uploaded image is used directly (rounded avatar), or - a **`LINKS`** field → a favicon is derived from the primary URL via the Twenty icons service (squared avatar), gated by `ALLOW_REQUESTS_TO_TWENTY_ICONS`. This lets any object type (Opportunity, a custom "Listing", etc.) define its own avatar/icon without code changes, and makes the field configurable/overridable for standard objects. ## ❓ Open question: also allow `TEXT` → direct image URL? Right now the image identifier is restricted to `FILES` (uploaded file) and `LINKS` (favicon). We deliberately left out `TEXT` → **direct image URL** (e.g. an imported/synced photo URL stored in a text field). There's precedent for it — Person's avatar was originally a `TEXT` `avatarUrl`, and WorkspaceMember still is — and it's unambiguous (a `TEXT` field has no favicon-vs-image ambiguity, and selecting it as the image identifier is itself the declaration of intent). It's a small, clean extension: - add `TEXT` to the allowed image-identifier types, - add an explicit `TEXT → raw URL` case - `getAvatarType`: `TEXT → rounded`. Caveats: it relies on admin assertion that the text values are image URLs (no data-level guarantee), and external image URLs load third-party content in the browser (IP-leak/hotlinking, same as favicons — a proxy/cache would be the more robust long-term answer). ### ✅ Resolution Decision: **we will not support `TEXT` as an image identifier.** Image identifiers stay restricted to `FILES` and `LINKS`, and any other type fails closed (returns no avatar) on both the frontend and backend. Instead, the legacy items that still rely on a `TEXT` avatar — Person's deprecated `avatarUrl` and WorkspaceMember's `avatarUrl` — will be migrated to `FILE` fields in a follow-up PR. Until then, WorkspaceMember remains an exception (its `avatarUrl` still resolves through the existing CorePicture path), and legacy Person `avatarUrl` values that haven't been migrated will show initials placeholders. <!-- This is an auto-generated description by cubic. --> <a href="https://cubic.dev/pr/twentyhq/twenty/pull/22644?utm_source=github" target="_blank" rel="noopener noreferrer" data-no-image-dialog="true"><picture><source media="(prefers-color-scheme: dark)" srcset="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"><source media="(prefers-color-scheme: light)" srcset="https://www.cubic.dev/buttons/review-in-cubic-light.svg"><img alt="Review in cubic" src="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"></picture></a> <!-- End of auto-generated description by cubic. --> |
||
|
Paul Rastoin
|
a28c3a905a |
Route pre-2.19 upgrade commands through a legacy validate-build path (#22884)
## Problem Since the centralized metadata side-effect engine landed in v2.19, `WorkspaceMigrationValidateBuildAndRunService.validateBuildAndRunWorkspaceMigrationFromRecord` runs `metadataSideEffectEngineService.expandWithSideEffects(...)` before building. As a result every historical upgrade command (`upgrade-version-command/1-21/*` … `2-18/*`), authored before the engine existed, now flows through it. Their operation matrix is no longer applied literally: the engine injects/cascades companions (system fields, `searchVector` field + GIN index, `searchFieldMetadata` rows, unique backing indexes) and can hard-fail on reserved-identifier collisions (`RESERVED_SYSTEM_UNIVERSAL_IDENTIFIER`). Two hazards for already-shipped commands: 1. **Collision → hard failure**: a command declaring a companion the engine now owns collides with the engine's deterministic `universalIdentifier`. 2. **Silent drift**: on object/field create/delete the engine adds/cascades companions the command author never intended, so workspaces upgraded now differ structurally from those upgraded incrementally before 2.19. Suspected real-world impact: a self-hosted user upgrading v2.6.1 → v2.21.0 hit `duplicate key value violates unique constraint "IDX_SEARCH_FIELD_METADATA_OBJECT_FIELD_UNIQUE"` in `upgrade:2-16:backfill-search-field-metadata`, because object-creating commands now cascade and pre-create the deterministic `searchFieldMetadata` rows the standalone backfill then re-inserts. ## Changes - `workspace-migration-validate-build-and-run-service.ts`: extract the shared compute-and-run tail into a private method, and add `validateBuildAndRunLegacyWorkspaceMigration` (marked `@deprecated`) that skips `expandWithSideEffects` and applies the matrix literally. The existing side-effect entry points are unchanged (the live API and application manifests depend on them). - Repoint **all** pre-2.19 upgrade command call sites (1-21 … 2-18, including `2-10 sync-call-recording-standard-objects`) to the legacy method. Only the four `2-20/*` commands (target version ≥ 2.19) remain on the side-effect path. - `2-16 backfill-search-field-metadata`: recompute `flatSearchFieldMetadataMaps` from the database before building the existing-rows dedupe set. The migration runner only invalidates the flat-maps keys a migration touched, so during a cross-version upgrade earlier commands can leave this map stale; a stale map breaks the dedupe and re-inserts rows, tripping `IDX_SEARCH_FIELD_METADATA_OBJECT_FIELD_UNIQUE`. This is the direct fix for the reported failure. - Export `FlatEntityMapsBundle` so the shared tail can be typed. - Document the side-effect vs legacy path and the selection rule in `packages/twenty-server/docs/UPGRADE_COMMANDS.md`. Selection rule: target version **< 2.19** → legacy path; **≥ 2.19** → side-effect path (default). No exceptions. ## Known gap / merge ordering The static twenty-standard definition declares all of `callRecording`'s fields (including the `searchVector` system field) but **not** its `searchVector` GIN index — every other searchable standard object declares its GIN index statically. On the legacy path, workspaces upgrading through `2-10 sync-call-recording-standard-objects` therefore create the `searchVector` column unindexed (`searchFieldMetadata` rows are created later in the same pipeline by the 2-16 backfill). The static GIN index declaration plus a backfill for already-upgraded workspaces land in a follow-up (twentyhq/core-team-issues#2672), which must ship in the same release as this PR. ## Out of scope (separate follow-ups) - `UpgradeMigrationService.getLastAttemptedInstanceCommand()` ordering. - callRecording `searchVector` GIN index static declaration + backfill (twentyhq/core-team-issues#2672, same-release dependency, see above). ## Test plan - `nx typecheck twenty-server` passes. - `nx lint:diff-with-main twenty-server` (oxlint + oxfmt) clean on changed files. - 2-20 command specs (which exercise the unchanged side-effect path) pass. --------- Co-authored-by: twenty <noreply@twenty.com> |
||
|
martmull
|
94192a2164 |
Resolve application registration logo and gallery image urls at query time (#22827)
## Context
Application registration logo and gallery image urls were baked into the
stored manifest and display columns at write time, with each source flow
doing it differently: npm catalog sync baked CDN urls, local dev sync
baked `public-assets` urls, and tarball uploads left raw manifest paths
that never displayed in the UI. The entity also carried a `logoUrl`
getter computed field.
This moves url generation to query time, the same way the workspace logo
works.
## What changed
**Read side**
- `ApplicationRegistrationAssetUrlService` builds display urls when
queried: stored files are served by fileId, absolute urls pass through
untouched, and not-yet-rehosted npm assets fall back to the registry CDN
from `sourcePackage@latestAvailableVersion`.
- The `logoUrl` getter on `ApplicationRegistrationEntity` is replaced by
`logoUrl` and `galleryImages` `@ResolveField`s on the metadata resolver,
the admin panel resolver, and a new resolver for
`ApplicationRegistrationSummary` (used by
`Application.applicationRegistration`).
- The marketplace detail/card DTOs and the public OAuth authorize DTO
(`findApplicationRegistrationByClientId`) go through the same url
builder.
- New public route `GET /file/application-registration/:id` streams
registration server files (these are instance-global marketplace assets,
also shown on the public OAuth authorize page).
`ServerFileStorageService.readServerFileById` now returns the mime type
alongside the stream.
**Write side**
- New `logoFileId` column on `applicationRegistration` (2.21 fast
instance command, constraint names match TypeORM naming), complementing
the fileIds already stored in the `galleryImages` jsonb.
- `ApplicationRegistrationAssetService` copies the manifest logo and
gallery images into instance-global server file storage, so all three
sources behave the same:
- **TARBALL**: from the uploaded package (previously only gallery images
were stored, never the logo).
- **LOCAL**: dev sync reads the already-uploaded public assets from
workspace storage (the CLI uploads files before syncing).
- **NPM**: catalog sync downloads the assets from the registry CDN.
Downloads are skipped when the package version is unchanged and the
files are already stored; failed or pending downloads fall back to CDN
urls at query time.
- Write-time url rewriting is removed
(`ManifestAssetUrlResolverService`, `resolveManifestAssetUrls`);
manifests now keep raw asset paths. Existing rows with baked absolute
urls keep working through the absolute-url passthrough, so no backfill
is needed.
- `updateFromManifest` and `upsertFromCatalog` preserve stored gallery
fileIds for unchanged paths, so installs and the hourly catalog sync no
longer clobber them.
## How it was verified
Against a local Postgres/Redis with the server running:
- Fresh database init runs the new instance command; column and FK/UQ
constraint names match TypeORM's generated names, and the CI
pending-migration check produces no diff.
- `findManyApplicationRegistrations { logoUrl galleryImages }` returns
fileId-served urls for a TARBALL registration (absolute urls passed
through), and null/[] for a LOCAL registration without assets.
- Ran `marketplace:catalog-sync` against the real npm registry: 14
packages synced, logos and gallery images rehosted from unpkg with
fileIds set; a second run re-downloaded nothing (version-unchanged
skip); `findMarketplaceAppDetail` for `twenty-linear` returns
fileId-served urls for the logo and all four gallery images.
- `GET /file/application-registration/:id` serves stored files with the
right content type (png and svg verified), 404s on unknown ids, and the
token-guarded generic `/file/:folder/:id` route still returns 403
without a token.
- Unit tests for the url builder and the assets-stored check; server
unit test suites for the application module pass; typecheck and lint
clean.
|
||
|
Paul Rastoin
|
de75be16e2 |
harden(server): backfill and enforce workspace.databaseSchema invariant with a check constraint (#22855)
## What & why `core.workspace.databaseSchema` is meant to be set for every workspace past the creation phase. It only started being written at creation time in 2.x (dual-write since 2026-03-28, direct write since 2026-04-10); older workspaces relied on the `1-21 backfill-datasource-to-workspace` instance command, which never effectively ran on some instances. On affected rows the column could be left `NULL`. A null value on a post-creation workspace is a real integrity problem — several paths trust the column: - **REST API**: `hydrateRestRequest` throws `No data sources found` for authenticated requests. - **GraphQL API**: `getOrComputeSchemaSDL` returns `null`, so `WorkspaceSchemaFactory` hands back an empty schema. - **GraphQL introspection** (direct execution) returns `null`. This PR makes the invariant impossible to silently violate, and repairs any instance still lagging. ### On the original "No data source, skipping" logs This investigation started from `BackfillActorSourceEnumValuesCommand` logging `No data source for workspace <id>, skipping` at high volume. **That symptom is not explained by this change, and this PR is not a fix for it.** Findings: - The workspace iterator only processes `ACTIVE` + `SUSPENDED` workspaces, and on the affected instance all of those already have `databaseSchema` set (only `PENDING_CREATION` rows are null, and those are never iterated). - `getGlobalWorkspaceDataSource()` never resolves to `undefined` (it returns a value or throws), so a defined-schema workspace should never hit the skip branch. - The upgrade-aware repository proxy was investigated as a possible cause (it can short-circuit `findOne` to `null` for entities marked unavailable during an upgrade) and **exonerated**: `WorkspaceEntity` and its `databaseSchema` column carry no `@WasIntroducedInUpgrade`/`@WasRemovedInUpgrade` decorators, so `resolveEntityShapeAtUpgradeCursor` always reports the entity available and the column visible at every cursor. In other words, current code should emit zero such skips for that instance's data, so the root cause of the observed logs remains undetermined and is tracked separately. See twentyhq/core-team-issues#2666. ## Changes - **Check constraint `workspace_requires_database_schema`** (the core of this PR): enforces `databaseSchema IS NOT NULL` for any workspace past creation (`activationStatus NOT IN ('PENDING_CREATION', 'ONGOING_CREATION')`). Declared on `WorkspaceEntity` and applied in the slow instance command's `up()`. Safe against the creation flow: `databaseSchema` is written in `WorkspaceManagerService.init` (right after schema creation) long before a workspace becomes `ACTIVE`. - **Defensive backfill** (`2-21` slow instance command): repopulates `databaseSchema` where it is `NULL`/empty, deriving the schema name deterministically from the workspace id (`getWorkspaceSchemaName`) and only setting it for workspaces whose schema actually exists in `information_schema.schemata` (so `PENDING_CREATION` rows without a provisioned schema are left untouched, and stay exempt via the constraint). No-op on instances already backfilled. - `runDataMigration` runs before `up()`, so the backfill repairs legacy rows before the constraint is enforced. Keeping both in the same slow command (rather than a standalone fast command) guarantees the constraint is never added ahead of the repair. - `checkSchemaExists` gets an explicit `: Promise<boolean>` return type. ## Notes - Backfill + constraint live in a **slow** instance command, so they only apply on upgrades run with `--include-slow`. - The constraint is added **`NOT VALID`**: the backfill repairs every workspace whose Postgres schema exists, but some legacy active/suspended workspaces (e.g. carried over from very old versions, as reproduced by the cross-version upgrade from v1.22) have a null `databaseSchema` with no schema to point at and are unrepairable. `NOT VALID` enforces the invariant on all future inserts/updates without failing the upgrade on that pre-existing corruption. - No production request path was changed — the iterator and `checkSchemaExists` keep trusting the (now backfilled + constrained) column. ## Test plan - [ ] Run `database:migrate:prod --include-slow` on an instance with null `databaseSchema` rows; verify rows whose schema exists get backfilled and `PENDING_CREATION` rows are left null. - [ ] Verify the `workspace_requires_database_schema` constraint exists on `core.workspace` and rejects nulling `databaseSchema` on an active workspace. - [ ] Verify a fresh workspace creation still succeeds (constraint does not fight the `PENDING_CREATION` → `ACTIVE` transition). |
||
|
Paul Rastoin
|
1b168ac1f7 |
fix(server): gate workspaceDiscoverability behind upgrade decorator (#22818)
## Context Needs to be patched on 2.20, will craft a 2.19 equivalent with fallback asap ( be it won't be merged unlike this one ) Fixes #22662. Follow-up to #22423, which introduced `workspaceDiscoverability`. A clean 2.18.x to 2.19 upgrade breaks on login with: ``` QueryFailedError: column workspaceDiscoverability does not exist ``` `workspaceDiscoverability` was added to `WorkspaceEntity` (in #22423) as a plain, always-selected, non-nullable column, so any workspace query (including the auth-path `findAvailableWorkspacesByEmail` lookup) fails as soon as the ORM selects it, before the `2.19` upgrade command that creates the column has run. Because the Docker entrypoint is fail-open, the API starts even if the upgrade is delayed, and users hit this on their first login. ## Changes - Add `@WasIntroducedInUpgrade` to `workspaceDiscoverability`, referencing the existing `2.19.0` fast instance command that creates the column. The upgrade-aware ORM then skips the column until the command has actually added it, keeping login working during the upgrade. - Keep the GraphQL `@Field` non-nullable and add a `workspaceDiscoverability` `@ResolveField` that falls back to `WorkspaceDiscoverability.PUBLIC` while the column is hidden, so the resolver never returns `null` for the non-nullable field during the upgrade window. This mirrors the existing pattern already applied to `FileEntity.status` and `FileEntity.applicationRegistrationId`, and the resolver-default pattern already used for `fastModel` / `smartModel` / `logo`. ## Cherry-pick This fix needs to be cherry-picked onto both the **2.19** and **2.20** release branches, since affected instances are upgrading into those versions. ## Test - `validate-upgrade-aware-entity-decorators` and `resolve-entity-shape-at-upgrade-cursor` unit tests pass (the referenced upgrade command name resolves correctly). - `upgrade-aware-repository.proxy` and `upgrade-aware-entity-metadata.adapter` specs pass. - `typecheck` and lint pass for `twenty-server` and `twenty-front`. - Regenerating the GraphQL schemas produces no diff (the field stays non-nullable). |
||
|
Paul Rastoin
|
652adc3c03 |
fix(server): backfill isSystemSideEffect on system fields provisioned before 2.15 (#22850)
## Context The `isSystemSideEffect` column was introduced in **2.15** via a fast instance command that added it with `DEFAULT false`. That stamped `false` onto every pre-existing `fieldMetadata` row — including the 8 engine-owned system fields (`id`, `createdAt`, `updatedAt`, `deletedAt`, `createdBy`, `updatedBy`, `position`, `searchVector`) of every object provisioned before 2.15, regardless of the creation path (API metadata **and** manifest sync). The per-workspace backfill that should have re-flagged those existing rows was explicitly deferred as out of scope in #21673 ("PR 2") and never shipped for `fieldMetadata`. Because `isSystemSideEffect` is configured with `toCompare: false`, no later sync ever repaired the stale value either. Since **2.20** the SDK no longer declares system fields in manifests. On an up-to-date instance, `twenty plan` against an unchanged app therefore diffs those stale-`false` system fields as **missing from the manifest**, and they fall through the `isSystemSideEffectFlatEntity` exclusion in `buildAllFlatEntityOperationRecordByMetadataNameFromFromTo`. Deletion inference then emits them as deletes, which the validator rejects: ``` Sync failed with 144 errors fieldMetadata: 144 errors 1..144. FIELD_MUTATION_NOT_ALLOWED: System fields cannot be deleted ``` (144 = 8 system fields × 18 custom objects, as reported on a production 2.20 instance.) ## What this PR does Adds a **2.21 workspace command** (`upgrade:2-21:backfill-system-field-is-system-side-effect`) that iterates active/suspended workspaces and flags the 8 system fields as `isSystemSideEffect: true`. - **Resolution by deterministic universal identifier**: for each object × reserved system field name it recomputes `getFieldUniversalIdentifier(applicationUID, objectUID, name)` and looks the row up in the flat maps. This is safe (and preferable to matching by `name`) because the 2.19 backfill already took over system field UIDs for every application, so an author-declared field reusing a reserved name keeps its own identifier and is never touched. An extra `isSystem` guard warn-and-skips any mismatch. - **All applications** are covered (installed apps, workspace custom app, twenty-standard): the stale flag is a function of *when* a row was provisioned, not *how*. Installed/custom apps are the acute `twenty plan` delete trap; twenty-standard has no trap today but flagging is a zero-diff no-op (`toCompare: false`) and a prerequisite for the end-state ownership invariant. - **`name` is intentionally excluded**: the 2.20 slow instance command deliberately flipped it to `false` (caller-provided default, not engine-owned); re-flagging it would undo that migration. - Supports `--dry-run`, updates only the collected rows, and invalidates the `flatFieldMetadataMaps` workspace cache after the write (a raw repository update does not invalidate it). ## Related - Resolves the pre-2.15 regression tail of twentyhq/core-team-issues#2635 - Follow-up to twentyhq/core-team-issues#2642 (system field side-effect engine migration) <!-- This is an auto-generated description by cubic. --> <a href="https://cubic.dev/pr/twentyhq/twenty/pull/22850?utm_source=github" target="_blank" rel="noopener noreferrer" data-no-image-dialog="true"><picture><source media="(prefers-color-scheme: dark)" srcset="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"><source media="(prefers-color-scheme: light)" srcset="https://www.cubic.dev/buttons/review-in-cubic-light.svg"><img alt="Review in cubic" src="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"></picture></a> <!-- End of auto-generated description by cubic. --> --------- Co-authored-by: Weiko <corentin@twenty.com> |
||
|
Thomas Trompette
|
c1b62334b7 |
fix(workflow): scope one-active-per-workflow index to workspace (#22795)
## Problem
The Phase 0 core index \`IDX_WORKFLOW_VERSION_ONE_ACTIVE_PER_WORKFLOW\`
is on \`(workflowId) WHERE status='ACTIVE'\`, with **no
\`workspaceId\`**. But \`core.workflowVersion\` is a shared multi-tenant
table, so this enforces "one active version per workflowId **globally
across all workspaces**" instead of per workspace.
The version backfill fails on staging with:
\`\`\`
duplicate key value violates unique constraint
"IDX_WORKFLOW_VERSION_ONE_ACTIVE_PER_WORKFLOW"
Detail: Key ("workflowId")=(8b213cac-...) already exists.
\`\`\`
across several different workspaces that share the same workflowId
(seeded/cloned data): workspace A's active version claims the
workflowId, and every other workspace's insert collides. Every other
index on this table includes \`workspaceId\`; this one dropped it when
copied from the per-tenant workspace entity.
## Fix
Index becomes \`(workspaceId, workflowId) WHERE status='ACTIVE'\` — one
active version per workflow **per workspace**, matching the table's
multi-tenant design and the intended invariant. New 2-20 fast instance
command drops and recreates the index (Phase 0's command is
merged/append-only).
## Test
Reset + reproduce the exact scenario against the fixed index:
- two workspaces with the same workflowId, both ACTIVE → **insert
succeeds** (previously collided)
- a second ACTIVE version for the same workflow within one workspace →
**still blocked** (invariant preserved)
Zero \`migrate:generate\` drift, typecheck + lint clean. After this
deploys, re-run \`upgrade:2-20:backfill-workflow-version-to-core\`.
<!-- This is an auto-generated description by cubic. -->
<a
href="https://cubic.dev/pr/twentyhq/twenty/pull/22795?utm_source=github"
target="_blank" rel="noopener noreferrer"
data-no-image-dialog="true"><picture><source
media="(prefers-color-scheme: dark)"
srcset="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"><source
media="(prefers-color-scheme: light)"
srcset="https://www.cubic.dev/buttons/review-in-cubic-light.svg"><img
alt="Review in cubic"
src="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"></picture></a>
<!-- End of auto-generated description by cubic. -->
|
||
|
martmull
|
23cae2040a |
Improve application asset management (#22564)
App manifests could point the logo and screenshots at either external
URLs or public folder paths, and that was handled inconsistently across
install, sync and the marketplace.
This makes assets always bundled files:
- Manifests now use `logo` and `galleryImages` (a `string[]` of public
folder paths) instead of `logoUrl` and `screenshots`. The old fields
still work but are deprecated. Gallery order comes from the array index.
Normalization (deprecated-field migration, and warning about + ignoring
external URLs) happens in `defineApplication`, so the warnings surface
at define time.
- Logo is stored as a File record (`logoFileId`).
- The registration gallery is configured via a `settings` jsonb column
on `applicationRegistration` (`{ galleryImages: string[] }`) — populated
from the manifest, read by the marketplace detail (falling back to the
legacy `screenshots` column, then the manifest). No dedicated gallery
table.
- The marketplace detail DTO and front now use `galleryImages`.
Verified against a local Postgres: the fast instance commands run with
no pending-migration diff, the schema is correct, and the server boots.
Typecheck, lint, codegen and the application unit tests pass.
Not included yet: rehosting assets into storage for npm catalog and
tarball registrations, versioned cache busting on the serving route, and
a backfill for existing installs.
<a
href="https://cubic.dev/pr/twentyhq/twenty/pull/22564?utm_source=github"
rel="nofollow noreferrer noopener" target="_blank">``<img alt="Review
in cubic"
src="https://www.cubic.dev/buttons/review-in-cubic-dark.svg">``</a>
|
||
|
Thomas Trompette
|
b327ab09a7 |
feat(workflow): add universalIdentifier + applicationId to core workflowVersion (#22747)
## workflowVersion core: syncable columns Adds `universalIdentifier` + `applicationId` (nullable) to `core.workflowVersion`, plus the FK to `core.application` and the `(workspaceId, universalIdentifier)` unique index, via a 2.20 add-columns fast command gated with `@WasIntroducedInUpgrade`. Nullable for now: the already-merged Phase A backfill (#22663) inserts version rows without these columns, so `applicationId` can't be NOT NULL yet. Flipping to NOT NULL + `extends SyncableEntity` comes once they're populated (backfill + dual-write follow-ups). Schema captured and verified via `migrate:generate` (zero drift). Independent of the core-workflow PR, but both add 2.20 upgrade commands, so this one (ts `…480`) must merge **after** the core-workflow PR (ts `…479`), or it gets re-timestamped on rebase. <!-- This is an auto-generated description by cubic. --> <a href="https://cubic.dev/pr/twentyhq/twenty/pull/22747?utm_source=github" target="_blank" rel="noopener noreferrer" data-no-image-dialog="true"><picture><source media="(prefers-color-scheme: dark)" srcset="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"><source media="(prefers-color-scheme: light)" srcset="https://www.cubic.dev/buttons/review-in-cubic-light.svg"><img alt="Review in cubic" src="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"></picture></a> <!-- End of auto-generated description by cubic. --> |
||
|
Thomas Trompette
|
6210389221 |
feat(workflow): add core workflow entity (syncable) + create-table (#22746)
## Core `workflow` entity (syncable) Part of the app-workflows work. Adds a core `WorkflowEntity extends SyncableEntity` (`name`, `lastPublishedVersionId`, plus `universalIdentifier`/`applicationId`/workspace from the base class) and its 2.20 create-table fast command, gated with `@WasIntroducedInUpgrade`. Schema was captured and verified via `migrate:generate` (zero drift, FK/index hashes correct), then run against a live DB. Backfill (populate from workspace `workflow` records) and the dual-write listener land in follow-ups. Independent of the version-syncable-columns PR, but note: both add 2.20 upgrade commands, so this one (ts `…479`) must merge **before** the version PR (ts `…480`) to satisfy the append-only guard. <!-- This is an auto-generated description by cubic. --> <a href="https://cubic.dev/pr/twentyhq/twenty/pull/22746?utm_source=github" target="_blank" rel="noopener noreferrer" data-no-image-dialog="true"><picture><source media="(prefers-color-scheme: dark)" srcset="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"><source media="(prefers-color-scheme: light)" srcset="https://www.cubic.dev/buttons/review-in-cubic-light.svg"><img alt="Review in cubic" src="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"></picture></a> <!-- End of auto-generated description by cubic. --> |
||
|
Paul Rastoin
|
60fd322b49 |
Centralize system field side effects + search field metadata (#22594)
## Introduction Closes twentyhq/core-team-issues#2635 and twentyhq/core-team-issues#2642 and twentyhq/core-team-issues#2589 Object system fields (`searchVector` + its GIN index + `searchFieldMetadata`, the reserved system fields, default relations) were provisioned through several scattered, path-specific code paths. As a result the **app-manifest sync path** authored objects with an empty/`NULL` `searchVector` and **zero `searchFieldMetadata`**, so app-owned objects shipped a broken generated search column (see #22657). The generation logic also lived partly in imperative services rather than in the metadata side-effect engine, and relied on non-deterministic (`v4`) universal identifiers that `twenty apply` could not converge, destroying manually backfilled rows. This PR centralizes every object-creation system side effect into the **metadata side-effect engine**, extends the engine to keep search metadata consistent on field delete and object relabel, makes the standard app's search identifiers deterministic, and ships upgrade commands to reconcile existing workspaces. ## What changed ### Side effects moved into the metadata side-effect engine New dedicated, self-contained handlers — so every write path (API and app manifest) gets identical results, and side effects never trigger other side effects. **Object create / delete** (`handlers/object-metadata`) * **`objectSystemFieldsOnCreate`** — generates the 7 reserved system fields (`id`, `createdAt`, `updatedAt`, `deletedAt`, `createdBy`, `updatedBy`, `position`). * **`objectSearchVectorOnCreate`** — provisions the full-text search surface as one unit: the `searchVector` `TS_VECTOR` field, its backing GIN index, and the `searchFieldMetadata` row (for searchable objects whose label identifier is a searchable field) that keeps `searchVector` populated instead of `NULL`. * **`objectSystemSideEffectsOnDelete`** — tears the above down on object deletion. **Search-metadata consistency on relabel / field delete** (new — these are what close the manifest-path gaps) * **`objectSearchVectorOnUpdate`** (`handlers/object-metadata`) — when a searchable object is relabeled onto a new searchable field, provisions the `searchFieldMetadata` row that indexes it. Relabeling is **additive**: existing rows (e.g. the provisioned `name` row) are preserved, so the previous label identifier stays searchable. Mirrors the API update path so a manifest re-sync that changes the label identifier reaches search parity. No-ops for junction objects (`id` label identifier) and non-searchable field types. * **`fieldSearchFieldMetadataOnDelete`** (`handlers/field-metadata`) — when a field is deleted, cascade-deletes every `searchFieldMetadata` row that indexes it. `searchFieldMetadata` is excluded from manifest deletion inference, so this explicit cascade is what covers **both the API and manifest paths** (the object-scoped DB cascade only fires on object deletion). Uses the `searchFieldMetadataUniversalIdentifiers` aggregator on the flat field for an O(k) lookup instead of scanning all rows. The **default `name` field and default relations are now caller-provided default fields** (SDK autocomplete on the manifest path, input transpiler on the API path) rather than system side effects — removing duplicate name generation, the imperative `build-default-*-for-custom-object` utilities, and the ad-hoc system-field integrity validator. ### Deterministic identifiers for the standard app The twenty-standard search GIN index and `searchFieldMetadata` now derive deterministic universal identifiers (`getIndexUniversalIdentifier` / `getSearchFieldUniversalIdentifier`) instead of `v4`, so `twenty apply` converges instead of recreating. ### Upgrade commands (`2-20`) to reconcile existing workspaces **Instance commands** (run once per instance; ordered fast → slow → workspace): 1. **`AddIsSystemSideEffectToSearchFieldMetadata`** (fast) — adds the `isSystemSideEffect` column to `core.searchFieldMetadata`. Defaults to `true`, which also correctly backfills every existing row since `searchFieldMetadata` is always system-derived (never user-authored). 2. **`BackfillNameFieldIsSystemSideEffect`** (slow) — re-flags existing `name` fields from `isSystemSideEffect: true` → `false`, since the default `name` field is now a caller-provided default like any other user-owned field (it was provisioned as `true` in 2.15 → 2.19). This is a pure data backfill, so the bulk `UPDATE` lives in `runDataMigration()` rather than `up()` — keeping it out of the fast schema transaction avoids holding an `ACCESS EXCLUSIVE` lock that could stall reads during the deploy. Slow instance commands still run before every workspace command of the version, so the fresh value is in place before the search-reconcile workspace commands recompute the `fieldMetadata` flat-entity cache. Scoping by name alone is safe (no engine-owned field is named `name`); `down()` is best-effort (pre-2.15 `false` rows are indistinguishable from flipped ones). **Workspace commands** (idempotent, dry-run supported): 1. **`reconcile-search-vector-gin-index-universal-identifier`** — re-owns every searchVector GIN index UID to its deterministic value (all applications), then backfills the missing GIN index for installed-app objects. 2. **`reconcile-search-field-metadata`** — re-owns every `searchFieldMetadata` UID (all applications), then backfills the missing rows for installed-app searchable objects. 3. **`rebuild-installed-app-search-vectors`** — rebuilds the `searchVector` column of every installed-app `TS_VECTOR` field, once the index and rows exist. Design notes: * **Re-own is global** (twenty-standard, workspace-custom, installed) — a UID convergence keyed on each row's own application. * **Backfill is installed-app only** — standard/custom objects already have these rows via the manifest funnel. * Re-own runs **before** backfill and is transaction-guarded; a failure aborts that workspace to avoid a unique-identifier collision. ## Tests * Integration: app manifest sync now asserts system fields + searchable objects (searchVector, GIN index, searchFieldMetadata) are created; a new relabel suite drives three manifest syncs and asserts records stay searchable through the old + new label identifiers and lose searchability when a field is removed; removed the obsolete system-fields-integrity suite/snapshots. * Unit: per-handler side-effect specs (including the new `objectSearchVectorOnUpdate` and `fieldSearchFieldMetadataOnDelete` handlers), and per-util specs for the re-own / backfill operation builders and the GIN-index classifier. ## Upgrade / migration notes * Existing workspaces converge on the next upgrade run via the `2-20` instance + workspace commands (idempotent, dry-run supported). * Backfill and rebuild go through the workspace-migration runner (automatic cache invalidation); the re-own step invalidates only the affected flat-entity maps directly. * The cross-version upgrade CI now flushes the cache before running the upgrade, so the new version recomputes every flat-entity map from the database instead of reading blobs the old version serialized in an older shape. ## Follow-up * `object-metadata.service.ts` still carries a `TODO: remove once default view fields move to the metadata side effect engine` — default view fields are the next candidate to move into the engine. * A single manifest sync cannot yet both create a field and relabel the object onto it, because `objectMetadata.update` is ordered before `fieldMetadata.create` in the migration runner. Tracked in twentyhq/core-team-issues#2655; to be fixed in a follow-up. |
||
|
Thomas Trompette
|
ca90a9358f |
feat(workflow): backfill workspace workflowVersion into core (phase A) (#22663)
## workflowVersion -> core, Phase A Follows #21674 (Phase 0, merged). Base: `main`. Populates core `workflowVersion` and keeps it in sync with the workspace object, so a later phase can switch reads to core. Reads stay on the workspace object in this PR. ### 1. Backfill (upgrade command) `BackfillWorkflowVersionToCoreCommand`, a `@RegisteredWorkspaceCommand('2.20.0', ...)`. Per workspace, reads all workspace `workflowVersion` records and upserts them into core, preserving ids (idempotent), dry-run aware. ### 2. Dual-write (always on, not flag-gated) `WorkflowVersionCoreDualWriteListener` hooks `@OnDatabaseBatchEvent('workflowVersion', CREATED/UPDATED/DELETED)` (same mechanism as the existing workflow-version status listener) and mirrors every mutation into core. Sync failures are logged, never break the user's write; drift is repaired by re-running the backfill command. Dual-write is deliberately not behind a flag: reading from core (next phase) is only safe if core has been continuously in sync since the backfill. An always-on mirror makes "core is fresh" an invariant, so the read switch becomes a plain flag flip. The cost is one extra upsert on infrequent workflowVersion writes. `IS_WORKFLOW_VERSION_IN_CORE_ENABLED` is reserved for the read switch (Phase B): dispatch from the `workflowAutomatedTriggerMaps` cache, runner and builder reading trigger/steps from core. Until then it gates nothing. Both the backfill and the listener go through a single `WorkflowVersionCoreSyncService` (`upsertToCore`/`deleteFromCore`): the workspace-to-core mapping (`trigger` -> `triggers[]`, plus `steps`, `status`, `workflowId`) and `workflowAutomatedTriggerMaps` invalidation live in one place. ### Rollout plan (following phases) - **B, read switch (flag per workspace):** reads move to core; writes keep flowing workspace -> listener -> core. Rollback = flip the flag back, workspace never stopped being source of truth. - **C, contract (code change):** write paths write trigger/steps to core directly; workspace `workflowVersion` stays as a thin shell (nav/relations/search) but drops the trigger/steps columns; listener and flag removed. ### Not in this PR - Reconciliation tooling beyond re-running the backfill. - The read switch (Phase B). |
||
|
martmull
|
0ea0c23556 |
refactor(app-marketplace): rename featured to vetted (#22674)
## What Renames the application-registration "featured" flag to "vetted" across the backend, frontend, GraphQL schema/DTOs, and the marketplace UI. "Vetted" better describes what the flag actually does today: it marks an app as reviewed and approved by the Twenty team (a trust signal), rather than "featured" which reads as spotlighting/promotion. The admin toggle description was already "Mark this app as reviewed and approved". ## How - Renamed `isFeatured` -> `isVetted` on the `ApplicationRegistration` entity, DTOs (`MarketplaceApp`, `MarketplaceAppDetail`, `UpdateApplicationRegistrationPayload`), services, GraphQL fragments, and the settings/admin UI (labels: "Featured" -> "Vetted", "Featured only" -> "Vetted only", etc.). - Renamed the `MARKETPLACE_FEATURED_APPLICATIONS` constant/file to `MARKETPLACE_VETTED_APPLICATIONS`. - Regenerated GraphQL client artifacts (`generated-metadata`, `generated-admin`, `twenty-client-sdk`). ### Database The `isFeatured` column is renamed in place to `isVetted` via a single 2.20 fast instance command (`ALTER TABLE ... RENAME COLUMN`). No new column, no data-copy backfill. - Since all 2.19 commands (including the existing `isFeatured` backfill) complete before any 2.20 command runs, the rename carries over the values that backfill set. - The entity uses `@WasRenamedInUpgrade` so the upgrade-aware layer queries the old column name until the rename step runs during an upgrade. ## Testing - `nx typecheck` and `nx lint:diff-with-main` pass for twenty-server and twenty-front. - Ran `database:reset` on a fresh dev DB: the 2.19 `isFeatured` backfill runs first, then the 2.20 rename; the column ends up as `isVetted` (and `isFeatured` no longer exists), values preserved. - Booted the server: the `@WasRenamedInUpgrade` decorator validates against the upgrade sequence, and GraphQL introspection confirms all four types expose `isVetted` and none expose `isFeatured`. - Ran the three `graphql:generate` configs and the SDK metadata client generator so the committed generated files match the generator output (field ordering included). ## Notes - The `api-breaking-changes` check flags the removal of the `isFeatured` GraphQL field — that is expected and inherent to this rename. - Translation catalogs (`locales/`) are intentionally not touched here since they are managed via Crowdin; new English strings render via Lingui's default-message fallback until translated. |
||
|
neo773
|
674de0056b |
feat(messaging): message campaign delivery stats + views (#22661)
Re-land of #22452 (reverted in #22627). Rebuilt on fresh main with upgrade commands isolated to 2-20 only; no other version's commands touched. <!-- This is an auto-generated description by cubic. --> <a href="https://cubic.dev/pr/twentyhq/twenty/pull/22661?utm_source=github" target="_blank" rel="noopener noreferrer" data-no-image-dialog="true"><picture><source media="(prefers-color-scheme: dark)" srcset="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"><source media="(prefers-color-scheme: light)" srcset="https://www.cubic.dev/buttons/review-in-cubic-light.svg"><img alt="Review in cubic" src="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"></picture></a> <!-- End of auto-generated description by cubic. --> |
||
|
Thomas Trompette
|
48730df0d2 |
feat(workflow): scaffold core workflowVersion entity + trigger cache (phase 0) (#21674)
## What **Phase 0 (scaffold)** of migrating `workflowVersion` data to **core**. Gated by `IS_WORKFLOW_VERSION_IN_CORE_ENABLED` with **no behavior change** — nothing reads or writes the new core entity yet. ## Plan `workflowVersion` becomes a thin **workspace shell** over a core entity (the `dashboard`/`pageLayout` pattern), so navigation, the metadata relations, and the record UI keep working while the heavy data (`triggers`, `steps`) lives in core. Trigger dispatch will derive from active core versions via a per-workspace cache, letting us **eliminate** the denormalized `workflowAutomatedTrigger` object. `workflow` and `workflowRun` stay as workspace objects. Phases: **0 — scaffold (this PR)** → A — backfill + dual-write → B — switch reads to core → C — drop the workspace `trigger`/`steps` columns + the `workflowAutomatedTrigger` object. ## Included - **Core `WorkflowVersionEntity`** (`extends WorkspaceRelatedEntity`) — stores version data, with triggers as an **array** (`triggers: WorkflowTrigger[]`), a long-due shape change. Storage only: dispatch reads the primary trigger, so behavior stays single-trigger for now. - **Fast create-table instance command** for `core."workflowVersion"` (v2.19.0). - **`IS_WORKFLOW_VERSION_IN_CORE_ENABLED`** feature flag. - **Per-workspace automated-trigger cache provider** deriving CRON/DATABASE_EVENT dispatch from the active version's trigger — groundwork for removing `workflowAutomatedTrigger`. ## Notes - `WorkspaceRelatedEntity`, **not** `SyncableEntity`: this is user runtime data (like `connectedAccount`/`apiKey`/`file`), not application-manifest metadata. - No frontend behavior; the generated `FeatureFlagKey` enums are updated to include the new flag. |
||
|
Etienne
|
4b9f393167 |
fix: add workspace command to backfill missing AGENT source enum values (#22593)
**What** Adds a 2.19 workspace upgrade command that backfills missing FieldActorSource enum values (AGENT) into the Postgres enums backing every ACTOR composite field (createdBy, updatedBy) across all existing workspaces. Each ACTOR field stores its source sub-field as a Postgres enum scoped to its own table and schema (e.g. workspace_abc.company_createdBySource_enum). Workspaces created before these enum values were introduced are missing them, which causes runtime errors when those sources are used. **How** buildActorSourceEnumBackfillTargets collects all ACTOR fields from the workspace metadata cache and maps each enum sub-property to its (tableName, columnName, enumName, expectedValues) tuple. Before iterating over all targets, the command performs a single fast pg_catalog.pg_enum lookup on company.createdBySource as a representative sentinel. If AGENT is already present there, the workspace is skipped entirely (idempotency fast-path). For each remaining target, ALTER TYPE … ADD VALUE IF NOT EXISTS is issued per missing value via WorkspaceSchemaEnumManagerService.addEnumValue, making the command fully idempotent and safe to re-run. Fixes https://discord.com/channels/1130383047699738754/1522507190949118003 <!-- This is an auto-generated description by cubic. --> <a href="https://cubic.dev/pr/twentyhq/twenty/pull/22593?utm_source=github" target="_blank" rel="noopener noreferrer" data-no-image-dialog="true"><picture><source media="(prefers-color-scheme: dark)" srcset="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"><source media="(prefers-color-scheme: light)" srcset="https://www.cubic.dev/buttons/review-in-cubic-light.svg"><img alt="Review in cubic" src="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"></picture></a> <!-- End of auto-generated description by cubic. --> |
||
|
martmull
|
b733a79821 |
feat(server): support server-scoped files via nullable workspaceId on file table (#22587)
Part of the app settings architecture cleanup (twentyhq/core-team-issues#2456) — PR 1 of the server-level documents plan, reworked after the revert of #22560 (#22579). Same capability, different shape: **no new entity** — server-level documents live in the existing `file` table with a nullable `workspaceId`. ## Problem All file storage is workspace-scoped (`FileEntity.workspaceId NOT NULL`, `{workspaceId}/{app}/…` storage keys). Server-level data like application-registration manifests and tarballs for ownerless catalog registrations has no first-class home, forcing raw-driver bypasses (`DefaultAiCatalogService`, prototype #22556). ## Changes (core storage layer only — no HTTP serving, no GraphQL exposure) **`FileEntity` gains server scope** (mirrors `KeyValuePairEntity`, which already supports both instance-level and per-workspace rows): - `workspaceId` uuid becomes **nullable** — NULL means server-scoped; the entity no longer extends `WorkspaceRelatedEntity` and declares its columns directly - `applicationRegistrationId` nullable FK (`onDelete: CASCADE`) — registration-owned documents follow their registration - ownership checks: `workspaceId IS NOT NULL OR applicationRegistrationId IS NOT NULL` and `workspaceId IS NULL OR applicationRegistrationId IS NULL` — every row has exactly one owner - `IDX_FILE_APPLICATION_REGISTRATION_ID_PATH_UNIQUE` UNIQUE (`applicationRegistrationId`, `path`) — mirrors the workspace unique-constraint pattern; workspace rows are exempt via their NULL `applicationRegistrationId` **New `ServerFileStorageService`** (`file-storage/services/`, exported from the global `FileStorageModule`; `FileStorageService` moved alongside it): - storage keys `server/{fileFolder}/{applicationRegistrationId}/{resourcePath}` — the registration segment is injected by the service itself, so paths cannot collide across registrations; scope-validation util mirroring `validateStoragePathIsWithinWorkspaceOrThrow`; new `ServerFileFolder` enum in twenty-shared - `writeServerFile` (upsert on (`applicationRegistrationId`, `path`) + driver write; throws on failure), `readServerFile`/`readServerFileById` (missing row or bytes surfaces `FILE_NOT_FOUND`), `checkServerFileExists`, `deleteServerFile`/`deleteByServerFileId` (bytes best-effort, row authoritative), `deleteByApplicationRegistrationId` - rows are accessed through a plain repository pinned to `workspaceId: IsNull()` on every query; workspace-file code paths still go through `WorkspaceScopedRepository`, which never sees NULL rows **Null-safety ripples** (workspaceId is now `string | null`): - `WorkspaceScopedEntity` bound widened to `workspaceId: string | null` (the wrapper always filters with a concrete id) - `list-and-delete-orphaned-workspace-entities` now skips `workspaceId IS NULL` rows — previously `NOT EXISTS` would have flagged server rows as orphans and deleted them - `PendingFileCleanupService` sweeps only `workspaceId IS NOT NULL` rows; `application-package-fetcher` pins its tarball lookup to workspace rows (tarball migration to server scope is a follow-up PR) **Migration**: `allow-server-scoped-file` ships as a **2-20 fast instance command** (2.20.0 is current since #22639; re-slotted from 2-19 per review). Command runs are tracked by name, so instances that already executed the 2-20 `standardOverrides` drop command still pick this one up. Its realistic timestamp sorts before that drop command's fabricated `1825000000000`, which the `ci:allow-upgrade-command-timestamp-exception` label covers. ## Next PRs in the plan - PR 2: HTTP serving + token type for server files - PR 3: application-registration manifests stored as versioned server files (rework of draft #22556) - PR 4 (optional): registration tarballs migrate to server scope ## Verification - New spec `server-file-storage.service.spec.ts` (traversal table, upsert conflict semantics, row-before-bytes reads, best-effort byte deletion, registration cascade) + scope-validation util spec; affected suites all green - Typecheck (server + shared), `lint:diff-with-main`, full `oxfmt --check src/` on both packages clean - Fresh `database:reset` on the re-slotted branch: the 2-20 command executes, generator then reports **no schema drift**; both ownership checks and the composite unique verified live (dual-owner insert and duplicate registration+path both rejected) |
||
|
martmull
|
435073e9c5 |
Display featured applications in marketplace (#22635)
## After <img width="1060" height="589" alt="image" src="https://github.com/user-attachments/assets/74dfadcf-8698-4404-81c6-b309cc4cbf79" /> <img width="732" alt="image" src="https://github.com/user-attachments/assets/0e1a3644-04bc-4208-aa77-3842d9db9cc8" /> <img width="797" alt="image" src="https://github.com/user-attachments/assets/0456ecce-607a-4705-8a89-c77029bfb6ac" /> - Remove IS_MARKETPLACE_SETTING_TAB_VISIBLE feature flag - add vetted toggle in admin app tab - added people data labs, last contact and call recorder to default vetted applications <!-- This is an auto-generated description by cubic. --> <a href="https://cubic.dev/pr/twentyhq/twenty/pull/22635?utm_source=github" target="_blank" rel="noopener noreferrer" data-no-image-dialog="true"><picture><source media="(prefers-color-scheme: dark)" srcset="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"><source media="(prefers-color-scheme: light)" srcset="https://www.cubic.dev/buttons/review-in-cubic-light.svg"><img alt="Review in cubic" src="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"></picture></a> <!-- End of auto-generated description by cubic. --> --------- Co-authored-by: prastoin <paul@twenty.com> |
||
|
Paul Rastoin
|
bd8bf89653 |
Revert "feat(messaging): message campaign delivery stats + views" (#22452) (#22627)
Revert "feat(messaging): message campaign delivery stats + views
(#22452)"
This reverts commit
|
||
|
neo773
|
2e1117d442 |
feat(messaging): message campaign delivery stats + views (#22452)
<!-- This is an auto-generated description by cubic. --> <a href="https://cubic.dev/pr/twentyhq/twenty/pull/22452?utm_source=github" target="_blank" rel="noopener noreferrer" data-no-image-dialog="true"><picture><source media="(prefers-color-scheme: dark)" srcset="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"><source media="(prefers-color-scheme: light)" srcset="https://www.cubic.dev/buttons/review-in-cubic-light.svg"><img alt="Review in cubic" src="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"></picture></a> <!-- End of auto-generated description by cubic. --> |
||
|
Paul Rastoin
|
6c40c7b91a |
Deterministic system field universal identifier (#22565)
# Introduction Close twentyhq/core-team-issues#2641 Auto-provisioned field metadata used to get its `universalIdentifier` from three unrelated sources: random `v4()` on the server when creating custom objects, hardcoded values in `STANDARD_OBJECTS`, and an ad-hoc `v5` derivation in the SDK manifest build. This PR unifies all of them behind the shared `getFieldUniversalIdentifier` derivation: ``` universalIdentifier = f(applicationUniversalIdentifier, objectUniversalIdentifier, fieldName) ``` ## Ownership model The rollout is built on an explicit split of who owns a field's universal identifier: - **The 8 system fields** (`id`, `createdAt`, `updatedAt`, `deletedAt`, `createdBy`, `updatedBy`, `position`, `searchVector`) are **server-owned**. Their universal identifiers are always the deterministic derivation, on **every** application (standard, workspace-custom, installed). Clients cannot provide custom values: a temporary check in `validateObjectMetadataSystemFieldsIntegrity` rejects any non-derived system field identifier at migration build time. This check stands in until system fields are generated exclusively server side by the metadata side-effect engine and stripped from client inputs — at which point it becomes structurally impossible to send one. - **`name` is a default field, not a system field**: it is auto-provisioned when absent (server side for custom objects, SDK side for application objects) but authors can define their own. It is only derived where it is guaranteed to be auto-provisioned. In particular, standard objects keep their **historical hardcoded** `name` identifiers: the standard app authors its `name` fields like any installed app would, and moving those identifiers would break every installed application referencing them (e.g. views on `opportunity.name`). - **User-created and author-provided fields** keep random / explicit identifiers, untouched. ## Server - `validateObjectMetadataSystemFieldsIntegrity` now validates, on top of the existing type/`isSystem` checks, that each system field's `universalIdentifier` equals the deterministic derivation. Runs for every object creation going through the migration orchestrator: app sync, custom object creation, standard provisioning - `build-default-flat-field-metadatas-for-custom-object.util.ts` derives the system field identifiers (and the auto-provisioned `name`) with `getFieldUniversalIdentifier` instead of `v4()` - `build-default-relation-flat-field-metadatas-for-custom-object.util.ts` derives both the forward and the reverse default relation field identifiers deterministically - `generateMorphOrRelationFlatFieldMetadataPair` accepts optional `sourceFieldUniversalIdentifier` / `targetFieldUniversalIdentifier` so callers can inject deterministic values; user-created relations still default to `v4()` ## twenty-shared - `STANDARD_OBJECTS` system field identifiers (the 8) are now computed at module load via `buildStandardObjectSystemFields`; `name` and every other identifier keep their hardcoded values - New snapshot test pinning **every** universal identifier of `STANDARD_OBJECTS`: any identifier change now requires an explicit snapshot update and should ship with a coordinated backfill ## SDK (breaking, pre-GA) - `generateDefaultFieldUniversalIdentifier` delegates to `getFieldUniversalIdentifier` and now requires `applicationUniversalIdentifier` - Reverse default relation field identifiers are derived from the field's real coordinates (standard object UID + actual field name, e.g. `targetRocket` on `attachment`) instead of the legacy custom-object UID + synthetic `${fieldName}Inverse` hash input. Field *names* are unchanged - The manifest build threads the application universal identifier through default field injection (two-pass over object configs) - `twenty dev:add` now resolves the application universal identifier upfront and refuses to scaffold anything until `defineApplication` declares one — no more `fill-later` placeholder for the app UID in generated files ## Upgrade A 2.19 **workspace command** backfills existing `fieldMetadata.universalIdentifier` rows to the deterministic derivation. Coverage follows the ownership model: - **The 8 system fields**: taken over for **every application**, whatever value they currently hold. This is both safe and required now that sync rejects non-derived values — leaving a row unconverged would make its application unsyncable - **`name`**: workspace-custom app → always taken over (server-generated, no author to clobber); installed applications → only rows still carrying the legacy SDK derivation are recomputed, author-provided identifiers are never touched; standard app → never touched (hardcoded in `STANDARD_OBJECTS`) - **Default relation fields**: workspace-custom app → forward fields on custom objects and reverse fields on the standard relation objects; installed applications → legacy-derivation probe only All identifiers of a workspace are updated inside a single transaction, then the command flushes the field-metadata-related workspace caches and bumps the metadata version. Stored `applicationRegistration.manifest` snapshots are intentionally **not** rewritten: installs and upgrades always sync from the `manifest.json` inside the resolved package (npm/tarball), the stored column is only used for display/marketplace purposes. ## Breaking behavior for old packages (fail closed) Packages built with an older SDK carry legacy system field identifiers in their tarball `manifest.json`. Installing or upgrading such a package now fails with an explicit `INVALID_SYSTEM_FIELD` validation error ("universal identifier is not deterministic") instead of silently mismatching against the backfilled rows and triggering a destructive delete+create. The remediation is to rebuild the package with the new SDK; the backfill has already converged the installed rows, so the rebuilt manifest syncs cleanly. ## Test plan - [x] `twenty-sdk` unit tests (526 tests) and typecheck - [x] `twenty-shared` unit tests (1635 tests) including the `STANDARD_OBJECTS` snapshot; `name` identifiers verified byte-for-byte identical to `main` - [x] Lint and typecheck clean on all touched packages - [x] Integration: create a custom object and verify system + default relation field identifiers match the deterministic derivation (`create-one-object-metadata-deterministic-field-universal-identifiers`, 13 assertions passing) - [x] Integration: `failing-sync-application-object-system-fields` extended with a non-derived system field identifier case; all identifiers in the spec pinned deterministically so snapshots embedding expected/actual values are stable across runs (verified with a double run) - [x] Integration: all application sync suites pass with the derived system field identifiers now required by the `buildDefaultObjectManifest` test helper (9 suites, 20 tests) - [x] Full test-database reset: standard app provisioning and seeded workspaces pass the new validation - [x] SDK manifest build verified on the postcard example app: all auto-generated default field identifiers match the derivation - [ ] Run `upgrade:2-19:backfill-deterministic-field-universal-identifiers` (dry-run then real) on a seeded workspace and verify identifier convergence with a rebuilt app manifest |
||
|
martmull
|
2327ae7122 |
Revert "feat(server): add instance-level file storage layer" (#22579)
Reverts twentyhq/twenty#22560 <!-- This is an auto-generated description by cubic. --> <a href="https://cubic.dev/pr/twentyhq/twenty/pull/22579?utm_source=github" target="_blank" rel="noopener noreferrer" data-no-image-dialog="true"><picture><source media="(prefers-color-scheme: dark)" srcset="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"><source media="(prefers-color-scheme: light)" srcset="https://www.cubic.dev/buttons/review-in-cubic-light.svg"><img alt="Review in cubic" src="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"></picture></a> <!-- End of auto-generated description by cubic. --> |
||
|
martmull
|
0baf213fa4 |
feat(server): add instance-level file storage layer (#22560)
Part of the app settings architecture cleanup
(twentyhq/core-team-issues#2456) — PR 1 of the instance-level documents
plan. Today all file storage is workspace-scoped
(`FileEntity.workspaceId NOT NULL`, `{workspaceId}/{app}/…` storage
keys, workspace-anchored tokens); instance-level data like
application-registration manifests and tarballs for ownerless catalog
registrations has no first-class home, forcing raw-driver bypasses
(`DefaultAiCatalogService`, prototype #22556).
## Changes (core storage layer only — no HTTP serving, no GraphQL
exposure)
**New `instanceFile` table** (`InstanceFileEntity`) — deliberately
separate from the workspace-scoped `file` table so nothing about the
existing system changes:
- `id`, `path` (unique, `{fileFolder}/{relativePath}` mirroring
FileEntity's convention), `size`, `mimeType`, timestamps
- nullable `applicationRegistrationId` FK (`onDelete: CASCADE`) —
registration-owned documents follow their registration
- no `workspaceId`, no `applicationId`; plain repository (added to the
`prefer-workspace-scoped-repository` lint rule's global-table
exemptions, as the rule's own message directs)
**New `InstanceFileStorageService`** (exported from the global
`FileStorageModule`):
- storage keys under a literal `instance/{fileFolder}/…` prefix —
collision-free with workspace prefixes (UUIDs); scope-validation util
mirroring `validateStoragePathIsWithinWorkspaceOrThrow`
- `writeInstanceFile` (upsert row on `path` conflict + driver write;
throws on failure — no swallowing),
`readInstanceFile`/`readInstanceFileById` (missing file surfaces
`FILE_NOT_FOUND` like `FileStorageService.readFile`),
`checkInstanceFileExists`, `deleteInstanceFile`/`deleteByInstanceFileId`
(bytes best-effort, row authoritative),
`deleteByApplicationRegistrationId` (lifecycle hook for
registration-owned files)
- same driver path as `FileStorageService` (`FileStorageDriverFactory` →
`ValidatedStorageDriver`)
**Migration**: fast instance command `add-instance-file-table` (2.19,
generator-produced; post-command `database:migrate:generate` reports no
pending changes).
## Next PRs in the plan
- PR 2: HTTP serving + token type for instance files (new route + guard;
workspace file endpoints untouched)
- PR 3: application-registration manifests stored as versioned instance
files (supersedes draft #22556)
- PR 4 (optional): registration tarballs migrate to instance scope,
removing the cross-workspace `FileEntity` read in
`application-package-fetcher` and the `ownerWorkspaceId` requirement on
`uploadTarball`
## Verification
- New specs: scope-validation util (traversal cases) + service (upsert
conflict, missing-file error, best-effort byte deletion, registration
cascade) — 16/16; `npx jest "application"` still 31 suites / 160 green
- Typecheck, `lint:diff-with-main`, full `oxfmt --check src/` (6421
files) and full type-aware oxlint clean
- Fast command executed against the local DB — table, unique index, and
CASCADE FK verified via psql; generator then reports no schema drift
- Server boots with the new provider; `generate-metadata-client
--skip-nx-cache` zero diff (no GraphQL change)
---
_Generated by [Claude
Code](https://claude.ai/code/session_011sST4rPLU1Koi2oVGi84ei)_
<!-- This is an auto-generated description by cubic. -->
<a
href="https://cubic.dev/pr/twentyhq/twenty/pull/22560?utm_source=github"
target="_blank" rel="noopener noreferrer"
data-no-image-dialog="true"><picture><source
media="(prefers-color-scheme: dark)"
srcset="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"><source
media="(prefers-color-scheme: light)"
srcset="https://www.cubic.dev/buttons/review-in-cubic-light.svg"><img
alt="Review in cubic"
src="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"></picture></a>
<!-- End of auto-generated description by cubic. -->
|
||
|
martmull
|
e5e3fadbbb |
feat(files): reap stale pending direct-upload files via hourly cron (#22531)
## Context Follow-up to #22449 (direct-to-storage upload endpoints). That PR introduced the `PENDING` → `UPLOADED` file lifecycle: `createFileUpload` inserts a file record in `PENDING`, the client uploads the bytes directly to storage, then `completeFileUpload` flips it to `UPLOADED`. A client that initiates an upload but never confirms — a crash, a closed tab, an expired presigned URL — leaves a `PENDING` file record and a possibly-partial storage object behind forever. This PR reaps them. ## What this does Adds an hourly cron that hard-deletes `PENDING` files older than 24h together with their storage objects, in bounded batches. - **`PendingFileCleanupService`** — finds `PENDING` files with `createdAt` older than `PENDING_FILE_MAX_AGE_MS` (24h), capped at `PENDING_FILE_CLEANUP_BATCH_SIZE` (200) per run, and deletes each via `FileStorageService.deleteByFileId` (which tolerates a missing object). A failure on one file is logged and skipped so the rest of the batch still gets cleaned. - **`PendingFileCleanupCronJob`** — `@Processor(cronQueue)` job that runs the service and reports exceptions. - **`PendingFileCleanupCronCommand`** — registers the job on the hourly pattern (`0 * * * *`). - Wired into `FileUploadModule` (providers + export) and registered in `cron:register:all`. ### Why 24h The reaper threshold sits well past the presigned URL expiry, so a `PENDING` file only becomes reapable long after any legitimate in-flight upload could still complete — the cleanup can never race a real upload. A file that was never confirmed is referenced by nothing; the client recovery path is simply re-uploading under a fresh `fileId`, so we never promote to `UPLOADED`. ## Tests `pending-file-cleanup.service.spec.ts` covers: the query shape (status + age threshold + batch cap), deleting each stale file and returning the count, continuing past a per-file deletion failure, and the empty-batch no-op. ## Scope Server-only, non-breaking, no user-facing change. Part of the incremental direct-upload rollout being split into small PRs. https://claude.ai/code/session_015UH8KWmsB9zdYaog8MFG1d --- _Generated by [Claude Code](https://claude.ai/code/session_015UH8KWmsB9zdYaog8MFG1d)_ <!-- This is an auto-generated description by cubic. --> <a href="https://cubic.dev/pr/twentyhq/twenty/pull/22531?utm_source=github" target="_blank" rel="noopener noreferrer" data-no-image-dialog="true"><picture><source media="(prefers-color-scheme: dark)" srcset="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"><source media="(prefers-color-scheme: light)" srcset="https://www.cubic.dev/buttons/review-in-cubic-light.svg"><img alt="Review in cubic" src="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"></picture></a> <!-- End of auto-generated description by cubic. --> |
||
|
martmull
|
d6b6962604 |
feat(files): content-verify direct uploads and pin pending files to octet-stream (#22533)
## Context Follow-up to #22449 (direct-to-storage upload endpoints). In that flow `createFileUpload` inserts a `PENDING` file record before any bytes exist, and until now it guessed the mime type from the **filename extension** — an untrusted, client-controlled value. This PR makes a pending file opaque and only trusts a mime type that was verified against the actual stored bytes. ## What this does **1. A pending file is always `application/octet-stream`.** `createFileUpload` records the pending file — and signs the presigned PUT — as `application/octet-stream`. The extension is still kept on the stored object name so the content can be checked against it later. **2. Content verification at completion.** `completeFileUpload`, after the existing size check, reads a **bounded prefix** of the stored object (`readReadablePrefix`, capped at 64 KiB — a large object is never buffered in full) and runs the existing `extractFileInfoOrThrow` util to detect the real mime type from the content. It: - writes the detected type alongside `status = UPLOADED`, and - rejects a file whose bytes don't match its declared extension (the record stays `PENDING`, so it can never be served or attached, and is reaped by the pending-file cleanup cron). Serving already overrides `Content-Type` from the DB record, so storing the object as octet-stream is fine. **3. A database constraint as backstop.** `CHK_FILE_PENDING_MIME_OCTET_STREAM` — `"status" != 'PENDING' OR "mimeType" = 'application/octet-stream'` — added to `FileEntity` and applied by a fast instance command (`2-19`). It is added `NOT VALID` on purpose: an instance freshly upgraded past #22449 may still hold `PENDING` rows whose mime came from the old extension-guess path, and `NOT VALID` enforces the invariant on every new/updated row without failing on that legacy backlog (those rows get overwritten to octet-stream when completed — `status` flips to `UPLOADED`, so the check passes — or are reaped while pending). ## Tests - `read-readable-prefix.spec.ts` — prefix reader: short source, early stop on a large source (asserts it tears the stream down without draining it), error propagation, empty stream. - `file-upload.service.spec.ts` — create records octet-stream; complete sniffs and sets the detected type, overrides a spoofed extension with the real content type, and rejects content that can't be matched to the declared extension. - `direct-file-upload.integration-spec.ts` — end-to-end case rejecting a `.png` upload whose bytes are plain text. ## Verification `typecheck` green, `lint:diff-with-main` clean, unit suites pass (17 tests). No GraphQL schema change, so no codegen drift. ## Scope Server-only, part of the incremental direct-upload rollout being split into small PRs. Independent of the reaper-cron PR (#22531). https://claude.ai/code/session_015UH8KWmsB9zdYaog8MFG1d --- _Generated by [Claude Code](https://claude.ai/code/session_015UH8KWmsB9zdYaog8MFG1d)_ <!-- This is an auto-generated description by cubic. --> <a href="https://cubic.dev/pr/twentyhq/twenty/pull/22533?utm_source=github" target="_blank" rel="noopener noreferrer" data-no-image-dialog="true"><picture><source media="(prefers-color-scheme: dark)" srcset="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"><source media="(prefers-color-scheme: light)" srcset="https://www.cubic.dev/buttons/review-in-cubic-light.svg"><img alt="Review in cubic" src="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"></picture></a> <!-- End of auto-generated description by cubic. --> |
||
|
Paul Rastoin
|
43730d7748 |
Centralized side effects devxp basis (#22295)
# Introduction
This PR introduces a centralized, strictly-typed **metadata side-effect
engine** that unifies how system metadata side effects are derived and
applied across both metadata entry points — the **metadata GraphQL API**
and the **application sync / manifest** flow — and migrates the first
side effect end-to-end: **a unique scalar field owns its backing
single-field `UNIQUE` index** (full create / update / delete lifecycle).
## New conventions
- **Engine-owned companions**: metadata flagged `isSystemSideEffect:
true` is owned by the engine. Its deletion is never inferred from
absence in a manifest — it results from PG-level cascade or from a
delete side effect (a side effect always has a cause, its parent
metadata).
- **Reserved deterministic identifiers**: apps cannot declare metadata
reusing an engine-owned deterministic `universalIdentifier`. Doing so
fails validation with `RESERVED_SYSTEM_UNIVERSAL_IDENTIFIER` (until an
explicit override API exists).
- **Record-native operation matrix**: the operation matrix is keyed by
`universalIdentifier` (`AllFlatEntityOperationRecordByMetadataName`)
instead of arrays, making parent resolution and deduplication O(1).
Array-based API callers are transpiled to records at the
validate-build-and-run boundary.
- Twenty-sdk user-facing experience with system fields will only be
related to overrides.
# What this PR does
## 1. Side-effect engine (foundation)
- `MetadataSideEffectEngineService.expandWithSideEffects(...)` takes the
intention-carrying record matrix and returns it expanded with derived
side effects, or a structured failure.
- Handlers are registered via a typed **decorator + registry** pattern
(`MetadataSideEffectHandler({ operation, metadataName, name, description
})`), with runtime duplicate-name detection. Multiple handlers per
(operation, metadataName) are supported.
- Handler contract mirrors the validator pattern:
- receives the trigger flat entity, the live record matrix, and
**strictly-typed related flat entity maps**
(`MetadataFlatEntityAndRelatedFlatEntityMapsForSideEffect<P>`, derived
from declared companion metadata names — no loose
`Partial<AllFlatEntityMaps>` context)
- returns `MetadataSideEffectResult`: `success` (operations record) |
`noop` | `fail` (structured failure)
- **Non-recursion is structural**: triggers are read from the original
caller input, never from the expanded matrix, so a side effect can never
trigger another side effect.
- **Deduplication + collision detection**: side effects are deduped by
`universalIdentifier` per operation; a caller-declared entity colliding
with an engine-owned deterministic identifier is recorded as a
collision.
- **Unified failure channel**: handler failures and reserved-identifier
collisions are merged into the same `OrchestratorFailureReport` contract
as builder validation errors, and the run short-circuits (fail-closed,
nothing is applied).
## 2. First migrated side effect — unique field → backing unique index
Three handlers own the complete lifecycle of the deterministic
single-field `UNIQUE` index backing a unique scalar field:
- **create**: unique scalar field → generate the deterministic backing
index (`fieldUniqueBackingIndexOnCreate`)
- **update**: `isUnique` flips and renames of still-unique fields (the
index name — and therefore its deterministic identifier — derives from
the field name, so a rename drops the stale index and recreates the
deterministic one) (`fieldUniqueBackingIndexOnUpdate`)
- **delete**: cascade-delete the backing index
(`fieldUniqueBackingIndexOnDelete`)
Supporting rules:
- The primary key `id` field never spawns a backing index (uniqueness
comes from the PK constraint) — explicit `isPrimaryKeyFlatFieldMetadata`
guard.
- Parent object resolution is **optimistic-first**: an object created or
updated in the same batch wins over the workspace cache (so e.g.
renaming an object while flipping a field to unique builds the index
from the post-rename object), resolved in O(1) via the record matrix.
- A missing parent object is reported as a structured side-effect
failure, never silently skipped.
## 3. Path convergence — manifest and API share one flow
- The manifest sync now derives a from→to **record matrix** from the
cache and feeds `validateBuildAndRunWorkspaceMigrationFromRecord`, the
same flow the API uses — both paths converge on the engine.
- Manifest-side unique-index generation and API transpiler
system-unique-index handling were removed (declared/composite/relation
indexes stay untouched).
- New `WorkspaceMigrationFlatEntityMapsService` mutualizes
flat-entity-maps computation between the side-effect engine and the
builder: cache keys are derived from the caller metadata names (+
validation- and side-effect-related closures) instead of hardcoded
loads.
- App-scoping and pruning are folded into one shared primitive
(`getSubAllFlatEntityMapsByApplicationIdsOrThrow`): slicing dependency
maps to the involved applications always prunes dangling one-to-many
aggregators — callers can no longer forget it.
- **Behavior change**: an app extending another app's view with a view
field now syncs successfully (cross-app view-field extension), covered
by a dedicated integration test.
## 4. Backfill upgrade command (2.19)
`upgrade:2-19:backfill-system-unique-index-universal-identifier`
rewrites legacy system unique-index `universalIdentifier`s to their
deterministic value so the engine can own pre-existing indexes. The
backfill is **driven from `isUnique: true` fields** (mirroring the
engine ownership predicate — excludes PK / morph / relation fields) and
resolves each field's backing index in O(1).
# Bugs fixed along the way
- `database:reset` seeding failed with
`INDEX_FIELD_INVALID_DEFAULT_VALUE`: the engine derived a backing
`UNIQUE` index for the default `id` primary key. Fixed with the explicit
primary-key guard.
- `isUnique` updates on system-flagged standard fields (e.g.
auto-created `name`) did not trigger the backing-index side effect.
- Manifest sync crashed with "Could not find flat entity with universal
identifier ..." when app-scoped slices left dangling aggregator
references — fixed by centralizing pruning in the shared slice primitive
|
||
|
martmull
|
1a85b88d38 |
feat(files): direct-to-storage upload endpoints with pending file lifecycle (#22449)
<img width="1484" height="404" alt="image" src="https://github.com/user-attachments/assets/b2d363bf-d9e1-49fb-9811-8cc98041aa79" /> ## Context Uploading large files currently OOMs the server: every upload resolver buffers the whole file in memory (`streamToBuffer`) before writing it to storage. This PR is the first of a series introducing direct client-to-storage uploads. It adds the server-side endpoints and driver support only — it is non-breaking and nothing consumes the new flow yet. Follow-up PRs will migrate the frontend upload paths, add a stale-pending-file cleanup cron, and cap the legacy buffered resolvers. ## What it does **New upload flow (initiate → PUT → confirm):** - `createFileUpload(filename, size, fileFolder, fieldMetadataId?)` validates the request (folder allowlist: `FilesField`/`Workflow`, max size, extension-derived mime type), creates the file record in a new `PENDING` status, and returns an upload target: - **S3 with presign enabled** → a presigned PUT URL with `Content-Type`/`Content-Length` pinned in the signature, so the client uploads straight to the bucket; - **local storage, or S3 without presign** → a token-authenticated streaming endpoint on the server (`PUT /file-upload/:id?token=…`, new `FILE_UPLOAD` JWT type) that pipes the request body to the storage driver with constant memory usage and a declared-size cap. - `completeFileUpload(fileId)` verifies the bytes actually landed in storage (HEAD + size match against the declared size) and flips the record to `UPLOADED`. Idempotent. **Pending lifecycle safety:** - New `status` column on `core.file` (`PENDING`/`UPLOADED`, default `UPLOADED` so all existing rows and the legacy upload path are unaffected) + fast instance command. - Files are refused by the serving endpoints and by FILES-field sync while `PENDING`. **Driver support (both drivers):** - `getPresignedUploadUrl` (S3: presigned PUT; local: `null` → server-endpoint fallback) - `writeFileStream` (local: `fs` pipeline with the existing symlink/containment hardening, partial-file cleanup on error; S3: `@aws-sdk/lib-storage` `Upload` for bounded-memory streaming) - `getFileMetadata` (HEAD/stat for confirm-time verification) ## Tests - `file-upload.service.spec.ts`: initiate validation (folder allowlist, size), presigned vs fallback target, confirm verification (missing object, size mismatch, happy path, idempotency) - `local.driver.spec.ts`: `writeFileStream` (content, symlink rejection, partial-file cleanup on stream error), `getFileMetadata` - `s3.driver.spec.ts`: `getPresignedUploadUrl` (disabled → null, PUT command with signed content-type/content-length) - `direct-file-upload.integration-spec.ts`: full end-to-end flow against the local driver (initiate → PUT → complete → download), plus error paths (complete without upload, oversized PUT → 413, invalid token → 403, unsupported folder, size above max) ## Notes for reviewers - The upload-size ceiling for direct uploads is `settings.storage.maxDirectUploadFileSize` (1GB), separate from the 10MB `maxFileSize` used for pictures. - Since content can't be sniffed before it reaches storage, the mime type is derived from the file extension (with the existing `TWENTY_MIME_POLICY` override) and unknown extensions fall back to `application/octet-stream`; the serving path already forces `Content-Disposition: attachment` for anything not on the inline-safe allowlist. - Self-hosters using S3 presign will need a bucket CORS policy allowing `PUT` from the frontend origin (config variable description updated). https://claude.ai/code/session_015UH8KWmsB9zdYaog8MFG1d --- _Generated by [Claude Code](https://claude.ai/code/session_015UH8KWmsB9zdYaog8MFG1d)_ <!-- This is an auto-generated description by cubic. --> <a href="https://cubic.dev/pr/twentyhq/twenty/pull/22449?utm_source=github" target="_blank" rel="noopener noreferrer" data-no-image-dialog="true"><picture><source media="(prefers-color-scheme: dark)" srcset="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"><source media="(prefers-color-scheme: light)" srcset="https://www.cubic.dev/buttons/review-in-cubic-light.svg"><img alt="Review in cubic" src="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"></picture></a> <!-- End of auto-generated description by cubic. --> |
||
|
Paul Rastoin
|
0db4ddd46e |
fix(server): tolerate metadata-physical index drift in legacy index name normalization (#22472)
## Context The 2.18 `NormalizeLegacyIndexNames` workspace command (`1799200000000`, introduced in #22053) fails several production workspaces with `42P01 relation … does not exist`, rolling back the whole per-workspace upgrade transaction and marking the workspace `Failed`. ## Root cause The command assumes the physical index in the workspace schema is named exactly as recorded in `core."indexMetadata"."name"`. For workspaces where the physical index was already rebuilt/renamed under the v2 deterministic name (only targeted phone/relation rebuilds got new names after #14567) while metadata kept the legacy hash, the rename source no longer exists, so `ALTER INDEX … RENAME` aborts the entire workspace upgrade. (The duplicate-drop path uses `DROP INDEX IF EXISTS` and is unaffected.) ## Fix - **`WorkspaceSchemaIndexManagerService`**: new `doesIndexExist` and `getIndexDefinition` helpers querying `pg_indexes` for a `(schema, index)` pair. `renameIndexWithoutRebuild` keeps its strict semantics (no `IF EXISTS`) — drift tolerance lives in the command, which is the only caller that expects it. - **`NormalizeLegacyIndexNamesCommand`** — the rename operation now reconciles drift instead of blindly renaming: - Target name already exists physically, source gone → skip the rename, just point `indexMetadata.name` at it (the common "physical already v2, metadata still legacy" case). - Both source and target exist physically → compare their `pg_indexes.indexdef` ignoring the name: if identical, drop the legacy duplicate (it would otherwise be orphaned forever since metadata stops referencing it, adding permanent write/maintenance cost); if the definitions differ, keep it in place and log a warning. - Source exists, target free → rename as before, then update metadata. - Neither exists → log a warning and update metadata so a future rebuild recreates the index under the expected v2 name. In every branch the metadata name ends up on the recomputed v2 name, and no missing physical index can abort the workspace transaction anymore. ## Tests - Regression tests on the command spec for the four drift cases (target-already-renamed, both-missing, both-present-identical → drop, both-present-different → keep); existing rename/duplicate/dry-run/rollback tests updated to declare the physical indexes present. - New spec for `WorkspaceSchemaIndexManagerService` covering the rename SQL, the `pg_indexes` existence check, and the definition lookup. - New spec for `areIndexDefinitionsEquivalent` (name-only diff, uniqueness, columns, where clause, malformed input). `npx jest` on all three specs (20 passed), `lint:diff-with-main` and `typecheck` green. |
||
|
martmull
|
0992d0b748 |
feat(server): promote registration display fields to first-class columns (#22513)
Part of the application settings architecture work: https://github.com/twentyhq/core-team-issues/issues/2456 — follow-up to #22453, delivering the promised removal of the temporary manifest load. Display data (description, author, category, websiteUrl, aboutDescription, termsUrl, emailSupport, issueReportUrl, screenshots) only existed inside the `manifest` jsonb, forcing hot paths to load it. This PR: - Promotes those 9 fields to first-class columns on `applicationRegistration`, populated at every ingestion point (`updateFromManifest`, both `upsertFromCatalog` branches) — fast command creates the columns at deploy, slow command backfills them from the manifest. - `findManyListedCatalogCards()` (marketplace list) now selects only scalar columns — the manifest jsonb is no longer loaded there. - `findPublicByClientId()` (OAuth consent page) now selects `id, name, logo, websiteUrl, oAuthScopes` — no manifest. - The narrow select used by `findMany`/`findAll`/`findOneById`/`findOneByIdGlobal` includes the new columns. - GraphQL surface unchanged (no new fields); the marketplace detail endpoint still reads the manifest and is slimmed in the next PR. Verified: migration applied via the real runner (both commands recorded completed), backfill SQL exercised against live rows (full + minimal manifests), migration generator reports no pending schema changes, typecheck, lint, unit suites (application-registration + marketplace 15/15). 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_011sST4rPLU1Koi2oVGi84ei --- _Generated by [Claude Code](https://claude.ai/code/session_011sST4rPLU1Koi2oVGi84ei)_ <!-- This is an auto-generated description by cubic. --> <a href="https://cubic.dev/pr/twentyhq/twenty/pull/22513?utm_source=github" target="_blank" rel="noopener noreferrer" data-no-image-dialog="true"><picture><source media="(prefers-color-scheme: dark)" srcset="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"><source media="(prefers-color-scheme: light)" srcset="https://www.cubic.dev/buttons/review-in-cubic-light.svg"><img alt="Review in cubic" src="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"></picture></a> <!-- End of auto-generated description by cubic. --> |
||
|
martmull
|
af1b89c788 |
feat(server): import application logo into file storage at install (#22437)
Installed apps stored the logo as the manifest's relative path but never imported the file, so the public-assets URL 404'd and logos went missing in the UI for npm/tarball sources. Import the logo (best-effort — a declared but unshipped logo is skipped, not fatal) and record it as a first-class logoFileId on the application so it can be served reliably. <!-- This is an auto-generated description by cubic. --> <a href="https://cubic.dev/pr/twentyhq/twenty/pull/22437?utm_source=github" target="_blank" rel="noopener noreferrer" data-no-image-dialog="true"><picture><source media="(prefers-color-scheme: dark)" srcset="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"><source media="(prefers-color-scheme: light)" srcset="https://www.cubic.dev/buttons/review-in-cubic-light.svg"><img alt="Review in cubic" src="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"></picture></a> <!-- End of auto-generated description by cubic. --> |
||
|
martmull
|
20f2e33702 |
feat(server): first-class logo on application registration + narrowed settings queries (#22453)
Part of the application settings architecture work: https://github.com/twentyhq/core-team-issues/issues/2456 Application-registration list queries loaded the entire `manifest` jsonb (potentially 100KB+/row) on every settings/marketplace list request because display data (logo, description, author, category) only exists inside it. This PR: - Adds a first-class nullable `logo` column on `applicationRegistration`, populated at every ingestion point (`updateFromManifest`, `upsertFromCatalog`) and backfilled from `manifest->application->>logoUrl` via a slow instance command (self-sufficient backfill since `runDataMigration` runs before `up`). - Backs the `logoUrl` GraphQL getter with the column (manifest fallback for un-backfilled rows) — **GraphQL surface unchanged**. - Narrows `findMany` / `findAll` / `findOneById` / `findOneByIdGlobal` to an explicit scalar select that excludes `manifest` and `oAuthClientSecretHash` (every caller audited — none needs them; OAuth verification paths are untouched). - Replaces `findManyListed()` with `findManyListedCatalogCards()`: a projection query that extracts the four display strings from the manifest in SQL (with explicit soft-delete filtering) instead of hydrating full entities, feeding `findManyMarketplaceApps`. Verified: typecheck, lint:diff-with-main, unit suites (application-registration 5/5, marketplace 10/10, instance-command 31/31), migration applied via the real runner, and the migration generator reports no pending schema changes. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_011sST4rPLU1Koi2oVGi84ei --- _Generated by [Claude Code](https://claude.ai/code/session_011sST4rPLU1Koi2oVGi84ei)_ <!-- This is an auto-generated description by cubic. --> <a href="https://cubic.dev/pr/twentyhq/twenty/pull/22453?utm_source=github" target="_blank" rel="noopener noreferrer" data-no-image-dialog="true"><picture><source media="(prefers-color-scheme: dark)" srcset="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"><source media="(prefers-color-scheme: light)" srcset="https://www.cubic.dev/buttons/review-in-cubic-light.svg"><img alt="Review in cubic" src="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"></picture></a> <!-- End of auto-generated description by cubic. --> |
||
|
martmull
|
25fe66565c |
feat(applications): add type and options to application variables (#22157)
## Before <img width="1452" height="709" alt="image" src="https://github.com/user-attachments/assets/cd384ffa-cbe6-49d5-a807-ca8d580f55a9" /> <img width="1074" height="452" alt="image" src="https://github.com/user-attachments/assets/720d38db-3495-4032-8831-17d24ec6a7e7" /> ## After <img width="1421" height="865" alt="image" src="https://github.com/user-attachments/assets/2275c996-c895-4800-8324-2aa2ddfddd43" /> <img width="1348" height="870" alt="image" src="https://github.com/user-attachments/assets/3e1a891d-6db0-4cbd-870a-2a5bbde4929d" /> ## Summary Adds typed application variables with optional select **options**. This is the other half of #22059, split out from the custom-settings-tab removal. ## Changes - **Shared types**: `ApplicationVariable` / `ServerVariables` gain an optional `type` (a `FieldMetadataType` subset — `TEXT`, `BOOLEAN`, `NUMBER`, `DATE`, `SELECT`, `MULTI_SELECT`, `RAW_JSON`, `RICH_TEXT`, `ARRAY`, …) and select `options`. New `serializeApplicationVariableValue` / `deserializeApplicationVariableValue` helpers convert typed values to/from the encrypted string storage. - **Server**: `type`/`options` columns on `applicationVariable` and `applicationRegistrationVariable` (entities + DTOs), a fast `2-17` instance command, manifest processing via the serialization helpers, and a `QueryDeepPartialEntity` cast where the manifest JSON column is persisted. - **Frontend**: a polymorphic `SettingsApplicationVariableInput` that renders the native `Form*` field component for each type (boolean, number, date/date-time, select, multi-select, array, raw JSON, rich text, text); fragment/query updates to fetch `type`/`options`. - **SDK**: `defineApplication` validates that `SELECT`/`MULTI_SELECT` variables declare non-empty `options` at build time (since `options` is kept structurally optional for TypeORM/SDK compatibility). Variables default to `TEXT` when no type is given, so existing manifests are unaffected. ## Notes The generated GraphQL artifacts (`type`/`options` on the variable types) are regenerated by codegen; that change accompanies this PR. https://claude.ai/code/session_013Z7UB35V2mvUozh55QHG23 --- _Generated by [Claude Code](https://claude.ai/code/session_013Z7UB35V2mvUozh55QHG23)_ <!-- This is an auto-generated description by cubic. --> <a href="https://cubic.dev/pr/twentyhq/twenty/pull/22157?utm_source=github" target="_blank" rel="noopener noreferrer" data-no-image-dialog="true"><picture><source media="(prefers-color-scheme: dark)" srcset="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"><source media="(prefers-color-scheme: light)" srcset="https://www.cubic.dev/buttons/review-in-cubic-light.svg"><img alt="Review in cubic" src="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"></picture></a> <!-- End of auto-generated description by cubic. --> |
||
|
Félix Malfait
|
024a9b4d94 |
fix(upgrade): re-slot all 2-19 upgrade commands to their real merge epochs and guard timestamps in CI (#22498)
# Fix broken 2-19 upgrade command sequence (dev incident: `lastStreamError` / `workspaceDiscoverability`) ## Incident The dev environment (tracking main) throws: - `Property "lastStreamError" was not found in "AgentChatThreadEntity"` - `Cannot return null for non-nullable field Workspace.workspaceDiscoverability` ## Root cause The 2-19 upgrade commands were committed with **fabricated future timestamps** (year-2027 epochs like `1820000000000`). The upgrade cursor (`upgrade-aware-entity-metadata.adapter.ts`) tracks a **single** most-recent applied step: it looks up the latest `core."upgradeMigration"` row's name in the sequence (sorted by timestamp within kind) and hides every `@WasIntroducedInUpgrade` column at or past that index. Two ways this breaks, and both were live on main: 1. **Cursor regression**: a command merged *later* with a *smaller* timestamp (e.g. `pendingQuestion` at `1811…` after `metadata-overrides` at `1820…` had run) sorts *before* already-applied steps. When migrate runs, the "latest" row now points earlier in the sequence, re-hiding columns that were already applied. 2. **Migrate not running at all** (Felix's hypothesis): if the deploy pipeline skipped `database:migrate:prod`, none of the 2-19 rows exist and every 2-19-gated column is hidden. Both hypotheses have the same fix path; the discriminating query is in the verification section below. ## Fix **Real timestamps** (per maintainer direction — no more fabricated epochs): | Command | Old (fabricated) | New (real merge epoch) | Introduced by | |---|---|---|---| | workspace: backfill-workspace-custom-application-registration | `1820000000000` | `1782853718000` | #22378 | | fast: add-metadata-overrides-column | `1820000100000` | `1782986475000` | #22417 | | slow: backfill-metadata-overrides | `1820000110000` | `1782986476000` (+1s to order after its fast pair) | #22417 | | fast: add-last-stream-error-to-agent-chat-thread | `1821000000000` | `1782996657000` | #22434 | | fast: add-pending-question-to-agent-chat-thread | `1811000000000` | `1782999138000` | #22346 | | fast: add-workspace-discoverability-to-workspace | `1820000001000` | `1783004140000` | #22423 | Each value is the committer epoch of the squash-merge commit that introduced the command on main (verified via `git log --diff-filter=A`). Sorted by real time, the fast sequence is strictly increasing, so the cursor can no longer regress. **Idempotency**: renaming a command changes its step name, so every one of these re-runs on any instance that already applied it under the old name (dev cluster, edge self-hosters — 2.19 is unreleased, so tagged releases are unaffected). All six are now safe to re-run: - `lastStreamError`, `pendingQuestion`, `metadata-overrides` fast: `ADD COLUMN IF NOT EXISTS` (already were) - `metadata-overrides` slow backfill: `WHERE … IS NULL` guard (already was) - workspace command: skips when `applicationRegistrationId` is already set (already did) - `workspaceDiscoverability`: **made idempotent in this PR** — `CREATE TYPE` wrapped in a `duplicate_object` handler, `ADD COLUMN IF NOT EXISTS` **CI guard** (replaces the append-only check added earlier on this branch): - Timestamps must be **real**: within `[now − 60 days, now + 2 days]`. This is the check that would have prevented the original sin — 2027 epochs can never pass. - Still **append-only** within the version directory, but computed from `git diff --name-status --find-renames` so renamed/copied files are checked too (cubic's P2), and files the PR deletes/renames away no longer count toward the existing max (otherwise a re-slotting PR like this one could never pass its own guard). - Covers `workspace-command-<ts>-` filenames, not just `instance-command-fast|slow-<ts>-`; skips `.spec.ts` files. - Failure message documents the escape path (re-slot the fabricated blocker to its real epoch + make it idempotent) and a bypass label `ci:allow-upgrade-command-timestamp-exception` for deliberate exceptions. ## Deploy sequencing (important) After this merges and deploys, `database:migrate:prod` **must run** before the API pods are relied on: the old step names no longer exist in the sequence, so until the renamed commands run once, the cursor resolves to 0 and *every* gated column is hidden. The commands are idempotent, so the re-run is harmless. Running API/worker pods only compute the cursor at boot — restart them after migrate. ## Verification / diagnosis on dev ```sql SELECT name, status, "createdAt" FROM core."upgradeMigration" WHERE "workspaceId" IS NULL ORDER BY "createdAt" DESC LIMIT 15; ``` - Latest rows named `…_182xxxxxxxxxx` (fabricated) and completed → migrate ran, cursor regressed (hypothesis 1). - No 2-19 rows at all → migrate never ran for 2-19 (hypothesis 2). - After the fix: latest row should be `2.19.0_AddWorkspaceDiscoverabilityToWorkspaceFastInstanceCommand_1783004140000`, status `completed`. https://claude.ai/code/session_01Lyi6zTema2FMVVh8MD6c38 |
||
|
Félix Malfait
|
a28887bba6 |
feat(server): workspace opt-out of root-domain directory listing (#22423)
## What Lets a workspace opt out of being surfaced in the multi-workspace root-domain (app.twenty.com) picker via **email-domain discovery**. Adds `isDirectoryListingEnabled` (default `true`) on the workspace. When `false`, the workspace is filtered out of the approved-access-domain branch of `findAvailableWorkspacesByEmail`, so a user whose email domain matches an approved access domain no longer sees the workspace in the sign-up picker. ## Scope of the opt-out (deliberately narrow) The filter is applied **only** to the approved-access-domain discovery source: - **Members** (`availableWorkspacesForSignIn`) — never filtered; they keep access. - **Explicit invitations** — never filtered; the intent is one-to-one. - **Approved-access-domain discovery** — the only "listing" source, gated by the flag. A hidden workspace stays fully reachable by members and invited users via the direct workspace subdomain; it just isn't advertised in the global picker. > Open question for review: do we also want a stronger mode that hides the workspace from the root-domain picker even for existing members (forcing them to use the subdomain directly)? That would additionally filter the member/invitation sources and is a larger behavior change — not included here. ## Changes **Backend** - `workspace.entity.ts` — new `isDirectoryListingEnabled` column (`@Field`, default `true`). - `user-workspace.service.ts` — filter the approved-access-domain branch on the flag. - `update-workspace-input.ts` — expose the field on `updateWorkspace`. - `workspace.service.ts` — `PermissionFlagType.SECURITY` (same as the other discovery/security toggles). - Fast instance command adding the column (default `true`, so no existing workspace is hidden). **Frontend** - Settings > Security: a **"List in workspace directory"** toggle (shown only in multi-workspace mode) that flips the flag via `updateWorkspace`, mirroring the existing `isInternalMessagesImportEnabled` toggle. - Threaded the field through the current-user fragment, `CurrentWorkspace` type, and mock data. - Regenerated the metadata + client-sdk GraphQL types (`generated-metadata`, `twenty-client-sdk/.../generated`) — generated against a server booted from this branch. ## Verification - `tsgo` typecheck: 0 errors. `oxlint`: 0/0. `oxfmt`: clean. - Codegen diff verified to contain **only** the new field (no unrelated drift). - Tests not run locally; CI covers unit/integration + the codegen/migration freshness checks. --------- Co-authored-by: claude[bot] <41898282+claude[bot]@users.noreply.github.com> |
||
|
Félix Malfait
|
4aaf171d63 |
feat(ai): add ask_questions interactive clarifying-question tool (#22346)
## What & why Adds an `ask_questions` tool that lets the in-app **Ask AI** assistant **pause a turn to ask the user one or more multiple-choice questions** (per the [Figma design](https://www.figma.com/design/xt8O9mFeLl46C5InWwoMrN/Twenty?node-id=105959-117153)) and resume once answered — instead of guessing on ambiguous/consequential decisions. The tool is **harness-only**: an interactive question UI is meaningless without a user to answer it, so it must be absent from MCP and from head-less workflow agents. ## Design — true tool-result resume (not a synthetic user message) The user's answer is a **structured tool result bound to the `toolCallId`**, and the **same agent turn resumes** — exactly how Anthropic (`tool_result` by `tool_use_id`) and OpenAI (`function_call_output`) model human-in-the-loop. The naive form of this (leave the tool call in `input-available` to mean "pending") is **impossible** here: `finalizeDanglingToolParts` rewrites `input-available` → `output-error` ("Tool execution was interrupted") on both the persist path (`addMessage`) and the model-reload path (`chat-execution.service.ts`). That util is a load-bearing safety net, so weakening it is the wrong move. Instead: - `ask_questions` is an **inline, chat-only tool with an `execute` that returns a `status: 'pending'` result immediately**, so the tool part is always `output-available` and **immune to `finalizeDanglingToolParts`**. `stopWhen(hasToolCall('ask_questions'))` halts the turn right after the call (the model never sees the placeholder). - A nullable **`thread.pendingQuestionMessageId`** marker records that a turn is awaiting an answer. - The new **`answerAgentChatQuestion`** mutation atomically *claims* the question (clears the marker, marks the thread streaming), **writes the answer onto the same tool part** (`status: 'answered'`), and **re-enqueues the turn via the existing `existingTurnId` plumbing** (`isResume` bypasses the per-turn dedup guard). On resume `finalizeDanglingToolParts` leaves the `output-available` part untouched and `convertToModelMessages` emits `assistant(tool_use)` + `tool_result(answers)`, so the model continues. This achieves the platform-aligned semantics **without** weakening the finalize safety net or inventing a fragile new part state. ### Meets the two requirements - **Survives refresh, scoped per-thread** — the pending state is a normal persisted `output-available` part + the thread marker; the frontend card is derived per-thread from the loaded messages, so it re-appears on reload and only on its own thread. - **Takes priority over the queue** — a unified `isBlocked = activeStreamId || pendingQuestionMessageId` gate is applied in both `sendChatMessage` (new messages queue) and `flushNextQueuedMessage` (the drain). The queue cannot unpile until the question is answered and the resumed turn completes. ### Harness-only by construction `ask_questions` is added **only** to the chat's inline `activeTools` (like `learn_tools`/`execute_tool`/`load_skills`). It never enters the tool registry/catalog, so it is invisible to MCP and to workflow agents — no `MCP_EXCLUDED_TOOL_NAMES` entry needed. ## UX While a question is pending, the **composer is replaced by the question card** (matching the Figma): question title + pager (`1/2`), numbered option rows (`IconSquareNumber*`) with per-option info-icon descriptions and a "Recommended" badge, and the normal composer as the free-text fallback ("Type anything to do differently."). The transcript shows a compact "Asking questions…" status line that becomes an answered summary. ## Changes **twenty-shared** - `ai/types/AskQuestionsToolTypes.ts` — `AskQuestionItem/Option/Answer/Result`, `ASK_QUESTIONS_TOOL_NAME`. **twenty-server** - `ai-chat/tools/ask-questions.tool.ts` — inline tool factory (pending-result `execute`, zod schema, 1–4 questions × 2–4 options). - `chat-execution.service.ts` — add to `activeTools` + `preloadedToolNames`; `hasToolCall` in `stopWhen`. - `chat-system-prompts.const.ts` — when-to-use guidance. - `entities/agent-chat-thread.entity.ts` — `pendingQuestionMessageId` column. - `stream-agent-chat.job.ts` — set the marker on a question pause; bypass the dedup guard on resume; suppress the no-text warning for question pauses. - `agent-chat-streaming.service.ts` — gate `flushNextQueuedMessage`; `enqueueResumeStream`. - `agent-chat.resolver.ts` — gate `sendChatMessage`; `answerAgentChatQuestion` mutation. - `agent-chat.service.ts` — `resolvePendingQuestion` (atomic claim + write answer). - `dtos/agent-chat-question-answer.input.ts`, `ai.exception.ts` (`QUESTION_NOT_PENDING`), `utils/find-pending-question-part.util.ts`. **twenty-front** - `components/AiChatQuestionCard.tsx` — the interactive card (matches Figma tokens) + `__stories__/AiChatQuestionCard.stories.tsx`. - `components/AiChatEditorSection.tsx` — swap the composer for the card while pending. - `components/AiChatQuestionStatusRenderer.tsx` + branch in `AiChatAssistantMessageRenderer.tsx`. - `states/selectors/agentChatPendingQuestionComponentSelector.ts`, `types/AgentChatPendingQuestion.ts`. - `hooks/useSubmitQuestionAnswer.ts` + `utils/markQuestionAnswered.ts` (optimistic) + `graphql/mutations/answerAgentChatQuestion.ts`. A design doc lives at `packages/twenty-server/docs/ASK_USER_QUESTION_TOOL_PLAN.md`. ## Migration Adds a nullable `pendingQuestionMessageId` (uuid) column to `core.agentChatThread`. Needs a generated **fast instance command** (`database:migrate:generate --name addThreadPendingQuestion --type fast`) — see "Verification status". ## Tests - Server: `ask-questions.tool.spec.ts` (pending echo + schema bounds), `find-pending-question-part.util.spec.ts`. - Front: `markQuestionAnswered.test.ts`, plus the Storybook story. ## Verification status (please read) This branch was authored in an environment where the monorepo `yarn install` repeatedly failed on transient TLS resets from the package registry, so I could **not** locally run the mechanical gates. The logic was reviewed by hand and the `ai@6.0.97` exports used (`hasToolCall`, `stepCountIs`, `generateId`) were confirmed against the package's type defs. Still **TODO** (will rely on CI / a follow-up once deps install): - [ ] `nx run twenty-shared:generateBarrels` (the `ai/index.ts` export was added by hand; regen to reconcile) - [ ] `nx run twenty-front:graphql:generate` (new mutation + input type) - [ ] generate the fast instance command (migration) for the new column - [ ] `typecheck` + `lint:diff-with-main` (front + server) — expect minor import-ordering autofixes - [ ] run the unit tests **Screenshots:** reproducing the live flow needs an AI provider API key (to get the model to actually call `ask_questions`), which isn't available here. The card can be screenshotted from its **Storybook story** (`AiChatQuestionCard.stories.tsx`) with no API key — I'll add that image once deps install, or a reviewer can run `nx storybook twenty-front`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01AArS8H3y3Z1Qwm763xhPLB --- _Generated by [Claude Code](https://claude.ai/code/session_01AArS8H3y3Z1Qwm763xhPLB)_ <!-- This is an auto-generated description by cubic. --> <a href="https://cubic.dev/pr/twentyhq/twenty/pull/22346?utm_source=github" target="_blank" rel="noopener noreferrer" data-no-image-dialog="true"><picture><source media="(prefers-color-scheme: dark)" srcset="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"><source media="(prefers-color-scheme: light)" srcset="https://www.cubic.dev/buttons/review-in-cubic-light.svg"><img alt="Review in cubic" src="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"></picture></a> <!-- End of auto-generated description by cubic. --> |
||
|
Félix Malfait
|
d709467902 |
feat(ai): surface AI chat stream failures through one typed error channel (#22434)
## Context Investigating a report where the AI chat showed only a `...` spinner while the network response clearly contained `No AI models are available`. Root cause: terminal stream failures reach the client on **two mismatched channels**. | Representation | Persisted (survives reload) | Rendered by client | |---|---|---| | AI-SDK `error` chunk (inside `stream-chunk`) | ✅ RPUSH'd to Redis | ❌ dropped by `readUIMessageStream` (no message part, no error state) | | typed `stream-error` event | ❌ never persisted | ✅ sets the error atom | Live, the `stream-error` event renders. But on reload, `chatStreamCatchupChunks` replays only the persisted **error chunk** — which the reducer discards — and the streaming indicator never clears. ## Change Collapse to a single typed error contract: - **Suppress the opaque `error` chunk** in the stream job; every failure is surfaced through the typed `stream-error` event. Errors are mapped via `mapErrorToStreamError` so an `AiException` keeps its `AiExceptionCode` (e.g. `API_KEY_NOT_CONFIGURED` → the existing "AI not configured" banner) instead of leaking a raw string. - **Persist the terminal error** next to the accumulated chunks and expose it as an explicit `error { code message }` field on `ChatStreamCatchupChunks`, so a client catching up after a reload recovers it — no dependency on the AI SDK's internal chunk shape. - **Reset per-thread stream state at job start**, so a failed turn's leftover chunks/error never replay on the next stream. - **Client replays the catchup error** as a terminal `stream-error` event, which clears the streaming indicator and renders the error (fixes the infinite spinner on a stream that ended in error). ## Notes - `ChatStreamError` is a new metadata GraphQL type; generated types (twenty-front metadata + client-sdk) were hand-updated to keep the tree consistent and will be reconciled by CI's `graphql:generate` check if anything differs. - Server unit test added for the error mapping. No schema/DB migration. ## Test plan - [ ] With no AI provider configured, send a chat message → error renders immediately (not a spinner). - [ ] Reload the thread → the error still renders (recovered from catchup), indicator not spinning. - [ ] Configure a provider and send again → normal streaming; no stale error from the previous failed turn. <!-- This is an auto-generated description by cubic. --> <a href="https://cubic.dev/pr/twentyhq/twenty/pull/22434?utm_source=github" target="_blank" rel="noopener noreferrer" data-no-image-dialog="true"><picture><source media="(prefers-color-scheme: dark)" srcset="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"><source media="(prefers-color-scheme: light)" srcset="https://www.cubic.dev/buttons/review-in-cubic-light.svg"><img alt="Review in cubic" src="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"></picture></a> <!-- End of auto-generated description by cubic. --> |
||
|
Félix Malfait
|
38fbff465f |
chore(server): ship the 2.20 standardOverrides drop as a dormant command (#22448)
Follow-up to #22417, per [this thread](https://github.com/twentyhq/twenty/pull/22417#discussion_r3512187719): migrate the `2-20/README.md` placeholder into a real command using the `TWENTY_NEXT_VERSIONS` mechanism. ### What - Add `DropMetadataStandardOverridesColumnFastInstanceCommand`, registered against `2.20.0`. It boots (`2.20.0` is in `TWENTY_ALL_VERSIONS`) but stays **dormant** — the upgrade sequence only runs `TWENTY_CROSS_UPGRADE_SUPPORTED_VERSIONS` (previous + current), so it never executes during the 2.19 deploy and activates automatically when `nx version:bump` promotes 2.20 to current. - Name constant + unit test (SQL parity, registration against `2.20.0`, name-constant parity). - Register it in `instance-commands.constant.ts`. - Update the `standardOverrides` `@deprecated` comments on object/field metadata to point at the shipped command. - Delete `2-20/README.md`. - Document the "ship a command for a future version" flow in `docs/UPGRADE_COMMANDS.md` and `.cursor/rules/server-migrations.mdc` (the mechanism was previously undocumented). ### Note / correction to the README's plan The old README implied both the command **and** `@WasRemovedInUpgrade` could be added at 2.20 time. Only the command can ship now: the decorator's validator runs against the active sequence, so referencing a still-dormant 2.20 step fails boot with `unknown-step-name`. So the entity keeps its `WasRemovedInUpgrade<T>` type wrapper for now; the decorator gets wired (one line, via the name constant) once 2.20 is current — same deferred-drop shape as `isUIReadOnly`. ### Verification Could not run `jest`/`typecheck`/`lint` in this environment: `yarn install` is blocked by egress policy on a git-based transitive dep (`github.com/electron/node-gyp.git`). Verified by review against the sibling 2-19 add-column and 2-12 drop commands. **Please let CI run before merge.** https://claude.ai/code/session_01KMArJvdEmsX3eAmJLbS1b6 --- _Generated by [Claude Code](https://claude.ai/code/session_01KMArJvdEmsX3eAmJLbS1b6)_ <!-- This is an auto-generated description by cubic. --> <a href="https://cubic.dev/pr/twentyhq/twenty/pull/22448?utm_source=github" target="_blank" rel="noopener noreferrer" data-no-image-dialog="true"><picture><source media="(prefers-color-scheme: dark)" srcset="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"><source media="(prefers-color-scheme: light)" srcset="https://www.cubic.dev/buttons/review-in-cubic-light.svg"><img alt="Review in cubic" src="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"></picture></a> <!-- End of auto-generated description by cubic. --> |
||
|
Félix Malfait
|
5a4ebca226 |
refactor(server): unify the two metadata override mechanisms into one (#22417)
## Unify the two metadata override mechanisms into one Twenty had **two** override mechanisms: - **`standardOverrides`** — a bespoke JSONB column on `objectMetadata`/`fieldMetadata` with typed DTOs and a per-locale `translations` map, resolved by two i18n-aware resolvers. - **`OverridableEntity.overrides`** — a flat, registry-driven JSONB blob on view / view-field / view-field-group / command-menu-item / page-layout-tab / page-layout-widget, resolved by a plain spread. This PR collapses them into **one** concept: a single `overrides` blob, one registry-driven overridable set, one i18n-aware read path, and one write path (`computeMetadataOverridesBlob`, extracted in #22404). Object/field **stay on `SyncableEntity`** (not reparented to `OverridableEntity`) so their `isActive` default stays **FALSE** — this sidesteps the `isActive` default conflict entirely. ### GraphQL breaking change (accepted) The `standardOverrides` field is **removed** with no deprecation alias — `overrides` (a `JSON` scalar) is exposed instead on `Object` and `Field`. Product confirmed negligible external usage; the front-end has no hand-written consumer (only generated types), which are regenerated here. ### Commit structure (reviewable commit-by-commit) 1. **Unified resolver + parity harness** — `resolveEffectiveEntityProperty` is a strict superset of the three legacy resolvers; a corpus parity spec compares it against a *frozen reference* of the old logic across every locale, `isStandardApp` branch and override shape. 2. **Registry-driven** — object/field presentation props tagged `isOverridable` + `translatable`; the overridable/translatable sets are derived from the registry (a test asserts they equal the legacy hardcoded lists). 3. **Rename + swap + delete** — `standardOverrides` → `overrides` across entities, DTOs, flat/universal types, producers, the ~12 resolve/write/create/sync call sites, mocks and specs; the reconciler's two compare entries collapse to one; the three legacy resolvers, both DTOs and the hardcoded constants/types are deleted. 4. **Migration (zero-downtime, two-phase)** — split across two releases so a rolling deploy never drops a column a previous-release pod still `SELECT`s: - **2.19 fast** — add the `overrides` column (schema only). - **2.19 slow** — backfill `overrides` from `standardOverrides` in `runDataMigration` (kept out of the schema transaction so the bulk write doesn't hold the ACCESS EXCLUSIVE lock; skipped on fresh installs, which have no data to copy). - **2.20 fast** — drop the legacy `standardOverrides` column (gated by `TWENTY_NEXT_VERSIONS`, so it stays dormant until the instance reaches 2.20). 5. **Front/client-SDK regen** — regenerated metadata GraphQL types. 6. **Integration specs + i18n** — updated the standard object/field update integration specs + snapshots, and the reworded validator message catalog entry. ### Rolling-deploy safety `standardOverrides` is retained through 2.19 and only dropped in 2.20, mirroring the codebase's deferred-drop convention (`isUIReadOnly`/`isCustom`). During the 2.19 rollout both columns exist, so old and new pods coexist without "column does not exist" errors. The backfill lives in a slow `runDataMigration` (per the `no-data-mutation-in-fast-instance-command` rule) so it doesn't stall reads. ### `isActive` guard The migration never reads or writes `isActive`; the backfill asserts the active-row count is unchanged and aborts otherwise. Verified on a real DB: apply + revert preserves the blob **and** the nested `translations` map, with `isActive` counts identical before/after. ### Verification (local) - `nx typecheck twenty-server` + `nx typecheck twenty-front` — green - `nx lint:diff-with-main twenty-server` (oxlint `--type-aware` + oxfmt) — green - `nx test twenty-server` — green (unit + parity + registry + migration tests) - `nx run twenty-server:test:integration:with-db-reset` — green - `database:reset` applies the 2.19 phases and leaves **both** columns present (2.20 drop stays dormant); backfill + revert round-trip verified on a real DB - Metadata integration suites (standard object/field update, application sync) pass end-to-end against the two-column schema - Metadata GraphQL types regenerated against a booted server; zero `standardOverrides` references remain in application code (only the migration commands + the legacy schema baseline) --------- Co-authored-by: prastoin <paul@twenty.com> |
||
|
Paul Rastoin
|
2b1417770e |
SearchVector derivation via migration-scoped index (alt to #2622 __warmedUpCache) (#22389)
## What this is close https://github.com/twentyhq/core-team-issues/issues/2622 A **POC / discussion branch** implementing the runner-scoped alternative to the `__warmedUpCache` design in [core-team-issues#2622](https://github.com/twentyhq/core-team-issues/issues/2622). Not for merge as-is — meant to diff against that plan. ## Problem `deriveSearchVectorAsExpressionForTsVectorField` scans the entire `flatSearchFieldMetadataMaps` (`Object.values(...).filter(...)`) once per object created in a migration. On install that's `O(objectsCreated × totalSearchFields)` — the quadratic #2622 targets. Only **one** of the three call sites is actually hot: - `create-object` (runner) — global maps, called per created object → the quadratic - export DDL — maps already built **per object** (O(k)) - `update-field` rebuild — one field, gated on `rebuildSearchVector` ## Approach Instead of a private `__warmedUpCache` side-channel on `FlatEntityMaps<T>` + drain-on-hydration, this keeps the index in the **consumer**: 1. `derive` now takes `targetSearchFieldMetadatas` (already scoped to the tsVector field) instead of scanning the map itself. 2. The runner builds a `Map<tsVectorFieldMetadataId, searchFields[]>` **once per migration**, lazily, and threads it through the action context. Safe because `searchFieldMetadata` creates are ordered before `objectMetadata` creates (`computeOrderedMigrationActions`), so the map is complete on first use. → `O(totalSearchFields)`. 3. `getTargetSearchFieldMetadatasForTsVectorField` (O(total) filter) stays as the fallback for the one-off callers (export, field-update) and when the accessor isn't provided. ## Why this over `__warmedUpCache` - **No `FlatEntityMaps<T>` type widening**, no convention-only privacy, no id/universalIdentifier drain to keep in sync. - **No referential-integrity obligation.** The index only ever contains entities present in the map, so the "search field created-then-deleted before its object hydrates" case (deferred as an edge in #2622) can't put a stale id into an aggregator and crash `derive` via the `-orThrow` lookup. - **One `derive` path**, not "aggregator + direct-filter fallback for export". - Blast radius: ~220 lines, mostly a new util + test. ## Benchmark (micro, isolated function) Median of 7 trials, 10 search fields per object, running the real shipped utils — old = `getTargetSearchFieldMetadatasForTsVectorField` once per object (identical to the old inline scan), new = `buildSearchFieldMetadatasByTsVectorFieldId` once + N lookups (both assert they resolve the same fields): | objects | total search fields | old (scan/obj) | new (index once) | speedup | |--------:|--------------------:|---------------:|-----------------:|--------:| | 50 | 500 | 2.08 ms | 0.06 ms | 33× | | 100 | 1,000 | 9.26 ms | 0.12 ms | 77× | | 200 | 2,000 | 36.2 ms | 0.23 ms | 160× | | 400 | 4,000 | 151 ms | 0.40 ms | 379× | | 800 | 8,000 | 701 ms | 0.92 ms | 766× | Confirms the old path is quadratic (~4× per doubling of object count) and the new path is linear (sub-ms throughout). **Caveats — read these before trusting the speedup:** - This is the **isolated derivation function**, no DB / DDL / inserts. In a real `create-object` action the derive is a small fraction of per-action cost, so the end-to-end win is far smaller than the ratios above. - A default workspace has ~20–30 objects, where the **old** code already costs only ~1–2 ms total across the whole install. The quadratic only becomes material (>50 ms, the runner's slow-action threshold) around **200–400 objects**. - The measurement that should actually gate this — `[install-perf] create:objectMetadata` on a real install against a real DB with a few hundred objects — has **not** been run yet. The micro-benchmark bounds the upside and locates the knee of the curve; it does not prove end-to-end payoff. ## Not done on purpose - **No end-to-end benchmark yet** — step 0 should still be measuring `[install-perf] create:objectMetadata` on a real large install to confirm the quadratic is worth removing at all. - Relies on the ordering invariant (commented at the build site). The fully self-contained variant is to put the object's search fields on `FlatCreateObjectAction` (builder change) — deliberately left out to keep this runner-scoped. ## Checks `nx typecheck twenty-server`, `nx lint:diff-with-main twenty-server`, new util spec + existing `generate-workspace-schema-ddl` spec all green. |
||
|
Félix Malfait
|
1df00698cf |
feat(server): make workspace Custom application carry an applicationRegistration so custom labels are translatable (#22378)
## Why Custom objects/fields belong to a per-workspace **Custom** application (`workspace.workspaceCustomApplicationId`). That application was created with `applicationRegistrationId = null`. Because the metadata label resolver loads a translation catalog from `core.applicationTranslation` **keyed by `applicationRegistrationId`** (`ApplicationTranslationCacheService.getCatalog` → `applicationTranslationCatalogLoader` → `resolveObjectMetadataStandardOverride` / `resolveFieldMetadataStandardOverride`), the Custom app had no catalog and custom labels always resolved to the raw source string. This is the foundational slice: it wires up the missing key so custom labels can be translated **exactly like any installed third-party app**. The read/resolve path already works once a catalog exists — confirmed end-to-end. `flatApplicationMaps` carries `applicationRegistrationId` straight from the entity column, so setting it + recomputing that cache is all that's needed. ## What changed - **`ApplicationService.createWorkspaceCustomApplication`** now creates a workspace-scoped `applicationRegistration` and links it to the Custom application. This covers both production creation sites (sign-in-up and the dev-seeder), which are the only callers. - **New idempotent workspace upgrade command** `upgrade:2-18:backfill-workspace-custom-application-registration` creates a registration for each existing workspace's Custom application that lacks one and links it. It delegates the registration lifecycle (create + link + `flatApplicationMaps` recompute) to `ApplicationService`, so the command only decides *which* workspaces need it. - New `WORKSPACE_CUSTOM_APPLICATION_NAME` constant; the registration creation lives in `ApplicationService.createWorkspaceCustomApplicationRegistration`. ## Design decisions - **Per-workspace registration (not a shared "custom" registration).** `applicationTranslation` is keyed *only* by `applicationRegistrationId` (cross-workspace). A shared registration would force every workspace's custom translations into one catalog keyed by `generateMessageId(sourceText)`, guaranteeing cross-workspace collisions and leakage (two workspaces both naming an object "Project" would clash). Each workspace's Custom app gets its own registration (`ownerWorkspaceId = workspaceId`, `universalIdentifier = the Custom app's per-workspace uuid`) and thus an isolated catalog — matching installed-app behaviour, where `application.universalIdentifier === registration.universalIdentifier`. - **Source-label keying kept** (`generateMessageId(sourceLabel)`). The resolve path and the third-party manifest pipeline both key catalogs this way. Re-keying by a stable `universalIdentifier` would require changing the shared resolver/dataloader for *all* apps and would break marketplace manifest translations — out of scope for this slice. Consequence: renaming a label orphans its catalog entry (it falls back to the source label until re-translated) — the same behaviour an installed app has when it changes a source string. Re-keying on rename can be handled later by the interactive write path. - **Workspace command (not instance command)** for the backfill: it is per-workspace data logic that must recompute the per-workspace `flatApplicationMaps` cache the resolver reads from. It is idempotent (skips Custom apps that already have a registration), supports `--dry-run`, and is forward-only by design. - **Interactive write path deferred** as an explicit follow-up. This slice proves the read/resolve path; an editor that writes custom translations into `applicationTranslation` (+ cache invalidation) is the natural next step. ## Tests - **Unit test** for the backfill command: creation + linking, idempotency, dry-run, and the skip paths. - **Integration test** (`custom-application-translation.integration-spec.ts`): on a freshly created workspace (so the registration's translation cache is guaranteed cold), it asserts the Custom application is created with a registration, seeds an `applicationTranslation` row, and verifies a custom object's label resolves from that catalog while a label with no catalog entry falls back to its source label. ## Notes for reviewers - No new entity columns or migrations beyond the workspace command — `ApplicationRegistrationEntity` already supports a workspace-scoped `workspaceId`. - The backfill follows the established upgrade-command pattern: it imports `ApplicationModule` and delegates to `ApplicationService` (consistent with the other version-command modules). https://claude.ai/code/session_018heTgu4ew4AJ99VVz4bjqd --- _Generated by [Claude Code](https://claude.ai/code/session_018heTgu4ew4AJ99VVz4bjqd)_ <!-- This is an auto-generated description by cubic. --> <a href="https://cubic.dev/pr/twentyhq/twenty/pull/22378?utm_source=github" target="_blank" rel="noopener noreferrer" data-no-image-dialog="true"><picture><source media="(prefers-color-scheme: dark)" srcset="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"><source media="(prefers-color-scheme: light)" srcset="https://www.cubic.dev/buttons/review-in-cubic-light.svg"><img alt="Review in cubic" src="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"></picture></a> <!-- End of auto-generated description by cubic. --> |
||
|
neo773
|
101b85db7b |
messaging remove dead workspace entities (#22366)
<!-- This is an auto-generated description by cubic. --> <a href="https://cubic.dev/pr/twentyhq/twenty/pull/22366?utm_source=github" target="_blank" rel="noopener noreferrer" data-no-image-dialog="true"><picture><source media="(prefers-color-scheme: dark)" srcset="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"><source media="(prefers-color-scheme: light)" srcset="https://www.cubic.dev/buttons/review-in-cubic-light.svg"><img alt="Review in cubic" src="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"></picture></a> <!-- End of auto-generated description by cubic. --> --------- Co-authored-by: prastoin <paul@twenty.com> |
