b5a1aed24b4ca54ed5a487db810b2cc18bdec04d
25 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Weiko
|
b7850a6c64 |
feat(metadata): deterministic universalIdentifiers for server-generated side-effects (#21949)
## Context
Server-generated "side-effect" entities created for every object (system
fields, INDEX view, record-page fields view + view fields, search-vector
index, navigation command, record page layout/tabs/widgets) were minted
with random v4() ids. Because they were non-deterministic, nothing could
reference them by id (e.g. point a view field at an object's createdAt
field).
This PR introduces a single shared rule for deriving these ids
deterministically via uuid v5, so the same (owner app, parent, kind)
always yields the same id, making side-effects referable and
reproducible.
This is the **forward-only foundation** (PR1). Follow-ups:
- PR2: SDK with optional universalIdentifier + expose helpers to app
authors.
- PR3: regenerate the standard-app constants to the same scheme +
workspace backfill.
## The rule
```ts
universalIdentifier = computeOwnerScopedUniversalIdentifier({ ownerAppUID, namespace, value })
= v5(value, v5(ownerAppUID, ENTITY_TYPE_NAMESPACE))
value = `${parentUID}:${discriminator}` // entity scoped under a parent
= `${discriminator}` // top-level, app-parented entity
```
- ownerAppUID: The application that owns the entity (already threaded
through every generator as applicationUniversalIdentifier); folded into
the namespace so it both owns and scopes
the id — two apps adding the same-named entity to a shared parent never
collide.
- namespace: Per entity type (ENTITY_TYPE_NAMESPACE_BY_TYPE), so
different types with the same parent+discriminator never collide.
- parentUID: The immediate parent's actual universalIdentifier (omitted
for top-level entities, since the owner app already scopes them).
- discriminator: A stable semantic key (field name, tab/widget title,
generated index name, select-option value, …).
Scope boundary: deterministic v5 applies to system side-effects (unique
by construction) and, later, app-authored manifest entities (uniqueness
enforced at SDK build time).
Entities created through the UI by the workspace "Custom" app (custom
objects/views/fields) keep v4, their natural keys aren't unique and
aren't enforced. A UI-created custom object keeps its v4 id; its
side-effects are deterministic relative to that v4 parent.
Changes
twenty-shared: new application/deterministic-identifier/ module:
- computeDeterministicUuid(value, namespace) primitive + a thin
computeOwnerScopedUniversalIdentifier wrapper (boilerplate only), and
frozen ENTITY_TYPE_NAMESPACE_BY_TYPE.
- One self-contained util per usecase (no central registry, no generic
engine): each util bakes in its own discriminator + namespace, so a key
lives next to the code that uses it and is individually testable. ~28
utils covering side-effect and (future) app-authored entities, e.g.
getFieldUniversalIdentifier, getIndexViewUniversalIdentifier,
getFieldsWidgetViewUniversalIdentifier, getViewFieldUniversalIdentifier,
getIndexUniversalIdentifier, getRecordPageLayoutUniversalIdentifier,
getPageLayoutTab/WidgetUniversalIdentifier,
getNavigationCommandUniversalIdentifier, plus the general
getViewUniversalIdentifier / getPageLayoutUniversalIdentifier and
app-authored
getObject/Role/PermissionFlag/Agent/Skill/…UniversalIdentifier.
- Golden snapshot test locking every util's output for fixed inputs,
plus a cross-type no-collision test.
twenty-server: side-effect generators now derive universalIdentifier via
the helpers (local id PKs stay v4()): system fields + name, INDEX view,
record-page fields (fields-widget) view, default view fields,
search-vector index, nav command, page layout/tabs/widgets. Index ids
key off the generated Postgres index name; extracted
computeFlatIndexNameOrThrow so the name (and therefore the id) is
computed once with no placeholder.
## Timeline
### What actually changes
- New objects (custom objects created via Settings/metadata API) and
fresh standard installs now get deterministic v5 universalIdentifiers
for all side-effect entities (system fields,
views, view fields, search index, nav command, page layout/tabs/widgets)
instead of random v4().
- The nav-command id formula changed (new owner-scoped) for new objects,
fresh standard installs, and the runtime lookup.
### What does NOT change
- Existing objects' side-effect ids — untouched (no migration;
forward-only).
- Standard object UIDs — untouched
- UI-created custom entities' own ids stay v4 (see scope boundary
above).
- Fresh installs are behaviorally a no-op — ids are internal; re-sync
produces no diff (verified). Nothing user-visible.
### The one real-world impact / risk (existing workspaces)
The nav-command runtime lookup (findNavigationCommandMenuItemForObject)
now computes the new formula, but existing workspaces' nav commands were
stored with the old formula. So on an upgraded existing workspace, until
the PR3 backfill:
- Object activate/deactivate toggle for existing objects won't find the
nav command → re-activating can create a duplicate nav command;
deactivating may no-op.
- Object deletion won't find/clean up the old nav command → orphaned
nav-command row.
### What app developers get right now
Nothing usable yet. The helpers exist in twenty-shared but aren't
re-exported from twenty-sdk (PR2), and app-authored objects still get
SDK-derived ids in the old format until PR2
re-mints them. So "reference a server entity by deterministic id"
doesn't work end-to-end until PR2
|
||
|
Weiko
|
3ee93b5ec9 |
feat(server): add isSystemSideEffect & merge createOneObject/createOneField side-effect migrations (#21673)
## Context When an object is created via the metadata API, `createOneObject` creates its side-effect entities (INDEX view + viewFields, indexes, navigation menu item, "go to" command menu item, record-page fields view, page layout/tabs/widgets) across **three separate `validateBuildAndRunWorkspaceMigration` calls**, purely because the protection behavior (mutations → overrides, delete → deactivate, reset → reactivate) was keyed on *"owned by the standard app"*, forcing the side effects into batches with different application owners. This misrepresents ownership and breaks atomicity. This PR separates two orthogonal concepts: - **Ownership** (`applicationId`), the true owner: the caller's application (the workspace custom app today, 3rd-party apps later). - **Protection** (`isSystemSideEffect`), the row was generated by the system, so user mutations route to overrides, deletion becomes deactivation, and reset restores defaults. Once side effects are re-owned to the caller, the old `applicationId === standardApp` check can no longer tell an original side-effect row from a user-added one so a dedicated `isSystemSideEffect` flag carries the protection instead. This is **PR 1 of 2** (forward-only). It makes newly created objects and fields correct; existing workspaces are handled by a follow-up backfill (see *Out of scope*). ## What this PR does - **`isSystemSideEffect` column** on the 8 affected entities (`view`, `viewField`, `indexMetadata`, `commandMenuItem`, `pageLayout`, `pageLayoutTab`, `pageLayoutWidget`, `fieldMetadata`), with `@WasIntroducedInUpgrade` + an entry in the flat-entity property configuration (`toCompare: true`, read-only). - **Single atomic migration in `createOneObject`**: the three `validateBuildAndRunWorkspaceMigration` calls are merged into one, owned by the caller (`resolvedOwnerFlatApplication`) and the record-page view/fields, page layout, and navigation command item are re-owned to the caller and flagged `isSystemSideEffect: true`. `buildNavigationFlatCommandMenuItem` is parameterized with `applicationUniversalIdentifier` (no longer hardcoded to the standard app). - **Field-creation side effects** (`createManyFields`/`createOneField` already run as a single caller-owned migration, so no re-ownership/merge was needed): the auto-created viewField is flagged `isSystemSideEffect: true`, and a new field now also propagates to the object's **INDEX/table view** (added there as a **hidden** column, `isVisible: false`) in addition to the record-page FIELDS widget. The INDEX view is targeted directly by `key = INDEX` (it is not a page-layout widget), de-duplicated per `(viewId, fieldMetadataUniversalIdentifier)` to respect the per-view unique index. The unique-field index is likewise flagged the inverse relation field stays unflagged (`isSystem: false`). - **Protection predicate** extended: `isCallerOverridingEntity` and the removal/reset split strategies now treat `isSystemSideEffect` rows as protected even when caller-owned (route to overrides / deactivate / reset) and the page-layout-reset guards allow resetting flagged entities. - **Standard compute maps** set the flag consistently so a re-sync produces no diff (standard-object side effects stay `false`; per-object nav command items and custom-object base fields are `true`). - **Read-only GraphQL exposure** of `isSystemSideEffect` on the view / view-field / page-layout / tab / widget / command-menu-item DTOs (not exposed on create/update inputs). => Todo: needs to take this new flag into account. This is fine for now because isSystem remains on object/field. - **Fast instance command** (`2-14`) adding the 8 columns (`NOT NULL DEFAULT false`). ## Scope decisions - **`pageLayout` is not an `OverridableEntity`**, its own row has nothing user-overridable (all customization lives on tabs/widgets). It's dual-purpose (`RECORD_PAGE` side-effect vs. user `DASHBOARD`), so it gets `isSystemSideEffect` for protection only, no `overrides` jsonb. - **`navigationMenuItem` is out of scope.**: Those are side effects only for the metadata API and not marked as "system" (they can be deleted/updated etc...) - **`viewFieldGroup` is not a side effect**, it's only created via the explicit view-field-group API, never by object/field creation, so it gets no flag. ## Out of scope (follow-ups) **PR 2** — slow per-workspace backfill (re-own + flag existing side effects, recreate missing ones) and deterministic v5 identifiers for base fields / pageLayout / tab. <!-- This is an auto-generated description by cubic. --> <a href="https://cubic.dev/pr/twentyhq/twenty/pull/21673?utm_source=github" target="_blank" rel="noopener noreferrer" data-no-image-dialog="true"><picture><source media="(prefers-color-scheme: dark)" srcset="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"><source media="(prefers-color-scheme: light)" srcset="https://www.cubic.dev/buttons/review-in-cubic-light.svg"><img alt="Review in cubic" src="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"></picture></a> <!-- End of auto-generated description by cubic. --> |
||
|
Weiko
|
9c66975520 |
isCustom deprecation for Objects and Fields (#21228)
## Context
`isCustom` was a legacy denormalized boolean on `ObjectMetadataEntity`
and `FieldMetadataEntity`.
Now that every metadata row carries `applicationId` (via
`SyncableEntity`), "is this custom" is fully derivable, and the stored
boolean was a redundant second source of truth that could drift.
The real meaning of `isCustom` is **"the owning application is not the
twenty-standard application"** — i.e. `!belongsToTwentyStandardApp`.
Note this is *not* "belongs to the workspace custom app" as I initially
thought: third-party-application
objects/fields are custom too.
The standard application has a globally stable `universalIdentifier`, so
the value derives with no per-workspace lookup.
## Changed
## `isCustom` checks — before → after
`isCustom` is no longer a stored column. The table below lists every
site that branched on it and how it resolves now. The unifying rule:
`isCustom ≡
!isTwentyStandardApplicationUniversalIdentifier(applicationUniversalIdentifier)`.
### Server — behavioural checks
| Location | Purpose | Before | Now |
|---|---|---|---|
| `utils/compute-object-target-table.util.ts` | Physical table name `_`
prefix | `computeTableName(nameSingular, objectMetadata.isCustom)` |
derives from `applicationUniversalIdentifier` (single source for all
table-name callers) |
| `twenty-orm/factories/entity-schema.factory.ts` +
`…/entity-schema-metadata.type.ts` | ORM table name (hot path) |
`object.isCustom` | `object.applicationId !== standardApplicationId`
(computed in `buildEntitySchemaMetadataMaps`) |
|
`twenty-orm/repository/workspace-{delete,soft-delete,update}-query-builder.ts`
| Table name for mutations | `computeTableName(nameSingular,
objectMetadata.isCustom)` | `computeObjectTargetTable(objectMetadata)` |
| `index-metadata/utils/generate-deterministic-index-name-v2.ts` | Index
name hash (must stay bit-identical) | `flatObjectMetadata.isCustom` |
derives from `applicationUniversalIdentifier` |
| `object-metadata/object-record-count.service.ts` | Table name for
record count | `computeTableName(nameSingular, isCustom)` |
`computeObjectTargetTable(flatObjectMetadata)` |
|
`workspace-manager/dev-seeder/data/services/dev-seeder-data.service.ts`
| Match seed config by table name | `computeTableName(item.nameSingular,
item.isCustom)` | `computeObjectTargetTable(item)` |
| `commands/workspace-export/workspace-export.service.ts` +
`…/utils/generate-workspace-schema-ddl.util.ts` | Export table name (raw
entity) | `objectMetadata.isCustom` |
`!isTwentyStandard…(objectMetadata.application?.universalIdentifier)` |
|
`flat-field-metadata/services/flat-field-metadata-type-validator.service.ts`
| Block users creating reserved field types |
`args.flatEntityToValidate.isCustom` |
`!args.flatEntityToValidate.isSystem` |
| `api/common/.../common-create-many-query-runner.service.ts` | Don't
let client overwrite system `createdBy` |
`createdByFieldMetadata.isCustom === false` |
`createdByFieldMetadata.isSystem === true` |
|
`field-metadata/utils/resolve-field-metadata-standard-override.util.ts`
| Skip i18n/overrides for custom fields | `if (fieldMetadata.isCustom)
return raw` | **removed** — falls through on
`isDefined(standardOverrides)` |
|
`object-metadata/utils/resolve-object-metadata-standard-override.util.ts`
| Skip i18n/overrides for custom objects | `if (objectMetadata.isCustom)
return raw` | **removed** — same fall-through |
|
`command-menu-item/utils/build-navigation-interpolation-context.util.ts`
| Override context for nav labels | passed `isCustom` into resolver |
dropped (resolver no longer needs it) |
| `api/common/.../data-arg-processor.service.ts` | `isCustom` for
record-position table name | `flatObjectMetadata.isCustom` | derives
from `applicationUniversalIdentifier` |
| `metadata-modules/minimal-metadata/minimal-metadata.service.ts` |
Minimal DTO + override context | `flatObjectMetadata.isCustom` | derives
from `applicationUniversalIdentifier` |
|
`commands/upgrade-version-command/1-23/…backfill-record-page-layouts.command.ts`
| Filter to custom objects | `objectMetadata.isCustom` |
`!isTwentyStandard…(applicationUniversalIdentifier)` |
### Server — DTO / API population
| Location | Before | Now |
|---|---|---|
|
`flat-object-metadata/utils/from-flat-object-metadata-to-object-metadata-dto.util.ts`
| passthrough `isCustom` | derives from `applicationUniversalIdentifier`
|
|
`flat-field-metadata/utils/from-flat-field-metadata-to-field-metadata-dto.util.ts`
| passthrough `isCustom` | derives from `applicationUniversalIdentifier`
|
|
`object-metadata/utils/from-object-metadata-entity-to-object-metadata-dto.util.ts`
(REST) | `entity.isCustom` | `entity.applicationId !==
standardApplicationId` |
|
`field-metadata/utils/from-field-metadata-entity-to-field-metadata-dto.util.ts`
(REST) | `entity.isCustom` | `entity.applicationId !==
standardApplicationId` |
| `dataloaders/dataloader.service.ts` | passed
`flatFieldMetadata.isCustom` into override resolver | dropped (resolver
no longer needs it) |
> REST controllers (`object-metadata.controller.ts`,
`field-metadata.controller.ts`) resolve `standardApplicationId` once per
request from the cached `flatApplicationMaps`.
### Frontend
| Location | Purpose | Before | Now |
|---|---|---|---|
| `settings/.../SettingsObjectFieldDisabledActionDropdown.tsx` | Whether
an inactive field is deletable | `isDeletable = isCustomField` |
`isDeletable = isCustomField && !isSystemField` |
### Unchanged (out of scope)
`isCustom` on `IndexMetadata` / `View` / `Skill` / `Agent` and their
guards still read the persisted column.
Breaking change is on the isCustom filter on field and object APIs, this
is never used in the FE and unlikely used by external consumers
|
||
|
Paul Rastoin
|
3d49642d12 |
[AUDIT] Run knip over twenty-server (#21159)
# Introduction Run [knip](https://knip.dev/) over twenty-server Used config: ```json { "$schema": "https://unpkg.com/knip@5/schema.json", "workspaces": { "packages/twenty-server": { "entry": [ "src/main.ts", "src/command/command.ts", "src/queue-worker/queue-worker.ts", "src/database/scripts/setup-db.ts", "src/database/scripts/truncate-db.ts", "src/database/clickHouse/migrations/run-migrations.ts", "src/database/clickHouse/seeds/run-seeds.ts", "src/instrument.ts", "lingui.config.ts", "test/integration/graphql/codegen/index.ts", "test/integration/utils/setup-test.ts", "test/integration/utils/teardown-test.ts", "scripts/**/*.ts", "**/*.spec.ts", "**/*.integration-spec.ts" ], "project": ["src/**/*.ts", "test/**/*.ts", "scripts/**/*.ts"], "ignore": [ "src/database/typeorm/**/migrations/**", "src/database/typeorm/**/*.entity.ts", "**/*.workspace-entity.ts", "**/logic-function-resource/constants/seed-project/**" ], "ignoreDependencies": ["@types/psl", "@types/aws-lambda"], "ignoreBinaries": ["nest", "lingui", "typeorm"] } } } ``` |
||
|
Félix Malfait
|
d602f35cbd |
feat(data-model): custom-indexes management UI and mutations (#20846)
## Summary
Brings indexes management into the per-object Settings tab as a section
under Search (no feature flag, advanced mode only). Admins can create /
delete non-unique indexes with the UI; apps can declare indexes in code
with `defineIndex`. Composite-typed fields are now indexable by picking
a specific sub-column (e.g. `Address > City`).
A few related polish items also land here (invite-user dropdown lands on
the Invite tab; standard warning callout above the new-index form).
## What ships
### UI — custom indexes on per-object Settings
- New section directly under Search, wrapped in
`AdvancedSettingsWrapper`.
- Filter dropdown on the search bar toggles system-index visibility
(shown by default since advanced mode).
- **+ Add Index** button (disabled with tooltip once the per-object cap
is reached) navigates to a dedicated `SettingsObjectNewIndex` page
(matches the field-creation pattern, not a modal):
- Field picker mirrors the webhook event-form layout (rows of dropdowns,
implicit trailing empty row).
- Composite fields surface their sub-properties (`Address > City`,
`Currency > Amount`, …).
- BTREE / GIN type selector.
- Standard warning Callout: "Use indexes sparingly — each one speeds
reads but slows writes."
- Trash icon on `isCustom: true` rows → confirmation modal →
`deleteOneIndex`.
### Server — `createOneIndex` / `deleteOneIndex` mutations
- Gated by `SettingsPermissionGuard(DATA_MODEL)`.
- `IndexMetadataService` wraps the existing migration runner via
`WorkspaceMigrationValidateBuildAndRunService` so the metadata row and
the SQL index land atomically.
- Validation: rejects empty fields, duplicate `(fieldMetadataId,
subFieldName)` pairs, fields not on the object, requires `subFieldName`
for composite parents, forbids `subFieldName` on scalar/relation,
enforces `MAX_CUSTOM_INDEXES_PER_OBJECT = 10`.
- Delete refuses on `isCustom: false` rows so system indexes can't be
removed via this API.
- Dedicated GraphQL exception handler maps each typed error to the right
transport error class.
### Composite sub-field indexing
- Adds `subFieldName: string | null` column to
`IndexFieldMetadataEntity` (fast instance command).
- The flat-entity flow (`UniversalFlatIndexFieldMetadata`,
`FlatIndexFieldMetadata`, `from-universal-flat-index-to-flat-index`,
runner column resolution) all carry `subFieldName` through.
- For composite parents, the runner uses
`computeCompositeColumnName({...}, property)` for the picked sub-column;
for non-composite parents, behavior is unchanged.
- The `'::'` separator encodes `(fieldMetadataId, subFieldName)` for
dedup on the wire; the frontend uses the same separator inside the
Select component's string value.
### Apps can declare indexes in code (`defineIndex`)
- New `IndexManifest` + `IndexFieldManifest` types in
`twenty-shared/application` wired into the `Manifest` type.
- `defineIndex` SDK helper + `IndexConfig`. CLI manifest builder +
extractor recognize `defineIndex` / `ManifestEntityKey.Indexes`.
- Server: `from-index-manifest-to-universal-flat-index` converter
resolves field IDs, validates composite/scalar `subFieldName` rules, and
delegates to `generateFlatIndexMetadataWithNameOrThrow` for the
deterministic name.
- Orchestrator wires the loop after the field-resolution pass;
per-object cap enforced inline against the manifest.
- Cascade on uninstall is automatic — when an app disappears its indexes
drop with it (universal-flat-entity diff handles it).
- Rich-app fixture ships a real `defineIndex` on `PostCard.status`,
exercising the full manifest → install path in CI.
### Closed for now (open later if needed)
- Apps cannot declare `isUnique` indexes — unique constraints stay with
the field-creation flow.
- Apps cannot use a partial-`indexWhereClause` — the UI surface keeps
the framework's hardcoded allowlist.
- UI cannot create unique or partial indexes either; same reasons.
### Cleanups along the way
- Reused the existing `getCompositeSubFieldLabel` +
`COMPOSITE_FIELD_SUB_FIELD_LABELS` (deleted the duplicates I'd created
early in the PR).
- Moved `MAX_CUSTOM_INDEXES_PER_OBJECT` to `twenty-shared/constants`
(single source for FE + BE).
- Replaced inline `isDefined(x) && x !== ''` with `isNonEmptyString`
(from `@sniptt/guards`).
- Hoisted the per-object fields Map + inlined the cap counter into the
indexes orchestrator loop (drops the install scan from O(indexes ×
totalFields) to O(totalFields + indexes)).
- Per design-feedback: page-based create flow (not a modal), filter
dropdown on the SearchInput (not a separate toggle), webhook-style
picker, field icons.
### Unrelated polish that lands here
- "Invite user" link in the multi-workspace dropdown now lands on the
Invite tab directly (`#invite`) instead of the first tab of the members
page.
## Test plan
- [ ] `npx nx typecheck twenty-server / twenty-front / twenty-sdk /
twenty-shared` — passes
- [ ] `npx nx lint:diff-with-main twenty-server / twenty-front` — clean
- [ ] `npx jest index-metadata.service.spec` — green
- [ ] `npx jest from-index-manifest-to-universal-flat-index` — green
(new converter spec, 8 cases)
- [ ] `npx vitest run
src/sdk/define/indexes/__tests__/define-index.spec.ts` (twenty-sdk) —
green (6 cases)
- [ ] `npx vitest run --config vitest.integration.config.ts -t
"rich-app"` — green (rich-app app-dev integration exercises the new
manifest path with the PostCard.status index)
- [ ] Advanced mode → Settings → any object → Settings tab → Indexes
section is visible under Search
- [ ] Create a single-field BTREE index, confirm SQL index exists
(verify via `pg_indexes`)
- [ ] Create a composite-field index (`Address > City`) and confirm the
column is `addressAddressCity`
- [ ] Create an index spanning two columns; column order matches the
picker order
- [ ] Attempt to create an 11th custom index → button is disabled with
tooltip
- [ ] Delete a custom index → confirmation modal → row disappears, PG
index dropped
- [ ] System indexes have no trash icon and are hidden by default
|
||
|
Charles Bochet
|
10876138d2 |
refactor: stop reading joinColumnName from relation field settings (#20304)
## Summary `joinColumnName` on relation field settings is always derivable from the field name (and the target object name for morph relations). This PR stops reading it from settings anywhere in production code; the stored value is no longer used. The settings field is **not** removed from data yet — a follow-up can drop it once we are confident nothing depends on the stored value. ## Helpers The helpers are split by layer because frontend and backend hold morph relations differently: the frontend has a base name plus a `morphRelations[]` array, the backend has one row per target with the name already morph-resolved. | Helper | Layer | When to use | |---|---|---| | `computeRelationGqlFieldJoinColumnName` | Shared / frontend (`gqlField`) | Non-morph relation on the frontend. | | `computeMorphRelationGqlFieldName` | Shared / frontend (`gqlField`) | Need the per-target morph gqlField name (e.g. `targetCompany`). | | `computeMorphRelationGqlFieldJoinColumnName` | Shared / frontend (`gqlField`) | Per-target morph join column on the frontend. Prefer over the non-morph helper for any morph field — it forces the per-target inputs. | | `computeMorphOrRelationFieldJoinColumnName` | Backend (`FlatFieldMetadata.name`) | Any backend read or write — the flat name is already morph-resolved, so one helper covers both cases. | | `computeMorphRelationFlatFieldName` | Backend (`FlatFieldMetadata.name`) | **Mutation paths only** (create / update / object rename). Reads consume the stored `field.name` and never call this. | ## Test plan - [x] Typecheck and lint (front, server, shared) - [x] Existing unit tests pass - [ ] CI green |
||
|
Paul Rastoin
|
8c951d3623 |
Migrate Views-xxx Index Field Object Skill to be fully universal ( all actions and metadata runner and builder ) + all metadata update actions runner (#17687)
# What this PR does Overall naming `universal` versus `flat` is not always the most updated and so on Will make a big cleaning tour after I've finished the whole migration Migrating all `view` and ( filter fields etc ) `field` `object` `index` to the universal pattern on all `services`, `builder` and `runner` levels ## Universal and flat optimistic tooling `addUniversalFlatEntityToUniversalFlatEntityAndRelatedEntityMapsThroughMutationOrThrow` and its delete counterpart maintain the consistency of `UniversalFlatEntityMaps` when an entity is created or removed. Beyond inserting/removing the entity from its own maps, they walk through `ALL_UNIVERSAL_METADATA_RELATIONS` to update the **aggregator arrays** on related parent entities — the add appends the new entity's `universalIdentifier` to the parent's aggregator (e.g. a new viewField's identifier gets appended to its parent view's `viewFieldUniversalIdentifiers`), and the delete filters it out. This keeps the maps in sync so that diff computations and relation lookups remain accurate throughout the migration building process. ## ALL_UNIVERSAL_METADATA_RELATIONS `ALL_UNIVERSAL_METADATA_RELATIONS` is the universal counterpart of `ALL_METADATA_RELATIONS`. It maps each metadata entity to its many-to-one and one-to-many relations using universal foreign keys (`*UniversalIdentifier`) instead of database IDs (`*Id`). This allows migration actions to reference related entities in a workspace-agnostic way. Relations that are workspace-specific (e.g. `workspace`, `dataSource`, `userWorkspace`) are set to `null` and skipped during resolution. ## `workspaceMigrationCreateIdEnrichment` Reserved to API metadata ( will be able to validate at app installation lvl ) - Workspace migration `create` actions now carry an optional `id` (and `fieldIdByUniversalIdentifier` for object/field actions) so that caller-provided IDs flow through the entire build-validate-run pipeline. - New `enrichCreateWorkspaceMigrationActionsWithIds` utility resolves `universalIdentifier → id` mappings after the builder runs and injects them into the migration actions before the runner persists entities. - Runner action handlers use the provided IDs instead of generating new UUIDs, enabling deterministic entity creation for synchronization workflows. ## `resolveUniversalUpdateRelationIdentifiersToIds` `resolveUniversalUpdateRelationIdentifiersToIds` converts universal identifiers (workspace-agnostic, stable keys) in a migration update payload into concrete database UUIDs, so the update can be applied to a specific workspace. It iterates over the many-to-one relations defined in `ALL_UNIVERSAL_METADATA_RELATIONS` for the given entity type, replaces each `*UniversalIdentifier` property with its corresponding `*Id` by looking up the target entity in `allFlatEntityMaps`, and throws if a non-null identifier can't be resolved. Used by all `update` action handlers in `transpileUniversalActionToFlatAction`, avoiding duplicated resolution logic across handlers. ## What this PR does not - Migrating twenty-standard declaration to universal - Migrating all the inputs transpilers to universal - Migrating all metadata to be fully universal ( we still need to de-scope the type of all of them and refactor their validator very close ) --------- Co-authored-by: Charles Bochet <charles@twenty.com> |
||
|
Paul Rastoin
|
e8d4f11d8e |
Standard index as code introduction + type refactor (#16426)
# Introduction Related https://github.com/twentyhq/core-team-issues/issues/1995 In this PR we're introducing the standard index tools for their declaration Also adding a common typing in order to have a simily consistency across standard builders Will implement remaining standard index builder in an other PR |
||
|
Etienne
|
be91d870f9 | Fix - revert standard sync logic on index + fix command (#16421) | ||
|
Etienne
|
68c429a54a | Null equivalence - remove feature flag (#16222) | ||
|
Etienne
|
208c0857ee |
common api - null equivalence (#15926)
closes https://github.com/twentyhq/core-team-issues/issues/1629 To do before requesting review : - filter update Migration to come in an other PR Strat : 1/ Null transformation - [x] Transform NULL equivalent value to NULL in field validation in common api - pre-query - with feature flag - [ ] Same logic in ORM (Not done, complex to handle feature flag here) - [x] Transform NULL value to equivalent in data formatting in ORM - post-query 2/ Migration (in other PR) for fieldMetadata not nullable with default defaultValue (empty string, ...) - [ ] Remove NOT NULL db constraint - [ ] Update record value to NULL - [ ] Update field metadata : isNullable:true - [ ] Update uniqueIndex whereClause (also for standard uniqueIndex) - [ ] Activate feature flag 3/ Update metadata creation - [x] No more default default value - [x] Update standard field nullability - [x] Remove index default whereClause for standard field 4/ Update filter - [x] When filtering on NULL or empty string, be sure all records are returned (the one with NULL + the one with "") 5/ Test - [ ] Strat. to do |
||
|
martmull
|
9880f192a5 | Move composite types to twenty-shared (#15741) | ||
|
Weiko
|
c0ed246a03 |
Add is unique in migration v2 + refactor inferDeletionFromMissingEntities (#15070)
## Goal - inferDeletionFromMissingEntities is now a map instead of a single bool, allowing us to parameterise it based on the entity we want to compare - Adding index creation/update when field isUnique is set to true and index deletion when isUnique is false (for now) close https://github.com/twentyhq/core-team-issues/issues/1346 |
||
|
Paul Rastoin
|
6188c72f74 |
Simplify and enhance v2 type devxp (#15032)
# Introduction This PR introduces a huge type refactor that will leverage dynamic intra entity optimistic flat maps update in the future and also a more granular cache invalidation enhancing performances close https://github.com/twentyhq/core-team-issues/issues/1717 close https://github.com/twentyhq/core-team-issues/issues/1716 close https://github.com/twentyhq/core-team-issues/issues/1643 ## What's done ### Comparators centralization Comparator is now done through global configuration as const for each metadata names Thanks to Note: Definition of standard is evolving, standard is now scoped to an app. Meaning that a manifest should be able to update its own standards objects but on other app standards ones ? Each synchronizable entities will have a standardOverrides ? ## Typing refactor ### `AllFlatEntityTypesByMetadataName` **Single source of truth for the complete type ecosystem**, mapping each metadata name to its entity types, flat entities, and migration actions: ```typescript export type AllFlatEntityTypesByMetadataName = { fieldMetadata: { actions: { created: CreateFieldAction; updated: UpdateFieldAction; deleted: DeleteFieldAction; }; flatEntity: FlatFieldMetadata; entity: FieldMetadataEntity; }; objectMetadata: { /* ... */ }; // ... all 10 metadata types }; ``` ### `ALL_METADATA_NAME_MANY_TO_ONE_RELATIONS` **Explicitly declares database relationships** between entities with compile-time validation: ```typescript export const ALL_METADATA_NAME_MANY_TO_ONE_RELATIONS = { viewField: { view: 'viewId', fieldMetadata: 'fieldMetadataId', }, cronTrigger: { serverlessFunction: 'serverlessFunctionId', }, // ... all relations } as const satisfies MetadataNameAndRelations; ``` ### `ALL_FLAT_ENTITY_CONFIGURATION` **Centralizes comparison and serialization logic** for each metadata type: ```typescript export const ALL_FLAT_ENTITY_CONFIGURATION = { fieldMetadata: { propertiesToCompare: ['name', 'type', 'label', 'defaultValue', /* ... */], propertiesToStringify: ['options', 'settings', 'defaultValue'], }, objectMetadata: { propertiesToCompare: ['nameSingular', 'namePlural', 'isActive', /* ... */], propertiesToStringify: [], }, // ... all metadata types } as const satisfies AllFlatEntityConfiguration; ``` ## Combined Impact These three configurations work together to create a **strongly-typed, centrally-managed metadata system**: 1. **`AllFlatEntityTypesByMetadataName`** defines *what exists* 2. **`ALL_METADATA_NAME_MANY_TO_ONE_RELATIONS`** defines *how they relate* 3. **`ALL_FLAT_ENTITY_CONFIGURATION`** defines *how to compare and serialize them* **Result:** Builders and validators become thin wrappers around type-safe, configuration-driven logic instead of containing scattered, error-prone manual implementations. ## What's next ### StandardOverrides standardization Every metadata entity can be a standard one for a workspace if it's an installed app, which means it might not expose the whole entity api to be editable through an import dynamically The standard overrides logic should not be applied to Fields and Objects but to every entities At the moment we have a logic of `EDITABLE_PROPERTIES` through the api, and also `STANDARD_OVERRIDEDABLE_PROPERTIES` This should be configuration centered like `propertiesToCompare` and `propertiesToStringify`. Scoping this PR to two last for the moment. As update dispatch to standardOverrides could be considered as a side effect prefer waiting to start the side effect refactor ### Granular Optimistic deprecation With this new grain at runtime we will be able to add a flat entity and dispatch its addition to related flat maps, so we don't have to describe an optimistic method for each flat entity operations See `addFlatEntityToFlatEntityAndRelatedEntityMapsOrThrow` Note: Still in wip and included in this PR but about to create a new one to integrate these utils and remove existing methods ### ValidateBuildAndRun dynamic args typed defintion We should restrain the devxp to send expected flat maps entity as at least from to or dependency as we now have the grain both a type lvl and runtime to do so It should not be possible in the devxp to forgot adding the views to the v2 builder when passing the view field anymore ( that would lead to permanent validation error in view field integrity checks ) ## Conclusion Thanks for reading and reviewing ! Any suggestions are more than welcomed ! ( same as for questions too ! ) |
||
|
Félix Malfait
|
e577c2d746 |
Update user friendly errors for translations (#15000)
Force msg typing instead of string for user friendly errors |
||
|
Paul Rastoin
|
170f6d27c5 |
Extract field metadata build out of object metadata build (#14763)
# Introduction Extracting the legacy fields build and dispatch out of the object one to follow the generic flat entity build, also update caches entries for object ## Main tasks - flat field map cache - flat field builder - refactored the dispatch matrix to return flat entity maps instead of flat entity arrays - orchestrator aggregator - removing legacy code - making universal identifier aka standardId of standard field for custom object deterministically dynamic - perfs debug logs for v2 ## TODO - [x] Refactor the generic entity builder to be dependency flat maps in order to main foreign keys list in flat parent - [x] Refactor the flat object metadata to contain the array of related fields and avoid costy find object fields - [ ] Improve the create field handler to handle multiple field at once - [ ] Refactor the dispatch to embbed the comparison - [ ] Improve perf by extracting from elements out of existing - [ ] Fix the labelIdentifierId validators on object before field creation ( integ tests are in failing mode ) ## Debug logs snippet ```ts [EntityBuilder fieldMetadata] matrix computation: 0.027ms [EntityBuilder fieldMetadata] creation validation: 0.001ms [EntityBuilder fieldMetadata] deletion validation: 0.293ms [EntityBuilder fieldMetadata] update validation: 0.006ms [EntityBuilder fieldMetadata] entity processing: 0.363ms [EntityBuilder fieldMetadata] validateAndBuild: 0.455ms [EntityBuilder index] matrix computation: 0.005ms [EntityBuilder index] creation validation: 0.001ms [EntityBuilder index] deletion validation: 0.146ms [EntityBuilder index] update validation: 0.004ms [EntityBuilder index] entity processing: 0.199ms [EntityBuilder index] validateAndBuild: 0.228ms [Runner] Initial cache retrieval: 0.549ms [BaseWorkspaceMigrationRunnerActionHandlerService] delete_index executeForWorkspaceSchema: 11.665ms [BaseWorkspaceMigrationRunnerActionHandlerService] delete_index executeForMetadata: 12.864ms [BaseWorkspaceMigrationRunnerActionHandlerService] delete_field executeForWorkspaceSchema: 1.476ms [BaseWorkspaceMigrationRunnerActionHandlerService] delete_field executeForMetadata: 6.816ms [BaseWorkspaceMigrationRunnerActionHandlerService] delete_field executeForWorkspaceSchema: 0.062ms [BaseWorkspaceMigrationRunnerActionHandlerService] delete_field executeForMetadata: 0.889ms [Runner] Transaction execution: 23.434ms [Runner] Cache invalidation: 316.662ms [Runner] Total execution: 340.767ms ``` As you can see cache invalidation is way to long, we could replace the cache by the optimistic in the end |
||
|
Paul Rastoin
|
fbd3286792 |
Index v2 side effects (#14567)
# Introduction Honestly this implem is a mess, discussing a potential side effect handler with @weiko before the build and run that would handle each side effect per entity and operation Handling: - [ ] unique index is generated when a field is updated with the `isUnique` - [x] an index is generated when a relation is created - [x] search vector index creation on custom object creation - [x] renaming a field metadata or an object should re-create all related indexes which are composed by their namings - [x] delete object should remove any related indexes - [x] delete field should update related indexes ( if index ends up empty it should be removed ) - [ ] on object renaming that contains morph fields -> triggers update field -> trigger index recompute - [x] on update name renaming should recompute all related indexes ## TODO - [x] Integration testing - [ ] Refactor the index maps cache to be storing a `idsByObjectMetadataId` - [x] Refactor deterministic name to use order sorting - [x] Remove flat index from flat object ## What's next Will handle morph indexes in a new dedicated PR for the moment will stick to this Same for the cache improvement and uniqueness |
||
|
Etienne
|
e06fc34cbe |
Unique fields - add unicity toggle on data model settings (#13592)
To review after https://github.com/twentyhq/twenty/pull/13539 closes https://github.com/twentyhq/core-team-issues/issues/1224 --------- Co-authored-by: Charles Bochet <charles@twenty.com> |
||
|
Etienne
|
7ce96049b8 |
Unique field - add unique property creation/deletion on field (#13539)
Done : - add isUnique prop availability on gql update/create fieldMetadata resolvers - add unique index creation logic at update & creation - add unique index deletion logic at update - update unique index if field name updated - edge cases : can't have default value and unique fields / standard default value excluded from index (where clause) / can't have composite unique fields / can't create unique fields on MORPH closes https://github.com/twentyhq/core-team-issues/issues/1222 |
||
|
Etienne
|
741924751b |
Connect - Import Relation (#13419)
re-opened https://github.com/twentyhq/twenty/pull/13213 |
||
|
Weiko
|
79f3fbb016 |
Revert "Connect - Relation on FE Importer (#13213)" (#13313)
This reverts commit
|
||
|
Etienne
|
cc71394863 |
Connect - Relation on FE Importer (#13213)
Done : - Relation connect on FE Importer - Remove templating on SpreadsheetMatchedColumn type - Remove useless files on import - Remove AvailableFieldsForImport type + Update SpreadsheetImportField type and SpreadsheetImportFieldOption To test : - Try import opportunities on Apple wk [using this file](https://github.com/user-attachments/files/21233720/Test.import.-.opportunities-sample.csv) closes : https://github.com/twentyhq/core-team-issues/issues/1090 |
||
|
Marie
|
1cb60f943e |
[field-level permissions] Upsert fieldPermission + use fieldPermission to compute permissions (#13050)
In this PR
- introduction of fieldPermission entity
- addition of upsertFieldPermission in role resolver
- computing of permissions taking fieldPermission into account. In order
to limit what is stored in Redis we only store fields restrictions. For
instance for objectMetadata with id XXX with a restriction on field with
id YYY we store:
`"XXX":{"canRead":true,"canUpdate":false,"canSoftDelete":false,"canDestroy":false,"restrictedFields":{"YYY":{"canRead":false,"canUpdate":null}}}`
---------
Co-authored-by: Charles Bochet <charlesBochet@users.noreply.github.com>
|
||
|
Etienne
|
ba67e0d5f4 |
Connect - Update Gql schema generation (#13001)
To test :
```
mutation testMutation {
createPerson(
data: {company: {connect: {where: {domainName: {primaryLinkUrl: "airbnb.com"}}}}}
) {
id
}
}
```
closes https://github.com/twentyhq/core-team-issues/issues/1167
|
||
|
Weiko
|
e13dc7a1fc |
[FlexibleSchema] Add IndexMetadata decorator (#5981)
## Context
Our Flexible Schema engine dynamically generates entities/tables/APIs
for us but was not flexible enough to build indexes in the DB. With more
and more features involving heavy queries such as Messaging, we are now
adding a new WorkspaceIndex() decorator for our standard objects (will
come later for custom objects). This decorator will give enough
information to the workspace sync metadata manager to generate the
proper migrations that will create or drop indexes on demand.
To be aligned with the rest of the engine, we are adding 2 new tables:
IndexMetadata and IndexFieldMetadata, that will store the info of our
indexes.
## Implementation
```typescript
@WorkspaceEntity({
standardId: STANDARD_OBJECT_IDS.person,
namePlural: 'people',
labelSingular: 'Person',
labelPlural: 'People',
description: 'A person',
icon: 'IconUser',
})
export class PersonWorkspaceEntity extends BaseWorkspaceEntity {
@WorkspaceField({
standardId: PERSON_STANDARD_FIELD_IDS.email,
type: FieldMetadataType.EMAIL,
label: 'Email',
description: 'Contact’s Email',
icon: 'IconMail',
})
@WorkspaceIndex()
email: string;
```
By simply adding the WorkspaceIndex decorator, sync-metadata command
will create a new index for that column.
We can also add composite indexes, note that the order is important for
PSQL.
```typescript
@WorkspaceEntity({
standardId: STANDARD_OBJECT_IDS.person,
namePlural: 'people',
labelSingular: 'Person',
labelPlural: 'People',
description: 'A person',
icon: 'IconUser',
})
@WorkspaceIndex(['phone', 'email'])
export class PersonWorkspaceEntity extends BaseWorkspaceEntity {
```
Currently composite fields and relation fields are not handled by
@WorkspaceIndex() and you will need to use this notation instead
```typescript
@WorkspaceIndex(['companyId', 'nameFirstName'])
export class PersonWorkspaceEntity extends BaseWorkspaceEntity {
```
<img width="700" alt="Screenshot 2024-06-21 at 15 15 45"
src="https://github.com/twentyhq/twenty/assets/1834158/ac6da1d9-d315-40a4-9ba6-6ab9ae4709d4">
Next step: We might need to implement more complex index expressions,
this is why we have an expression column in IndexMetadata.
What I had in mind for the decorator, still open to discussion
```typescript
@WorkspaceIndex(['nameFirstName', 'nameLastName'], { expression: "$1 || ' ' || $2"})
export class PersonWorkspaceEntity extends BaseWorkspaceEntity {
```
---------
Co-authored-by: Charles Bochet <charles@twenty.com>
|
