af0af0a237a9fbba0f5c759d5799f221eefc4586
61 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Marie
|
77409b6eb2 |
[Requires "warm" cache flush (no immediate downtime before flush)] Migrate viewGroup.fieldMetadataId -> view.mainGroupByFieldMetadataId (1/3) (#16206)
In this PR (1/3) - introduce view.mainGroupByFieldMetadataId as the new reference determining which fieldMetadataId is used in a grouped view, in order to deprecate viewGroup.fieldMetadataId which creates inconsistencies. view.mainGroupByFieldMetadataId is now filled at every view creation, though not in use yet. - Introduce a command to backfill view.mainGroupByFieldMetadataId for existing views + delete all viewGroup.fieldMetadataId with a fieldMetadataId that is not view.mainGroupByFieldMetadataId. (It should concern 37 active workspaces) - Temporarily disable the option to change a grouped view's fieldMetadataId as for now it creates inconsistencies. This feature can be reintroduced when we have done the full migration. In a next PR - (2/3) use view.mainGroupByFieldMetadataId instead of viewGroup.fieldMetadataId. In FE we may keep viewGroup.fieldMetadataId as a state (TBD). View groups will now be created / deleted as a side effect of view's mainGroupByFieldMetadataId update. - (3/3) remove viewGroup.fieldMetadataId --------- Co-authored-by: Charles Bochet <charles@twenty.com> |
||
|
Paul Rastoin
|
ea3c5d2d45 |
Migrate role and role target to v2 (#16009)
# Introduction close https://github.com/twentyhq/core-team-issues/issues/1930 close https://github.com/twentyhq/core-team-issues/issues/1929 Migrating role and roleTarget entities to the v2 core engine, allowing v2 caching leverage and allow migrating agent to v2 that needs role target in prior After agent we should be able to pass twenty standard app totally though workspace migration ## Role target assignation Please note that role target have 3 creation entrypoints: - Agent - User workspace - ApiKey Refactored all 3 of them to pass through a new role-target.service.ts that consumes the v2 under the hood. --------- Co-authored-by: Weiko <corentin@twenty.com> |
||
|
Paul Rastoin
|
3514054235 |
V2 centralize relation optimistic logic (#15552)
# Introduction
This PR aims to deprecate having to manually handle optimistic side
effect foreign key addition in the whole v2 experience.
This PR implements the strong basis + builder refactor of the optimistic
computation of a given flat entity maps with its related flat entity
maps ( runner needs a small refactor on actions type definition first )
Flat entity maps updates through mutations are now only scoped to the
generic entity builder ( very isolated )
## What's next
- Refactor actions v2 type definition to gain grain over `metadataName`
and action operation ( `create` `delete` `update` ).
from `{type: 'create_view_field'}` to `{metadataName: 'view_field',
type: 'create' }`
- Use new optimistic tool computation tools
- Only invalidate impacted flat maps cache
## New tools
Strictly dynamically typed new flat entity maps tools
- `addFlatEntityToFlatEntityAndRelatedEntityMapsThroughMutationOrThrow`
-
`deleteFlatEntityFromFlatEntityAndRelatedEntityMapsThroughMutationOrThrow`
## Unit test
Adding basic unit testing coverage to introduced tools
## `FlatEntityValidationArgs`
From
```ts
export type FlatEntityValidationArgs<T extends AllMetadataName> = {
flatEntityToValidate: MetadataFlatEntity<T>;
optimisticFlatEntityMaps: MetadataFlatEntityMaps<T>;
mutableDependencyOptimisticFlatEntityMaps: MetadataValidationRelatedFlatEntityMaps<T>;
workspaceId: string;
remainingFlatEntityMapsToValidate: MetadataFlatEntityMaps<T>;
buildOptions: WorkspaceMigrationBuilderOptions;
};
```
To
```ts
export type FlatEntityValidationArgs<T extends AllMetadataName> = {
flatEntityToValidate: MetadataFlatEntity<T>;
optimisticFlatEntityMapsAndRelatedFlatEntityMaps: MetadataFlatEntityAndRelatedFlatEntityMapsForValidation<T>;
workspaceId: string;
remainingFlatEntityMapsToValidate: MetadataFlatEntityMaps<T>;
buildOptions: WorkspaceMigrationBuilderOptions;
};
```
|
||
|
Paul Rastoin
|
d640b93096 |
Improve v2 and cache invalidation perfs (#15467)
# Introduction Log are debug logs of `packages/twenty-server/test/integration/metadata/suites/object-metadata/create-delete-and-create-object-metadata-v2.integration-spec.ts`run ten times in a row on clean db reset ## Next Will improve cache computation to lighter invalidation. RelationLoad `query` does not seem to work with typeorm so I'll continue the custom integration i've started in https://github.com/twentyhq/twenty/tree/optimize-cache-read-v2 ## Integration tests duration Significant test duration improvement too ### Before <img width="2632" height="1402" alt="image" src="https://github.com/user-attachments/assets/2f1f0ccf-44de-4856-bfe1-4f45a351763a" /> ### After <img width="2632" height="1402" alt="image" src="https://github.com/user-attachments/assets/4bc9e6db-3046-48b9-b903-1464053936c4" /> ## What's next - The legacy cache invalidation removal - Factorizing redis calls in only one operation ## Autogenerated performance comparison ( including mutation refactor too ) [Before](https://gist.github.com/prastoin/3c1e21fa9e3b3ce4b0716902ff4a2dd6) [After](https://gist.github.com/prastoin/7bfddd14bfded2e4991a9378970a026d) The optimized implementation shows **dramatic performance improvements** across all metrics: - 🚀 **Cache Invalidation**: 156.3ms → 76.8ms (**50.9% faster**) - 🚀 **Builder Operations**: 21.3ms → 14.2ms (**33.3% faster**) - ⚡ **Consistency**: 16.4% more predictable performance --- ## 1. Overall Performance Summary | Component | Before (avg) | After (avg) | Best (After) | Worst (After) | Improvement | |-----------|-------------|-------------|--------------|---------------|-------------| | **Total Execution Time** | 180.2ms | 110.5ms | 52.3ms | 585.9ms | **38.7% faster** ⚡⚡ | | **Cache Invalidation** | 156.3ms | 76.8ms | 47.8ms | 285.1ms | **50.9% faster** ⚡⚡⚡ | | **Transaction Execution** | 22.4ms | 22.1ms | 0.99ms | 314.2ms | Similar | | **Initial Cache Retrieval** | 0.81ms | 2.08ms | 0.21ms | 8.24ms | Similar | | **Entity Builder (total)** | 21.3ms | 14.2ms | 0.36ms | 42.5ms | **33.3% faster** ⚡ | ### Total Execution Time Distribution #### Before (Legacy Sequential) ``` Time (ms) Count Percentage Visualization < 150 18 16% ████ 150-180 32 29% ███████ 180-210 35 32% ████████ 210-250 17 15% ████ 250-350 6 5% █ > 350 2 2% ▌ ``` #### After (Optimized Parallel) ``` Time (ms) Count Percentage Visualization < 70 28 31% ████████ 70-100 31 34% █████████ 100-150 18 20% █████ 150-200 8 9% ██ 200-300 4 4% █ > 300 2 2% ▌ ``` --- ## 2. Builder Performance Breakdown ### Field Metadata Builder | Operation | Before (avg) | After (avg) | Improvement | |-----------|-------------|-------------|-------------| | Matrix computation | 3.5ms | 3.4ms | Similar | | Creation validation | 15.2ms | 2.1ms | **86% faster** ⚡⚡⚡ | | Deletion validation | 0.08ms | 0.06ms | Similar | | Update validation | 1.3ms | 0.09ms | **93% faster** ⚡⚡⚡ | | Entity processing | 18.6ms | 11.8ms | **37% faster** ⚡ | | **Total validateAndBuild** | **21.3ms** | **14.2ms** | **33% faster** ⚡ | #### Performance Distribution ``` Before: ▁▂▄█████▆▄▂▁ (wide spread, 15-28ms range) After: ▁▁▃█████▃▁▁ (tight clustering, 10-18ms range) ``` ## 4. Cache Invalidation Performance Breakdown ### Cache Invalidation Summary | Metric | Before (Legacy) | After (Optimized) | Improvement | |--------|-----------------|-------------------|-------------| | **Best Time** | 131.965ms | 47.833ms | **63.7% faster** ⚡⚡⚡ | | **10th Percentile** | 140.2ms | 51.7ms | **63.1% faster** ⚡⚡⚡ | | **25th Percentile** | 146.1ms | 54.4ms | **62.7% faster** ⚡⚡⚡ | | **Median (50th)** | 155.1ms | 63.4ms | **59.1% faster** ⚡⚡⚡ | | **Average** | 156.3ms | 76.8ms | **50.9% faster** ⚡⚡⚡ | | **75th Percentile** | 160.2ms | 90.2ms | **43.7% faster** ⚡⚡ | | **90th Percentile** | 191.7ms | 100.2ms | **47.7% faster** ⚡⚡ | | **95th Percentile** | 235.6ms | 110.3ms | **53.2% faster** ⚡⚡⚡ | | **99th Percentile** | 278.2ms | 224.9ms | **19.1% faster** ⚡ | | **Worst Time** | 383.914ms | 285.102ms | **25.7% faster** ⚡ | ### Cache Invalidation Time Distribution #### Before (Legacy Sequential) ``` Time (ms) Count Percentage Visualization 130-140 3 3% ▊ 140-150 15 14% ████ 150-160 48 44% ███████████ 160-180 31 28% ███████ 180-220 8 7% ██ 220-280 3 3% ▊ > 280 2 2% ▌ ``` #### After (Optimized Parallel + Intersection) ``` Time (ms) Count Percentage Visualization < 50 3 3% ▊ 50-60 27 25% ███████ 60-70 25 23% ██████ 70-90 25 23% ██████ 90-100 15 14% ████ 100-120 8 7% ██ 120-150 3 3% ▊ > 150 4 4% █ ``` ## 6. Performance Consistency Analysis ### Standard Deviation & Variance | Metric | Before | After | Improvement | |--------|--------|-------|-------------| | **Cache Invalidation Std Dev** | 42.1ms | 35.2ms | **16.4% more consistent** ⚡ | | **Total Execution Std Dev** | 68.3ms | 89.1ms | Slightly more variable | | **Coefficient of Variation (Cache)** | 26.9% | 45.8% | More variance | | **Outliers (> 2σ)** | 5 cases | 3 cases | **40% fewer outliers** ⚡ | --- |
||
|
Paul Rastoin
|
267af42412 |
Centralize v2 errors types in twenty-shared (#15358)
# Introduction Followup of https://github.com/twentyhq/twenty/pull/15331 ( Reducing size by concerns ) This PR centralizes v2 format error types in `twenty-shared` and consuming them in the existing v2 error format logic in `twenty-server` ## Next This https://github.com/twentyhq/twenty/pull/15360 handles the frontend v2 format error refactor ## Conclusion Related to https://github.com/twentyhq/core-team-issues/issues/1776 |
||
|
Paul Rastoin
|
2e84c11eae |
[v2_FIX] Update standard object/field (#15233)
# Introduction Refactoring the standard overrides dispatcher to only pass over fields to has to be dispatched in the standardOverrides entry and let the other side effects resulting from out of standard overrides mutation trigger Related to https://github.com/twentyhq/core-team-issues/issues/1753 ## This allows - standard field settings, options etc updates and so on ## Remark - Determine what we should do on object deactivation ( right now in production we can still access deactivated object relation properties and so on e.g deactivate opportunities still accessible from a view field on company ( still have to re-create it as it has been deleted ) => decided to leave as it is right now, `isActive` could be considered as uiDeactivated in the end - We should also add forbidden standard field mutations validation inside the builder itself ( here we want to early return in the api input transpiler too as we don't want to spread invalid side effects ) => or in the end we could just centralize both but it will generate several errors ## Coverage ```ts PASS test/integration/metadata/suites/object-metadata/successful-update-one-standard-object-metadata.integration-spec.ts PASS test/integration/metadata/suites/field-metadata/successful-update-one-standard-field-metadata.integration-spec.ts PASS test/integration/metadata/suites/object-metadata/failing-update-one-standard-object-metadata.integration-spec.ts PASS test/integration/metadata/suites/field-metadata/failing-update-one-standard-field-metadata.integration-spec.ts Test Suites: 4 passed, 4 total Tests: 18 passed, 18 total Snapshots: 16 passed, 16 total Time: 8.721 s, estimated 10 s ``` ## Update post review Faced a behavior where updating back the company label to its original value would result in storing this value in the standard overrides Refactored both field and object transpilation behavior to rather remove the standard override value instead and let fallback on original value Yes it's quite duplicated will factorize once we move this inside the builder |
||
|
Paul Rastoin
|
45473218d3 |
Field deactivation side effect views calendar kanban viewFields (#15180)
# Introduction
Handling both:
- field deactivation side effect on view fields, view filters and views
- field deactivation side effect on view that targets it as
`kanbanAggregateFieldMetadataId`
- field deactivation side effect on view that targets it as
`calendarFieldMetadataId`
## Coverage
added coverage
```ts
PASS test/integration/metadata/suites/field-metadata/kanban-aggregate-field-deactivation-deletes-views.integration-spec.ts (13.132 s)
kanban-aggregate-field-deactivation-nullifies-kanban-properties
✓ should nullify kanban properties when field used as kanbanAggregateOperationFieldMetadataId is deactivated (3923 ms)
✓ should not modify views when field not used as kanbanAggregateOperationFieldMetadataId is deactivated (2958 ms)
✓ should nullify kanban properties on multiple views when they all use the same field as kanbanAggregateOperationFieldMetadataId (2542 ms)
✓ should nullify kanban properties when views have different aggregate operations on same field (3380 ms)
Test Suites: 1 passed, 1 total
Tests: 4 passed, 4 total
Snapshots: 0 total
Time: 13.154 s
```
```ts
PASS test/integration/metadata/suites/field-metadata/view-group-field-deactivation-deletes-views.integration-spec.ts (12.639 s)
view-group-field-deactivation-deletes-views
✓ should delete view when field used in view group is deactivated (3469 ms)
✓ should not delete view when field not used in view group is deactivated (3109 ms)
✓ should delete multiple views when they all use the same field in view groups (2741 ms)
✓ should handle deactivation when view has multiple view groups with different fields (3008 ms)
Test Suites: 1 passed, 1 total
Tests: 4 passed, 4 total
Snapshots: 0 total
Time: 12.664 s
```
```ts
PASS test/integration/metadata/suites/field-metadata/calendar-field-deactivation-deletes-views.integration-spec.ts (14.579 s)
calendar-field-deactivation-deletes-views
✓ should delete view when field used as calendarFieldMetadataId is deactivated (3388 ms)
✓ should not delete view when field not used as calendarFieldMetadataId is deactivated (2438 ms)
✓ should delete multiple views when they all use the same field as calendarFieldMetadataId (2635 ms)
✓ should handle deactivation when views have different calendar layouts on same field (3195 ms)
✓ should delete calendar view but not other view types when calendar field is deactivated (2682 ms)
Test Suites: 1 passed, 1 total
Tests: 5 passed, 5 total
Snapshots: 0 total
Time: 14.601 s, estimated 15 s
```
## View soft deletion
We decided to remove the soft deletion grain on all the views, in this
PR context we've only removed soft deleted validation requirement on any
view entities
## Conclusion
close https://github.com/twentyhq/core-team-issues/issues/1754
|
||
|
Weiko
|
cceeb6ed4d |
Add applicationId to syncableEntity and fix syncApp deletion (#15170)
## Context - All flatEntity should extend SyncableEntity - SyncableEntity should now have applicationId and application relation - Fix syncApp deletion, should now properly use migration v2 to delete syncable entities |
||
|
martmull
|
b16ab1b7c9 |
1518 extensibility front add an application section in settings (#15056)
Protected by IS_APPLICATION_ENABLED featureFlag Add `Application` section in settings <img width="301" height="137" alt="image" src="https://github.com/user-attachments/assets/ee53bdd2-36f6-45c6-8646-17b1e08abf00" /> A `settings/applications` route listing all installed applications <img width="661" height="428" alt="image" src="https://github.com/user-attachments/assets/69d534c4-4e9e-452a-a3d9-ded0223bb457" /> Introduce a new Tag for application managed items <img width="885" height="759" alt="image" src="https://github.com/user-attachments/assets/19767be5-61e5-4bd2-a51d-54ed9bfb1923" /> A `settings/applications/<application_id>` details setting page listing all objects, serverlessFunctions and agents created by the application: <img width="917" height="778" alt="image" src="https://github.com/user-attachments/assets/7fc056a6-1d73-4242-b2eb-6f8955d8597d" /> A `settings/applications/<application_id>/<serverless_function_id>` <img width="905" height="652" alt="image" src="https://github.com/user-attachments/assets/56ca0021-26bf-42cb-9abf-34879f16050a" /> Add trigger tab in serverless function details (readonly for now) <img width="899" height="724" alt="image" src="https://github.com/user-attachments/assets/5eeefa35-f2a4-4fd8-a640-7b5c5891f226" /> Set object, serverless and agent setting detail pages readonly for managed items <img width="1075" height="859" alt="image" src="https://github.com/user-attachments/assets/57c73d69-4980-47a2-b752-8dc5ab494530" /> <img width="648" height="582" alt="image" src="https://github.com/user-attachments/assets/5ad5f3f7-3bc3-4e40-870a-4981c6492524" /> <img width="982" height="692" alt="image" src="https://github.com/user-attachments/assets/7ad756c4-5d33-4a0a-9eb8-416c040362b9" /> <img width="1077" height="647" alt="image" src="https://github.com/user-attachments/assets/e086b9f5-4062-4d10-82a9-4023de3cad3f" /> |
||
|
Paul Rastoin
|
7d747c9876 |
[REQUIRES_CACHE_FLUSH][GQL_VIEW_GROUP_API_BREAKING_CHANGE] ViewGroup in v2 (#15052)
# Introduction Adding view-group to core engine v2 Following https://github.com/twentyhq/twenty/pull/15010 ( same pattern ) ## What's done - Created flat-view-group - flat view group runner - flat view group builder - create view group service v2 and input transpilers - refactor the existing view group resolver to fix standard ( BREAKING_CHANGE on graphql api update especially ) REST stays the same - refactored the front to consume the mutations autogenerated close https://github.com/twentyhq/core-team-issues/issues/1665 |
||
|
Paul Rastoin
|
6188c72f74 |
Simplify and enhance v2 type devxp (#15032)
# Introduction This PR introduces a huge type refactor that will leverage dynamic intra entity optimistic flat maps update in the future and also a more granular cache invalidation enhancing performances close https://github.com/twentyhq/core-team-issues/issues/1717 close https://github.com/twentyhq/core-team-issues/issues/1716 close https://github.com/twentyhq/core-team-issues/issues/1643 ## What's done ### Comparators centralization Comparator is now done through global configuration as const for each metadata names Thanks to Note: Definition of standard is evolving, standard is now scoped to an app. Meaning that a manifest should be able to update its own standards objects but on other app standards ones ? Each synchronizable entities will have a standardOverrides ? ## Typing refactor ### `AllFlatEntityTypesByMetadataName` **Single source of truth for the complete type ecosystem**, mapping each metadata name to its entity types, flat entities, and migration actions: ```typescript export type AllFlatEntityTypesByMetadataName = { fieldMetadata: { actions: { created: CreateFieldAction; updated: UpdateFieldAction; deleted: DeleteFieldAction; }; flatEntity: FlatFieldMetadata; entity: FieldMetadataEntity; }; objectMetadata: { /* ... */ }; // ... all 10 metadata types }; ``` ### `ALL_METADATA_NAME_MANY_TO_ONE_RELATIONS` **Explicitly declares database relationships** between entities with compile-time validation: ```typescript export const ALL_METADATA_NAME_MANY_TO_ONE_RELATIONS = { viewField: { view: 'viewId', fieldMetadata: 'fieldMetadataId', }, cronTrigger: { serverlessFunction: 'serverlessFunctionId', }, // ... all relations } as const satisfies MetadataNameAndRelations; ``` ### `ALL_FLAT_ENTITY_CONFIGURATION` **Centralizes comparison and serialization logic** for each metadata type: ```typescript export const ALL_FLAT_ENTITY_CONFIGURATION = { fieldMetadata: { propertiesToCompare: ['name', 'type', 'label', 'defaultValue', /* ... */], propertiesToStringify: ['options', 'settings', 'defaultValue'], }, objectMetadata: { propertiesToCompare: ['nameSingular', 'namePlural', 'isActive', /* ... */], propertiesToStringify: [], }, // ... all metadata types } as const satisfies AllFlatEntityConfiguration; ``` ## Combined Impact These three configurations work together to create a **strongly-typed, centrally-managed metadata system**: 1. **`AllFlatEntityTypesByMetadataName`** defines *what exists* 2. **`ALL_METADATA_NAME_MANY_TO_ONE_RELATIONS`** defines *how they relate* 3. **`ALL_FLAT_ENTITY_CONFIGURATION`** defines *how to compare and serialize them* **Result:** Builders and validators become thin wrappers around type-safe, configuration-driven logic instead of containing scattered, error-prone manual implementations. ## What's next ### StandardOverrides standardization Every metadata entity can be a standard one for a workspace if it's an installed app, which means it might not expose the whole entity api to be editable through an import dynamically The standard overrides logic should not be applied to Fields and Objects but to every entities At the moment we have a logic of `EDITABLE_PROPERTIES` through the api, and also `STANDARD_OVERRIDEDABLE_PROPERTIES` This should be configuration centered like `propertiesToCompare` and `propertiesToStringify`. Scoping this PR to two last for the moment. As update dispatch to standardOverrides could be considered as a side effect prefer waiting to start the side effect refactor ### Granular Optimistic deprecation With this new grain at runtime we will be able to add a flat entity and dispatch its addition to related flat maps, so we don't have to describe an optimistic method for each flat entity operations See `addFlatEntityToFlatEntityAndRelatedEntityMapsOrThrow` Note: Still in wip and included in this PR but about to create a new one to integrate these utils and remove existing methods ### ValidateBuildAndRun dynamic args typed defintion We should restrain the devxp to send expected flat maps entity as at least from to or dependency as we now have the grain both a type lvl and runtime to do so It should not be possible in the devxp to forgot adding the views to the v2 builder when passing the view field anymore ( that would lead to permanent validation error in view field integrity checks ) ## Conclusion Thanks for reading and reviewing ! Any suggestions are more than welcomed ! ( same as for questions too ! ) |