Commit Graph

9 Commits

Author SHA1 Message Date
Paul Rastoin 98a9ae2a0e Migrate view filter group to v2 (#16876)
## Introduction
On the side hanlded PR with a // agents on another repo
Made several iterations to fix behavior and direction
Find below auto-generated PR description

closes https://github.com/twentyhq/core-team-issues/issues/2037

Created generic tooling for entity circular dep checking, will be useful
for permissions validation too @Weiko


## Migrate `viewFilterGroup` entity to v2 flat architecture

### Summary
Migrates the `viewFilterGroup` entity from v1 to the v2 flat entity
architecture, following the established patterns for other v2 entities
like `viewFilter`, `view`, and `viewField`.

### Changes

**Types & Constants**
- Added `FlatViewFilterGroup` and `FlatViewFilterGroupMaps` types
- Added editable properties constant for `viewFilterGroup`
- Registered `viewFilterGroup` in `ALL_METADATA_NAME`,
`ALL_METADATA_RELATION_PROPERTIES`,
`ALL_METADATA_MANY_TO_ONE_RELATIONS`, and related constants

**Cache Service**
- Created `WorkspaceFlatViewFilterGroupMapCacheService` with proper
relation loading for `viewFilters` and `childViewFilterGroups`
- Updated `WorkspaceFlatViewMapCacheService` to load `viewFilterGroups`
relation

**Builder & Validator**
- Created `WorkspaceMigrationV2ViewFilterGroupActionsBuilderService`
- Created `FlatViewFilterGroupValidatorService` with creation, update,
and deletion validation
- Integrated validation into the orchestrator service (runs before
`viewFilter` validation)

**Action Handlers**
- Created create, update, and delete action handlers for
`viewFilterGroup`

**Service Migration**
- Rewrote `ViewFilterGroupService` to use v2 migration pattern with
`WorkspaceMigrationValidateBuildAndRunService`
- Created utility functions for transforming DTOs to flat entities

**Database Migration**
- Added migration to make `parentViewFilterGroupId` foreign key
deferrable (handles self-referential parent/child insertions)

**ViewFilter Integration**
- Added `viewFilterGroupId` validation in
`FlatViewFilterValidatorService`
- Updated `viewFilter` many-to-one relations to include
`viewFilterGroup`

**Tests**
- Added integration tests for successful creation, update, deletion, and
destruction
- Added failing test cases for non-existent entities and invalid
references
- Added failing test for `viewFilter` creation with non-existent
`viewFilterGroupId`

### Breaking Changes
None - existing API contracts are preserved.
2026-01-02 14:16:01 +01:00
Weiko 15b21570ae Row level permissions - POC 1 (#16599)
## Context
This PR adds the core structure for RLS implementation:
- RLS data model
- RLS service layer
- RLS WorkspaceMigration and Syncable Entity + cache + Validations
- RLS resolver layer
- ORM layer with RLS Predicate to ORM WHERE clause conversion with
workspaceMember record transposition

Tests are missing though

<!-- CURSOR_SUMMARY -->
---

> [!NOTE]
> Establishes core row-level permissions infrastructure and enforcement
across the stack.
> 
> - Backend: new `rowLevelPermissionPredicate` and
`rowLevelPermissionPredicateGroup` entities, TypeORM migration, feature
flag `IS_ROW_LEVEL_PERMISSION_PREDICATES_ENABLED`, flat-entity
maps/cache wiring, services and GraphQL resolvers for CRUD, and
inclusion of `workspaceMember` in auth context
> - ORM: applies row-level permission predicates to SELECT, DELETE, and
SOFT DELETE query builders; propagates context through
GlobalWorkspaceOrmManager/EntityManager
> - GraphQL: generated schema/types/queries/mutations for
creating/updating/deleting/fetching predicates and groups
> - Frontend: settings page adds a gated "Record-level" section
(placeholder) and metadata error handler labels for new entities
> 
> <sup>Written by [Cursor
Bugbot](https://cursor.com/dashboard?tab=bugbot) for commit
fe955cc4588a92157afa6795fb574189a4be1e93. This will update automatically
on new commits. Configure
[here](https://cursor.com/dashboard?tab=bugbot).</sup>
<!-- /CURSOR_SUMMARY -->

---------

Co-authored-by: Félix Malfait <felix.malfait@gmail.com>
2025-12-31 16:15:17 +01:00
Paul Rastoin 47a8a15598 Metadata modules for PageLayoutTab PageLayoutWidget (#16662)
# Introduction
Creating dedicated folders and module for both `page-layout-tab` and
`page-layout-widget`

The addition diff with deletion is due to the module being added
2025-12-18 11:15:29 +00:00
Raphaël Bosi 5fb7e76005 Migrate page layout to v2 (#16364) 2025-12-08 14:11:24 +00:00
Raphaël Bosi 077be7644c Migrate page layout widget to v2 of the API (#16323) 2025-12-05 13:04:06 +00:00
Paul Rastoin 59672e3e34 Migrate agent v2 (#16214)
# Introduction
Closes https://github.com/twentyhq/core-team-issues/issues/1980

In this PR we migrate the agent from v1 to v2.

## New FlatRoleTargetByAgentIdMaps
Derivated the `flatRoleTargetMaps` to be building a
`flatRoleTargetByAgentIdMaps` to ease retrieving a roleId to associate
to an agent

## Coverage
Added strong coverage on both failing and successful CRU agents
operations

---------

Co-authored-by: Weiko <corentin@twenty.com>
2025-12-03 11:51:19 +01:00
Weiko 1eb2e44058 Refactor workspace cache service (#16208)
## Context
We've recently introduced a new workspace cache service which now acts
as a cache access and local storage for all workspace related data,
deprecating the individual specific services.
- Better performance through multiple caching/fetching strategies
- Consistent data access patterns across the codebase
- Reduced redis queries through MGET/MSET/PIPELINE with multiple cache
keys
2025-12-01 17:08:21 +01:00
Paul Rastoin d640b93096 Improve v2 and cache invalidation perfs (#15467)
# Introduction
Log are debug logs of
`packages/twenty-server/test/integration/metadata/suites/object-metadata/create-delete-and-create-object-metadata-v2.integration-spec.ts`run
ten times in a row on clean db reset

## Next
Will improve cache computation to lighter invalidation.
RelationLoad `query` does not seem to work with typeorm so I'll continue
the custom integration i've started in
https://github.com/twentyhq/twenty/tree/optimize-cache-read-v2

## Integration tests duration
Significant test duration improvement too
### Before
<img width="2632" height="1402" alt="image"
src="https://github.com/user-attachments/assets/2f1f0ccf-44de-4856-bfe1-4f45a351763a"
/>

### After
<img width="2632" height="1402" alt="image"
src="https://github.com/user-attachments/assets/4bc9e6db-3046-48b9-b903-1464053936c4"
/>


## What's next
- The legacy cache invalidation removal
- Factorizing redis calls in only one operation

## Autogenerated performance comparison ( including mutation refactor
too )

[Before](https://gist.github.com/prastoin/3c1e21fa9e3b3ce4b0716902ff4a2dd6)

[After](https://gist.github.com/prastoin/7bfddd14bfded2e4991a9378970a026d)

The optimized implementation shows **dramatic performance improvements**
across all metrics:

- 🚀 **Cache Invalidation**: 156.3ms → 76.8ms (**50.9% faster**)
- 🚀 **Builder Operations**: 21.3ms → 14.2ms (**33.3% faster**)
-  **Consistency**: 16.4% more predictable performance

---

## 1. Overall Performance Summary

| Component | Before (avg) | After (avg) | Best (After) | Worst (After)
| Improvement |

|-----------|-------------|-------------|--------------|---------------|-------------|
| **Total Execution Time** | 180.2ms | 110.5ms | 52.3ms | 585.9ms |
**38.7% faster**  |
| **Cache Invalidation** | 156.3ms | 76.8ms | 47.8ms | 285.1ms | **50.9%
faster**  |
| **Transaction Execution** | 22.4ms | 22.1ms | 0.99ms | 314.2ms |
Similar |
| **Initial Cache Retrieval** | 0.81ms | 2.08ms | 0.21ms | 8.24ms |
Similar |
| **Entity Builder (total)** | 21.3ms | 14.2ms | 0.36ms | 42.5ms |
**33.3% faster**  |

### Total Execution Time Distribution

#### Before (Legacy Sequential)
```
Time (ms)    Count  Percentage  Visualization
< 150         18      16%       ████
150-180       32      29%       ███████
180-210       35      32%       ████████
210-250       17      15%       ████
250-350        6       5%       █
> 350          2       2%       ▌
```

#### After (Optimized Parallel)
```
Time (ms)    Count  Percentage  Visualization
< 70          28      31%       ████████
70-100        31      34%       █████████
100-150       18      20%       █████
150-200        8       9%       ██
200-300        4       4%       █
> 300          2       2%       ▌
```
---

## 2. Builder Performance Breakdown

### Field Metadata Builder

| Operation | Before (avg) | After (avg) | Improvement |
|-----------|-------------|-------------|-------------|
| Matrix computation | 3.5ms | 3.4ms | Similar |
| Creation validation | 15.2ms | 2.1ms | **86% faster**  |
| Deletion validation | 0.08ms | 0.06ms | Similar |
| Update validation | 1.3ms | 0.09ms | **93% faster**  |
| Entity processing | 18.6ms | 11.8ms | **37% faster**  |
| **Total validateAndBuild** | **21.3ms** | **14.2ms** | **33% faster**
 |

#### Performance Distribution
```
Before:  ▁▂▄█████▆▄▂▁     (wide spread, 15-28ms range)
After:   ▁▁▃█████▃▁▁       (tight clustering, 10-18ms range)
```

## 4. Cache Invalidation Performance Breakdown

### Cache Invalidation Summary

| Metric | Before (Legacy) | After (Optimized) | Improvement |
|--------|-----------------|-------------------|-------------|
| **Best Time** | 131.965ms | 47.833ms | **63.7% faster**  |
| **10th Percentile** | 140.2ms | 51.7ms | **63.1% faster**  |
| **25th Percentile** | 146.1ms | 54.4ms | **62.7% faster**  |
| **Median (50th)** | 155.1ms | 63.4ms | **59.1% faster**  |
| **Average** | 156.3ms | 76.8ms | **50.9% faster**  |
| **75th Percentile** | 160.2ms | 90.2ms | **43.7% faster**  |
| **90th Percentile** | 191.7ms | 100.2ms | **47.7% faster**  |
| **95th Percentile** | 235.6ms | 110.3ms | **53.2% faster**  |
| **99th Percentile** | 278.2ms | 224.9ms | **19.1% faster**  |
| **Worst Time** | 383.914ms | 285.102ms | **25.7% faster**  |

### Cache Invalidation Time Distribution

#### Before (Legacy Sequential)
```
Time (ms)    Count  Percentage  Visualization
130-140       3       3%        ▊
140-150      15      14%        ████
150-160      48      44%        ███████████
160-180      31      28%        ███████
180-220       8       7%        ██
220-280       3       3%        ▊
> 280         2       2%        ▌
```

#### After (Optimized Parallel + Intersection)
```
Time (ms)    Count  Percentage  Visualization
< 50          3       3%        ▊
50-60        27      25%        ███████
60-70        25      23%        ██████
70-90        25      23%        ██████
90-100       15      14%        ████
100-120       8       7%        ██
120-150       3       3%        ▊
> 150         4       4%        █
```

## 6. Performance Consistency Analysis

### Standard Deviation & Variance

| Metric | Before | After | Improvement |
|--------|--------|-------|-------------|
| **Cache Invalidation Std Dev** | 42.1ms | 35.2ms | **16.4% more
consistent**  |
| **Total Execution Std Dev** | 68.3ms | 89.1ms | Slightly more variable
|
| **Coefficient of Variation (Cache)** | 26.9% | 45.8% | More variance |
| **Outliers (> 2σ)** | 5 cases | 3 cases | **40% fewer outliers**  |

---
2025-10-31 16:20:51 +00:00
Paul Rastoin 6188c72f74 Simplify and enhance v2 type devxp (#15032)
# Introduction
This PR introduces a huge type refactor that will leverage dynamic intra
entity optimistic flat maps update in the future and also a more
granular cache invalidation enhancing performances

close https://github.com/twentyhq/core-team-issues/issues/1717
close https://github.com/twentyhq/core-team-issues/issues/1716
close https://github.com/twentyhq/core-team-issues/issues/1643

## What's done

### Comparators centralization
Comparator is now done through global configuration as const for each
metadata names
Thanks to 

Note:
Definition of standard is evolving, standard is now scoped to an app.
Meaning that a manifest should be able to update its own standards
objects but on other app standards ones ? Each synchronizable entities
will have a standardOverrides ?

## Typing refactor

### `AllFlatEntityTypesByMetadataName`

**Single source of truth for the complete type ecosystem**, mapping each
metadata name to its entity types, flat entities, and migration actions:

```typescript
export type AllFlatEntityTypesByMetadataName = {
  fieldMetadata: {
    actions: { created: CreateFieldAction; updated: UpdateFieldAction; deleted: DeleteFieldAction; };
    flatEntity: FlatFieldMetadata;
    entity: FieldMetadataEntity;
  };
  objectMetadata: { /* ... */ };
  // ... all 10 metadata types
};
```

### `ALL_METADATA_NAME_MANY_TO_ONE_RELATIONS`

**Explicitly declares database relationships** between entities with
compile-time validation:

```typescript
export const ALL_METADATA_NAME_MANY_TO_ONE_RELATIONS = {
  viewField: {
    view: 'viewId',
    fieldMetadata: 'fieldMetadataId',
  },
  cronTrigger: {
    serverlessFunction: 'serverlessFunctionId',
  },
  // ... all relations
} as const satisfies MetadataNameAndRelations;
```

### `ALL_FLAT_ENTITY_CONFIGURATION`

**Centralizes comparison and serialization logic** for each metadata
type:

```typescript
export const ALL_FLAT_ENTITY_CONFIGURATION = {
  fieldMetadata: {
    propertiesToCompare: ['name', 'type', 'label', 'defaultValue', /* ... */],
    propertiesToStringify: ['options', 'settings', 'defaultValue'],
  },
  objectMetadata: {
    propertiesToCompare: ['nameSingular', 'namePlural', 'isActive', /* ... */],
    propertiesToStringify: [],
  },
  // ... all metadata types
} as const satisfies AllFlatEntityConfiguration;
```

## Combined Impact

These three configurations work together to create a **strongly-typed,
centrally-managed metadata system**:

1. **`AllFlatEntityTypesByMetadataName`** defines *what exists*
2. **`ALL_METADATA_NAME_MANY_TO_ONE_RELATIONS`** defines *how they
relate*
3. **`ALL_FLAT_ENTITY_CONFIGURATION`** defines *how to compare and
serialize them*

**Result:** Builders and validators become thin wrappers around
type-safe, configuration-driven logic instead of containing scattered,
error-prone manual implementations.

## What's next

### StandardOverrides standardization
Every metadata entity can be a standard one for a workspace if it's an
installed app, which means it might not expose the whole entity api to
be editable through an import dynamically
The standard overrides logic should not be applied to Fields and Objects
but to every entities

At the moment we have a logic of `EDITABLE_PROPERTIES` through the api,
and also `STANDARD_OVERRIDEDABLE_PROPERTIES`
This should be configuration centered like `propertiesToCompare` and
`propertiesToStringify`.
Scoping this PR to two last for the moment. As update dispatch to
standardOverrides could be considered as a side effect prefer waiting to
start the side effect refactor

### Granular Optimistic deprecation
With this new grain at runtime we will be able to add a flat entity and
dispatch its addition to related flat maps, so we don't have to describe
an optimistic method for each flat entity operations

See `addFlatEntityToFlatEntityAndRelatedEntityMapsOrThrow`

Note: Still in wip and included in this PR but about to create a new one
to integrate these utils and remove existing methods

### ValidateBuildAndRun dynamic args typed defintion
We should restrain the devxp to send expected flat maps entity as at
least from to or dependency as we now have the grain both a type lvl and
runtime to do so
It should not be possible in the devxp to forgot adding the views to the
v2 builder when passing the view field anymore ( that would lead to
permanent validation error in view field integrity checks )

## Conclusion
Thanks for reading and reviewing !
Any suggestions are more than welcomed ! ( same as for questions too ! )
2025-10-13 10:31:34 +02:00