476bdf764c5faa5f19c5c81abb6beccdc717d302
4 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Paul Rastoin
|
476bdf764c |
Refactor flat entity maps to be universal oriented (#17665)
# Introduction
In preparation of the workspace agnostic builder, we're migrating
`FlatEntityMaps` to be universal identifier oriented and based
As in the builder context there're won't be any ids at all
Please also note that the FlatEntity is a UniversalFlatEntity superset
From
```ts
import { type SyncableFlatEntity } from 'src/engine/metadata-modules/flat-entity/types/flat-entity-from.type';
export type FlatEntityMaps<T extends SyncableFlatEntity> = {
byId: Partial<Record<string, T>>;
idByUniversalIdentifier: Partial<Record<string, string>>;
universalIdentifiersByApplicationId: Partial<Record<string, string[]>>;
};
```
To
```ts
export type FlatEntityMaps<
T extends SyncableFlatEntity | UniversalSyncableFlatEntity,
> = {
byUniversalIdentifier: Partial<Record<string, T>>;
universalIdentifierById: Partial<Record<string, string>>;
universalIdentifiersByApplicationId: Partial<Record<string, string[]>>; // this might make more sense to be migrated to universalIdentifiersByApplicationUniversalIdentifier but it's the main topic of this PR
};
```
## Low level maps tools
Had to refactor find | create | delete | replace | find-many | get-sub
tools ( through mutations and or throw equivalent )
|
||
|
Paul Rastoin
|
4c93ab5259 |
Introduce UniversalFlatEntityFrom (#17367)
# Introduction
Creating a `UniversalFlatEntityFrom` that strips out all the relation
and foreignKey properties in order to replace them with
`UniversalIdentifier` suffix
This data type will be major for the workspace migration workspace
agnostic refactor
## Chore
- renamed `flat-entity.type` to `flat-entity-from.type.ts` ( more
accurate to exported module )
- create static test type over the field metadata entity on quite
complex utils as both coverage and documentation
## Example
Here's an example of a `UniversalFlatEntityFrom<FieldMetadataEntity>`
```ts
const universalFlatFieldMetadata: UniversalFlatFieldMetadata<FieldMetadataType.RELATION> = {
// Base properties (from FieldMetadataEntity, excluding relations and applicationId)
universalIdentifier: '550e8400-e29b-41d4-a716-446655440001',
applicationUniversalIdentifier: '5800681c-088e-4e2b-9fc3-bcf6e8ec2051',
type: FieldMetadataType.RELATION,
name: 'firstName',
label: 'First Name',
defaultValue: null,
description: 'The first name of the person',
icon: 'IconUser',
standardOverrides: null,
options: null,
settings: {
relationType: RelationType.ONE_TO_MANY,
},
isCustom: false,
isActive: true,
isSystem: false,
isUIReadOnly: false,
isNullable: true,
isUnique: false,
isLabelSyncedWithName: true,
morphId: null,
// Date properties cast to string
createdAt: '2024-01-15T10:30:00.000Z',
updatedAt: '2024-01-15T10:30:00.000Z',
// ManyToOne relation universal identifiers (from FieldMetadataEntity relations)
relationTargetFieldMetadataUniversalIdentifier:
'550e8400-e29b-41d4-a716-446655440012',
relationTargetObjectMetadataUniversalIdentifier:
'550e8400-e29b-41d4-a716-446655440013',
// Join column universal identifiers (foreignKey -> universalIdentifier)
objectMetadataUniversalIdentifier: '550e8400-e29b-41d4-a716-446655440010',
// OneToMany relation universal identifiers (array of related entity identifiers)
viewFieldUniversalIdentifiers: [
'550e8400-e29b-41d4-a716-446655440020',
'550e8400-e29b-41d4-a716-446655440021',
],
viewFilterUniversalIdentifiers: ['550e8400-e29b-41d4-a716-446655440030'],
kanbanAggregateOperationViewUniversalIdentifiers: [],
calendarViewUniversalIdentifiers: [],
mainGroupByFieldMetadataViewUniversalIdentifiers: [],
};
```
## Settings
Will hop on the settings typing next. Might not be dynamic but
declarative though
|
||
|
Charles Bochet
|
a18203934c |
Fix flat entity maps date serialization (#16420)
Changes: - as we store date in redis as serialized, let's make all flatEntity dates as string. This requires changing FlatEntity types and making sure that entity are converted to flatEntity and flatEntity to dtos |
||
|
Paul Rastoin
|
d640b93096 |
Improve v2 and cache invalidation perfs (#15467)
# Introduction Log are debug logs of `packages/twenty-server/test/integration/metadata/suites/object-metadata/create-delete-and-create-object-metadata-v2.integration-spec.ts`run ten times in a row on clean db reset ## Next Will improve cache computation to lighter invalidation. RelationLoad `query` does not seem to work with typeorm so I'll continue the custom integration i've started in https://github.com/twentyhq/twenty/tree/optimize-cache-read-v2 ## Integration tests duration Significant test duration improvement too ### Before <img width="2632" height="1402" alt="image" src="https://github.com/user-attachments/assets/2f1f0ccf-44de-4856-bfe1-4f45a351763a" /> ### After <img width="2632" height="1402" alt="image" src="https://github.com/user-attachments/assets/4bc9e6db-3046-48b9-b903-1464053936c4" /> ## What's next - The legacy cache invalidation removal - Factorizing redis calls in only one operation ## Autogenerated performance comparison ( including mutation refactor too ) [Before](https://gist.github.com/prastoin/3c1e21fa9e3b3ce4b0716902ff4a2dd6) [After](https://gist.github.com/prastoin/7bfddd14bfded2e4991a9378970a026d) The optimized implementation shows **dramatic performance improvements** across all metrics: - 🚀 **Cache Invalidation**: 156.3ms → 76.8ms (**50.9% faster**) - 🚀 **Builder Operations**: 21.3ms → 14.2ms (**33.3% faster**) - ⚡ **Consistency**: 16.4% more predictable performance --- ## 1. Overall Performance Summary | Component | Before (avg) | After (avg) | Best (After) | Worst (After) | Improvement | |-----------|-------------|-------------|--------------|---------------|-------------| | **Total Execution Time** | 180.2ms | 110.5ms | 52.3ms | 585.9ms | **38.7% faster** ⚡⚡ | | **Cache Invalidation** | 156.3ms | 76.8ms | 47.8ms | 285.1ms | **50.9% faster** ⚡⚡⚡ | | **Transaction Execution** | 22.4ms | 22.1ms | 0.99ms | 314.2ms | Similar | | **Initial Cache Retrieval** | 0.81ms | 2.08ms | 0.21ms | 8.24ms | Similar | | **Entity Builder (total)** | 21.3ms | 14.2ms | 0.36ms | 42.5ms | **33.3% faster** ⚡ | ### Total Execution Time Distribution #### Before (Legacy Sequential) ``` Time (ms) Count Percentage Visualization < 150 18 16% ████ 150-180 32 29% ███████ 180-210 35 32% ████████ 210-250 17 15% ████ 250-350 6 5% █ > 350 2 2% ▌ ``` #### After (Optimized Parallel) ``` Time (ms) Count Percentage Visualization < 70 28 31% ████████ 70-100 31 34% █████████ 100-150 18 20% █████ 150-200 8 9% ██ 200-300 4 4% █ > 300 2 2% ▌ ``` --- ## 2. Builder Performance Breakdown ### Field Metadata Builder | Operation | Before (avg) | After (avg) | Improvement | |-----------|-------------|-------------|-------------| | Matrix computation | 3.5ms | 3.4ms | Similar | | Creation validation | 15.2ms | 2.1ms | **86% faster** ⚡⚡⚡ | | Deletion validation | 0.08ms | 0.06ms | Similar | | Update validation | 1.3ms | 0.09ms | **93% faster** ⚡⚡⚡ | | Entity processing | 18.6ms | 11.8ms | **37% faster** ⚡ | | **Total validateAndBuild** | **21.3ms** | **14.2ms** | **33% faster** ⚡ | #### Performance Distribution ``` Before: ▁▂▄█████▆▄▂▁ (wide spread, 15-28ms range) After: ▁▁▃█████▃▁▁ (tight clustering, 10-18ms range) ``` ## 4. Cache Invalidation Performance Breakdown ### Cache Invalidation Summary | Metric | Before (Legacy) | After (Optimized) | Improvement | |--------|-----------------|-------------------|-------------| | **Best Time** | 131.965ms | 47.833ms | **63.7% faster** ⚡⚡⚡ | | **10th Percentile** | 140.2ms | 51.7ms | **63.1% faster** ⚡⚡⚡ | | **25th Percentile** | 146.1ms | 54.4ms | **62.7% faster** ⚡⚡⚡ | | **Median (50th)** | 155.1ms | 63.4ms | **59.1% faster** ⚡⚡⚡ | | **Average** | 156.3ms | 76.8ms | **50.9% faster** ⚡⚡⚡ | | **75th Percentile** | 160.2ms | 90.2ms | **43.7% faster** ⚡⚡ | | **90th Percentile** | 191.7ms | 100.2ms | **47.7% faster** ⚡⚡ | | **95th Percentile** | 235.6ms | 110.3ms | **53.2% faster** ⚡⚡⚡ | | **99th Percentile** | 278.2ms | 224.9ms | **19.1% faster** ⚡ | | **Worst Time** | 383.914ms | 285.102ms | **25.7% faster** ⚡ | ### Cache Invalidation Time Distribution #### Before (Legacy Sequential) ``` Time (ms) Count Percentage Visualization 130-140 3 3% ▊ 140-150 15 14% ████ 150-160 48 44% ███████████ 160-180 31 28% ███████ 180-220 8 7% ██ 220-280 3 3% ▊ > 280 2 2% ▌ ``` #### After (Optimized Parallel + Intersection) ``` Time (ms) Count Percentage Visualization < 50 3 3% ▊ 50-60 27 25% ███████ 60-70 25 23% ██████ 70-90 25 23% ██████ 90-100 15 14% ████ 100-120 8 7% ██ 120-150 3 3% ▊ > 150 4 4% █ ``` ## 6. Performance Consistency Analysis ### Standard Deviation & Variance | Metric | Before | After | Improvement | |--------|--------|-------|-------------| | **Cache Invalidation Std Dev** | 42.1ms | 35.2ms | **16.4% more consistent** ⚡ | | **Total Execution Std Dev** | 68.3ms | 89.1ms | Slightly more variable | | **Coefficient of Variation (Cache)** | 26.9% | 45.8% | More variance | | **Outliers (> 2σ)** | 5 cases | 3 cases | **40% fewer outliers** ⚡ | --- |