Improve v2 and cache invalidation perfs (#15467)

# Introduction
Log are debug logs of
`packages/twenty-server/test/integration/metadata/suites/object-metadata/create-delete-and-create-object-metadata-v2.integration-spec.ts`run
ten times in a row on clean db reset

## Next
Will improve cache computation to lighter invalidation.
RelationLoad `query` does not seem to work with typeorm so I'll continue
the custom integration i've started in
https://github.com/twentyhq/twenty/tree/optimize-cache-read-v2

## Integration tests duration
Significant test duration improvement too
### Before
<img width="2632" height="1402" alt="image"
src="https://github.com/user-attachments/assets/2f1f0ccf-44de-4856-bfe1-4f45a351763a"
/>

### After
<img width="2632" height="1402" alt="image"
src="https://github.com/user-attachments/assets/4bc9e6db-3046-48b9-b903-1464053936c4"
/>


## What's next
- The legacy cache invalidation removal
- Factorizing redis calls in only one operation

## Autogenerated performance comparison ( including mutation refactor
too )

[Before](https://gist.github.com/prastoin/3c1e21fa9e3b3ce4b0716902ff4a2dd6)

[After](https://gist.github.com/prastoin/7bfddd14bfded2e4991a9378970a026d)

The optimized implementation shows **dramatic performance improvements**
across all metrics:

- 🚀 **Cache Invalidation**: 156.3ms → 76.8ms (**50.9% faster**)
- 🚀 **Builder Operations**: 21.3ms → 14.2ms (**33.3% faster**)
-  **Consistency**: 16.4% more predictable performance

---

## 1. Overall Performance Summary

| Component | Before (avg) | After (avg) | Best (After) | Worst (After)
| Improvement |

|-----------|-------------|-------------|--------------|---------------|-------------|
| **Total Execution Time** | 180.2ms | 110.5ms | 52.3ms | 585.9ms |
**38.7% faster**  |
| **Cache Invalidation** | 156.3ms | 76.8ms | 47.8ms | 285.1ms | **50.9%
faster**  |
| **Transaction Execution** | 22.4ms | 22.1ms | 0.99ms | 314.2ms |
Similar |
| **Initial Cache Retrieval** | 0.81ms | 2.08ms | 0.21ms | 8.24ms |
Similar |
| **Entity Builder (total)** | 21.3ms | 14.2ms | 0.36ms | 42.5ms |
**33.3% faster**  |

### Total Execution Time Distribution

#### Before (Legacy Sequential)
```
Time (ms)    Count  Percentage  Visualization
< 150         18      16%       ████
150-180       32      29%       ███████
180-210       35      32%       ████████
210-250       17      15%       ████
250-350        6       5%       █
> 350          2       2%       ▌
```

#### After (Optimized Parallel)
```
Time (ms)    Count  Percentage  Visualization
< 70          28      31%       ████████
70-100        31      34%       █████████
100-150       18      20%       █████
150-200        8       9%       ██
200-300        4       4%       █
> 300          2       2%       ▌
```
---

## 2. Builder Performance Breakdown

### Field Metadata Builder

| Operation | Before (avg) | After (avg) | Improvement |
|-----------|-------------|-------------|-------------|
| Matrix computation | 3.5ms | 3.4ms | Similar |
| Creation validation | 15.2ms | 2.1ms | **86% faster**  |
| Deletion validation | 0.08ms | 0.06ms | Similar |
| Update validation | 1.3ms | 0.09ms | **93% faster**  |
| Entity processing | 18.6ms | 11.8ms | **37% faster**  |
| **Total validateAndBuild** | **21.3ms** | **14.2ms** | **33% faster**
 |

#### Performance Distribution
```
Before:  ▁▂▄█████▆▄▂▁     (wide spread, 15-28ms range)
After:   ▁▁▃█████▃▁▁       (tight clustering, 10-18ms range)
```

## 4. Cache Invalidation Performance Breakdown

### Cache Invalidation Summary

| Metric | Before (Legacy) | After (Optimized) | Improvement |
|--------|-----------------|-------------------|-------------|
| **Best Time** | 131.965ms | 47.833ms | **63.7% faster**  |
| **10th Percentile** | 140.2ms | 51.7ms | **63.1% faster**  |
| **25th Percentile** | 146.1ms | 54.4ms | **62.7% faster**  |
| **Median (50th)** | 155.1ms | 63.4ms | **59.1% faster**  |
| **Average** | 156.3ms | 76.8ms | **50.9% faster**  |
| **75th Percentile** | 160.2ms | 90.2ms | **43.7% faster**  |
| **90th Percentile** | 191.7ms | 100.2ms | **47.7% faster**  |
| **95th Percentile** | 235.6ms | 110.3ms | **53.2% faster**  |
| **99th Percentile** | 278.2ms | 224.9ms | **19.1% faster**  |
| **Worst Time** | 383.914ms | 285.102ms | **25.7% faster**  |

### Cache Invalidation Time Distribution

#### Before (Legacy Sequential)
```
Time (ms)    Count  Percentage  Visualization
130-140       3       3%        ▊
140-150      15      14%        ████
150-160      48      44%        ███████████
160-180      31      28%        ███████
180-220       8       7%        ██
220-280       3       3%        ▊
> 280         2       2%        ▌
```

#### After (Optimized Parallel + Intersection)
```
Time (ms)    Count  Percentage  Visualization
< 50          3       3%        ▊
50-60        27      25%        ███████
60-70        25      23%        ██████
70-90        25      23%        ██████
90-100       15      14%        ████
100-120       8       7%        ██
120-150       3       3%        ▊
> 150         4       4%        █
```

## 6. Performance Consistency Analysis

### Standard Deviation & Variance

| Metric | Before | After | Improvement |
|--------|--------|-------|-------------|
| **Cache Invalidation Std Dev** | 42.1ms | 35.2ms | **16.4% more
consistent**  |
| **Total Execution Std Dev** | 68.3ms | 89.1ms | Slightly more variable
|
| **Coefficient of Variation (Cache)** | 26.9% | 45.8% | More variance |
| **Outliers (> 2σ)** | 5 cases | 3 cases | **40% fewer outliers**  |

---
This commit is contained in:
Paul Rastoin
2025-10-31 17:20:51 +01:00
committed by GitHub
parent 2d1a89c4c2
commit d640b93096
61 changed files with 1120 additions and 846 deletions
@@ -6,7 +6,7 @@ import { FieldMetadataType } from 'twenty-shared/types';
import { getFlatFieldMetadataMock } from 'src/engine/metadata-modules/flat-field-metadata/__mocks__/get-flat-field-metadata.mock';
import { getFlatObjectMetadataMock } from 'src/engine/metadata-modules/flat-object-metadata/__mocks__/get-flat-object-metadata.mock';
import { EMPTY_ORCHESTRATOR_ACTIONS_REPORT } from 'src/engine/workspace-manager/workspace-migration-v2/constant/empty-orchestrator-actions-report.constant';
import { createEmptyOrchestratorActionsReport } from 'src/engine/workspace-manager/workspace-migration-v2/constant/empty-orchestrator-actions-report.constant';
import { type OrchestratorActionsReport } from 'src/engine/workspace-manager/workspace-migration-v2/types/workspace-migration-orchestrator.type';
import { aggregateOrchestratorActionsReportCreateObjectAndCreateFieldActions } from 'src/engine/workspace-manager/workspace-migration-v2/utils/aggregate-orchestrator-actions-report-create-object-and-create-field-actions.util';
import { type CreateFieldAction } from 'src/engine/workspace-manager/workspace-migration-v2/workspace-migration-builder-v2/builders/field/types/workspace-migration-field-action-v2';
@@ -26,7 +26,7 @@ describe('aggregateOrchestratorActionsReportCreateObjectAndCreateFieldActions',
'should aggregate single object with multiple fields into one object action',
context: {
input: {
...EMPTY_ORCHESTRATOR_ACTIONS_REPORT,
...createEmptyOrchestratorActionsReport(),
objectMetadata: {
created: [
{
@@ -83,7 +83,7 @@ describe('aggregateOrchestratorActionsReportCreateObjectAndCreateFieldActions',
'should keep separate field actions when no matching object action exists',
context: {
input: {
...EMPTY_ORCHESTRATOR_ACTIONS_REPORT,
...createEmptyOrchestratorActionsReport(),
objectMetadata: {
created: [],
updated: [],
@@ -147,7 +147,7 @@ describe('aggregateOrchestratorActionsReportCreateObjectAndCreateFieldActions',
title: 'should handle multiple objects with their respective fields',
context: {
input: {
...EMPTY_ORCHESTRATOR_ACTIONS_REPORT,
...createEmptyOrchestratorActionsReport(),
objectMetadata: {
created: [
{
@@ -228,7 +228,7 @@ describe('aggregateOrchestratorActionsReportCreateObjectAndCreateFieldActions',
'should handle mixed scenario with some fields merged and some kept separate',
context: {
input: {
...EMPTY_ORCHESTRATOR_ACTIONS_REPORT,
...createEmptyOrchestratorActionsReport(),
objectMetadata: {
created: [
{
@@ -293,7 +293,7 @@ describe('aggregateOrchestratorActionsReportCreateObjectAndCreateFieldActions',
'should aggregate multiple field actions for the same object when no create object action exists',
context: {
input: {
...EMPTY_ORCHESTRATOR_ACTIONS_REPORT,
...createEmptyOrchestratorActionsReport(),
objectMetadata: {
created: [],
updated: [],
@@ -356,7 +356,7 @@ describe('aggregateOrchestratorActionsReportCreateObjectAndCreateFieldActions',
{
title: 'should handle empty actions report',
context: {
input: EMPTY_ORCHESTRATOR_ACTIONS_REPORT,
input: createEmptyOrchestratorActionsReport(),
expected: {
expectCreateFieldActionPerObjectMetadataId: {},
expectCreateObjectActionPerObjectMetadataId: {},
@@ -3,7 +3,7 @@ import {
type EachTestingContext,
} from 'twenty-shared/testing';
import { EMPTY_ORCHESTRATOR_ACTIONS_REPORT } from 'src/engine/workspace-manager/workspace-migration-v2/constant/empty-orchestrator-actions-report.constant';
import { createEmptyOrchestratorActionsReport } from 'src/engine/workspace-manager/workspace-migration-v2/constant/empty-orchestrator-actions-report.constant';
import { type OrchestratorActionsReport } from 'src/engine/workspace-manager/workspace-migration-v2/types/workspace-migration-orchestrator.type';
import { aggregateOrchestratorActionsReportDeleteObjectAndDeleteFieldActions } from 'src/engine/workspace-manager/workspace-migration-v2/utils/aggregate-orchestrator-actions-report-delete-object-and-delete-field.util';
import { type DeleteFieldAction } from 'src/engine/workspace-manager/workspace-migration-v2/workspace-migration-builder-v2/builders/field/types/workspace-migration-field-action-v2';
@@ -23,7 +23,7 @@ describe('aggregateOrchestratorActionsReportDeleteObjectAndDeleteFieldActions',
title: 'should remove field actions when parent object is being deleted',
context: {
input: {
...EMPTY_ORCHESTRATOR_ACTIONS_REPORT,
...createEmptyOrchestratorActionsReport(),
objectMetadata: {
created: [],
updated: [],
@@ -63,7 +63,7 @@ describe('aggregateOrchestratorActionsReportDeleteObjectAndDeleteFieldActions',
title: 'should keep field actions when no parent object is being deleted',
context: {
input: {
...EMPTY_ORCHESTRATOR_ACTIONS_REPORT,
...createEmptyOrchestratorActionsReport(),
objectMetadata: {
created: [],
updated: [],
@@ -94,7 +94,7 @@ describe('aggregateOrchestratorActionsReportDeleteObjectAndDeleteFieldActions',
'should handle mixed scenario with some fields removed and some kept',
context: {
input: {
...EMPTY_ORCHESTRATOR_ACTIONS_REPORT,
...createEmptyOrchestratorActionsReport(),
objectMetadata: {
created: [],
updated: [],
@@ -136,7 +136,7 @@ describe('aggregateOrchestratorActionsReportDeleteObjectAndDeleteFieldActions',
title: 'should handle multiple objects with mixed field deletions',
context: {
input: {
...EMPTY_ORCHESTRATOR_ACTIONS_REPORT,
...createEmptyOrchestratorActionsReport(),
objectMetadata: {
created: [],
updated: [],
@@ -187,7 +187,7 @@ describe('aggregateOrchestratorActionsReportDeleteObjectAndDeleteFieldActions',
{
title: 'should handle empty actions report',
context: {
input: EMPTY_ORCHESTRATOR_ACTIONS_REPORT,
input: createEmptyOrchestratorActionsReport(),
expected: {
expectDeleteFieldActionPerObjectMetadataId: {},
expectDeleteObjectActionPerObjectMetadataId: {},
@@ -0,0 +1,54 @@
import { isDefined } from 'twenty-shared/utils';
import {
FlatEntityMapsException,
FlatEntityMapsExceptionCode,
} from 'src/engine/metadata-modules/flat-entity/exceptions/flat-entity-maps.exception';
import { type FlatEntityMaps } from 'src/engine/metadata-modules/flat-entity/types/flat-entity-maps.type';
import { type FlatEntity } from 'src/engine/metadata-modules/flat-entity/types/flat-entity.type';
type AddFlatEntityToFlatEntityMapsThroughMutationOrThrowArgs<
T extends FlatEntity,
> = {
flatEntity: T;
flatEntityMapsToMutate: FlatEntityMaps<T>;
};
export const addFlatEntityToFlatEntityMapsThroughMutationOrThrow = <
T extends FlatEntity,
>({
flatEntity,
flatEntityMapsToMutate,
}: AddFlatEntityToFlatEntityMapsThroughMutationOrThrowArgs<T>): void => {
if (isDefined(flatEntityMapsToMutate.byId[flatEntity.id])) {
throw new FlatEntityMapsException(
'addFlatEntityToFlatEntityMapsThroughMutationOrThrow: flat entity to add already exists',
FlatEntityMapsExceptionCode.ENTITY_ALREADY_EXISTS,
);
}
flatEntityMapsToMutate.byId[flatEntity.id] = flatEntity;
flatEntityMapsToMutate.idByUniversalIdentifier[
flatEntity.universalIdentifier
] = flatEntity.id;
if (isDefined(flatEntity.applicationId)) {
const existingUniversalIdentifiers =
flatEntityMapsToMutate.universalIdentifiersByApplicationId[
flatEntity.applicationId
];
if (isDefined(existingUniversalIdentifiers)) {
if (
!existingUniversalIdentifiers.includes(flatEntity.universalIdentifier)
) {
existingUniversalIdentifiers.push(flatEntity.universalIdentifier);
}
} else {
flatEntityMapsToMutate.universalIdentifiersByApplicationId[
flatEntity.applicationId
] = [flatEntity.universalIdentifier];
}
}
};
@@ -0,0 +1,60 @@
import { isDefined } from 'twenty-shared/utils';
import {
FlatEntityMapsException,
FlatEntityMapsExceptionCode,
} from 'src/engine/metadata-modules/flat-entity/exceptions/flat-entity-maps.exception';
import { type FlatEntityMaps } from 'src/engine/metadata-modules/flat-entity/types/flat-entity-maps.type';
import { type FlatEntity } from 'src/engine/metadata-modules/flat-entity/types/flat-entity.type';
export type DeleteFlatEntityFromFlatEntityMapsThroughMutationOrThrowArgs<
T extends FlatEntity,
> = {
entityToDeleteId: string;
flatEntityMapsToMutate: FlatEntityMaps<T>;
};
export const deleteFlatEntityFromFlatEntityMapsThroughMutationOrThrow = <
T extends FlatEntity,
>({
flatEntityMapsToMutate,
entityToDeleteId,
}: DeleteFlatEntityFromFlatEntityMapsThroughMutationOrThrowArgs<T>): void => {
const entityToDelete = flatEntityMapsToMutate.byId[entityToDeleteId];
if (!isDefined(entityToDelete)) {
throw new FlatEntityMapsException(
'deleteFlatEntityFromFlatEntityMapsThroughMutationOrThrow: entity to delete not found',
FlatEntityMapsExceptionCode.ENTITY_NOT_FOUND,
);
}
delete flatEntityMapsToMutate.byId[entityToDeleteId];
delete flatEntityMapsToMutate.idByUniversalIdentifier[
entityToDelete.universalIdentifier
];
if (isDefined(entityToDelete.applicationId)) {
const universalIdentifiers =
flatEntityMapsToMutate.universalIdentifiersByApplicationId[
entityToDelete.applicationId
];
if (isDefined(universalIdentifiers)) {
const index = universalIdentifiers.indexOf(
entityToDelete.universalIdentifier,
);
if (index !== -1) {
universalIdentifiers.splice(index, 1);
}
if (universalIdentifiers.length === 0) {
delete flatEntityMapsToMutate.universalIdentifiersByApplicationId[
entityToDelete.applicationId
];
}
}
}
};
@@ -1,16 +1,16 @@
import { type AllMetadataName } from 'twenty-shared/metadata';
import { type FromTo } from 'twenty-shared/types';
import { isDefined } from 'twenty-shared/utils';
import { type AllMetadataName } from 'twenty-shared/metadata';
import { ALL_FLAT_ENTITY_PROPERTIES_TO_COMPARE_AND_STRINGIFY } from 'src/engine/metadata-modules/flat-entity/constant/all-flat-entity-properties-to-compare-and-stringify.constant';
import { EMPTY_FLAT_ENTITY_MAPS } from 'src/engine/metadata-modules/flat-entity/constant/empty-flat-entity-maps.constant';
import { createEmptyFlatEntityMaps } from 'src/engine/metadata-modules/flat-entity/constant/create-empty-flat-entity-maps.constant';
import { type FlatEntityPropertiesToCompare } from 'src/engine/metadata-modules/flat-entity/types/flat-entity-properties-to-compare.type';
import { type FlatEntityPropertiesToStringify } from 'src/engine/metadata-modules/flat-entity/types/flat-entity-properties-to-stringify.type';
import { type FlatEntityPropertiesUpdates } from 'src/engine/metadata-modules/flat-entity/types/flat-entity-properties-updates.type';
import { type MetadataFlatEntityMaps } from 'src/engine/metadata-modules/flat-entity/types/metadata-flat-entity-maps.type';
import { type MetadataFlatEntity } from 'src/engine/metadata-modules/flat-entity/types/metadata-flat-entity.type';
import { addFlatEntityToFlatEntityMapsOrThrow } from 'src/engine/metadata-modules/flat-entity/utils/add-flat-entity-to-flat-entity-maps-or-throw.util';
import { compareTwoFlatEntity } from 'src/engine/metadata-modules/flat-entity/utils/compare-two-flat-entity.util';
import { addFlatEntityToFlatEntityMapsThroughMutationOrThrow } from 'src/engine/workspace-manager/workspace-migration-v2/utils/add-flat-entity-to-flat-entity-maps-through-mutation-or-throw.util';
import { type WorkspaceMigrationBuilderOptions } from 'src/engine/workspace-manager/workspace-migration-v2/workspace-migration-builder-v2/types/workspace-migration-builder-options.type';
export type DeletedCreatedUpdatedMatrix<T extends AllMetadataName> = {
@@ -46,8 +46,8 @@ export const flatEntityDeletedCreatedUpdatedMatrixDispatcher = <
buildOptions,
}: FlatEntityDeletedCreatedUpdatedMatrixDispatcherArgs<T>): DeletedCreatedUpdatedMatrix<T> => {
const initialDispatcher: DeletedCreatedUpdatedMatrix<T> = {
createdFlatEntityMaps: EMPTY_FLAT_ENTITY_MAPS,
deletedFlatEntityMaps: EMPTY_FLAT_ENTITY_MAPS,
createdFlatEntityMaps: createEmptyFlatEntityMaps(),
deletedFlatEntityMaps: createEmptyFlatEntityMaps(),
updatedFlatEntityMaps: { byId: {} },
};
@@ -59,11 +59,10 @@ export const flatEntityDeletedCreatedUpdatedMatrixDispatcher = <
if (toMap.has(universalIdentifier)) {
continue;
}
initialDispatcher.deletedFlatEntityMaps =
addFlatEntityToFlatEntityMapsOrThrow({
flatEntity: fromEntity,
flatEntityMaps: initialDispatcher.deletedFlatEntityMaps,
});
addFlatEntityToFlatEntityMapsThroughMutationOrThrow({
flatEntity: fromEntity,
flatEntityMapsToMutate: initialDispatcher.deletedFlatEntityMaps,
});
}
}
@@ -71,11 +70,10 @@ export const flatEntityDeletedCreatedUpdatedMatrixDispatcher = <
if (fromMap.has(universalIdentifier)) {
continue;
}
initialDispatcher.createdFlatEntityMaps =
addFlatEntityToFlatEntityMapsOrThrow({
flatEntity: toFlatEntity,
flatEntityMaps: initialDispatcher.createdFlatEntityMaps,
});
addFlatEntityToFlatEntityMapsThroughMutationOrThrow({
flatEntity: toFlatEntity,
flatEntityMapsToMutate: initialDispatcher.createdFlatEntityMaps,
});
}
const { propertiesToCompare, propertiesToStringify } =
@@ -0,0 +1,28 @@
import { type FlatEntityMaps } from 'src/engine/metadata-modules/flat-entity/types/flat-entity-maps.type';
import { type FlatEntity } from 'src/engine/metadata-modules/flat-entity/types/flat-entity.type';
import { addFlatEntityToFlatEntityMapsThroughMutationOrThrow } from 'src/engine/workspace-manager/workspace-migration-v2/utils/add-flat-entity-to-flat-entity-maps-through-mutation-or-throw.util';
import { deleteFlatEntityFromFlatEntityMapsThroughMutationOrThrow } from 'src/engine/workspace-manager/workspace-migration-v2/utils/delete-flat-entity-from-flat-entity-maps-through-mutation-or-throw.util';
export type ReplaceFlatEntityInFlatEntityMapsThroughMutationOrThrowArgs<
T extends FlatEntity,
> = {
flatEntity: T;
flatEntityMapsToMutate: FlatEntityMaps<T>;
};
export const replaceFlatEntityInFlatEntityMapsThroughMutationOrThrow = <
T extends FlatEntity,
>({
flatEntity,
flatEntityMapsToMutate,
}: ReplaceFlatEntityInFlatEntityMapsThroughMutationOrThrowArgs<T>): void => {
deleteFlatEntityFromFlatEntityMapsThroughMutationOrThrow({
flatEntityMapsToMutate,
entityToDeleteId: flatEntity.id,
});
addFlatEntityToFlatEntityMapsThroughMutationOrThrow({
flatEntity,
flatEntityMapsToMutate,
});
};