Improve v2 and cache invalidation perfs (#15467)
# Introduction Log are debug logs of `packages/twenty-server/test/integration/metadata/suites/object-metadata/create-delete-and-create-object-metadata-v2.integration-spec.ts`run ten times in a row on clean db reset ## Next Will improve cache computation to lighter invalidation. RelationLoad `query` does not seem to work with typeorm so I'll continue the custom integration i've started in https://github.com/twentyhq/twenty/tree/optimize-cache-read-v2 ## Integration tests duration Significant test duration improvement too ### Before <img width="2632" height="1402" alt="image" src="https://github.com/user-attachments/assets/2f1f0ccf-44de-4856-bfe1-4f45a351763a" /> ### After <img width="2632" height="1402" alt="image" src="https://github.com/user-attachments/assets/4bc9e6db-3046-48b9-b903-1464053936c4" /> ## What's next - The legacy cache invalidation removal - Factorizing redis calls in only one operation ## Autogenerated performance comparison ( including mutation refactor too ) [Before](https://gist.github.com/prastoin/3c1e21fa9e3b3ce4b0716902ff4a2dd6) [After](https://gist.github.com/prastoin/7bfddd14bfded2e4991a9378970a026d) The optimized implementation shows **dramatic performance improvements** across all metrics: - 🚀 **Cache Invalidation**: 156.3ms → 76.8ms (**50.9% faster**) - 🚀 **Builder Operations**: 21.3ms → 14.2ms (**33.3% faster**) - ⚡ **Consistency**: 16.4% more predictable performance --- ## 1. Overall Performance Summary | Component | Before (avg) | After (avg) | Best (After) | Worst (After) | Improvement | |-----------|-------------|-------------|--------------|---------------|-------------| | **Total Execution Time** | 180.2ms | 110.5ms | 52.3ms | 585.9ms | **38.7% faster** ⚡⚡ | | **Cache Invalidation** | 156.3ms | 76.8ms | 47.8ms | 285.1ms | **50.9% faster** ⚡⚡⚡ | | **Transaction Execution** | 22.4ms | 22.1ms | 0.99ms | 314.2ms | Similar | | **Initial Cache Retrieval** | 0.81ms | 2.08ms | 0.21ms | 8.24ms | Similar | | **Entity Builder (total)** | 21.3ms | 14.2ms | 0.36ms | 42.5ms | **33.3% faster** ⚡ | ### Total Execution Time Distribution #### Before (Legacy Sequential) ``` Time (ms) Count Percentage Visualization < 150 18 16% ████ 150-180 32 29% ███████ 180-210 35 32% ████████ 210-250 17 15% ████ 250-350 6 5% █ > 350 2 2% ▌ ``` #### After (Optimized Parallel) ``` Time (ms) Count Percentage Visualization < 70 28 31% ████████ 70-100 31 34% █████████ 100-150 18 20% █████ 150-200 8 9% ██ 200-300 4 4% █ > 300 2 2% ▌ ``` --- ## 2. Builder Performance Breakdown ### Field Metadata Builder | Operation | Before (avg) | After (avg) | Improvement | |-----------|-------------|-------------|-------------| | Matrix computation | 3.5ms | 3.4ms | Similar | | Creation validation | 15.2ms | 2.1ms | **86% faster** ⚡⚡⚡ | | Deletion validation | 0.08ms | 0.06ms | Similar | | Update validation | 1.3ms | 0.09ms | **93% faster** ⚡⚡⚡ | | Entity processing | 18.6ms | 11.8ms | **37% faster** ⚡ | | **Total validateAndBuild** | **21.3ms** | **14.2ms** | **33% faster** ⚡ | #### Performance Distribution ``` Before: ▁▂▄█████▆▄▂▁ (wide spread, 15-28ms range) After: ▁▁▃█████▃▁▁ (tight clustering, 10-18ms range) ``` ## 4. Cache Invalidation Performance Breakdown ### Cache Invalidation Summary | Metric | Before (Legacy) | After (Optimized) | Improvement | |--------|-----------------|-------------------|-------------| | **Best Time** | 131.965ms | 47.833ms | **63.7% faster** ⚡⚡⚡ | | **10th Percentile** | 140.2ms | 51.7ms | **63.1% faster** ⚡⚡⚡ | | **25th Percentile** | 146.1ms | 54.4ms | **62.7% faster** ⚡⚡⚡ | | **Median (50th)** | 155.1ms | 63.4ms | **59.1% faster** ⚡⚡⚡ | | **Average** | 156.3ms | 76.8ms | **50.9% faster** ⚡⚡⚡ | | **75th Percentile** | 160.2ms | 90.2ms | **43.7% faster** ⚡⚡ | | **90th Percentile** | 191.7ms | 100.2ms | **47.7% faster** ⚡⚡ | | **95th Percentile** | 235.6ms | 110.3ms | **53.2% faster** ⚡⚡⚡ | | **99th Percentile** | 278.2ms | 224.9ms | **19.1% faster** ⚡ | | **Worst Time** | 383.914ms | 285.102ms | **25.7% faster** ⚡ | ### Cache Invalidation Time Distribution #### Before (Legacy Sequential) ``` Time (ms) Count Percentage Visualization 130-140 3 3% ▊ 140-150 15 14% ████ 150-160 48 44% ███████████ 160-180 31 28% ███████ 180-220 8 7% ██ 220-280 3 3% ▊ > 280 2 2% ▌ ``` #### After (Optimized Parallel + Intersection) ``` Time (ms) Count Percentage Visualization < 50 3 3% ▊ 50-60 27 25% ███████ 60-70 25 23% ██████ 70-90 25 23% ██████ 90-100 15 14% ████ 100-120 8 7% ██ 120-150 3 3% ▊ > 150 4 4% █ ``` ## 6. Performance Consistency Analysis ### Standard Deviation & Variance | Metric | Before | After | Improvement | |--------|--------|-------|-------------| | **Cache Invalidation Std Dev** | 42.1ms | 35.2ms | **16.4% more consistent** ⚡ | | **Total Execution Std Dev** | 68.3ms | 89.1ms | Slightly more variable | | **Coefficient of Variation (Cache)** | 26.9% | 45.8% | More variance | | **Outliers (> 2σ)** | 5 cases | 3 cases | **40% fewer outliers** ⚡ | ---
This commit is contained in:
Paul Rastoin
+1
-1
@@ -1,5 +1,5 @@
|
||||
import { type ExtractPropertiesThatEndsWithId } from 'twenty-shared/types';
|
||||
import { type AllMetadataName } from 'twenty-shared/metadata';
|
||||
import { type ExtractPropertiesThatEndsWithId } from 'twenty-shared/types';
|
||||
|
||||
import { type MetadataEntity } from 'src/engine/metadata-modules/flat-entity/types/metadata-entity.type';
|
||||
|
||||
|
||||
+19
@@ -0,0 +1,19 @@
|
||||
import {
|
||||
ALL_METADATA_NAME,
|
||||
type AllMetadataName,
|
||||
} from 'twenty-shared/metadata';
|
||||
|
||||
import { createEmptyFlatEntityMaps } from 'src/engine/metadata-modules/flat-entity/constant/create-empty-flat-entity-maps.constant';
|
||||
import { type AllFlatEntityMaps } from 'src/engine/metadata-modules/flat-entity/types/all-flat-entity-maps.type';
|
||||
import { getMetadataFlatEntityMapsKey } from 'src/engine/metadata-modules/flat-entity/utils/get-metadata-flat-entity-maps-key.util';
|
||||
|
||||
export const createEmptyAllFlatEntityMaps = (): AllFlatEntityMaps =>
|
||||
(
|
||||
Object.keys(ALL_METADATA_NAME) as AllMetadataName[]
|
||||
).reduce<AllFlatEntityMaps>(
|
||||
(acc, metadataName) => ({
|
||||
...acc,
|
||||
[getMetadataFlatEntityMapsKey(metadataName)]: createEmptyFlatEntityMaps(),
|
||||
}),
|
||||
{} as AllFlatEntityMaps,
|
||||
);
|
||||
+6
-5
@@ -1,8 +1,9 @@
|
||||
import { type FlatEntityMaps } from 'src/engine/metadata-modules/flat-entity/types/flat-entity-maps.type';
|
||||
import { type FlatEntity } from 'src/engine/metadata-modules/flat-entity/types/flat-entity.type';
|
||||
|
||||
export const EMPTY_FLAT_ENTITY_MAPS = {
|
||||
byId: {},
|
||||
idByUniversalIdentifier: {},
|
||||
universalIdentifiersByApplicationId: {},
|
||||
} as const satisfies FlatEntityMaps<FlatEntity>;
|
||||
export const createEmptyFlatEntityMaps = () =>
|
||||
({
|
||||
byId: {},
|
||||
idByUniversalIdentifier: {},
|
||||
universalIdentifiersByApplicationId: {},
|
||||
}) as const satisfies FlatEntityMaps<FlatEntity>;
|
||||
-18
@@ -1,18 +0,0 @@
|
||||
import {
|
||||
ALL_METADATA_NAME,
|
||||
type AllMetadataName,
|
||||
} from 'twenty-shared/metadata';
|
||||
|
||||
import { EMPTY_FLAT_ENTITY_MAPS } from 'src/engine/metadata-modules/flat-entity/constant/empty-flat-entity-maps.constant';
|
||||
import { type AllFlatEntityMaps } from 'src/engine/metadata-modules/flat-entity/types/all-flat-entity-maps.type';
|
||||
import { getMetadataFlatEntityMapsKey } from 'src/engine/metadata-modules/flat-entity/utils/get-metadata-flat-entity-maps-key.util';
|
||||
|
||||
export const EMPTY_ALL_FLAT_ENTITY_MAPS = (
|
||||
Object.keys(ALL_METADATA_NAME) as AllMetadataName[]
|
||||
).reduce<AllFlatEntityMaps>(
|
||||
(acc, metadataName) => ({
|
||||
...acc,
|
||||
[getMetadataFlatEntityMapsKey(metadataName)]: EMPTY_FLAT_ENTITY_MAPS,
|
||||
}),
|
||||
{} as AllFlatEntityMaps,
|
||||
);
|
||||
+25
-16
@@ -33,23 +33,32 @@ export class WorkspaceManyOrAllFlatEntityMapsCacheService {
|
||||
? flatMapsKeys
|
||||
: ALL_FLAT_ENTITY_MAPS_PROPERTIES;
|
||||
|
||||
for (const flatMapKey of keysToProcess) {
|
||||
try {
|
||||
const service = this.cacheRegistry.getCacheServiceOrThrow(
|
||||
flatMapKey as K[number],
|
||||
);
|
||||
const results = await Promise.allSettled(
|
||||
keysToProcess.map(async (flatMapKey) => {
|
||||
try {
|
||||
const service = this.cacheRegistry.getCacheServiceOrThrow(
|
||||
flatMapKey as K[number],
|
||||
);
|
||||
|
||||
await action({
|
||||
flatMapKey: flatMapKey,
|
||||
service,
|
||||
});
|
||||
} catch (error) {
|
||||
this.logger.error(
|
||||
`Failed to run action on flat entity maps of ${flatMapKey}`,
|
||||
error,
|
||||
);
|
||||
throw error;
|
||||
}
|
||||
return await action({
|
||||
flatMapKey: flatMapKey,
|
||||
service,
|
||||
});
|
||||
} catch (error) {
|
||||
this.logger.error(
|
||||
`Failed to run action on flat entity maps of ${flatMapKey}`,
|
||||
error,
|
||||
);
|
||||
throw error;
|
||||
}
|
||||
}),
|
||||
);
|
||||
|
||||
const failures = results.filter((result) => result.status === 'rejected');
|
||||
|
||||
if (failures.length > 0) {
|
||||
this.logger.error(`${failures.length} operations failed`);
|
||||
throw new Error(`Failed to process ${failures.length} flat entity maps`);
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
+6
@@ -0,0 +1,6 @@
|
||||
import { type AllMetadataName } from 'twenty-shared/metadata';
|
||||
|
||||
import { type ALL_METADATA_RELATED_METADATA_BY_FOREIGN_KEY } from 'src/engine/metadata-modules/flat-entity/constant/all-metadata-many-to-one-relations.constant';
|
||||
|
||||
export type MetadataManyToOneJoinColumn<T extends AllMetadataName> =
|
||||
keyof (typeof ALL_METADATA_RELATED_METADATA_BY_FOREIGN_KEY)[T];
|
||||
+2
-1
@@ -1,9 +1,10 @@
|
||||
import { type AllMetadataName } from 'twenty-shared/metadata';
|
||||
|
||||
import { type ALL_METADATA_RELATED_METADATA_BY_FOREIGN_KEY } from 'src/engine/metadata-modules/flat-entity/constant/all-metadata-many-to-one-relations.constant';
|
||||
import { type MetadataManyToOneJoinColumn } from 'src/engine/metadata-modules/flat-entity/types/metadata-many-to-one-join-column.type';
|
||||
|
||||
export type MetadataManyToOneRelatedMetadataNames<T extends AllMetadataName> =
|
||||
Extract<
|
||||
(typeof ALL_METADATA_RELATED_METADATA_BY_FOREIGN_KEY)[T][keyof (typeof ALL_METADATA_RELATED_METADATA_BY_FOREIGN_KEY)[T]],
|
||||
(typeof ALL_METADATA_RELATED_METADATA_BY_FOREIGN_KEY)[T][MetadataManyToOneJoinColumn<T>],
|
||||
AllMetadataName
|
||||
>;
|
||||
|
||||
+2
-2
@@ -1,4 +1,4 @@
|
||||
import { EMPTY_FLAT_ENTITY_MAPS } from 'src/engine/metadata-modules/flat-entity/constant/empty-flat-entity-maps.constant';
|
||||
import { createEmptyFlatEntityMaps } from 'src/engine/metadata-modules/flat-entity/constant/create-empty-flat-entity-maps.constant';
|
||||
import { type FlatEntityMaps } from 'src/engine/metadata-modules/flat-entity/types/flat-entity-maps.type';
|
||||
import { type FlatEntity } from 'src/engine/metadata-modules/flat-entity/types/flat-entity.type';
|
||||
import { addFlatEntityToFlatEntityMapsOrThrow } from 'src/engine/metadata-modules/flat-entity/utils/add-flat-entity-to-flat-entity-maps-or-throw.util';
|
||||
@@ -21,5 +21,5 @@ export const getSubFlatEntityMapsOrThrow = <T extends FlatEntity>({
|
||||
flatEntity,
|
||||
flatEntityMaps: acc,
|
||||
});
|
||||
}, EMPTY_FLAT_ENTITY_MAPS);
|
||||
}, createEmptyFlatEntityMaps());
|
||||
};
|
||||
|
||||
Reference in New Issue
Block a user