Improve v2 and cache invalidation perfs (#15467)
# Introduction Log are debug logs of `packages/twenty-server/test/integration/metadata/suites/object-metadata/create-delete-and-create-object-metadata-v2.integration-spec.ts`run ten times in a row on clean db reset ## Next Will improve cache computation to lighter invalidation. RelationLoad `query` does not seem to work with typeorm so I'll continue the custom integration i've started in https://github.com/twentyhq/twenty/tree/optimize-cache-read-v2 ## Integration tests duration Significant test duration improvement too ### Before <img width="2632" height="1402" alt="image" src="https://github.com/user-attachments/assets/2f1f0ccf-44de-4856-bfe1-4f45a351763a" /> ### After <img width="2632" height="1402" alt="image" src="https://github.com/user-attachments/assets/4bc9e6db-3046-48b9-b903-1464053936c4" /> ## What's next - The legacy cache invalidation removal - Factorizing redis calls in only one operation ## Autogenerated performance comparison ( including mutation refactor too ) [Before](https://gist.github.com/prastoin/3c1e21fa9e3b3ce4b0716902ff4a2dd6) [After](https://gist.github.com/prastoin/7bfddd14bfded2e4991a9378970a026d) The optimized implementation shows **dramatic performance improvements** across all metrics: - 🚀 **Cache Invalidation**: 156.3ms → 76.8ms (**50.9% faster**) - 🚀 **Builder Operations**: 21.3ms → 14.2ms (**33.3% faster**) - ⚡ **Consistency**: 16.4% more predictable performance --- ## 1. Overall Performance Summary | Component | Before (avg) | After (avg) | Best (After) | Worst (After) | Improvement | |-----------|-------------|-------------|--------------|---------------|-------------| | **Total Execution Time** | 180.2ms | 110.5ms | 52.3ms | 585.9ms | **38.7% faster** ⚡⚡ | | **Cache Invalidation** | 156.3ms | 76.8ms | 47.8ms | 285.1ms | **50.9% faster** ⚡⚡⚡ | | **Transaction Execution** | 22.4ms | 22.1ms | 0.99ms | 314.2ms | Similar | | **Initial Cache Retrieval** | 0.81ms | 2.08ms | 0.21ms | 8.24ms | Similar | | **Entity Builder (total)** | 21.3ms | 14.2ms | 0.36ms | 42.5ms | **33.3% faster** ⚡ | ### Total Execution Time Distribution #### Before (Legacy Sequential) ``` Time (ms) Count Percentage Visualization < 150 18 16% ████ 150-180 32 29% ███████ 180-210 35 32% ████████ 210-250 17 15% ████ 250-350 6 5% █ > 350 2 2% ▌ ``` #### After (Optimized Parallel) ``` Time (ms) Count Percentage Visualization < 70 28 31% ████████ 70-100 31 34% █████████ 100-150 18 20% █████ 150-200 8 9% ██ 200-300 4 4% █ > 300 2 2% ▌ ``` --- ## 2. Builder Performance Breakdown ### Field Metadata Builder | Operation | Before (avg) | After (avg) | Improvement | |-----------|-------------|-------------|-------------| | Matrix computation | 3.5ms | 3.4ms | Similar | | Creation validation | 15.2ms | 2.1ms | **86% faster** ⚡⚡⚡ | | Deletion validation | 0.08ms | 0.06ms | Similar | | Update validation | 1.3ms | 0.09ms | **93% faster** ⚡⚡⚡ | | Entity processing | 18.6ms | 11.8ms | **37% faster** ⚡ | | **Total validateAndBuild** | **21.3ms** | **14.2ms** | **33% faster** ⚡ | #### Performance Distribution ``` Before: ▁▂▄█████▆▄▂▁ (wide spread, 15-28ms range) After: ▁▁▃█████▃▁▁ (tight clustering, 10-18ms range) ``` ## 4. Cache Invalidation Performance Breakdown ### Cache Invalidation Summary | Metric | Before (Legacy) | After (Optimized) | Improvement | |--------|-----------------|-------------------|-------------| | **Best Time** | 131.965ms | 47.833ms | **63.7% faster** ⚡⚡⚡ | | **10th Percentile** | 140.2ms | 51.7ms | **63.1% faster** ⚡⚡⚡ | | **25th Percentile** | 146.1ms | 54.4ms | **62.7% faster** ⚡⚡⚡ | | **Median (50th)** | 155.1ms | 63.4ms | **59.1% faster** ⚡⚡⚡ | | **Average** | 156.3ms | 76.8ms | **50.9% faster** ⚡⚡⚡ | | **75th Percentile** | 160.2ms | 90.2ms | **43.7% faster** ⚡⚡ | | **90th Percentile** | 191.7ms | 100.2ms | **47.7% faster** ⚡⚡ | | **95th Percentile** | 235.6ms | 110.3ms | **53.2% faster** ⚡⚡⚡ | | **99th Percentile** | 278.2ms | 224.9ms | **19.1% faster** ⚡ | | **Worst Time** | 383.914ms | 285.102ms | **25.7% faster** ⚡ | ### Cache Invalidation Time Distribution #### Before (Legacy Sequential) ``` Time (ms) Count Percentage Visualization 130-140 3 3% ▊ 140-150 15 14% ████ 150-160 48 44% ███████████ 160-180 31 28% ███████ 180-220 8 7% ██ 220-280 3 3% ▊ > 280 2 2% ▌ ``` #### After (Optimized Parallel + Intersection) ``` Time (ms) Count Percentage Visualization < 50 3 3% ▊ 50-60 27 25% ███████ 60-70 25 23% ██████ 70-90 25 23% ██████ 90-100 15 14% ████ 100-120 8 7% ██ 120-150 3 3% ▊ > 150 4 4% █ ``` ## 6. Performance Consistency Analysis ### Standard Deviation & Variance | Metric | Before | After | Improvement | |--------|--------|-------|-------------| | **Cache Invalidation Std Dev** | 42.1ms | 35.2ms | **16.4% more consistent** ⚡ | | **Total Execution Std Dev** | 68.3ms | 89.1ms | Slightly more variable | | **Coefficient of Variation (Cache)** | 26.9% | 45.8% | More variance | | **Outliers (> 2σ)** | 5 cases | 3 cases | **40% fewer outliers** ⚡ | ---
This commit is contained in:
Paul Rastoin
+2
-2
@@ -1,4 +1,4 @@
|
||||
import { EMPTY_FLAT_ENTITY_MAPS } from 'src/engine/metadata-modules/flat-entity/constant/empty-flat-entity-maps.constant';
|
||||
import { createEmptyFlatEntityMaps } from 'src/engine/metadata-modules/flat-entity/constant/create-empty-flat-entity-maps.constant';
|
||||
import { addFlatEntityToFlatEntityMapsOrThrow } from 'src/engine/metadata-modules/flat-entity/utils/add-flat-entity-to-flat-entity-maps-or-throw.util';
|
||||
import { ALL_FLAT_OBJECT_METADATA_MOCKS } from 'src/engine/metadata-modules/flat-object-metadata/__mocks__/all-flat-object-metadatas.mock';
|
||||
|
||||
@@ -10,5 +10,5 @@ export const FLAT_OBJECT_METADATA_MAPS_MOCKS = [
|
||||
flatEntity: flatObjectMetadata,
|
||||
flatEntityMaps: flatObjectMaps,
|
||||
}),
|
||||
EMPTY_FLAT_ENTITY_MAPS,
|
||||
createEmptyFlatEntityMaps(),
|
||||
);
|
||||
|
||||
+66
-31
@@ -6,14 +6,18 @@ import { Repository } from 'typeorm';
|
||||
import { InjectCacheStorage } from 'src/engine/core-modules/cache-storage/decorators/cache-storage.decorator';
|
||||
import { CacheStorageService } from 'src/engine/core-modules/cache-storage/services/cache-storage.service';
|
||||
import { CacheStorageNamespace } from 'src/engine/core-modules/cache-storage/types/cache-storage-namespace.enum';
|
||||
import { EMPTY_FLAT_ENTITY_MAPS } from 'src/engine/metadata-modules/flat-entity/constant/empty-flat-entity-maps.constant';
|
||||
import { FieldMetadataEntity } from 'src/engine/metadata-modules/field-metadata/field-metadata.entity';
|
||||
import { createEmptyFlatEntityMaps } from 'src/engine/metadata-modules/flat-entity/constant/create-empty-flat-entity-maps.constant';
|
||||
import { FlatEntityMaps } from 'src/engine/metadata-modules/flat-entity/types/flat-entity-maps.type';
|
||||
import { addFlatEntityToFlatEntityMapsOrThrow } from 'src/engine/metadata-modules/flat-entity/utils/add-flat-entity-to-flat-entity-maps-or-throw.util';
|
||||
import { FlatObjectMetadata } from 'src/engine/metadata-modules/flat-object-metadata/types/flat-object-metadata.type';
|
||||
import { fromObjectMetadataEntityToFlatObjectMetadata } from 'src/engine/metadata-modules/flat-object-metadata/utils/from-object-metadata-entity-to-flat-object-metadata.util';
|
||||
import { IndexMetadataEntity } from 'src/engine/metadata-modules/index-metadata/index-metadata.entity';
|
||||
import { ObjectMetadataEntity } from 'src/engine/metadata-modules/object-metadata/object-metadata.entity';
|
||||
import { ViewEntity } from 'src/engine/metadata-modules/view/entities/view.entity';
|
||||
import { WorkspaceFlatMapCache } from 'src/engine/workspace-flat-map-cache/decorators/workspace-flat-map-cache.decorator';
|
||||
import { WorkspaceFlatMapCacheService } from 'src/engine/workspace-flat-map-cache/services/workspace-flat-map-cache.service';
|
||||
import { regroupEntitiesByRelatedEntityId } from 'src/engine/workspace-flat-map-cache/utils/regroup-entities-by-related-entity-id';
|
||||
import { addFlatEntityToFlatEntityMapsThroughMutationOrThrow } from 'src/engine/workspace-manager/workspace-migration-v2/utils/add-flat-entity-to-flat-entity-maps-through-mutation-or-throw.util';
|
||||
|
||||
@Injectable()
|
||||
@WorkspaceFlatMapCache('flatObjectMetadataMaps')
|
||||
@@ -26,6 +30,12 @@ export class WorkspaceFlatObjectMetadataMapCacheService extends WorkspaceFlatMap
|
||||
|
||||
@InjectRepository(ObjectMetadataEntity)
|
||||
private readonly objectMetadataRepository: Repository<ObjectMetadataEntity>,
|
||||
@InjectRepository(FieldMetadataEntity)
|
||||
private readonly fieldMetadataRepository: Repository<FieldMetadataEntity>,
|
||||
@InjectRepository(IndexMetadataEntity)
|
||||
private readonly indexMetadataRepository: Repository<IndexMetadataEntity>,
|
||||
@InjectRepository(ViewEntity)
|
||||
private readonly viewRepository: Repository<ViewEntity>,
|
||||
) {
|
||||
super(cacheStorageService);
|
||||
}
|
||||
@@ -35,36 +45,61 @@ export class WorkspaceFlatObjectMetadataMapCacheService extends WorkspaceFlatMap
|
||||
}: {
|
||||
workspaceId: string;
|
||||
}): Promise<FlatEntityMaps<FlatObjectMetadata>> {
|
||||
const objectMetadatas = await this.objectMetadataRepository.find({
|
||||
where: {
|
||||
workspaceId,
|
||||
},
|
||||
withDeleted: true,
|
||||
select: {
|
||||
fields: {
|
||||
id: true,
|
||||
},
|
||||
indexMetadatas: {
|
||||
id: true,
|
||||
},
|
||||
views: {
|
||||
id: true,
|
||||
},
|
||||
},
|
||||
relations: ['fields', 'indexMetadatas', 'views'],
|
||||
});
|
||||
const [objectMetadatas, fields, indexMetadatas, views] = await Promise.all([
|
||||
this.objectMetadataRepository.find({
|
||||
where: { workspaceId },
|
||||
withDeleted: true,
|
||||
}),
|
||||
this.fieldMetadataRepository.find({
|
||||
where: { workspaceId },
|
||||
select: ['id', 'objectMetadataId'],
|
||||
withDeleted: true,
|
||||
}),
|
||||
this.indexMetadataRepository.find({
|
||||
where: { workspaceId },
|
||||
select: ['id', 'objectMetadataId'],
|
||||
withDeleted: true,
|
||||
}),
|
||||
this.viewRepository.find({
|
||||
where: { workspaceId },
|
||||
select: ['id', 'objectMetadataId'],
|
||||
withDeleted: true,
|
||||
}),
|
||||
]);
|
||||
|
||||
return objectMetadatas.reduce(
|
||||
(flatObjectMetadataMaps, objectMetadataEntity) => {
|
||||
const flatObjectMetadata =
|
||||
fromObjectMetadataEntityToFlatObjectMetadata(objectMetadataEntity);
|
||||
const [fieldsByObjectId, indexesByObjectId, viewsByObjectId] = (
|
||||
[
|
||||
{
|
||||
entities: fields,
|
||||
foreignKey: 'objectMetadataId',
|
||||
},
|
||||
{
|
||||
entities: indexMetadatas,
|
||||
foreignKey: 'objectMetadataId',
|
||||
},
|
||||
{
|
||||
entities: views,
|
||||
foreignKey: 'objectMetadataId',
|
||||
},
|
||||
] as const
|
||||
).map(regroupEntitiesByRelatedEntityId);
|
||||
|
||||
return addFlatEntityToFlatEntityMapsOrThrow({
|
||||
flatEntity: flatObjectMetadata,
|
||||
flatEntityMaps: flatObjectMetadataMaps,
|
||||
});
|
||||
},
|
||||
EMPTY_FLAT_ENTITY_MAPS,
|
||||
);
|
||||
const flatObjectMetadataMaps = createEmptyFlatEntityMaps();
|
||||
|
||||
for (const objectMetadataEntity of objectMetadatas) {
|
||||
const flatObjectMetadata = fromObjectMetadataEntityToFlatObjectMetadata({
|
||||
...objectMetadataEntity,
|
||||
fields: fieldsByObjectId.get(objectMetadataEntity.id) || [],
|
||||
indexMetadatas: indexesByObjectId.get(objectMetadataEntity.id) || [],
|
||||
views: viewsByObjectId.get(objectMetadataEntity.id) || [],
|
||||
} as ObjectMetadataEntity);
|
||||
|
||||
addFlatEntityToFlatEntityMapsThroughMutationOrThrow({
|
||||
flatEntity: flatObjectMetadata,
|
||||
flatEntityMapsToMutate: flatObjectMetadataMaps,
|
||||
});
|
||||
}
|
||||
|
||||
return flatObjectMetadataMaps;
|
||||
}
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user