When starting the app on a fresh database reset the cache would be
filled with empty flat field metadata maps
Because the reset command hack through the repository directly in order
to create views and stuff
implemented an iso flush as the one existing initially
added it to a workspace deletion tambien
```
{
byId: {],
universalIdById: {}
}
```
# Introduction
Extracting the legacy fields build and dispatch out of the object one to
follow the generic flat entity build, also update caches entries for
object
## Main tasks
- flat field map cache
- flat field builder
- refactored the dispatch matrix to return flat entity maps instead of
flat entity arrays
- orchestrator aggregator
- removing legacy code
- making universal identifier aka standardId of standard field for
custom object deterministically dynamic
- perfs debug logs for v2
## TODO
- [x] Refactor the generic entity builder to be dependency flat maps in
order to main foreign keys list in flat parent
- [x] Refactor the flat object metadata to contain the array of related
fields and avoid costy find object fields
- [ ] Improve the create field handler to handle multiple field at once
- [ ] Refactor the dispatch to embbed the comparison
- [ ] Improve perf by extracting from elements out of existing
- [ ] Fix the labelIdentifierId validators on object before field
creation ( integ tests are in failing mode )
## Debug logs snippet
```ts
[EntityBuilder fieldMetadata] matrix computation: 0.027ms
[EntityBuilder fieldMetadata] creation validation: 0.001ms
[EntityBuilder fieldMetadata] deletion validation: 0.293ms
[EntityBuilder fieldMetadata] update validation: 0.006ms
[EntityBuilder fieldMetadata] entity processing: 0.363ms
[EntityBuilder fieldMetadata] validateAndBuild: 0.455ms
[EntityBuilder index] matrix computation: 0.005ms
[EntityBuilder index] creation validation: 0.001ms
[EntityBuilder index] deletion validation: 0.146ms
[EntityBuilder index] update validation: 0.004ms
[EntityBuilder index] entity processing: 0.199ms
[EntityBuilder index] validateAndBuild: 0.228ms
[Runner] Initial cache retrieval: 0.549ms
[BaseWorkspaceMigrationRunnerActionHandlerService] delete_index executeForWorkspaceSchema: 11.665ms
[BaseWorkspaceMigrationRunnerActionHandlerService] delete_index executeForMetadata: 12.864ms
[BaseWorkspaceMigrationRunnerActionHandlerService] delete_field executeForWorkspaceSchema: 1.476ms
[BaseWorkspaceMigrationRunnerActionHandlerService] delete_field executeForMetadata: 6.816ms
[BaseWorkspaceMigrationRunnerActionHandlerService] delete_field executeForWorkspaceSchema: 0.062ms
[BaseWorkspaceMigrationRunnerActionHandlerService] delete_field executeForMetadata: 0.889ms
[Runner] Transaction execution: 23.434ms
[Runner] Cache invalidation: 316.662ms
[Runner] Total execution: 340.767ms
```
As you can see cache invalidation is way to long, we could replace the
cache by the optimistic in the end
## Context
Adding a new service that provides an abstract caching system for
FlatEntityMaps.
This takes care of cache invalidation and storing local and remote cache
for the map with retrieval after a comparison with map hash between
local and remote (redis).
## Implementation
Remote (redis) keys
- Flat map data:
`engine:workspace:flat-maps:{flatMapKey}:{workspaceId}:flat-map`
- Content hash:
`engine:workspace:flat-maps:{flatMapKey}:{workspaceId}:hash`
**Local Cache Hit**: If local hash matches Redis hash, return local data
**Remote Cache Hit**: If Redis has data with different hash, update
local cache
**Remote Cache Miss**: Recompute from database, store in Remote and
locally
**Invalidation**: Remove from Remote, triggering recomputation on next
access
## Usage
```typescript
@WorkspaceFlatMapCache('view') // redis key
export class WorkspaceFlatViewMapCacheService extends WorkspaceFlatMapCacheService<FlatViewMaps> {
constructor(
@InjectCacheStorage(CacheStorageNamespace.EngineWorkspace)
cacheStorageService: CacheStorageService,
@InjectRepository(ViewEntity)
private readonly viewRepository: Repository<ViewEntity>,
) {
super(cacheStorageService);
}
// only method to implement
public async computeFlatMap(workspaceId: string): Promise<FlatViewMaps> {
const views = await this.viewRepository.find({
where: { workspaceId },
relations: ['viewFields'],
select: { viewFields: { id: true } },
});
return generateFlatViewMaps(views);
}
}
```
```typescript
// 2 public methods, getExistingOrRecomputeFlatMaps to fetch the map and invalidateCache after a mutation
await this.workspaceFlatViewMapCacheService.invalidateCache(
viewData.workspaceId,
);
const flatViewMaps =
await this.workspaceFlatViewMapCacheService.getExistingOrRecomputeFlatMaps(
workspaceId,
);
```
## Multi-Pod Synchronization
- Each pod maintains local cache for performance
- SHA256 hash of flatMap content used for version comparison
- `invalidateCache()` reset Redis data, forcing other pods to refresh
when calling getExistingOrRecomputeFlatMaps
---
<img width="1072" height="325" alt="Screenshot 2025-09-11 at 15 04 48"
src="https://github.com/user-attachments/assets/ed6ce82c-db35-4a1b-8a4e-247b694e2ddf"
/>
<img width="1030" height="328" alt="Screenshot 2025-09-11 at 15 04 40"
src="https://github.com/user-attachments/assets/43a15789-7f27-4104-a9bb-bef19908a5cd"
/>
Next step: Implement a locking mechanism by reusing existing WithLock
decorator
Paul Rastoin