Commit Graph

2 Commits

Author SHA1 Message Date
Paul Rastoin 371c26bc9b Flat entity maps cache generic service + runner dynamically retrieving invalidating update cache + view service v2 refactor (#14508)
# Introduction
Migrate previous runner only iterating on `flatObjectMetadataMaps` to
`allFlatEntityMaps`.
Refactored the optimistic to be handled inside the actions handler

## Workspace flat map cache
Introducing a new service and registry, that will dynamically retrieve
and or recompute requested cache when called

## Runner refactor
Runner now dynamically invalidate updated cache at the end of the
transaction

close
https://github.com/orgs/twentyhq/projects/1/views/8?pane=issue&itemId=129136356&issue=twentyhq%7Ccore-team-issues%7C1492
close
https://github.com/orgs/twentyhq/projects/1/views/8?pane=issue&itemId=129136210&issue=twentyhq%7Ccore-team-issues%7C1494
2025-09-16 14:39:42 +02:00
Weiko d9a6e7e0b1 Add flat map cache service (#14418)
## Context
Adding a new service that provides an abstract caching system for
FlatEntityMaps.
This takes care of cache invalidation and storing local and remote cache
for the map with retrieval after a comparison with map hash between
local and remote (redis).

## Implementation
Remote (redis) keys
- Flat map data:
`engine:workspace:flat-maps:{flatMapKey}:{workspaceId}:flat-map`
- Content hash:
`engine:workspace:flat-maps:{flatMapKey}:{workspaceId}:hash`

**Local Cache Hit**: If local hash matches Redis hash, return local data
**Remote Cache Hit**: If Redis has data with different hash, update
local cache
**Remote Cache Miss**: Recompute from database, store in Remote and
locally
**Invalidation**: Remove from Remote, triggering recomputation on next
access

## Usage
```typescript
@WorkspaceFlatMapCache('view') // redis key
export class WorkspaceFlatViewMapCacheService extends WorkspaceFlatMapCacheService<FlatViewMaps> {
  constructor(
    @InjectCacheStorage(CacheStorageNamespace.EngineWorkspace)
    cacheStorageService: CacheStorageService,
    @InjectRepository(ViewEntity)
    private readonly viewRepository: Repository<ViewEntity>,
  ) {
    super(cacheStorageService);
  }

  // only method to implement
  public async computeFlatMap(workspaceId: string): Promise<FlatViewMaps> {
    const views = await this.viewRepository.find({
      where: { workspaceId },
      relations: ['viewFields'],
      select: { viewFields: { id: true } },
    });

    return generateFlatViewMaps(views);
  }
}
```

```typescript
// 2 public methods, getExistingOrRecomputeFlatMaps to fetch the map and invalidateCache after a mutation 
  await this.workspaceFlatViewMapCacheService.invalidateCache(
    viewData.workspaceId,
  );

  const flatViewMaps =
    await this.workspaceFlatViewMapCacheService.getExistingOrRecomputeFlatMaps(
      workspaceId,
    );
```

## Multi-Pod Synchronization
- Each pod maintains local cache for performance
- SHA256 hash of flatMap content used for version comparison
- `invalidateCache()` reset Redis data, forcing other pods to refresh
when calling getExistingOrRecomputeFlatMaps

--- 
<img width="1072" height="325" alt="Screenshot 2025-09-11 at 15 04 48"
src="https://github.com/user-attachments/assets/ed6ce82c-db35-4a1b-8a4e-247b694e2ddf"
/>
<img width="1030" height="328" alt="Screenshot 2025-09-11 at 15 04 40"
src="https://github.com/user-attachments/assets/43a15789-7f27-4104-a9bb-bef19908a5cd"
/>

Next step: Implement a locking mechanism by reusing existing WithLock
decorator
2025-09-11 16:54:21 +02:00