feat(front): persist metadata store cache in IndexedDB instead of localStorage (#21586)

## Problem The metadata store cache (object/field metadata, views, page layouts, command menu items, …) is persisted client-side to power **cache-first boot**: the app renders instantly from the cache, then `MinimalMetadataLoadEffect` revalidates per-collection hashes and only refetches what's stale. It was persisted to **localStorage**, which Safari/WebKit caps at **~5 MB per origin, counted in UTF-16 (2 bytes/char)** → an effective ceiling of ~2.5 M characters. Measured on the seeded demo workspace (33 objects, 612 fields): | Bucket | Safari quota (UTF-16) | |---|---| | `metadataStoreState__*` (26 keys) | **1.9 MB — 37%** | | Whole origin | **2.47 MB — 48%** | A workspace ~2.5× the demo's schema blows past 5 MB, and there is **no `QuotaExceededError` handling** — `setItem` throws and breaks the app. This is what large-workspace users on Safari have been hitting. ## Fix Move **only the metadata store** to **IndexedDB** (multi-GB, disk-based quota), keeping a **fully synchronous read path** so the ~24 consumers that read these atoms with `useAtomValue` never suspend. The auth/UI atoms (incl. the synchronously-read `tokenPair`) stay on localStorage — intentionally scoped. - **`createIndexedDbBackedJotaiStorage.ts`** — a synchronous Jotai storage facade backed by an in-memory map, hydrated once from IndexedDB at boot and written through on every set. IndexedDB access uses the **`idb-keyval`** library (by the IndexedDB spec co-author, ~0.6 KB) rather than a hand-rolled wrapper. Each cache gets its own database + BroadcastChannel (`twenty-front-<cacheName>`), so it's safely reusable. Swallowed errors are surfaced via `logError`. When IndexedDB is unavailable the cache stays in memory only (re-fetched each boot). - **`createAtomFamilyState`** — gains an optional `storage` param; `metadataStoreState` uses the IndexedDB-backed storage. - **`index.tsx`** — awaits hydration before mounting so atoms (`getOnInit: true`) read the persisted snapshot synchronously → cache-first boot preserved. - **No migration**: the facade does not touch localStorage at all. Pre-existing localStorage snapshots are ignored — on first boot of the new code the IndexedDB cache is empty and atoms re-fetch from the network (a one-time reconnect). Old `metadataStoreState__*` localStorage keys are left in place (cleared by the existing logout/reset cleanup); new writes only ever go to IndexedDB. - **Cross-tab sync**: the old localStorage atoms synced across tabs for free via `storage` events; the IndexedDB facade had no equivalent, so a schema change in one tab left others stale until reload. Restored by implementing the Jotai storage `subscribe` contract over a **`BroadcastChannel`** — writes broadcast to other tabs, which update their in-memory map and notify `atomWithStorage` subscribers so mounted atoms re-render live. (BroadcastChannel doesn't echo to the sender, so no feedback loop; guarded for environments without it.) ## Why a synchronous facade (not async `atomWithStorage`) Consumers use `useAtomValue` directly; an async storage would make the atoms resolve to Promises and **suspend** every reader. The in-memory facade keeps reads synchronous (zero ripple on consumers) and confines the async part to a single bulk read at boot, which the existing `MinimalMetadataGater` loader already covers. ## Tests ### Automated - Unit test (10 cases) for the storage facade: synchronous read/write, IndexedDB write-through, hydration from IndexedDB, `removeItem`/`clear`, per-cache DB namespacing, persist-failure logging, in-memory-only behaviour when IndexedDB is unavailable, distinguishing a stored `undefined` from a missing key, and cross-tab subscriber registration. - Existing metadata-store tests (`useIsLayoutCustomizationDirty`, `useDefaultHomePagePath`) still pass. - `nx typecheck twenty-front` and `nx lint:diff-with-main twenty-front` clean. ### Manual (local seeded workspace, two tabs, Playwright) Storage: - After login the metadata cache lives in **IndexedDB (24 keys, ~945 KB)** and **localStorage drops 48% → 11%** of the Safari quota (the remainder is `currentUserState` + auth, out of scope). - Reload boots from the cache (no heavy refetch). Scenarios: | Scenario | Result | |---|---| | **Sign out** | auth cleared, redirect to sign-in, no leftover localStorage, no errors | | **Sign back in** | metadata `up-to-date`, company table renders, token restored | | **Add object** (`Gadget`) | write-through to IndexedDB; survives reload via cache-first hydration | | **Add view** (`QA Cross Tab View`, TABLE) | persisted to the `views` collection (`up-to-date`) | | **Two tabs open** | second tab boots cleanly from the shared IndexedDB — no lock/crash under concurrent access | | **Cross-tab live sync** | creating an object in tab A makes it appear in tab B's open settings object list **without a reload** | Verified by design (no regression): - Runtime sign-out (`clearSession`) clears session keys and does a full `window.location.assign` reload; the metadata-clearing path (`resetJotaiStore`) is test-only, so there's no async-`clear()`-vs-sign-in race. Metadata persisting across sign-out is unchanged from the old localStorage behavior (it's schema, revalidated by hash on next login). ## Notes / follow-ups (not in this PR) - **IndexedDB query capabilities** are not used yet: the cache stores one blob per collection (as it did in localStorage), so this is still a pure key-value use (`idb-keyval`). If we later want to query individual metadata records — e.g. fields by `objectMetadataId` via an index/cursor, or partial hydration — that means record-level storage and a richer wrapper (`idb` for a thin near-native layer, or **Dexie** for a full query API + reactive `liveQuery` that could also replace the BroadcastChannel sync). - IndexedDB still has a (large) quota and Safari ITP eviction applies to both stores — the cache-first design already tolerates eviction by revalidating. - Complementary "load less" wins remain: the denormalized per-field `relation` block (~700 chars/field of pure duplication) and persisting `currentUser.workspaceMembers` (the ~0.5 MB still in localStorage).  <a href="https://cubic.dev/pr/twentyhq/twenty/pull/21586?utm_source=github" target="_blank" rel="noopener noreferrer" data-no-image-dialog="true"><picture><source media="(prefers-color-scheme: dark)" srcset="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"><source media="(prefers-color-scheme: light)" srcset="https://www.cubic.dev/buttons/review-in-cubic-light.svg"><img alt="Review in cubic" src="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"></picture></a>
2026-06-15 15:25:53 +02:00
parent 018af36cfc
commit 88b9294afd
11 changed files with 391 additions and 12 deletions
@@ -94,6 +94,7 @@
    "graphql": "16.8.1",
    "graphql-sse": "^2.5.4",
    "graphql-tag": "^2.12.6",
+    "idb-keyval": "^6.2.5",
    "immer": "^10.1.1",
    "input-otp": "^1.4.2",
    "jotai": "^2.17.1",
@@ -2,6 +2,7 @@ import ReactDOM from 'react-dom/client';

 import { App } from '@/app/components/App';
 import { migrateTokenPairCookieToLocalStorage } from '@/auth/utils/migrateTokenPairCookieToLocalStorage';
+import { hydrateMetadataStore } from '@/metadata-store/storage/metadataStoreStorage';
 import 'react-loading-skeleton/dist/skeleton.css';
 import 'twenty-ui-deprecated/style.css';
 import 'twenty-ui-deprecated/theme-light.css';
@@ -15,8 +16,12 @@ import './index.css';
 // legacy cookie to localStorage (legacy cookie has a 180-day expiry).
 migrateTokenPairCookieToLocalStorage();

-const root = ReactDOM.createRoot(
-  document.getElementById('root') ?? document.body,
-);
+const renderApp = () => {
+  const root = ReactDOM.createRoot(
+    document.getElementById('root') ?? document.body,
+  );

-root.render(<App />);
+  root.render(<App />);
+};
+
+hydrateMetadataStore().then(renderApp, renderApp);
@@ -1,18 +1,19 @@
 import { safeRemoveLocalStorageItems } from '@/auth/utils/safeRemoveLocalStorageItems';
 import {
  ALL_METADATA_ENTITY_KEYS,
+  METADATA_STORE_KEY_PREFIX,
  type MetadataEntityKey,
 } from '@/metadata-store/states/metadataStoreState';
+import { clearMetadataStoreStorage } from '@/metadata-store/storage/metadataStoreStorage';
 import { clearSessionLocalStorageKeys } from './clearSessionLocalStorageKeys';

-const METADATA_STORE_PREFIX = 'metadataStoreState__';
-
 const getMetadataStoreKeys = (): string[] =>
  ALL_METADATA_ENTITY_KEYS.map(
-    (key: MetadataEntityKey) => `${METADATA_STORE_PREFIX}${key}`,
+    (key: MetadataEntityKey) => `${METADATA_STORE_KEY_PREFIX}${key}`,
  );

 export const clearAllSessionLocalStorageKeys = () => {
  clearSessionLocalStorageKeys();
+  void clearMetadataStoreStorage();
  safeRemoveLocalStorageItems(getMetadataStoreKeys());
 };
@@ -1,3 +1,4 @@
+import { metadataStoreStorage } from '@/metadata-store/storage/metadataStoreStorage';
 import { createAtomFamilyState } from '@/ui/utilities/state/jotai/utils/createAtomFamilyState';

 export type MetadataEntityStoreStatus =
@@ -51,12 +52,14 @@ const METADATA_STORE_ITEM_INITIAL_VALUE: MetadataStoreItem = {
  status: 'empty',
 };

+export const METADATA_STORE_KEY_PREFIX = 'metadataStoreState__';
+
 export const metadataStoreState = createAtomFamilyState<
  MetadataStoreItem,
  MetadataEntityKey
 >({
  key: 'metadataStoreState',
  defaultValue: METADATA_STORE_ITEM_INITIAL_VALUE,
-  useLocalStorage: true,
+  storage: metadataStoreStorage,
  localStorageOptions: { getOnInit: true },
 });
@@ -0,0 +1,8 @@
+import { type MetadataStoreItem } from '@/metadata-store/states/metadataStoreState';
+import { createIndexedDbBackedJotaiStorage } from '@/ui/utilities/state/jotai/utils/createIndexedDbBackedJotaiStorage';
+
+export const {
+  storage: metadataStoreStorage,
+  hydrate: hydrateMetadataStore,
+  clear: clearMetadataStoreStorage,
+} = createIndexedDbBackedJotaiStorage<MetadataStoreItem>('metadata-store');
@@ -0,0 +1,10 @@
+export type JotaiSyncStorage<ValueType> = {
+  getItem: (key: string, initialValue: ValueType) => ValueType;
+  setItem: (key: string, newValue: ValueType) => void;
+  removeItem: (key: string) => void;
+  subscribe?: (
+    key: string,
+    callback: (value: ValueType) => void,
+    initialValue: ValueType,
+  ) => () => void;
+};
@@ -0,0 +1,153 @@
+import { clear, createStore, del, entries, set } from 'idb-keyval';
+
+import { createIndexedDbBackedJotaiStorage } from '@/ui/utilities/state/jotai/utils/createIndexedDbBackedJotaiStorage';
+import { isIndexedDbAvailable } from '@/ui/utilities/state/jotai/utils/isIndexedDbAvailable';
+import { logError } from '~/utils/logError';
+
+jest.mock('idb-keyval', () => ({
+  createStore: jest.fn(() => ({ store: 'mock' })),
+  set: jest.fn(() => Promise.resolve()),
+  del: jest.fn(() => Promise.resolve()),
+  clear: jest.fn(() => Promise.resolve()),
+  entries: jest.fn(() => Promise.resolve([])),
+}));
+jest.mock('@/ui/utilities/state/jotai/utils/isIndexedDbAvailable');
+jest.mock('~/utils/logError');
+
+const mockedSet = jest.mocked(set);
+const mockedDel = jest.mocked(del);
+const mockedClear = jest.mocked(clear);
+const mockedEntries = jest.mocked(entries);
+const mockedCreateStore = jest.mocked(createStore);
+const mockedIsIndexedDbAvailable = jest.mocked(isIndexedDbAvailable);
+const mockedLogError = jest.mocked(logError);
+
+const flushMicrotasks = () => new Promise((resolve) => setTimeout(resolve, 0));
+
+type Item = { value: number };
+
+const INITIAL: Item = { value: 0 };
+
+describe('createIndexedDbBackedJotaiStorage', () => {
+  beforeEach(() => {
+    jest.clearAllMocks();
+    localStorage.clear();
+    mockedIsIndexedDbAvailable.mockReturnValue(true);
+    mockedEntries.mockResolvedValue([]);
+  });
+
+  it('should use a database namespaced to the cache name', () => {
+    createIndexedDbBackedJotaiStorage<Item>('metadata-store');
+
+    expect(mockedCreateStore).toHaveBeenCalledWith(
+      'twenty-front-metadata-store',
+      'keyval',
+    );
+  });
+
+  it('should read synchronously from the in-memory map', () => {
+    const { storage } = createIndexedDbBackedJotaiStorage<Item>('test');
+
+    expect(storage.getItem('k', INITIAL)).toBe(INITIAL);
+
+    storage.setItem('k', { value: 1 });
+
+    expect(storage.getItem('k', INITIAL)).toEqual({ value: 1 });
+  });
+
+  it('should distinguish a stored undefined value from a missing key', () => {
+    const { storage } = createIndexedDbBackedJotaiStorage<number | undefined>(
+      'test',
+    );
+
+    expect(storage.getItem('k', 0)).toBe(0);
+
+    storage.setItem('k', undefined);
+
+    expect(storage.getItem('k', 0)).toBeUndefined();
+  });
+
+  it('should write through to IndexedDB on set', () => {
+    const { storage } = createIndexedDbBackedJotaiStorage<Item>('test');
+
+    storage.setItem('k', { value: 2 });
+
+    expect(mockedSet).toHaveBeenCalledWith(
+      'k',
+      { value: 2 },
+      expect.anything(),
+    );
+  });
+
+  it('should hydrate the in-memory map from IndexedDB', async () => {
+    mockedEntries.mockResolvedValue([['k', { value: 7 }]]);
+
+    const { storage, hydrate } =
+      createIndexedDbBackedJotaiStorage<Item>('test');
+
+    await hydrate();
+
+    expect(storage.getItem('k', INITIAL)).toEqual({ value: 7 });
+  });
+
+  it('should delete from the map and IndexedDB on removeItem', () => {
+    const { storage } = createIndexedDbBackedJotaiStorage<Item>('test');
+
+    storage.setItem('k', { value: 1 });
+    storage.removeItem('k');
+
+    expect(storage.getItem('k', INITIAL)).toBe(INITIAL);
+    expect(mockedDel).toHaveBeenCalledWith('k', expect.anything());
+  });
+
+  it('should clear the map and IndexedDB', async () => {
+    const { storage, clear: clearStorage } =
+      createIndexedDbBackedJotaiStorage<Item>('test');
+
+    storage.setItem('k', { value: 1 });
+    await clearStorage();
+
+    expect(storage.getItem('k', INITIAL)).toBe(INITIAL);
+    expect(mockedClear).toHaveBeenCalled();
+  });
+
+  it('should register and unregister cross-tab subscribers cleanly', () => {
+    const { storage } = createIndexedDbBackedJotaiStorage<Item>('test');
+
+    const callback = jest.fn();
+    const unsubscribe = storage.subscribe?.('k', callback, INITIAL);
+
+    expect(typeof unsubscribe).toBe('function');
+    expect(() => unsubscribe?.()).not.toThrow();
+  });
+
+  it('should keep the value in memory and log when a persist fails', async () => {
+    mockedSet.mockRejectedValueOnce(new Error('write failed'));
+
+    const { storage } = createIndexedDbBackedJotaiStorage<Item>('test');
+
+    expect(() => storage.setItem('k', { value: 1 })).not.toThrow();
+    expect(storage.getItem('k', INITIAL)).toEqual({ value: 1 });
+
+    await flushMicrotasks();
+
+    expect(mockedLogError).toHaveBeenCalled();
+  });
+
+  describe('when IndexedDB is unavailable', () => {
+    beforeEach(() => {
+      mockedIsIndexedDbAvailable.mockReturnValue(false);
+    });
+
+    it('should keep values in memory only, without persisting anywhere', () => {
+      const { storage } = createIndexedDbBackedJotaiStorage<Item>('test');
+
+      storage.setItem('k', { value: 5 });
+
+      expect(storage.getItem('k', INITIAL)).toEqual({ value: 5 });
+      expect(mockedCreateStore).not.toHaveBeenCalled();
+      expect(mockedSet).not.toHaveBeenCalled();
+      expect(localStorage.getItem('k')).toBeNull();
+    });
+  });
+});
@@ -1,18 +1,22 @@
 import { atom } from 'jotai';
 import { atomWithStorage } from 'jotai/utils';
+import { isDefined } from 'twenty-shared/utils';

 import { type FamilyState } from '@/ui/utilities/state/jotai/types/FamilyState';
+import { type JotaiSyncStorage } from '@/ui/utilities/state/jotai/types/JotaiSyncStorage';

 export const createAtomFamilyState = <ValueType, FamilyKey>({
  key,
  defaultValue,
  useLocalStorage = false,
  localStorageOptions,
+  storage,
 }: {
  key: string;
  defaultValue: ValueType;
  useLocalStorage?: boolean;
  localStorageOptions?: { getOnInit?: boolean };
+  storage?: JotaiSyncStorage<ValueType>;
 }): FamilyState<ValueType, FamilyKey> => {
  const atomCache = new Map<
    string,
@@ -32,14 +36,30 @@ export const createAtomFamilyState = <ValueType, FamilyKey>({
    }

    const atomKey = `${key}__${cacheKey}`;
-    const baseAtom = useLocalStorage
-      ? atomWithStorage<ValueType>(
+
+    const buildBaseAtom = () => {
+      if (isDefined(storage)) {
+        return atomWithStorage<ValueType>(
+          atomKey,
+          defaultValue,
+          storage,
+          localStorageOptions ?? { getOnInit: true },
+        );
+      }
+
+      if (useLocalStorage) {
+        return atomWithStorage<ValueType>(
          atomKey,
          defaultValue,
          undefined,
          localStorageOptions ?? undefined,
-        )
-      : atom(defaultValue);
+        );
+      }
+
+      return atom(defaultValue);
+    };
+
+    const baseAtom = buildBaseAtom();
    baseAtom.debugLabel = atomKey;
    atomCache.set(cacheKey, baseAtom);

@@ -0,0 +1,163 @@
+import * as idb from 'idb-keyval';
+import { isDefined } from 'twenty-shared/utils';
+
+import { type JotaiSyncStorage } from '@/ui/utilities/state/jotai/types/JotaiSyncStorage';
+import { isIndexedDbAvailable } from '@/ui/utilities/state/jotai/utils/isIndexedDbAvailable';
+import { logError } from '~/utils/logError';
+
+const INDEXED_DB_STORE_NAME = 'keyval';
+
+type CrossTabMessage<ValueType> =
+  | { type: 'set'; key: string; value: ValueType }
+  | { type: 'remove'; key: string };
+
+type CrossTabSubscriber<ValueType> = {
+  callback: (value: ValueType) => void;
+  initialValue: ValueType;
+};
+
+type IndexedDbBackedJotaiStorage<ValueType> = {
+  storage: JotaiSyncStorage<ValueType>;
+  hydrate: () => Promise<void>;
+  clear: () => Promise<void>;
+};
+
+const createIndexedDbStore = (cacheName: string): idb.UseStore | undefined => {
+  if (!isIndexedDbAvailable()) {
+    return undefined;
+  }
+
+  return idb.createStore(`twenty-front-${cacheName}`, INDEXED_DB_STORE_NAME);
+};
+
+const createCrossTabChannel = (cacheName: string): BroadcastChannel | null => {
+  if (typeof BroadcastChannel === 'undefined') {
+    return null;
+  }
+
+  try {
+    return new BroadcastChannel(`twenty-front-${cacheName}-sync`);
+  } catch (error) {
+    logError(error);
+    return null;
+  }
+};
+
+export const createIndexedDbBackedJotaiStorage = <ValueType>(
+  cacheName: string,
+): IndexedDbBackedJotaiStorage<ValueType> => {
+  const memoryMap = new Map<string, ValueType>();
+  const idbStore = createIndexedDbStore(cacheName);
+  const broadcastChannel = createCrossTabChannel(cacheName);
+  const subscribers = new Map<string, Set<CrossTabSubscriber<ValueType>>>();
+  let isHydrated = false;
+
+  const persist = (operation: Promise<unknown>): void => {
+    void operation.catch(logError);
+  };
+
+  if (broadcastChannel !== null) {
+    broadcastChannel.onmessage = (
+      event: MessageEvent<CrossTabMessage<ValueType>>,
+    ) => {
+      const message = event.data;
+
+      if (message.type === 'set') {
+        memoryMap.set(message.key, message.value);
+      } else {
+        memoryMap.delete(message.key);
+      }
+
+      const keySubscribers = subscribers.get(message.key);
+
+      if (isDefined(keySubscribers)) {
+        for (const subscriber of keySubscribers) {
+          subscriber.callback(
+            message.type === 'set' ? message.value : subscriber.initialValue,
+          );
+        }
+      }
+    };
+  }
+
+  const broadcast = (message: CrossTabMessage<ValueType>): void => {
+    try {
+      broadcastChannel?.postMessage(message);
+    } catch (error) {
+      logError(error);
+    }
+  };
+
+  const storage: JotaiSyncStorage<ValueType> = {
+    getItem: (key, initialValue) =>
+      memoryMap.has(key) ? (memoryMap.get(key) as ValueType) : initialValue,
+    setItem: (key, newValue) => {
+      memoryMap.set(key, newValue);
+      broadcast({ type: 'set', key, value: newValue });
+
+      if (isDefined(idbStore)) {
+        persist(idb.set(key, newValue, idbStore));
+      }
+    },
+    removeItem: (key) => {
+      memoryMap.delete(key);
+      broadcast({ type: 'remove', key });
+
+      if (isDefined(idbStore)) {
+        persist(idb.del(key, idbStore));
+      }
+    },
+    subscribe: (key, callback, initialValue) => {
+      const keySubscribers = subscribers.get(key) ?? new Set();
+      const subscriber: CrossTabSubscriber<ValueType> = {
+        callback,
+        initialValue,
+      };
+      keySubscribers.add(subscriber);
+      subscribers.set(key, keySubscribers);
+
+      return () => {
+        keySubscribers.delete(subscriber);
+
+        if (keySubscribers.size === 0) {
+          subscribers.delete(key);
+        }
+      };
+    },
+  };
+
+  const hydrate = async (): Promise<void> => {
+    if (isHydrated || !isDefined(idbStore)) {
+      isHydrated = true;
+      return;
+    }
+
+    try {
+      const persistedEntries = await idb.entries<string, ValueType>(idbStore);
+
+      for (const [key, value] of persistedEntries) {
+        memoryMap.set(key, value);
+      }
+    } catch (error) {
+      logError(error);
+    }
+
+    isHydrated = true;
+  };
+
+  const clear = async (): Promise<void> => {
+    memoryMap.clear();
+
+    if (!isDefined(idbStore)) {
+      return;
+    }
+
+    try {
+      await idb.clear(idbStore);
+    } catch (error) {
+      logError(error);
+    }
+  };
+
+  return { storage, hydrate, clear };
+};
@@ -0,0 +1,7 @@
+export const isIndexedDbAvailable = (): boolean => {
+  try {
+    return typeof indexedDB !== 'undefined' && indexedDB !== null;
+  } catch {
+    return false;
+  }
+};
@@ -36608,6 +36608,13 @@ __metadata:
  languageName: node
  linkType: hard

+"idb-keyval@npm:^6.2.5":
+  version: 6.2.5
+  resolution: "idb-keyval@npm:6.2.5"
+  checksum: 10c0/58055106e2447cd4a91c62f1c6d4122d8e6d141c86c6e79860dbb6af0c6462bc9520e84c1d56dc9e5fb66c817dffd04b2a2432efada1273b89db955fc49d374b
+  languageName: node
+  linkType: hard
+
 "identifier-regex@npm:^1.0.0":
  version: 1.0.1
  resolution: "identifier-regex@npm:1.0.1"
@@ -53509,6 +53516,7 @@ __metadata:
    graphql: "npm:16.8.1"
    graphql-sse: "npm:^2.5.4"
    graphql-tag: "npm:^2.12.6"
+    idb-keyval: "npm:^6.2.5"
    immer: "npm:^10.1.1"
    input-otp: "npm:^1.4.2"
    jest: "npm:29.7.0"