Create the necessary tooling to listen to metadata events and plug it to
the front components. Now we have a hot reload like experience when we
edit a component in an app.
## Backend
- Split `EventWithQueryIds` into `ObjectRecordEventWithQueryIds` and
`MetadataEventWithQueryIds`
- Publish metadata event batches to active SSE streams in
`MetadataEventsToDbListener`
## Frontend
- Create a metadata event dispatching pipeline: SSE metadata events are
grouped by metadata name, transformed into
`MetadataOperationBrowserEventDetail` objects, and dispatched as browser
`CustomEvents`
- Add `useListenToMetadataOperationBrowserEvent` hook for consuming
metadata operation events filtered by metadata name and operation type
- Rename `useListenToObjectRecordEventsForQuery` to
`useListenToEventsForQuery`, now accepting both
`RecordGqlOperationSignature` and `MetadataGqlOperationSignature`
- Implement `useOnFrontComponentUpdated` which subscribes to front
component metadata events and updates the Apollo cache when the
component is modified
- Add `builtComponentChecksum` to the front component query and appends
it to the component URL for browser cache invalidation
This PR handles SSE event stream edge cases.
- When a pod restarts, the front clients have to reconnect to SSE
- When the dev server restarts or is hot reloaded, the front client has
to reconnect to SSE
- When redis server restarts or the redis key is cleared for any reason,
the server has to recreate the event stream in redis, this can happen
when navigating for example.
- Log in / log out flow
With this PR we avoid error messages in the front end due to TTL or pod
crash, we implement a resilient way of reconnecting silently.
To avoid DDoSing our servers if pods crash or a full restart of the
cluster is made, we evenly space retry attempts to reconnect from all
the clients, to avoid n clients reconnection at the same time, we use a
random wait time between 0 and a constant max wait time (set to 2 mins
for now). This is the cheapest and most effective solution, clients who
want to force reconnect have to refresh or navigate to another page.
Fixes https://github.com/twentyhq/core-team-issues/issues/2045
This PR update the redis subscription iterator wrapper with an heartbeat
system. Heartbeat interval are based on event stream TTL. Those refresh
the event stream TTL and active streams in redis.
I also cleaned a few functions that were not used anymore.
---------
Co-authored-by: cubic-dev-ai[bot] <191113872+cubic-dev-ai[bot]@users.noreply.github.com>
- new channel EVENT_STREAM_CHANNEL based on event stream id
- on event, perform the matching and publish only to the right streams
- store a list of active streams per workspace
- store the user id along with the queries for each stream
Bonus:
- remove onSubscriptionMatch
- moved a few gql types to twenty shared to re-use in server
- added a new endpoint, onEventSubscription that expect a streamId to
create a connection
- two new endpoints to store queries in Redis
- updated the existing batch channel to directly use object record type
Thomas Trompette