## Summary
- Fixes object `color` to use the `standardOverrides` mechanism for
standard objects, matching how `label`, `description`, and `icon`
already work
- Previously, color was written directly to the `objectMetadata.color`
column for **both** standard and custom objects, which meant user color
customizations on standard objects could be overwritten during metadata
syncs
- Custom objects continue to have `color` updated directly on the entity
(no change)
## Changes
| File | What changed |
|------|-------------|
| `object-metadata-standard-overrides-properties.constant.ts` | Added
`'color'` to `OBJECT_METADATA_STANDARD_OVERRIDES_PROPERTIES` so
`sanitizeRawUpdateObjectInput` routes color into `standardOverrides` for
standard objects |
| `resolve-object-metadata-standard-override.util.ts` | Extended to
support `'color'` as a key — handled like `icon` (no i18n/translation,
just direct override check) |
| `object-metadata.resolver.ts` | Added `@ResolveField` for `color` that
resolves through `resolveObjectMetadataStandardOverride`, matching the
existing `labelSingular`/`labelPlural`/`description`/`icon` resolve
fields |
| `flat-object-metadata-validator.service.ts` | Removed `'color'` from
`allowedOverrideKeys` for system objects since it now flows through
`standardOverrides` |
| `resolve-object-metadata-standard-override.util.spec.ts` | Added test
cases for custom object color, standard object color override, and
standard object color fallback |
| `successful-update-one-standard-object-metadata.integration-spec.ts` |
Added `'when updating color'` test case, included `color` in GraphQL
queries and `standardOverrides` fragment, reset color in `afterEach`
cleanup |
## How it works now
| Object type | Color update flow |
|---|---|
| **Custom** | Written directly to `objectMetadata.color` column |
| **Standard** | Stored in `objectMetadata.standardOverrides.color`,
resolved via `@ResolveField` at query time |
This is identical to how `label`, `description`, and `icon` have always
worked.
## Test plan
- [x] Unit tests pass
(`resolve-object-metadata-standard-override.util.spec.ts` — 21 tests)
- [x] Typecheck passes (`npx nx typecheck twenty-server`)
- [x] Lint passes (`npx nx lint:diff-with-main twenty-server`)
- [ ] Integration test snapshot regenerates correctly
(`successful-update-one-standard-object-metadata`)
- [ ] Verify standard object color editing from sidebar persists via
`standardOverrides`
- [ ] Verify custom object color editing from sidebar persists directly
on entity
Made with [Cursor](https://cursor.com)
## Summary
- **Optimistic metadata store updates**: Replace `refetchQueries` with
direct `addToDraft`/`applyChanges` calls in create, update, and delete
navigation menu item mutation hooks for instant UI feedback. Client-side
UUID generation enables optimistic creates before the server responds.
- **SSE event enrichment with `targetRecordIdentifier`**: Introduce
`NavigationMenuItemRecordIdentifierService` to resolve record display
info (label, image) and enrich SSE metadata events at emission time, so
the sidebar shows record names immediately without a page refresh.
- **Centralized role permission resolution**: Add
`resolveRolePermissionConfigFromAuthContext` to `PermissionsService`,
removing duplicated role resolution logic from individual services.
- **Mutation fragments include `targetRecordIdentifier`**: Switch
create/update/delete mutations from `NavigationMenuItemFields` to
`NavigationMenuItemQueryFields` so the mutation response includes
`targetRecordIdentifier`, preventing a brief gap where RECORD favorites
are invisible in the sidebar.
- **Folder UI fixes**: Remove transparent border on
`StyledFolderContainer` that caused a 1px size inconsistency between
folder and non-folder items in Favorites. Make the folder kebab menu
hover-only instead of always visible.
- enrich SSE events with relations
- remove queries from sse metadata events
- on sse event, manage store
- on object/field metadata changes, manage store
TODO left:
- fix other metadata items
---------
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions <github-actions@twenty.com>
Co-authored-by: Charles Bochet <charles@twenty.com>
## Summary
Replace the single `metadataVersion` integer with per-entity-type
**collection hashes** for granular metadata staleness detection. The
backend already generates a UUID per flat entity map on each cache
recompute (`crypto.randomUUID()` in `WorkspaceCacheService`); we now
expose these via the minimal metadata endpoint and SSE events so the
frontend can compare and know exactly which entity types are stale.
### Key changes
**Backend:**
- `WorkspaceCacheService.getCacheHashes()` — new public method that
reads only `:hash` keys from Redis without fetching full data
- `MinimalMetadataDTO` — added `collectionHashes: Record<string,
string>` (JSON scalar mapping `AllMetadataName` → collection hash),
removed `metadataVersion`
- `MetadataEventDTO` — added optional `updatedCollectionHash` field to
SSE events
- `MetadataEventsToDbListener` — reads the collection hash for the
affected entity type after cache invalidation and attaches it to the SSE
event before publishing
- `MinimalMetadataService` — no longer queries the workspace table; uses
`getCacheHashes()` for all flat entity maps and maps cache keys to
`AllMetadataName` locally
**Frontend:**
- `metadataCollectionHashesState` — new Jotai atom with
`atomWithStorage` + `getOnInit: true` storing
`Partial<Record<MetadataEntityKey, string>>`
- `mapAllMetadataNameToEntityKey()` — explicit mapping from backend
`AllMetadataName` to frontend `MetadataEntityKey` (23 entries)
- `useLoadMinimalMetadata` — stores `collectionHashes` from server,
computes `staleEntityKeys` by comparing local vs server hashes
- `patchMetadataStoreFromSSEEvent()` — accepts optional
`updatedCollectionHash` and updates `metadataCollectionHashesState`
- All 11 SSE effect components — pass
`eventDetail.updatedCollectionHash` through to the patch function
- `useStaleMetadataEntities` — new hook returning entity keys missing
from collection hashes (not yet loaded/synced)
- `resetMetadataStore()` — also clears collection hashes
- Deleted `metadataVersionState` (superseded by collection hashes)
### Design decisions
- **No change to hash generation** — existing `crypto.randomUUID()` is
sufficient. Hashes are persisted in Redis, survive server restarts, and
change only on `invalidateAndRecompute`.
- **"Collection hash" naming** — used consistently to clarify the hash
represents an entire entity collection (e.g., all views), not a single
record.
- **Mapping localized** — backend `WorkspaceCacheKeyName` →
`AllMetadataName` mapping lives in the minimal metadata service.
Frontend `AllMetadataName` → `MetadataEntityKey` mapping lives in a
local utility. Nothing in `twenty-shared`.
- **Backward compatible** — `collectionHashes` is additive;
`updatedCollectionHash` is nullable.
# Introduction
Previously the auth jwt stragegy would lod the whole user entity in the
auth user context
On an exception it would completely get logged on the pods
## Security layer
- 0/ Updating the type system ( devxp only though )
- 1/ The jwt auth stragegy only load a specific sub set of the user
entity
- 2/ Sanitizing at the exception log level directly in case of a user
context
- 3/ Sanitizing at the console driver
The last two sanitization could sound a bit redundant though they're
still good fallback to keep in case new path occurs in the cb
Create the necessary tooling to listen to metadata events and plug it to
the front components. Now we have a hot reload like experience when we
edit a component in an app.
## Backend
- Split `EventWithQueryIds` into `ObjectRecordEventWithQueryIds` and
`MetadataEventWithQueryIds`
- Publish metadata event batches to active SSE streams in
`MetadataEventsToDbListener`
## Frontend
- Create a metadata event dispatching pipeline: SSE metadata events are
grouped by metadata name, transformed into
`MetadataOperationBrowserEventDetail` objects, and dispatched as browser
`CustomEvents`
- Add `useListenToMetadataOperationBrowserEvent` hook for consuming
metadata operation events filtered by metadata name and operation type
- Rename `useListenToObjectRecordEventsForQuery` to
`useListenToEventsForQuery`, now accepting both
`RecordGqlOperationSignature` and `MetadataGqlOperationSignature`
- Implement `useOnFrontComponentUpdated` which subscribes to front
component metadata events and updates the Apollo cache when the
component is modified
- Add `builtComponentChecksum` to the front component query and appends
it to the component URL for browser cache invalidation
Fixes https://github.com/twentyhq/core-team-issues/issues/2192
This PR implements what is necessary to re-create the query that we
build on the frontend to obtain the returned object record from a
mutation, but on the backend, which was only partially implemented for
REST API.
Usually we want to have relations with only their id and label
identifier field to have lighter payloads.
In the event we only had depth 0 fields, with this PR we have all events
with depth 1 relations.
We have depth 2 for many-to-many cases, like updateOne or updateMany
result :
- Junction tables
- Activity target tables
## Fix resolver schema leaking between `/metadata` and `/graphql`
endpoints
### Summary
- Patch `@nestjs/graphql` to support a `resolverSchemaScope` option that
filters resolvers at both schema generation and runtime, preventing
cross-endpoint leaking
- Introduce `@CoreResolver()` and `@MetadataResolver()` decorators to
explicitly scope each resolver to its endpoint
- Move most resolvers (auth, billing, workspace, user, etc.) to the
metadata schema where the frontend expects them; only workflow and
timeline calendar/messaging resolvers remain on `/graphql`
- Fix frontend `SSEQuerySubscribeEffect` to use the default (metadata)
Apollo client instead of the core client
### Problem
NestJS GraphQL's module-based resolver discovery traverses transitive
imports, causing resolvers from `/metadata` modules to leak into the
`/graphql` schema and vice versa. This made the schemas unpredictable
and tightly coupled to module import order.
### Approach
- Added `resolverSchemaScope` to `GqlModuleOptions` via a patch on
`@nestjs/graphql`, filtering in both `filterResolvers()` (runtime
binding) and `getAllCtors()` (schema generation)
- Each resolver is explicitly decorated with `@CoreResolver()` or
`@MetadataResolver()`
- Organized decorator, constant, and type files under `graphql-config/`
following project conventions
Core GQL Schema: (see: no more fields!)
<img width="827" height="894" alt="image"
src="https://github.com/user-attachments/assets/668f3f0f-485e-43f0-92be-4345aeccacb6"
/>
Metadata GQL Schema (see no more getTimelineCalendarEventsFromCompany)
<img width="827" height="894" alt="image"
src="https://github.com/user-attachments/assets/443913db-e5fe-4161-b0e7-4a971cc80a71"
/>
# Introduction
In preparation of the workspace agnostic builder, we're migrating
`FlatEntityMaps` to be universal identifier oriented and based
As in the builder context there're won't be any ids at all
Please also note that the FlatEntity is a UniversalFlatEntity superset
From
```ts
import { type SyncableFlatEntity } from 'src/engine/metadata-modules/flat-entity/types/flat-entity-from.type';
export type FlatEntityMaps<T extends SyncableFlatEntity> = {
byId: Partial<Record<string, T>>;
idByUniversalIdentifier: Partial<Record<string, string>>;
universalIdentifiersByApplicationId: Partial<Record<string, string[]>>;
};
```
To
```ts
export type FlatEntityMaps<
T extends SyncableFlatEntity | UniversalSyncableFlatEntity,
> = {
byUniversalIdentifier: Partial<Record<string, T>>;
universalIdentifierById: Partial<Record<string, string>>;
universalIdentifiersByApplicationId: Partial<Record<string, string[]>>; // this might make more sense to be migrated to universalIdentifiersByApplicationUniversalIdentifier but it's the main topic of this PR
};
```
## Low level maps tools
Had to refactor find | create | delete | replace | find-many | get-sub
tools ( through mutations and or throw equivalent )
This PR handles SSE event stream edge cases.
- When a pod restarts, the front clients have to reconnect to SSE
- When the dev server restarts or is hot reloaded, the front client has
to reconnect to SSE
- When redis server restarts or the redis key is cleared for any reason,
the server has to recreate the event stream in redis, this can happen
when navigating for example.
- Log in / log out flow
With this PR we avoid error messages in the front end due to TTL or pod
crash, we implement a resilient way of reconnecting silently.
To avoid DDoSing our servers if pods crash or a full restart of the
cluster is made, we evenly space retry attempts to reconnect from all
the clients, to avoid n clients reconnection at the same time, we use a
random wait time between 0 and a constant max wait time (set to 2 mins
for now). This is the cheapest and most effective solution, clients who
want to force reconnect have to refresh or navigate to another page.
Fixes https://github.com/twentyhq/core-team-issues/issues/2045
This PR adds what is required to handle soft-delete and restore SSE
events in virtualized table and board.
Restore is not handled in board for now as it requires respecting sorts
when inserting record ids.
Since virtualized table is refetching small chunks, we just refetch for
now.
The long term goal is to handle event handling without refetching in all
main components, and also handle SSE events that have the same origin
that the current tab. But for now we implement what is easily doable.
# QA
Delete between table and board (delete only) :
https://github.com/user-attachments/assets/715dd44a-007a-44ab-bf49-5ef039cd57c3
Delete and restore between table and table :
https://github.com/user-attachments/assets/f2122519-e969-491f-b71c-018d0f85bd86
This PR update the redis subscription iterator wrapper with an heartbeat
system. Heartbeat interval are based on event stream TTL. Those refresh
the event stream TTL and active streams in redis.
I also cleaned a few functions that were not used anymore.
---------
Co-authored-by: cubic-dev-ai[bot] <191113872+cubic-dev-ai[bot]@users.noreply.github.com>
Fetch and send full workspace member to handle dynamic predicate. I only
tested that it did not break the current logic since the frontend does
not yet send queries we dynamic predicates.
Flow:
- fetch user role from context stored in cache - do not support api
context yet
- check object permissions
- filter restricted fields on event
- fetch role rls predicate and combine it with the query filter
Do not support dynamic predicates yet.
This PR implements SSE update events across the main components of the
application : tables, boards, calendars, show pages.
There is still work to do on other event type and on making sure
everything works fine, but this first implementation should be robust
enough to start with.
Some problems encountered along the way :
- Events are returning raw Postgres output, because they are not
normalized by the GraphQL layer, so we ended up with amountMicros as
string values, which the frontend does not like, so I implemented a
small util in our `formatResult` generic pipeline to turn amountMicros
to a number if it's a string value. We could implement other formatters
for composite fields if we see problems with events.
-`action` property was missing in SSE events, which is required by the
frontend to know what kind of event it is.
# QA
https://github.com/user-attachments/assets/393b45ac-59d2-48b0-855b-2ce9c4b8ae57https://github.com/user-attachments/assets/cb214e7a-1595-4b85-bdb6-87630993d2e2
- new channel EVENT_STREAM_CHANNEL based on event stream id
- on event, perform the matching and publish only to the right streams
- store a list of active streams per workspace
- store the user id along with the queries for each stream
Bonus:
- remove onSubscriptionMatch
This PR is a follow-up of https://github.com/twentyhq/twenty/pull/16966
and implements a new mechanism to handle SSE events.
It creates only one event stream per browser tab, then use mutations to
tell the backend which query to listen to, without re-mounting the event
stream connexion.
Then each event that comes from this unique subscription is then
dispatched in a new JavaScript CustomEvent, per queryId, on which
specific hooks add an event listener.
This PR introduces the generic tooling as well as the handling of update
events on table.
- moved a few gql types to twenty shared to re-use in server
- added a new endpoint, onEventSubscription that expect a streamId to
create a connection
- two new endpoints to store queries in Redis
- updated the existing batch channel to directly use object record type
## Context
Deprecating the old objectMetadataMap type in favour of split flat
entities to match with our new caching.
In the long run, trying to achieve:
- Better performance through caching
- Consistent data access patterns across the codebase
- Reduced database queries
Now that everything is based on flat entities, which are cached, we can
finish the refactoring of workspace context cache which should already
improve performances.
Then the last step will be to consume that new cache in the new global
datasource to get rid of the many workspace datasources stored in the
server
- on draft creation, do not fetch the full version. Avoid the fetch of
steps and trigger
- on activation, we were performing 8 queries/mutations synchronously +
2 additional for automated triggers. I refacto the call it it gets
reduced to 4 queries/mutations + 2 additional for automated triggers
## Context
1. We are debugging the experience on messaging and calendar sync. I'm
adding two scripts to relaunch messaging and calendar channels
2. We are seeing errors in Sentry regarding microsoft drivers errors. I
was about to fix them but I'm refactoring / simplifying a bit the logic
first. It will be a betters starting point
In this PR:
- refactor timelineActivity computation to batch it
- make sure to pass the authContext to insert-query-builder in order to
pass user information to timelineActivity
- refactor message-participant /calendar-participant creation to batch
more
- fix favorite on view race condition (FE)
- deprecate PARTIAL_CALENDAR_EVENT_FETCH_LIST syncStage as we will
deprecate partial vs full notion (we will just leverage cursor emptyness
or not)
- introduce calendar / messging SCHEDULED syncStage that will allow
better performance granularity later
- activate quick message import after message list fetch to speed
performance on small message lists
This PR aims to improve performances as we are making A LOT of queries
after having added events emission to the ORM layer. While investigating
issues, I've come across multiple problems I will describe below
## Add index on workspace.activationStatus as we are querying it a lot
As per title
## Add logs on core datasource destroy
It seems that we have postgres connection pool destruction in
production. I cannot reproduce locally but I suspect the Query Timeouts
to be the root cause. I'm fixing most of the Query Timeouts cause in
this PR but I'm adding the logs so we have more information in case in
keeps happening in production.
## GraphQL query runner createMany
It was using a for loop on each record. This is as issue has we will
emit an event separately for each record => we should always try to
batch events.
Replacing by a save on all records. Note that this is not perfect as we
should avoid using save (bad performances), and use insert + updateMany
instead. As I have follow up discussions regarding permissions, I
haven't replaced it by insert and updateMany yet. Using save instead of
a for loop makes the problem less worrying.
## Introduce updateMany in ORM @Weiko @ijreilly FYI
.save(manyRecords) is bad as it's querying the data for no good reason
(and doing a select for each record...).
We already have .insert(), i'm introducing .updateMany()
I think our ORM layer should be simplified a lot but I'm not starting
the refacto yet
## Fixing ORM @Weiko @ijreilly FYI
- Fixing events emission in delete function
- Fixing events emission in insert function
- make sure everything is batched
## Events performance @Weiko @ijreilly FYI
Do not emit timelineActivity db events as this does not seem useful and
is quite heavy
## Messaging and Calendar performance @bosiraphael FYI
Rework many functions to make sure they are batched. This does not touch
the driver layer and I have heavily tested it. (multiple account, with
multiple channels, common thread, common messages, etc...)
## Workflow Trigger relation Fetch performance @martmull FYI
Improved performance by batching
## Context
Add an eventEmitter instance to twenty datasources so we can emit DB
events.
Add input and output formatting to twenty orm (formatData, formatResult)
Those 2 elements simplified existing logic when we interact with the
ORM, input will be formatted by the ORM so we can directly use
field-like structure instead of column-like. The output will be
formatted, for builder queries it will be in `result.generatedMaps`
where `result.raw` preserves the previous column-like structure.
Important change: We now have an authContext that we can pass when we
get a repository, this will be used for the different events emitted in
the ORM. We also removed the caching for repositories as it was not
scaling well and not necessary imho
Note: An upcoming PR should handle the onDelete: cascade behavior where
we send DESTROY events in cascade when there is an onDelete: CASCADE on
the FK.
---------
Co-authored-by: Charles Bochet <charles@twenty.com>
Fixes https://github.com/twentyhq/twenty/issues/8300
## Context
API events were created too late and were already formatted as Gql
responses (including nesting with edges/node/type + formatting that
should not exist in an event payload). This PR moves the emit logic to
the resolver where we actually do the DB query
Note: Also added RESTORED events
Thomas Trompette