Fixes https://github.com/twentyhq/twenty/issues/16624
Original issue:
- while persisting a field (calling useUpdateOne), the response from the
backend is missing the taskTargets many to many (same for note). As we
optimistically update the cache, we lose the "Relations" in the UI
- I'm changing the behavior of useUpsertInRecordStore to accept
recordGqlFields to only update the fields we want in the record store
(this way, we are not losing the targets information in our case)
Avoid fetching full steps and trigger for versions that are not the
current version. Because those won't be used anyway. Better for
performances.
Only difficulty was for the `createDraft` mutation. I needed to return
the full created version so I can store it in cache and use it as new
`currentVersion`. Otherwise the current version is considered as
incomplete for a short time, since workflow is fetched separately from
the current version.
---------
Co-authored-by: Charles Bochet <charles@twenty.com>
## Problem
With Twenty usage growing, we are facing challenges on server side to
respond to the demand. The current bottleneck we are facing is server
CPU.
While investigating CPU performances, we figured out that loading all
relation fields was the biggest issue.
### About graphql query response size
**Example:** on `People Index` Table page:
we query: `person.company` and we query **all non relation fields** on
company relation field.
`{ person { company { id, name, domainName, employees, address ... } }`
However we only need **company.id, company.name and company.domainName**
to be able to render the Company Chip in the company column (RELATION)
in the table.
`{ person { company { id, name, domainName } }`
We initially assumed that the querying all non relation fields was not
an issue because it was not adding additional load on database (which is
also partially true: if a field is containing a huge json this will add
load on postgres as data needs to be transfered)
This assumption is wrong on CPU side:
- when loading one to many relations (company.people), this quickly ads
up and we end up with response of 20kB quite quickly, even without
adding any custom field on person.
- even when loading a many to one relation, if the user is storing big
data in a given field (note.body for instance, or workflowRun.state),
this starts also being an issue.
- Worst case scenario are starting to happen: on a production workspace
with 20 active workflows. Loading workflow table while displaying
workflow runs commands (20 workflows x 60 workflowRuns x
workflowRun.state which is a big JSON) result in a response of... 125MB.
### Why is it an issue?
When yoga (graphql engine) parses a response to send it back in server
response, it's using `JSON.parse `(or stringify depending on the case).
Parsing data is CPU instensive (as you need to validate, transform) and
this is more or less `O(n)`.
This means returning 125MB is very intense on the CPU and will likely
use 100% of the CPU for ~1sec in our production servers.
NodeJs (nodeV8) is single threaded. It's able to process multiple
requests in parallel but to do that, it will cut them in "microTasks"
and process each microTasks (that can belong to different requests) one
after the other. Exactly like a single threaded CPU would allocate some
time to a process and then to the next one, etc.
This means that this big response request will actually block the other
request.
As a result, all requests are slow and our infrastructure is multi
tenant so some customers are impacting others.
This is even a bigger issue when our health checks start to fail and
containers are starting being considered as unhealthy and killed by the
orchestrator
## How to fix this
1. On Front end side, we should only query what we need. In this PR, I'm
forcing the relations to only query: `id`, `labelIdentifier`,
`imageIdentifier`
2. (later) as we are API first, we also need to do something to protect
the servers from this side too. To do that, Graphql APIs can associate a
cost to each graphql field (maybe 1 for a test fixed, 2 for a JSON
field, 10 for relations, etc...) and we can throttle based on that.
## Changes in this PR
It's mainly about refactor the tooling to generate `RecordGqlFields`
(`generateDepthOneGqlFields`). this tooling is used to generate the list
of fields we want to query. Most of the time we want to query a record
with one level of nesting.
1) Refactor to remove duplicate code => `generateDepthOne` become
`generateDepth` and can handle both `depth = 0 | 1`
2) Introduce `generateDepthRecordGqlFieldsFromFields.ts` which is the
base and can give you a list of gqlFields based on FieldMetadataItems
3) Introduce `generateDepthRecordGqlFieldsFromObject` which is a
shortcut for an object
4) Introduce `generateDepthRecordGqlFieldsFromRecords` which is the
intersection between `generateDepthRecordGqlFieldsFromObject` and a
given record (useful for cache tooling)
5) Replace all usages + introduce a hook
`useGenerateDepthRecordGqlFieldsFromObject` to ease usage
The goal of this PR is to test if Twenty can support a large number of
members, a question which was raised by a large company that is
considering moving away from Salesforce.
I was expecting the currentWorkspaceMembersState to cause a lot more
issue. It would be very hard to get rid of it in the context of actors,
I think that would require a big refactoring. I thought we'd have to do
it but it turns out the perf are pretty good. One thing we need to
improve is the pagination on the roles page, we'll wait for @Bonapara to
update that
This PR removes board field definitions usage in favor of record fields,
like we've already done on table with table columns.
This PR also introduces a first step in the refactoring of the
RecordIndexContext as a new way to store in-memory a normalized cache
for some states used pretty much everywhere like field metadata items or
field definition (for now but it will soon be removed).
Since most of those states do not change if we don't modify the
metadata, then we can safely store them in a context, which has the
fastest access time in React.
Co-authored-by: Charles Bochet <charles@twenty.com>
## Add Morph Relation Field Display Support for Tables (one to many &
many to one)
**Features:**
- Display morph relation fields (one-to-many, many-to-one) in table
cells
- Multi-field record retrieval for improved performance
- Enhanced table cell rendering with morph relation support
**Components Added:**
- `MorphRelationManyToOneFieldDisplay` - Shows many-to-one morph
relations
- `MorphRelationOneToManyFieldDisplay` - Shows one-to-many morph
relations
TODO :
handle notes and tasks case if we choose these kind of objects as
targets/source for a morph relation
---------
Co-authored-by: Charles Bochet <charles@twenty.com>
Issue: https://github.com/twentyhq/twenty/issues/13913
Motivation/Problem:
Bulk delete and query paths were composing filters differently, causing
mismatches. In some cases this led to invalid or empty GraphQL filters
(e.g. {"and":[{}]}), breaking delete operations.
Fix:
Unify filter composition (combinedFilter) across queries and bulk
delete, ensuring consistent handling of base filters + soft-deleted
clause. Also adjusted record filter grouping logic to avoid dropping
filters when no groups exist.
Result:
Filtered queries and bulk deletes now behave consistently and reliably
without producing broken filters.
---------
Co-authored-by: root <root@DESKTOP-E2VOJGE>
Co-authored-by: Charles Bochet <charles@twenty.com>
This PR adds a new RecordField abstraction to mimick the behavior of
RecordFilter, RecordSort and other similar concepts that allow to have
something that is not related directly to views.
It is closely following the same pattern with an Effect component to
initialize it from views and CRUD hooks to modify it.
Although for this concept we save to view fields automatically for every
modification.
This PR does not implement saving to view fields, it is just a parallel
code path that is meant to be built to see if everything behaves the
same then we'll remove the old code path at the end by saving record
fields instead of the many states for columns, fields definitions, etc.
This PR removes any V2 naming in state management logic and some minor
utils and hooks.
It has a lot of changes but nearly all of them were made by the rename
functionality of vscode which is deterministic, so it shouldn't
introduce any regression.
QA has been made on this PR on the main features of the app without any
noticeable issue.
Also renamed some other v2 naming related items :
- TextInputV2 => TextInput
- TextInput => SettingsTextInput
- ObjectFilterDropdownFilterSelectMenuItemV2 =>
ObjectFilterDropdownFilterSelectMenuItem
- useInitDraftValueV2 => useInitDraftValue
- useOpenRecordTableCellV2 => useOpenRecordTableCell
In our Graphql schema, ids are typed as UUID.
Our FE codegen is typing it as 'any', unless we say otherwise. Fixing it
and fixing the impacts in the codebase
This PR removes every legacy state management logic left in the code
base, we only keep our last component state management logic.
Removed :
- Scoped states logic
- Dropdown scope logic
- Scope id naming
- Component state v1
- Component state v2_alpha
# QA
| Component | Comments |
| --- | --- |
| AttachementDropdown | Ok |
| ObjectTasks | Ok |
| ActivityRichTextEditor | Ok |
| App | Ok dialogs work |
| AppRouterProviders | Ok dialogs work |
| RecordBoardScrollToFocusedCardEffect | Ok |
| RecordBoard | Ok |
| RecordBoardCard | OK |
| useSelectAllCards | Ok |
| RecordInlineCell | Ok |
| useInlineCell | Ok |
| RecordDetailRelationRecordsListItem | Ok |
| RecordDetailRelationSectionDropdownToMany | Ok |
| RecordDetailRelationSectionDropdownToOne | Ok |
| useHandleContainerMouseEnter | Ok |
| useResetTableRowSelection | Ok |
| useSelectAllRows | Ok |
| useSetRecordTableData | Ok |
| useRecordTable | Ok |
| useRecordTableMoveFocusedCell | Ok |
| useRecordTableMoveFocusedRow | Ok |
| useMoveHoverToCurrentCell | OK |
| useTriggerActionMenuDropdown | Ok |
| RecordTableBodyDragDropContextProvider | Ok |
| RecordTableHeaderCell | Ok |
| useSetCurrentRowSelected | Ok |
| useRecordTitleCell | Ok |
| SettingsObjectFieldActiveActionDropdown | Ok |
| SettingsObjectFieldInactiveActionDropdown | Ok |
| SettingsObjectFieldItemTableRow | Ok |
| SettingsObjectInactiveMenuDropDown | Ok |
| SettingsRoleEditEffect | Ok |
| SettingsRolesQueryEffect | Ok |
| Dropdown | Ok |
| useSelectableList | Ok |
| useSelectableListHotKeys | Ok |
| useSelectableListListenToEnterHotkeyOnItem | Ok |
| useNavigationSection | Ok |
| useCreateViewFromCurrentState | Ok |
| useDeleteViewFromCurrentState | Ok |
| useUpdateViewFromCurrentState | Ok |
| ViewPickerFavoriteFoldersDropdown | Ok |
| useCreateViewFromCurrentView | Ok |
| ViewBarDetails | Ok |
| WorkflowRunVisualizer | Ok |
| useHandleWorkflowRunDiagramCanvasInit | Ok |
| WorkflowDiagramCanvasBase | Ok |
| WorkflowRunVisualizerEffect | Ok |
| useRemoveStepFilter | Ok |
| useRemoveStepFilterGroup | Ok |
| useUpsertStepFilterSettings | Ok |
| WorkflowFindRecordsFilters | Ok |
| SettingsObjects | Ok |
This PR refactors the snackbar modules that was using legacy versions of
our state management.
We replace the old states with new ones and also the old scoped context
with component instance context.
# Introduction
In this PR we've migrated `twenty-shared` from a `vite` app
[libary-mode](https://vite.dev/guide/build#library-mode) to a
[preconstruct](https://preconstruct.tools/) "atomic" application ( in
the future would like to introduce preconstruct to handle of all our
atomic dependencies such as `twenty-emails` `twenty-ui` etc it will be
integrated at the monorepo's root directly, would be to invasive in the
first, starting incremental via `twenty-shared`)
For more information regarding the motivations please refer to nor:
- https://github.com/twentyhq/core-team-issues/issues/587
-
https://github.com/twentyhq/core-team-issues/issues/281#issuecomment-2630949682
close https://github.com/twentyhq/core-team-issues/issues/589
close https://github.com/twentyhq/core-team-issues/issues/590
## How to test
In order to ease the review this PR will ship all the codegen at the
very end, the actual meaning full diff is `+2,411 −114`
In order to migrate existing dependent packages to `twenty-shared` multi
barrel new arch you need to run in local:
```sh
yarn tsx packages/twenty-shared/scripts/migrateFromSingleToMultiBarrelImport.ts && \
npx nx run-many -t lint --fix -p twenty-front twenty-ui twenty-server twenty-emails twenty-shared twenty-zapier
```
Note that `migrateFromSingleToMultiBarrelImport` is idempotent, it's atm
included in the PR but should not be merged. ( such as codegen will be
added before merging this script will be removed )
## Misc
- related opened issue preconstruct
https://github.com/preconstruct/preconstruct/issues/617
## Closed related PR
- https://github.com/twentyhq/twenty/pull/11028
- https://github.com/twentyhq/twenty/pull/10993
- https://github.com/twentyhq/twenty/pull/10960
## Upcoming enhancement: ( in others dedicated PRs )
- 1/ refactor generate barrel to export atomic module instead of `*`
- 2/ generate barrel own package with several files and tests
- 3/ Migration twenty-ui the same way
- 4/ Use `preconstruct` at monorepo global level
## Conclusion
As always any suggestions are welcomed !
This PR simply implements record filter group states and context, as we
did for record filter and record sort.
We use a separate context for record filter and record filter group,
we'll see later if it can be merged in practice, but better be cautious
for now.
This PR is simple, it creates states for record sort, mirroring record
filter states.
It also implements RecordSortsComponentInstanceContext everywhere
RecordFiltersComponentInstanceContext is used.
This could be later merged into a common RecordContext concept but we
first need to decide how to handle the existing ContextStore and
RecordIndexContext and ideally end up with a unique context (or a
context provider component that wraps in all those contexts at once).
Some bugs are already present on main when trying to delete a sort, they
will be fixed in the next PRs.
In this huge (sorry!) PR:
- introducing objectMetadataItem in contextStore instead of
objectMetadataId which is more convenient
- splitting some big hooks into smaller parts to avoid re-renders
- removing Effects to avoid re-renders (especially onViewChange)
- making the view prefetch separate from favorites to avoid re-renders
- making the view prefetch load a state and add selectors on top of it
to avoir re-renders
As a result, the performance is WAY better (I suspect the favorite
implementation to trigger a lot of re-renders unfortunately).
However, we are still facing a random app freeze on view creation. I
could not investigate the root cause. As this seems to be already there
in the precedent release, we can move forward but this seems a urgent
follow up to me ==> EDIT: I've found the root cause after a few ours of
deep dive... an infinite loop in RecordTableNoRecordGroupBodyEffect...
prastoin edit: close https://github.com/twentyhq/twenty/issues/10253
---------
Co-authored-by: Lucas Bordeau <bordeau.lucas@gmail.com>
Co-authored-by: prastoin <paul@twenty.com>
# Introduction
While importing records encountering missing expected fields when
writting a fragment from apollo cache
## Updates
### 1/ `createdBy` Default value
When inserting in cache in create single or many we will now make
optimistic behavior on the createdBy value
### 2/ `createRecordInCache` dynamically create `recordGrqlFields`
When creating an entry in cache, we will now dynamically generate fields
to be written in the fragment instead of expecting all of them. As by
nature record could be partial
### 3/ Strictly typed `RecordGqlFields`
# Conclusion
closes#9927
This PR essentially removes the usage of filterDefinition.type, by
replacing it with fieldMetadataItem.type derivation. Thus allowing to
completely remove filterDefinition later on.
In computeFilterRecordGqlOperationFilter, emptyOperationFilter is now
returned before going into the big switch case. This avoids repeating
the same exact call to getEmptyRecordGqlOperationFilter for each type.
Fixed some tests that need
getJestMetadataAndApolloMocksAndActionMenuWrapper to have record filters
properly working with the new implementation. We'll probably want to
refactor the record context store, record index context, etc.
Co-authored-by: Charles Bochet <charlesBochet@users.noreply.github.com>
# Introduction
At the moment when updating any record cache occurence, we will build a
fragment that will expect all of the object metadata item fields to be
provided.
Which result in the following traces: ( in the video companies aren't
fetch with companyId and other missing fields )
https://github.com/user-attachments/assets/56eab7c1-8f01-45ff-8f5d-78737b788b92
By definition as we're using graphql we might not request every record's
fields each time we wanna consume them.
In this way we will now dynamically compute or expect depending on the
CRUD operation specific fields to be written in the cache, and not all
of them
Tested all optimistic and failure management use cases
## Covering cache
Added coverage only for the `deleteOne` and `deleteMany` hooks, it cover
only the record record cache and not its relations hydratation ( for the
moment )
## Why not closing #9927
Unless I'm mistaken everything done here have fixed the same logs/traces
issue for updates and deletion but not creation.
Which means we still need to investigate the mass upload from import and
prefillRecord behavior
In a nutshell: went over each `updateRecordFromCache` calls, still need
to do all `createRecordInCache` calls
related to #9927
## Conlusion
Sorry for the big PR should have ejected into a specific one for the
`MinimalRecord` refactor
Will also continue covering others hooks later in my week as for the
`deleteOne`
As always any suggestions are welcomed !
# Introduction
Added the `RecordAction` destroy multiple record
## Repro
Select multiples `deletedRecords`, you should be able to see the
`Destroy` pinned CTA ( iso short label with the destroy one ), open
control panel and fin new CTA `Permanently delete records`
https://github.com/user-attachments/assets/31ee8738-9d61-4dec-9a1f-41bb6785e018
## TODO
- [ ] Gain granularity within tests to assert the action should be
registered only when filtering by deleted
## Conclusion
Closes https://github.com/twentyhq/core-team-issues/issues/110
The global record filter refactor will derive everything at runtime from
objectMetadataItemsState, thus removing the need for a filter definition
concept.
Here we don't yet remove available filter definition usage but we
replace the available filter definitions states, we now derive the same
value from objectMetadataItemsState.
This will allow us to progressively remove the usage of the concept of
filter definition, at the end it will then be easy to just remove from
the codebase because nothing will use it anymore.
This PR implements a first real use case, now currentRecordFilters
component state acts as the global record filter reference.
It is set by the view initially and can be reset to view filters state
at any point.
This new state is also modified by two new upsertRecordFilter /
removeRecordFilter hooks that will be drop-in replacement of the actual
upsertCombinedViewFilter and removeCombinediewFilter hooks.
This PR implements the logic to manipulate record filters but only reads
it to make the table find many request, all other features are still
relying on the old view filter implementation.
Advanced filters are ignored because they are hidden and because this
effort is made precisely to allow the completion of the advanced filters
feature.
Closes#9260
- Refactored multiple record actions and no selection record actions to
use config file
- Simplified actions registration logic
- Updated tests
## Context
Refactored all single record actions so they can be defined by a config
file.
This refactoring is made with the idea that later the actions will be
stored in the database, so we needed a way to serialize them.
For each object we can define a config file, if an object has no config
file, it falls back to the default config.
I introduced action hooks, which return:
- `shouldBeRegistered`: `boolean` Whether the action should be
registered.
- `onClick`: `() => void` The code that will be executed when we click
on an action
- `ConfirmationModal`?: `React.ReactNode` (optional) The confirmation
modal which will be displayed on click
This PR also closes#8973
## Next steps
- Refactor multiple records actions
- Refactor no selection actions
- Add tests
TODO:
- [ ] It should not be possible to add tasks, notes, files, etc.
Fix for https://github.com/twentyhq/twenty/issues/7172
Note for reviewer:
- With these changes, `deletedAt` is now always included in
`recordGqlFields`.
--- Edit from Charles --
In this PR:
1) As mentionned by @pau-not-paul, we are adding deletedAt to our
recordGqlFields for board and table
2) I'm removing cellReadOnly logic, it is now fully computed using
useIsFieldValueReadonly like it's done in other places, there is no need
to maintain two read only systems
3) refactoring useIsFieldValueReadonly to take all the business logics
into one place together. It's now a function of the 5 factors (isRemote,
isDeleted, objectName, fieldName, etc...). Later it's likely to get back
to a function of Pick<FieldMetadata>, Pick<ObjectMetadata>,
record.isDeleted but we are not there yet
Note: as all cells are listening to the record (for isDeleted), updating
a field will trigger a re-rendering of the row which is not an issue
right now. It will be if we introduce bulk edition. In this case we
would need to use a selector on fields on top of record store
---------
Co-authored-by: Charles Bochet <charles@twenty.com>
Closes#7499
- Modifies context store states to be component states
- Introduces the concept of `mainContextStore` which will dictate the
available actions inside the command K
- Adds contextual actions inside the right drawer
- Creates a new type of modal variant
Fix all the broken CIs :p
This includes an ongoing effort to simplify test maintenance by having 1
unique source of truth about metadata and data mocks (that will later be
generated from a unique source of seeds: dev = demo = test)
Regressions:
- Unit line coverage: 60 > 55
- Storybook Pages branch coverage: 40 > 35
We will need to write tests to increase those coverage
- RelationFieldDisplay perf: 0.2ms to 0.22ms > We might have a
regression here
- Removed perf story about RawJSON > We will need to re-add it
## Setup
This PR can be tested only if some feature flags have specific values:
- `IsWorkflowEnabled` equals `true`
- `IsQueryRunnerTwentyORMEnabled` equals `false`
These feature flags weren't committed to don't break other branches.
## What this PR brings
- Display buttons to activate and deactivate a workflow version and a
button to discard the current draft version. I also scaffolded a "Test"
button, which doesn't do anything for now.
- Wired the activate, deactivate and discard draft buttons to the
backend.
- Made it possible to "edit" active and deactivated versions by
automatically creating a new draft version when the user tries to edit
the version.
- Hide the "Discard Draft", button if the current version is not a draft
or is the first version ever created.
- On the backend, don't consider discarded drafts when checking if a new
draft version can be created.
- On the backend, disallow deleting the first created workflow version.
Otherwise, we will end up with a blank canvas in the front end, and it
will be impossible to recover from it.
- On the backend, disallow running deactivation steps if the workflow
version is not currently active. Previously, we were throwing, which is
unnecessary as it's a valid case.
## Spotted bugs that we must dive into
### Duplicate workflow versions in Apollo cache
https://github.com/user-attachments/assets/7cfffd06-11e0-417a-8da0-f9a5f43b84e2
---------
Co-authored-by: Charles Bochet <charles@twenty.com>
## Query depth deprecation
I'm deprecating depth parameter in our graphql query / cache tooling.
They were obsolete since we introduce the possibility to provide
RecordGqlFields
## Refactor combinedFindManyRecordHook
The hook can now take an array of operationSignatures
## Fix tasks issues
Fix optimistic rendering issue. Note that we still haven't handle
optimisticEffect on creation properly
* Use new ObjectRecordConnection
* Use new records without connection in GraphQLView
* Added playwright for storybook tests
* Fixed lint
* Fixed test and tsc
* Fixed storybook tests
* wip tests
* Added useMapConnectionToRecords unit test
---------
Co-authored-by: Charles Bochet <charles@twenty.com>
Charles Bochet