Following https://github.com/twentyhq/twenty/pull/14869
Dynamically clearing the cache, as one entry was forgotten
It seems like a permission test deletes a role that's expected to be
existing in following tests, as now cache is getting invalidated tests
are failing
Seems to be related to
https://discord.com/channels/1130383047699738754/1423768505911869460
## Singleton local cache key collision
When set for the first time caches local keys looks like
`workspaceId:undefined", singleton is shared between several cache
instances
If the given key is undefined it will read on other entity cache entry.
Updates of the hello-world application
- remove the old hello-world application
- adds an object "postCard"
- adds a serverlessFunction `create-new-post-card` that calls the twenty
api to create a new postCard record
- add a route trigger /post-card/create?recipient=John
## What
In this PR, we are fixing two issues:
- while deleting an object, the views are not properly refreshed leading
to FE bug. This is due to the fact that we were refetching **views
associated to the object** which has been deleted. As views are properly
deleted in the BE, the FE gets no views from this call and cannot
optimistically react. In this case, I'm triggering a complete view
refetch
- messaging error handling had a hole
When starting the app on a fresh database reset the cache would be
filled with empty flat field metadata maps
Because the reset command hack through the repository directly in order
to create views and stuff
implemented an iso flush as the one existing initially
added it to a workspace deletion tambien
```
{
byId: {],
universalIdById: {}
}
```
## Improvements
- Add logs to all gql operations and rest calls to help debug CPU issues
on the backend. These are temporary and should be removed
- Remove nested relations from workflowVersions load (used to add manual
triggers in the side bar). On some workspaces this call result in a
response of 4MB which is heavy on CPU
- Investigated Redis Usage ==> made a few improvements, we are should
still migrate to the new cache service once available
- investigated db calls in messaging / calendar fetch list + workflow
enqueue run cron jobs. Everything seems to be properly batched
Closes [#1583](https://github.com/twentyhq/core-team-issues/issues/1583)
- Removed `messageId` column from `File` table and its references in
code (unrelated to this PR)
- Updated AI chat to use React Context API with persistent provider in
`CommandMenuContainer`
- Converted all AI chat component states to regular Recoil atoms (no
instance context needed)
---------
Co-authored-by: Félix Malfait <felix@twenty.com>
This PR is the first part of the creation of the Chart editor.
https://github.com/user-attachments/assets/8b0af8ea-be41-4506-84cb-e40f5521cbf0
Done:
- Bar chart settings (except filters)
- Line chart settings (except filters)
In progress:
- Pie chart settings
- Number chart settings
- Gauge chart settings
Left to do:
- Loosen the backend validation to allow the user to save a partial
configuration, validate the graph configuration in the frontend and
display a error friendly message in the graph if the config is not
completed yet
- Implement the filter edition
- Finish the other graph types settings
## Context
Add routes to migration V2
- Resolvers
- Service v2
- Builder
- Validator
- Action runner
Next PR: Add to twenty-cli to sync routes with serverless
## Context
We are experiencing bad performance on Twenty. One of the root cause
hypothesis is that computing `currentUser.currentWorkspace.views` is CPU
consuming. Without views, the GetCurrentUser response is ~1000 lines.
With its ~10000 lines.
As graphql is going through all fields recursively this can be quite
heavy on CPU. We had a similar issues on ObjectMetadataItems 2 years ago
and came with storing the response in redis.
Note: I thought there was also a cache in RAM but this is not the case,
so to invalidate the cache we can just empty redis.
## How
- Extract getting all views from GetCurrentUser and update frontend to
perform both queries
- Add views to cached graphql operations
- invalidate the cache manually on view or related core entities update
/ create / delete / destroy
## Tests
I have tested a lot on v1
# Introduction
Extracting the legacy fields build and dispatch out of the object one to
follow the generic flat entity build, also update caches entries for
object
## Main tasks
- flat field map cache
- flat field builder
- refactored the dispatch matrix to return flat entity maps instead of
flat entity arrays
- orchestrator aggregator
- removing legacy code
- making universal identifier aka standardId of standard field for
custom object deterministically dynamic
- perfs debug logs for v2
## TODO
- [x] Refactor the generic entity builder to be dependency flat maps in
order to main foreign keys list in flat parent
- [x] Refactor the flat object metadata to contain the array of related
fields and avoid costy find object fields
- [ ] Improve the create field handler to handle multiple field at once
- [ ] Refactor the dispatch to embbed the comparison
- [ ] Improve perf by extracting from elements out of existing
- [ ] Fix the labelIdentifierId validators on object before field
creation ( integ tests are in failing mode )
## Debug logs snippet
```ts
[EntityBuilder fieldMetadata] matrix computation: 0.027ms
[EntityBuilder fieldMetadata] creation validation: 0.001ms
[EntityBuilder fieldMetadata] deletion validation: 0.293ms
[EntityBuilder fieldMetadata] update validation: 0.006ms
[EntityBuilder fieldMetadata] entity processing: 0.363ms
[EntityBuilder fieldMetadata] validateAndBuild: 0.455ms
[EntityBuilder index] matrix computation: 0.005ms
[EntityBuilder index] creation validation: 0.001ms
[EntityBuilder index] deletion validation: 0.146ms
[EntityBuilder index] update validation: 0.004ms
[EntityBuilder index] entity processing: 0.199ms
[EntityBuilder index] validateAndBuild: 0.228ms
[Runner] Initial cache retrieval: 0.549ms
[BaseWorkspaceMigrationRunnerActionHandlerService] delete_index executeForWorkspaceSchema: 11.665ms
[BaseWorkspaceMigrationRunnerActionHandlerService] delete_index executeForMetadata: 12.864ms
[BaseWorkspaceMigrationRunnerActionHandlerService] delete_field executeForWorkspaceSchema: 1.476ms
[BaseWorkspaceMigrationRunnerActionHandlerService] delete_field executeForMetadata: 6.816ms
[BaseWorkspaceMigrationRunnerActionHandlerService] delete_field executeForWorkspaceSchema: 0.062ms
[BaseWorkspaceMigrationRunnerActionHandlerService] delete_field executeForMetadata: 0.889ms
[Runner] Transaction execution: 23.434ms
[Runner] Cache invalidation: 316.662ms
[Runner] Total execution: 340.767ms
```
As you can see cache invalidation is way to long, we could replace the
cache by the optimistic in the end
twenty-cli serverless triggers follow up. Fixes:
- eventName don't support wildcard
- universalIdentifier not used to create or update trigger : update does
not work properly (does deletion then creation)
- add a base project in twenty-cli that is copied when creating a new
app
In this PR https://github.com/twentyhq/twenty/pull/14785 we deprecated
viewFilterOperand (from camelCase to capital snakeCase values), but we
had not migrated the existing workflow steps values. Hence they cannot
be executed!
Let's fix that by dealing with both operands, new and deprecated, to
mitigate the issue. Then we will add a command to migrate the values.
Tested :
- server level ok
- workspace level ok
- server level without canImpersonate or without allow impersonation -
fail ok
- workspace level without permission - fail ok
- add serverlessFunction schema in twenty-cli
- add trigger schema in twenty-cli
- update serverless function code save and get
- sync serverless function
Legacy from early v2 implementation, we don't want to store relation ids
in the cache as the new implementation handle that with flat entities
properly already.
This causes some issues because when we return the updated view, it
contains an array of viewField objects with only the id in it so gql
complains because some fields are non-nullable in the output and the
resolver can't resolve viewField directly because viewField is in the
output (not complete though) already so it just ignores it.
Following https://github.com/twentyhq/twenty/pull/14762
In this PR
- moved logic around any field filter formatting to twenty-shared
- added any field filter to the filters applied to groupBy when viewId
is defined
- added integration test
In the BE we have stored filter operand as IS_EMPTY, IS_NOT_EMPTY, etc.
For some reason in the FE we were manipulating IsEmpty, IsNotEmpty, etc.
(maybe because they were used before in Views before they were moved to
core)
So we were converting the operands in the FE from IS to Is
(convertViewFilterOperandFromCore) to read and manipulate viewFilters,
and then back to BE version to send mutations etc., from Is to IS
(convertViewFilterOperandToCore).
The migration is now over, so we can remove and simplify that code.
(In the 1-5:migrate-views-to-core command we still do the migration from
Is format to IS format.)
When all items have been processed, iterator should:
- store last iteration in history
- return the final result saying iteration have been successful. So it
gets stored by the executor
It was not doing the first step, so we were losing the last iteration.
Splitting into separated function + adding tests
Paul Rastoin