This PR improves the general UX and DX of boards, by modifying the query
effect to only use paged group by queries.
In this PR we implement two more things in the backend for group by
queries :
- Fixed ORDER BY in the PARTITION BY sub-query (this wasn't working
because it was applied in the main query, so it sorted randomly picked
records, which was a correct sort on an incorrect dataset returned by
the sub-query)
- Added offset paging in PARTITION BY
Miscellaneous, various bug fixes and improvements along the way :
- Throttled loading of cards to avoid React freeze
- Handling of drag & drop
- Handling of create / delete / update
- Reworked skeleton (the library slows down a lot with hundreds of
skeleton for a spinning effect that is hardly noticed)
- Fixed refetch of aggregate queries (I included the new group by
aggregates query we use in the existing refetch mechanism)
- Re-trigger queries on filters and sorts changes
- Unselect all record ids when deleting / restoring / detroying
- Fetch only groups that still have records to lighten the group by
query.
# What remains to be done
This is still a naïve fetch more implementation that will work for a few
fetch more rounds, but if you scroll and load say 200 cards per column
on a board, React will re-render all 200 cards of each column each time.
We would probably need to virtualize the board with paged queries as we
did for the table, this could be done after this PR but seems less
urgent.
What's nice is that this new query pattern is well designed for
virtualization also, drawing from our experience with table
virtualization, and adapted to a multi-column request pattern, like a
2:2 matrix of records, for our boards.
So the remaining work would be to design a UI solution for virtualizing
this matrix of records, which could be quite different from our table
virtualization mechanism.
## Context
Deprecating the old objectMetadataMap type in favour of split flat
entities to match with our new caching.
In the long run, trying to achieve:
- Better performance through caching
- Consistent data access patterns across the codebase
- Reduced database queries
Now that everything is based on flat entities, which are cached, we can
finish the refactoring of workspace context cache which should already
improve performances.
Then the last step will be to consume that new cache in the new global
datasource to get rid of the many workspace datasources stored in the
server
Closes https://github.com/twentyhq/core-team-issues/issues/1600.
Two remarks
- This `limit` variable does not reduce postgre's work at it still needs
to scan the whole table. It did not seem possible to me to optimize this
as we cannot foresee which dimensions will be used by the user, and an
optimization could only result from an index on the dimension(s) (e.g.:
group companies by addressCity limit 50 can be optimized if we have an
index on companies.addressCity + we had a default orderBy on
adressCity). But this will still optimize the FE which at the moment
receives all groups and truncates the result.
- I have not done the work on the FE as the addition of limit is a
breaking change, and will break until the workspaces' schema is rebuilt,
so we need to flush the cache. I think this could be acceptable as the
feature is in the lab but I preferred not doing it yet as it would have
no impact since in the BE I added a default limit to 50 groups, and I
expect more FE work will be done to allow the user to choose their own
limit
As title
- adds decorators in twenty-sdk
- update twenty-cli load-manifest to it gets @FieldMetadata infos +
testing
- update twenty-server so it CRUD fields properly, using
universalIdentifier
- Fix UI so we can update managed objects records
- move FieldMetadata items from twenty-server to twenty-shared
This big PR implements table virtualization with an offset paging,
allowing a way more fluid UX.
It is a v1 that should be improved in the future with partial data
loading and optimization of the browser display performance of a row.
But with this PR we have the solid enough technical foundation, both
frontend and backend, to get to a smooth table UX.
Fixes and improvements after first successful round of development
(needed to have main clean) :
- [x] Delete should refresh virtualized portion only and reset all table
- [x] Fix add new : top and bottom
- [x] Table empty shouldn’t show when first loading
- [x] Fix d&d
- [x] Fix sorts
- [x] Fix drag when scrolling after a full virtual page (it throws an
error)
- [x] Si update mais qu’on a un sort ou filter, alors il faut trigger le
refresh
- [x] Reset scroll position between tables
- [x] Reset scroll shadows between tables
- [x] Setup d&n for virtual list :
https://github.com/hello-pangea/dnd/blob/main/docs/patterns/virtual-lists.md
- [x] Full table re-render when entering edit mode
- [x] Clean code and prepare for merge
Fixes https://github.com/twentyhq/core-team-issues/issues/1613 that
contains other bugs to be fixed before merge
---------
Co-authored-by: Charles Bochet <charles@twenty.com>
Closes
https://github.com/twentyhq/core-team-issues/issues/248?issue=twentyhq%7Ccore-team-issues%7C1543
In this PR
- Implementation of groupBy resolver, without pagination, without viewId
parameter
- introduction of new GqlInputTypeDefinitionKind `OrderByWithGroupBy`
which differs from `OrderBy` as it should use aggregations as order by
conditions (e.g.: OrderByWithGroupBy should use avgEmployees and not
employees which does not make sense)
- Minor refacto of all orderBy expressions to use orderBy signature with
`OrderByCondition` which is usable for both groupBy queries and
non-groupBy queries. groupBy queries differ because their orderBy
conditions are based on aggregatedFields (such as avgEmployees) for
which we can't use the same orderBy signature
<img width="728" height="216" alt="Capture d’écran 2025-09-24 à 15 07
14"
src="https://github.com/user-attachments/assets/aa5c0875-cf38-4d4b-be70-f4d64da77a41"
/>
- Minor refacto to add table name in aggregates expressions
`(AVG("employees") -> AVG("company"."employees")` to keep having the
table names in the orderBy expression as we already had. Since we use
common utils for orderBy and aggregates on findMany this change also
impacts findMany queries with aggregates. It also impacts field
permission check where we extract the name of the selected fields to
check permission on, so I also updated that (see
`extractColumnNamesFromAggregateExpression`)
# Morph Relation needs some work on the core API
Especially on the Morph types
<img width="682" height="394" alt="Screenshot 2025-07-31 at 12 19 40"
src="https://github.com/user-attachments/assets/8b185ae3-3fe7-4886-97b1-ec143bb1b0f4"
/>
## Focus Areas:
- GraphQL schema generation
- Query parsing and field selection
- Morph relation field support
- Input type definition preparation
In this PR we adapt field permission to two things
1. Connect: the recent insertion of the connect feature introduces the
possibility to have "connect" objects in typeORM's expressionMap's
valueSet, where we previously only had column names (if i followed
correctly). For instance for a person object that a N - 1 relationship
to company, person will have both companyId and company as possible
valueSet keys for the upsert. We need to reflect that in our
`getColumnNameToFieldMetadataIdMap` util that returns a map containing
every possible value we could encounter in valueSet. In an attempt to
tie this to the schema generation where this is introduced, I created
the shallow util extractGraphQLRelationFieldNames (probably ill-named -
im willing to update the name if @etiennejouan has a better idea?) to
remind us that these two are linked.
2. CreateMany: When calling query builders methods directly on custom
object (like we do in graphql-create-many-resolver), we need to be
careful to call them with a selection of readable fields. We had a
debate on whether we should compute this selection containing all
readable fields by default under the hood when no selected fields are
indicated. I am still not 100% convinced as I think it should remain the
caller's responsibility, but this case reminds us that it could easily
be forgotten by developers - although it is all the more the case as we
don't have seeds yet that help us realize that ([PR on the
way](https://github.com/twentyhq/twenty/pull/13646)).
Fixes https://github.com/twentyhq/core-team-issues/issues/1262
In this PR we add the update permission check layer by
- for the graphql api: extracting columns to update from the
expressionMap
- for rest api: .save() is used so we need to add the permission layer
to .save directly. We also take advantage of this PR to filter out
non-readable fields from save response (other save returns the whole
entity) - this was planned in
https://github.com/twentyhq/core-team-issues/issues/1216
The current solution does not work with rest api depth 2 queries, but
this seem to already not work on main (for timeout reasons though, so
different). I offer to create a ticket to fix it altogether later.
Context :
Large PR with 600+ test files. Enable connect and disconnect logic in
createMany (upsert true) / updateOne / updateMany resolvers
- Add disconnect logic
- Gather disconnect and connect logic -> called relation nested queries
- Move logic to query builder (insert and update one) with a preparation
step in .set/.values and an execution step in .execute
- Add integration tests
Test :
- Test API call on updateMany, updateOne, createMany (upsert:true) with
connect/disconnect
# Introduction
Following https://github.com/twentyhq/twenty/pull/13310
> After this PR merge will create a new one removing the type and
replacing it to ObjectMetadataEntity.
This is it !
# Introduction
Following https://github.com/twentyhq/twenty/pull/13264
> After this PR merge will create a new one removing the type and
replacing it to FieldMetadataEntity.
This is it !
# Introduction
From the moment replaced the FieldMetadataInterface definition to:
```ts
import { FieldMetadataType } from 'twenty-shared/types';
import { FieldMetadataEntity } from 'src/engine/metadata-modules/field-metadata/field-metadata.entity';
export type FieldMetadataInterface<
T extends FieldMetadataType = FieldMetadataType,
> = FieldMetadataEntity<T>;
```
After this PR merge will create a new one removing the type and
replacing it to `FieldMetadataEntity`.
Did not renamed it here to avoid conflicts on naming + type issues fixs
within the same PR
## Field metadata entity RELATION or MORPH
Relations fields cannot be null for those field metadata entity instance
anymore, but are never for the others see
`packages/twenty-server/src/engine/metadata-modules/field-metadata/types/field-metadata-entity-test.type.ts`
( introduced TypeScript tests )
## Concerns
- TS_VECTOR is the most at risk with the `generatedType` and
`asExpression` removal from interface
## What's next
- `FielMetadataInterface` removal and rename ( see introduction )
- Depcrecating `ObjectMetadataInterface`
- Refactor `FieldMetadataEntity` optional fiels to be nullable only
- TO DIG `never` occurences on settings, defaultValue etc
- Some interfaces will be replaced by the `FlatFieldMetadata` when
deprecating the current sync and comparators tools
Large PR, sorry for that. Don't hesitate to reach me to have full
context (env. 500lines for integration and unit tests)
- Add connect logic in Workspace Entity Manager
- Update QueryDeepPartialEntity type to enable dev to use connect
- Add integration test on createOne / createMany
- Add unit test to cover main utils
- Remove feature flag on connect
closes https://github.com/twentyhq/core-team-issues/issues/1148
closes https://github.com/twentyhq/core-team-issues/issues/1147
# Introduction
Added a no-explicit-any rule to the twenty-server, not applicable to
tests and integration tests folder
Related to https://github.com/twentyhq/core-team-issues/issues/975
Discussed with Charles
## In case of conflicts
Until this is approved I won't rebased and handle conflict, just need to
drop two latest commits and re run the scripts etc
## Legacy
We decided not to handle the existing lint error occurrences and
programmatically ignored them through a disable next line rule comment
## Open question
We might wanna activate the
[no-explicit-any](https://typescript-eslint.io/rules/no-explicit-any/)
`ignoreRestArgs` for our use case ?
```
ignoreRestArgs?: boolean;
```
---------
Co-authored-by: etiennejouan <jouan.etienne@gmail.com>
This is a first PR to remove old relation logic
Next steps:
- remove relationMetadata from cache
- remove relationMetadata table content and structure
- refactor relationDefinition to leverage field.settings instead
# Introduction
In this PR we've migrated `twenty-shared` from a `vite` app
[libary-mode](https://vite.dev/guide/build#library-mode) to a
[preconstruct](https://preconstruct.tools/) "atomic" application ( in
the future would like to introduce preconstruct to handle of all our
atomic dependencies such as `twenty-emails` `twenty-ui` etc it will be
integrated at the monorepo's root directly, would be to invasive in the
first, starting incremental via `twenty-shared`)
For more information regarding the motivations please refer to nor:
- https://github.com/twentyhq/core-team-issues/issues/587
-
https://github.com/twentyhq/core-team-issues/issues/281#issuecomment-2630949682
close https://github.com/twentyhq/core-team-issues/issues/589
close https://github.com/twentyhq/core-team-issues/issues/590
## How to test
In order to ease the review this PR will ship all the codegen at the
very end, the actual meaning full diff is `+2,411 −114`
In order to migrate existing dependent packages to `twenty-shared` multi
barrel new arch you need to run in local:
```sh
yarn tsx packages/twenty-shared/scripts/migrateFromSingleToMultiBarrelImport.ts && \
npx nx run-many -t lint --fix -p twenty-front twenty-ui twenty-server twenty-emails twenty-shared twenty-zapier
```
Note that `migrateFromSingleToMultiBarrelImport` is idempotent, it's atm
included in the PR but should not be merged. ( such as codegen will be
added before merging this script will be removed )
## Misc
- related opened issue preconstruct
https://github.com/preconstruct/preconstruct/issues/617
## Closed related PR
- https://github.com/twentyhq/twenty/pull/11028
- https://github.com/twentyhq/twenty/pull/10993
- https://github.com/twentyhq/twenty/pull/10960
## Upcoming enhancement: ( in others dedicated PRs )
- 1/ refactor generate barrel to export atomic module instead of `*`
- 2/ generate barrel own package with several files and tests
- 3/ Migration twenty-ui the same way
- 4/ Use `preconstruct` at monorepo global level
## Conclusion
As always any suggestions are welcomed !
Fixes https://github.com/twentyhq/twenty/issues/8300
## Context
API events were created too late and were already formatted as Gql
responses (including nesting with edges/node/type + formatting that
should not exist in an event payload). This PR moves the emit logic to
the resolver where we actually do the DB query
Note: Also added RESTORED events
First step of #https://github.com/twentyhq/twenty/issues/3298.
Here we update the search endpoint to allow for a filter argument, which
we currently use in the relation pickers to restrict or exclude ids from
search.
In a future PR we will try to simplify the search logic in the FE
Lucas Bordeau