In this PR, I'm fixing a bug introduced in recent performance work on
the cache.
Bug context: https://github.com/twentyhq/twenty/issues/12865
Related PR opened by a contributor:
https://github.com/twentyhq/twenty/pull/13003
## Root cause
We cache all objectMetadataItems at graphql level : see
`useCachedMetadata` hook:
- instead of going through the regular resolvers, we direlcty load data
from the cache. However this data must be localized regarding labels and
descriptions
In a precedent refactoring, we introduced the notion of locale in the
cache key. However, the user locale was not properly taken into account
as we did not have the information in this hook.
## Fix
1. **Introduce locale in userWorkspace entity**. The locale is stored on
workspaceMember in each postgres workspaceSchema (workspace_xxx) which
is the alter ego of userWorkspace in postgres core schema. Note that we
can't store it in user as a user can be part of multiple workspaces (the
locale already there must be seen as a default for this user), and we
cannot rely on workspaceMember as we would need to query the
workspaceSchema in the authentication layer which we want to avoid for
performance reasons.
2. During request hydration from token (containing the userWorkspaceId),
we fetch the userWorkspace and store it in the Request (this impact both
AuthContext and Request interface)
3. Leverage userWorkspace.locale in the useCachedMetadata hook
## Additional notes
There is no need to change the way we store and retrieve the
object-metadata-maps object itself which is different from the graphql
layer cache. object-metadadata-maps are not localized
Currently, when a server query or mutation from the front-end fails, the
error message defined server-side is displayed in a snackbar in the
front-end.
These error messages usually contain technical details that don't belong
to the user interface, such as "ObjectMetadataCollection not found" or
"invalid ENUM value for ...".
**BE**
In addition to the original error message that is still needed (for the
request response, debugging, sentry monitoring etc.), we add a
`displayedErrorMessage` that will be used in the snackbars. It's only
relevant to add it for the messages that will reach the FE (ie. not in
jobs or in rest api for instance) and if it can help the user sort out /
fix things (ie. we do add displayedErrorMessage for "Cannot create
multiple draft versions for the same workflow" or "Cannot delete
[field], please update the label identifier field first", but not
"Object metadata does not exist"), even if in practice in the FE users
should not be able to perform an action that will not work (ie should
not be able to save creation of multiple draft versions of the same
workflows).
**FE**
To ease the usage we replaced enqueueSnackBar with enqueueErrorSnackBar
and enqueueSuccessSnackBar with an api that only requires to pass on the
error.
If no displayedErrorMessage is specified then the default error message
is `An error occured.`
In this PR:
## Improve recompute metadata cache performance. We are aiming for
~100ms
Deleting relationMetadata table and FKs pointing on it
Fetching indexMetadata and indexFieldMetadata in a separate query as
typeorm is suboptimizing
## Remove caching lock
As recomputing the metadata cache is lighter, we try to stop preventing
multiple concurrent computations. This also simplifies interfaces
## Introduce self recovery mecanisms to recompute cache automatically if
corrupted
Aka getFreshObjectMetadataMaps
## custom object resolver performance improvement: 1sec to 200ms
Double check queries and indexes used while creating a custom object
Remove the queries to db to use the cached objectMetadataMap
## reduce objectMetadataMaps to 500kb
<img width="222" alt="image"
src="https://github.com/user-attachments/assets/2370dc80-49b6-4b63-8d5e-30c5ebdaa062"
/>
We used to stored 3 fieldMetadataMaps (byId, byName, byJoinColumnName).
While this is great for devXP, this is not great for performances.
Using the same mecanisme as for objectMetadataMap: we only keep byIdMap
and introduce two otherMaps to idByName, idByJoinColumnName to make the
bridge
## Add dataloader on IndexMetadata (aka indexMetadataList in the API)
## Improve field resolver performances too
## Deprecate ClientConfig
We were using a global ValidationPipe in main.ts. This is an issue as
@Controllers should return HttpExecption and @Resolvers should return
GraphqlErrors
Removing the global pipe and creating a ResolverValidationPipe able to
generate GraphqlError. We also need to handle the exception in a filter
to avoid nest to think it's unhandled and make it flow to logs
Next step:
- it would be nice to have both @UsePipes(ResolverValidationPipe) +
@UseFilters(GraphqlValidationExceptionFilter) come together. This should
be possible if we create a @GraphQLResolver annotation
In this PR
1. fix workflow step creation by adding forgotten
`shouldBypassPermissionChecks` in WorkflowVersionStepWorkspaceService
2. clarify the rule for twentyORMGlobalManager: do not add unnecessary
`shouldBypassPermissionChecks` for system objects (there are no
object-records permission checks on system objects, they are dealt with
at resolver level)
# What
Fully deprecate old relations because we have one bug tied to it and it
make the codebase complex
# How I've made this PR:
1. remove metadata datasource (we only keep 'core') => this was causing
extra complexity in the refactor + flaky reset
2. merge dev and demo datasets => as I needed to update the tests which
is very painful, I don't want to do it twice
3. remove all code tied to RELATION_METADATA /
relation-metadata.resolver, or anything tied to the old relation system
4. Remove ONE_TO_ONE and MANY_TO_MANY that are not supported
5. fix impacts on the different areas : see functional testing below
# Functional testing
## Functional testing from the front-end:
1. Database Reset ✅
2. Sign In ✅
3. Workspace sign-up ✅
5. Browsing table / kanban / show ✅
6. Assigning a record in a one to many / in a many to one ✅
7. Deleting a record involved in a relation ✅ => broken but not tied to
this PR
8. "Add new" from relation picker ✅ => broken but not tied to this PR
9. Creating a Task / Note, Updating a Task / Note relations, Deleting a
Task / Note (from table, show page, right drawer) ✅ => broken but not
tied to this PR
10. creating a relation from settings (custom / standard x oneToMany /
manyToOne) ✅
11. updating a relation from settings should not be possible ✅
12. deleting a relation from settings (custom / standard x oneToMany /
manyToOne) ✅
13. Make sure timeline activity still work (relation were involved
there), espacially with Task / Note => to be double checked ✅ => Cannot
convert undefined or null to object
14. Workspace deletion / User deletion ✅
15. CSV Import should keep working ✅
16. Permissions: I have tested without permissions V2 as it's still hard
to test v2 work and it's not in prod yet ✅
17. Workflows global test ✅
## From the API:
1. Review open-api documentation (REST) ✅
2. Make sure REST Api are still able to fetch relations ==> won't do, we
have a coupling Get/Update/Create there, this requires refactoring
3. Make sure REST Api is still able to update / remove relation => won't
do same
## Automated tests
1. lint + typescript ✅
2. front unit tests: ✅
3. server unit tests 2 ✅
4. front stories: ✅
5. server integration: ✅
6. chromatic check : expected 0
7. e2e check : expected no more that current failures
## Remove // Todos
1. All are captured by functional tests above, nothing additional to do
## (Un)related regressions
1. Table loading state is not working anymore, we see the empty state
before table content
2. Filtering by Creator Tim Ap return empty results
3. Not possible to add Tasks / Notes / Files from show page
# Result
## New seeds that can be easily extended
<img width="1920" alt="image"
src="https://github.com/user-attachments/assets/d290d130-2a5f-44e6-b419-7e42a89eec4b"
/>
## -5k lines of code
## No more 'metadata' dataSource (we only have 'core)
## No more relationMetadata (I haven't drop the table yet it's not
referenced in the code anymore)
## We are ready to fix the 6 months lag between current API results and
our mocked tests
## No more bug on relation creation / deletion
---------
Co-authored-by: Weiko <corentin@twenty.com>
Co-authored-by: Félix Malfait <felix@twenty.com>
We must separate the concept of hydratation which happens at the request
level (take the token and pass auth/user context), from the concept of
authorization which happens at the query/endpoint/mutation level.
Previously, hydratation exemption happened at the operation name level
which is not correct because the operation name is meaningless and
optional. Still this gave an impression of security by enforcing a
blacklist. So in this PR we introduce linting rule that aim to achieve a
similar behavior, now every api method has to have a guard. That way if
and endpoint is not protected by AuthUserGuard or AuthWorspaceGuard,
then it has to be stated explicitly next to its code.
---------
Co-authored-by: Charles Bochet <charles@twenty.com>
# Summary
Enhanced the Google OAuth flow to better handle missing permissions and
improved user experience by redirecting to settings/account page.
## Changes
- Added new google-apis-scopes.ts service for better scope management
- Updated Google APIs auth controller for better flow control
- New tests for this logic
## User request
From @bonapara email test and need to better handle user flow during the
connect email flow
Before :
<img width="574" alt="Screenshot 2025-05-28 at 17 58 59"
src="https://github.com/user-attachments/assets/fd54625b-e211-4b2f-b76a-48bcb08b5222"
/>
After :
<img width="1143" alt="Screenshot 2025-05-28 at 16 29 05"
src="https://github.com/user-attachments/assets/8f3d1f2c-9e02-4d25-b949-fe2b20f048f4"
/>
## Reference :
For google specialities, I added this link in the `export const
getGoogleApisOauthScopes` in order to keep that in mind
https://developers.google.com/identity/protocols/oauth2/scopes
Context :
Plan choice [on pricing page on website](https://twenty.com/pricing)
should redirect you the right plan on app /plan-required page (after
sign in), thanks to query parameters and BillingCheckoutSessionState
sync.
With email verification, an other session starts at CTA click in
verification email. Initial BillingCheckoutSessionState is lost and user
can't submit to the plan he choose.
Solution :
Pass a nextPath query parameter in email verification link
To test :
- Modify .env to add IS_BILLING_ENABLED (+ reset db + sync billing) +
IS_EMAIL_VERIFICATION_REQUIRED
- Start test from this page
http://app.localhost:3001/welcome?billingCheckoutSession={%22plan%22:%22ENTERPRISE%22,%22interval%22:%22Year%22,%22requirePaymentMethod%22:true}
- After verification, check you arrive on /plan-required page with
Enterprise plan on a yearly interval (default is Pro/monthly).
closes https://github.com/twentyhq/twenty/issues/12288
# Gmail OAuth authentication flow issues
### TLDR
This error is not an error and therefore should be treated as a simple
redirect with a snackbar.
### More details
Fixing incomplete OAuth token exchange processes and improving error
handling for empty Gmail inboxes.
The changes include modifications to OAuth guards, to ensure that if a
user clicks "cancel" instead of completing the authentication workflow
if fails
## Before:
Redirection from `/settings/accounts` to `app.twenty.com` with an
`UNAUTHORIZED` error
## After :
<img width="948" alt="Screenshot 2025-05-26 at 18 04 37"
src="https://github.com/user-attachments/assets/62c8721e-c2b3-4e3d-ad0b-e4059dfb7a98"
/>
Fixes https://github.com/twentyhq/twenty/issues/11895
---------
Co-authored-by: Charles Bochet <charles@twenty.com>
### Solution
> After discussion with charles & weiko, we chose the long term
solution.
>
> Fix FE to request checkUserExists resolver with lowercased emails
> Add a decorator on User (and AppToken for invitation), to lowercase
email at user (appToken) creation. ⚠️ It works for TypeOrm .save method
only (there is no user email update in codebase, but in future it
could..)
> Add email lowercasing logic in external auth controller
> Fix FE to request sendInvitations resolver with lowercased emails
> Add migration command to lowercase all existing user emails and
invitation emails
> For other BE resolvers, we let them permissive. For example, if you
made a request on CheckUserExists resolver with uppercased email, you
will not found any user. We will not transform input before checking for
existence.
[link to comment
](https://github.com/twentyhq/twenty/pull/12130#discussion_r2098062093)
### Test 🚧
- sign-in and up from main subdomain and workspace sub domain > Google
Auth (lowercased email) ✔️ | Microsoft Auth (uppercased email ✔️ &
lowercased email) | LoginPassword (uppercased email ✔️& lowercased
email✔️)
- invite flow with uppercased and lowercased ✔️
- migration command + sign-in ( former uppercased microsoft email ✔️) /
sign-up ( former uppercased invited email ✔️)
closes https://github.com/twentyhq/private-issues/issues/278, closes
https://github.com/twentyhq/private-issues/issues/275, closes
https://github.com/twentyhq/private-issues/issues/279
Follow-up on https://github.com/twentyhq/twenty/pull/12007
In this PR
- adding a filter on HttpExceptionHandlerService to filter out 4xx
errors from driver handling (as we do for graphQL errors: see
useGraphQLErrorHandler hook - only filteredIssues are sent to`
exceptionHandlerService.captureExceptions()`.)
- grouping together more missing metadata issues
- attempting to use error codes as issues names in sentry to improve UI;
for now it says "Error" all the time
# Introduction
Added a no-explicit-any rule to the twenty-server, not applicable to
tests and integration tests folder
Related to https://github.com/twentyhq/core-team-issues/issues/975
Discussed with Charles
## In case of conflicts
Until this is approved I won't rebased and handle conflict, just need to
drop two latest commits and re run the scripts etc
## Legacy
We decided not to handle the existing lint error occurrences and
programmatically ignored them through a disable next line rule comment
## Open question
We might wanna activate the
[no-explicit-any](https://typescript-eslint.io/rules/no-explicit-any/)
`ignoreRestArgs` for our use case ?
```
ignoreRestArgs?: boolean;
```
---------
Co-authored-by: etiennejouan <jouan.etienne@gmail.com>
## Description
When a calendar channel fails, its status is not reset during the
reconnect step.
In some cases, resetting is necessary—especially when we’ve deployed a
fix and users try to reconnect their accounts after the patch.
## Why reseting to initial state
Our processes are idempotent, so we can safely set the status to null to
restart the flow. This approach covers all cases and avoids edge
conditions caused by inconsistent statuses.
Fixes : https://github.com/twentyhq/twenty/issues/12026
This PR attemps at improving sentry grouping and filtering by
- Using the exceptionCode as the fingerprint when the error is a
customException. For this to work in this PR we are now throwing
customExceptions instead of internalServerError deprived of their code.
They will still be converted to Internal server errors when sent back as
response
- Filtering 4xx issues where it was missing (for emailVerification
because errors were not handled, for invalid captcha and billing errors
because they are httpErrors and not graphqlErrors)
---------
Co-authored-by: Félix Malfait <felix@twenty.com>
In this PR we are
1. cleaning typeORM service by removing connectToDataSource method
2. using workspaceDataSource instead of mainDataSource when possible,
and replacing raw SQL with workspaceRepository methods to use
First and main step of
https://github.com/twentyhq/core-team-issues/issues/747
We are implementing a permission check layer in our custom
WorkspaceEntityManager by overriding all the db-executing methods (this
PR only overrides some as a POC, the rest will be done in the next PR).
Our custom repositories call entity managers under the hood to interact
with the db so this solves the repositories case too.
This is still behind the feature flag IsPermissionsV2Enabled.
In the next PR
- finish overriding all the methods required in WorkspaceEntityManager
- add tests
In this PR:
- this should fix the sync metadata for new relation system
This goes with the recent PR:
https://github.com/twentyhq/twenty/pull/11725
What we want:
- ONE_TO_MANY relations should have no joinColumn and no onDelete
- MANY_TO_ONE should have both
Extracted isWorkEmail check into a variable for reusability and adjusted
subdomain generation to conditionally include email. This enhances code
readability and maintains logic consistency.
Done :
- move metrics and health cache services from health module to metrics
module
- refactor metrics counter from specific method to set up from enum keys
- add OpenTelemetry (Otel) instrumentation for metrics
- set up Otel SDK to send metrics to Otel collector
To do later :
- implement Otel instrumentation for traces + plug Sentry on top
Implemented fallback logic to associate a user with a workspace when
none is found. Introduced new GraphQL types and mutations for roles and
permissions management. Simplified and refactored URL-building logic for
email verification, improving code maintainability and flexibility.
Updated the method to properly fetch and return workspace data with
related approved access domains. This ensures the correct handling of
workspace lookup and fixes potential issues with undefined returns.
Fix#10982
# Introduction
In this PR we've migrated `twenty-shared` from a `vite` app
[libary-mode](https://vite.dev/guide/build#library-mode) to a
[preconstruct](https://preconstruct.tools/) "atomic" application ( in
the future would like to introduce preconstruct to handle of all our
atomic dependencies such as `twenty-emails` `twenty-ui` etc it will be
integrated at the monorepo's root directly, would be to invasive in the
first, starting incremental via `twenty-shared`)
For more information regarding the motivations please refer to nor:
- https://github.com/twentyhq/core-team-issues/issues/587
-
https://github.com/twentyhq/core-team-issues/issues/281#issuecomment-2630949682
close https://github.com/twentyhq/core-team-issues/issues/589
close https://github.com/twentyhq/core-team-issues/issues/590
## How to test
In order to ease the review this PR will ship all the codegen at the
very end, the actual meaning full diff is `+2,411 −114`
In order to migrate existing dependent packages to `twenty-shared` multi
barrel new arch you need to run in local:
```sh
yarn tsx packages/twenty-shared/scripts/migrateFromSingleToMultiBarrelImport.ts && \
npx nx run-many -t lint --fix -p twenty-front twenty-ui twenty-server twenty-emails twenty-shared twenty-zapier
```
Note that `migrateFromSingleToMultiBarrelImport` is idempotent, it's atm
included in the PR but should not be merged. ( such as codegen will be
added before merging this script will be removed )
## Misc
- related opened issue preconstruct
https://github.com/preconstruct/preconstruct/issues/617
## Closed related PR
- https://github.com/twentyhq/twenty/pull/11028
- https://github.com/twentyhq/twenty/pull/10993
- https://github.com/twentyhq/twenty/pull/10960
## Upcoming enhancement: ( in others dedicated PRs )
- 1/ refactor generate barrel to export atomic module instead of `*`
- 2/ generate barrel own package with several files and tests
- 3/ Migration twenty-ui the same way
- 4/ Use `preconstruct` at monorepo global level
## Conclusion
As always any suggestions are welcomed !
In [a previous PR](https://github.com/twentyhq/twenty/pull/11023) I
fixed the issue where users where re-assigned the default role when
signin up using SSO.
The "fix" actually introduced another error, calling
`assignRoleToUserWorkspace` only when a user is signing up to twenty,
forgetting about the case where an existing twenty user signs up to a
different workspace. They should still be assigned the role in that
case.
To fix this and improve clarity, I moved assignRoleToUserWorkspace to
addUserToWorkspace and renamed addUserToWorkspace to
addUserToWorkspaceIfUserNotInWorkspace since this is at each sign in but
does nothing if user is already in the workspace.
I think ideally we should refactor this part to improve readability and
understandability, maybe with different flows for each case: signIn and
signUp, to twenty or to a workspace
When logging using a SSO method, we call signInUp service in which we
were wrongfully assigning a role to the user even if the user is signing
in and not signin up.
This went unnoticed during our QA as a different sign-in method is
called when logging with the credentials.
---------
Co-authored-by: Weiko <corentin@twenty.com>
driver implementation for sending emails with microsoft
---------
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
Closes https://github.com/twentyhq/core-team-issues/issues/469 and
https://github.com/twentyhq/core-team-issues/issues/500
In this PR
1. stop conditioning permission initialization for a workspace to env
variable value. Instead we want to create and assign permissions and
roles in all new workspaces. For now that will be totally silent.
2. temporarily, the default role is set to the admin role for new
workspaces. it will also be the case for existing workspaces through the
backfill command. Member role is still being created though. (when we
will do the final roll-out we will update this so that future workspaces
have the member role as default role. our goal here is not to break any
current behaviour for users, that today have all have the equivalent of
admin rights).
Charles Bochet