This PR aims to improve performances as we are making A LOT of queries
after having added events emission to the ORM layer. While investigating
issues, I've come across multiple problems I will describe below
## Add index on workspace.activationStatus as we are querying it a lot
As per title
## Add logs on core datasource destroy
It seems that we have postgres connection pool destruction in
production. I cannot reproduce locally but I suspect the Query Timeouts
to be the root cause. I'm fixing most of the Query Timeouts cause in
this PR but I'm adding the logs so we have more information in case in
keeps happening in production.
## GraphQL query runner createMany
It was using a for loop on each record. This is as issue has we will
emit an event separately for each record => we should always try to
batch events.
Replacing by a save on all records. Note that this is not perfect as we
should avoid using save (bad performances), and use insert + updateMany
instead. As I have follow up discussions regarding permissions, I
haven't replaced it by insert and updateMany yet. Using save instead of
a for loop makes the problem less worrying.
## Introduce updateMany in ORM @Weiko @ijreilly FYI
.save(manyRecords) is bad as it's querying the data for no good reason
(and doing a select for each record...).
We already have .insert(), i'm introducing .updateMany()
I think our ORM layer should be simplified a lot but I'm not starting
the refacto yet
## Fixing ORM @Weiko @ijreilly FYI
- Fixing events emission in delete function
- Fixing events emission in insert function
- make sure everything is batched
## Events performance @Weiko @ijreilly FYI
Do not emit timelineActivity db events as this does not seem useful and
is quite heavy
## Messaging and Calendar performance @bosiraphael FYI
Rework many functions to make sure they are batched. This does not touch
the driver layer and I have heavily tested it. (multiple account, with
multiple channels, common thread, common messages, etc...)
## Workflow Trigger relation Fetch performance @martmull FYI
Improved performance by batching
# Introduction
Following https://github.com/twentyhq/twenty/pull/13264
> After this PR merge will create a new one removing the type and
replacing it to FieldMetadataEntity.
This is it !
Fixes: https://github.com/twentyhq/twenty/issues/13110
I'm deprecating note.body and task.body to remove confusion between body
and bodyV2
What will be left but should be done later to avoid breaking changes:
- re-add a body field in the graphql API only that points to the same
bodyV2 field in SQL (need to be handled in fields and filter for note
and task)
- (wait some time)
- remove bodyV2 field
In the BE we throw custom errors with precise error codes (e.g.
"LABEL_ALREADY_EXISTS") before catching them in filters and rethrowing
BaseGraphQLErrors (standard errors such as NotFoundError, UserInputError
etc.).
In the FE we were grouping sentries based on the error codes but we were
actually grouping by very broad codes such as "NOT_FOUND" or
"BAD_USER_INPUT", extracted from the BaseGraphQLErrors.
To fix that, we update the BaseGraphQLError constructor api to allow to
pass on the CustomError directly and retrieve from it the original code
and store it in existing property `subCode` that we will use in the FE
to send errors to sentry.
This new api also eases usage of `userFriendlyMessage` that is passed on
to the api response and therefore to the FE when CustomError is passed
on directly to the BaseGraphQLError constructor.
Fixes https://github.com/twentyhq/twenty/issues/12867
Issue:
when you have a variable `toto` which is: `Record<string, MyType>` and
you do toto['xxx'], this will be typed as `MyType` instead of `MyType |
undefined`
Solutions:
- activate `noUncheckedIndexedAccess` check in tsconfig, this is the
preferred solution but will take time to get there (this raises 600+
errors)
- use a Map: cf https://github.com/twentyhq/twenty/pull/13125/files
- set the type to Partial<Record<string, MyType>>. Drawback is that when
you do Object.values(toto), you'll get `Array<MyType | undefined>`.
Hence why we have to filter these behind
<img width="1512" alt="image"
src="https://github.com/user-attachments/assets/d0a0bfed-c441-4e53-84c2-2da98ccbcf50"
/>
Large PR, sorry for that. Don't hesitate to reach me to have full
context (env. 500lines for integration and unit tests)
- Add connect logic in Workspace Entity Manager
- Update QueryDeepPartialEntity type to enable dev to use connect
- Add integration test on createOne / createMany
- Add unit test to cover main utils
- Remove feature flag on connect
closes https://github.com/twentyhq/core-team-issues/issues/1148
closes https://github.com/twentyhq/core-team-issues/issues/1147
Context :
- IndexFieldMetadata was no longer available on 'objects' gql query
([since this PR](https://github.com/twentyhq/twenty/pull/12785)). Then,
unicity checks on import do not work anymore.
Fix :
- Add a dataloader logic in indexFieldMetadata
- Add extra check in unicity hook on import
In this PR:
## Improve recompute metadata cache performance. We are aiming for
~100ms
Deleting relationMetadata table and FKs pointing on it
Fetching indexMetadata and indexFieldMetadata in a separate query as
typeorm is suboptimizing
## Remove caching lock
As recomputing the metadata cache is lighter, we try to stop preventing
multiple concurrent computations. This also simplifies interfaces
## Introduce self recovery mecanisms to recompute cache automatically if
corrupted
Aka getFreshObjectMetadataMaps
## custom object resolver performance improvement: 1sec to 200ms
Double check queries and indexes used while creating a custom object
Remove the queries to db to use the cached objectMetadataMap
## reduce objectMetadataMaps to 500kb
<img width="222" alt="image"
src="https://github.com/user-attachments/assets/2370dc80-49b6-4b63-8d5e-30c5ebdaa062"
/>
We used to stored 3 fieldMetadataMaps (byId, byName, byJoinColumnName).
While this is great for devXP, this is not great for performances.
Using the same mecanisme as for objectMetadataMap: we only keep byIdMap
and introduce two otherMaps to idByName, idByJoinColumnName to make the
bridge
## Add dataloader on IndexMetadata (aka indexMetadataList in the API)
## Improve field resolver performances too
## Deprecate ClientConfig
# Introduction
close https://github.com/twentyhq/twenty/issues/12343
Adding a transform step for any field phone in order to infer country
code and calling code from the number if they're provided
## Edges cases
```ts
RecordTransformerExceptionCode.INVALID_PHONE_NUMBER:
RecordTransformerExceptionCode.INVALID_PHONE_COUNTRY_CODE:
RecordTransformerExceptionCode.CONFLICTING_PHONE_COUNTRY_CODE:
RecordTransformerExceptionCode.CONFLICTING_PHONE_CALLING_CODE:
RecordTransformerExceptionCode.CONFLICTING_PHONE_CALLING_CODE_AND_COUNTRY_CODE:
RecordTransformerExceptionCode.INVALID_PHONE_CALLING_CODE:
RecordTransformerExceptionCode.INVALID_URL:
```
## Coverage
Note: Will handle REST api integration testing pivot and UPDATE
operation later in the afternoon, critical bug appeared that I prefer
handling before improving this PR coverage, also would be too many
updates
Note2: Haven't fuzzed all of the string inputs, would seem overkill for
such a use case, to be debated
```ts
PASS test/integration/metadata/suites/field-metadata/phone/create-one-field-metadata-phone.integration-spec.ts (23.609 s)
Phone field metadata tests suite
✓ It should succeed create primary phone field (1397 ms)
✓ It should succeed create primary phone field with number and other information (930 ms)
✓ It should succeed create primary phone field with full international format and other information (893 ms)
✓ It should succeed create primary phone field with full international and infer other information from it but not the countryCode as its shared (825 ms)
✓ It should succeed create primary phone field with full international and infer other information from it (818 ms)
✓ It should succeed create primary phone field with empty payload (827 ms)
✓ It should succeed create additional phone field with number and other information (894 ms)
✓ It should succeed create additional phone field with full international format and other information (1024 ms)
✓ It should succeed create additional phone field with full international and infer other information from it but not the countryCode as its shared (808 ms)
✓ It should succeed create additional phone field with full international and infer other information from it (751 ms)
✓ It should succeed create additional phone field with empty payload (739 ms)
✓ It should fail to create primary phone field without country or calling code at all (776 ms)
✓ It should fail to create primary phone field with invalid country code (782 ms)
✓ It should fail to create primary phone field with invalid calling code (858 ms)
✓ It should fail to create primary phone field with conflicting country code and calling code (872 ms)
✓ It should fail to create primary phone field with invalid phone number format (1489 ms)
✓ It should fail to create primary phone field with conflicting phone number country code (1425 ms)
✓ It should fail to create primary phone field with conflicting phone number calling code (1553 ms)
✓ It should fail to create primary phone field without country or calling code at all (814 ms)
✓ It should fail to create primary phone field with invalid country code (813 ms)
✓ It should fail to create primary phone field with invalid calling code (742 ms)
✓ It should fail to create primary phone field with conflicting country code and calling code (783 ms)
✓ It should fail to create primary phone field with invalid phone number format (731 ms)
✓ It should fail to create primary phone field with conflicting phone number country code (947 ms)
✓ It should fail to create primary phone field with conflicting phone number calling code (822 ms)
Test Suites: 1 passed, 1 total
Tests: 25 passed, 25 total
Snapshots: 14 passed, 14 total
Time: 23.627 s
```
Fix inconsistent domain URL formats : removing the last / that was
caused by URL method
Standardize URL formatting to ensure consistent links storage and
retrieval of domain URLs across the application. Will improve the
dedpulicates in the links
Note: there is another temporary issue from google that was solved on
the 13th of june https://groups.google.com/g/adwords-api/c/tRSQMRZrJYM
but we consider this out of this scope
Fixes#12621
Closes#12303
### What’s Changed
- Replace auto‐save with explicit Save / Cancel
Webhook forms now use manual “Save” and “Cancel” buttons instead of the
old debounced auto‐save/update.
- Separate “New” and “Detail” routes
Two dedicated paths `/settings/webhooks/new` for creation and
/`settings/webhooks/:webhookId` for editing, making the UX clearer.
- URL hint & normalization
If a user omits the http(s):// scheme, we display a “Will be saved as
https://…” hint and automatically default to HTTPS.
- Centralized validation with Zod
Introduced a `webhookFormSchema` for client‐side URL, operations, and
secret validation.
- Storybook coverage
Added stories for both “New Webhook” and “Webhook Detail”
- Unit tests
Added tests for the new `useWebhookForm` hook
In this PR
- Determine object record permissions on workflows objects (workflow,
workflowVersion, workflowRun) base on settings permissions @Weiko
- Add Workflow permission guards on workflow resolvers @thomtrp . **Any
method within a resolver that has the SettingsPermission Guard is only
callable by a apiKey or a user that has the permission** (so not by
external parties).
- Add checks bypass in workflow services since 1) for actions gated by
settings permissions, the gate should be done at resolver level, so it
will have been done before the call to the service 2) some service
methods may be called by workflowTriggerController which is callable by
external parties without permissions (ex:
workflowCommonWorkspaceService.getWorkflowVersionOrFail). This is
something we may want to change in the future (still to discuss), by
removing the guard at resolver-level and relying on
shouldBypassPermissionChecks at getRepository and made in a way that we
only bypass for external parties.
- Add checks bypass for actions performed by workflows since they should
not be restricted in our current vision
- Add tests
Closes https://github.com/twentyhq/core-team-issues/issues/748
In the frame of the work on permissions we
- remove all raw queries possible to use repositories instead
- forbid usage workspaceDataSource.executeRawQueries()
- restrict usage of workspaceDataSource.query() to force developers to
pass on shouldBypassPermissionChecks to use it.
---------
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
This PR has several objectives:
- Ignore invalid and empty links in the frontend
- Ignore empty links when creating or updating a link field in the
backend
- Throw an error when trying to create or update a link field with an
invalid link
The logic is mostly the same in the frontend and the backend: we take
the initial primaryLink and the secondaryLinks, we discard all the empty
links (with `url === '' || url === null`), and the primaryLink becomes
the first remaining link.
## Frontend
There are three parts in the frontend where we have to remove the empty
links:
- LinksDisplay
- LinksFieldInput
- isFieldValueEmpty; used in RecordInlineCell
## Backend
I put the logic in
`packages/twenty-server/src/engine/core-modules/record-transformer/services/record-input-transformer.service.ts`
as it's used by the REST API, the GraphQL API, and by Create Record and
Update Record actions in the workflows.
Follow-up on https://github.com/twentyhq/twenty/pull/12007
In this PR
- adding a filter on HttpExceptionHandlerService to filter out 4xx
errors from driver handling (as we do for graphQL errors: see
useGraphQLErrorHandler hook - only filteredIssues are sent to`
exceptionHandlerService.captureExceptions()`.)
- grouping together more missing metadata issues
- attempting to use error codes as issues names in sentry to improve UI;
for now it says "Error" all the time
# Introduction
Added a no-explicit-any rule to the twenty-server, not applicable to
tests and integration tests folder
Related to https://github.com/twentyhq/core-team-issues/issues/975
Discussed with Charles
## In case of conflicts
Until this is approved I won't rebased and handle conflict, just need to
drop two latest commits and re run the scripts etc
## Legacy
We decided not to handle the existing lint error occurrences and
programmatically ignored them through a disable next line rule comment
## Open question
We might wanna activate the
[no-explicit-any](https://typescript-eslint.io/rules/no-explicit-any/)
`ignoreRestArgs` for our use case ?
```
ignoreRestArgs?: boolean;
```
---------
Co-authored-by: etiennejouan <jouan.etienne@gmail.com>
This PR attemps at improving sentry grouping and filtering by
- Using the exceptionCode as the fingerprint when the error is a
customException. For this to work in this PR we are now throwing
customExceptions instead of internalServerError deprived of their code.
They will still be converted to Internal server errors when sent back as
response
- Filtering 4xx issues where it was missing (for emailVerification
because errors were not handled, for invalid captcha and billing errors
because they are httpErrors and not graphqlErrors)
---------
Co-authored-by: Félix Malfait <felix@twenty.com>
Several users have complained about not being able to read their emails
anymore.
This is because the find-messages post query hook is expecting
ObjectRecord[] as an input but is actually getting a graphql Connection
Typing was wrong. This PR fixes the typing and make sure the post query
hook always get an ObjectRecord[]
This is a first PR to remove old relation logic
Next steps:
- remove relationMetadata from cache
- remove relationMetadata table content and structure
- refactor relationDefinition to leverage field.settings instead
No need to audit log workflow runs as it's already a form of audit log.
Add more audit log for other objects
Rename MessagingTelemetry to MessagingMonitoring
Merge Analytics and Audit in one (Audit)
---------
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
related to https://github.com/twentyhq/core-team-issues/issues/601
## Done
- add a `onDbEvent` `Subscription` graphql endpoint to listen to
database_event using what we have done with webhooks:
- you can subscribe to any `action` (created, updated, ...) for any
`objectNameSingular` or a specific `recordId`. Parameters are nullable
and treated as wildcards when null.
- returns events with following shape
```typescript
@Field(() => String)
eventId: string;
@Field()
emittedAt: string;
@Field(() => DatabaseEventAction)
action: DatabaseEventAction;
@Field(() => String)
objectNameSingular: string;
@Field(() => GraphQLJSON)
record: ObjectRecord;
@Field(() => [String], { nullable: true })
updatedFields?: string[];
```
- front provide a componentEffect `<ListenRecordUpdatesEffect />` that
listen for an `objectNameSingular`, a `recordId` and a list of
`listenedFields`. It subscribes to record updates and updates its apollo
cached value for specified `listenedFields`
- subscription is protected with credentials
## Result
Here is an application with `workflowRun`
https://github.com/user-attachments/assets/c964d857-3b54-495f-bf14-587ba26c5a8c
---------
Co-authored-by: prastoin <paul@twenty.com>
### Remove unnecessary `await` from `encodeFileToken` calls (now
synchronous) #11611
#### Context
In [PR #11385 – commit
26c17f3](https://github.com/twentyhq/twenty/pull/11385/commits/26c17f3205eb0cf4b93dd37cfec97638596ed263),
`FileService.encodeFileToken()` was updated to be a **synchronous**
method. However, several places in the codebase were still calling it
using `await`.
#### Changes
This PR cleans up those redundant `await` usages to:
- Improve clarity
- Avoid confusion (no longer awaiting a non-Promise)
- Slightly reduce overhead in affected functions
- Removed `await` from calls to `this.fileService.encodeFileToken(...)`
Fixes issue #10606.
This PR makes `RICH_TEXT_V2` field behavior in REST API matche the
current behavior in GraphQL API:
Currently both `markdown` and `blocknote` fields must be included in the
request, one of them can be `null`. The field with a `null` value will
be filled by the converted value of the other field.
In other words, this works:
```
curl http://localhost:3000/rest/notes \
--request POST \
--header 'Content-Type: application/json' \
--header 'Authorization: Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiIyMDIwMjAyMC0xYzI1LTRkMDItYmYyNS02YWVjY2Y3ZWE0MTkiLCJ0eXBlIjoiQVBJX0tFWSIsIndvcmtzcGFjZUlkIjoiMjAyMDIwMjAtMWMyNS00ZDAyLWJmMjUtNmFlY2NmN2VhNDE5IiwiaWF0IjoxNzQxODA1MzQyLCJleHAiOjQ4OTU0MDUzNDEsImp0aSI6ImZlMzU0NTBkLTlhMDMtNGE2ZS04ODVjLTBlNTU3M2Y3YTE0NiJ9.6_g8cwoSE7ZCX1Zzsw44gZIyBdLKNsnDqMOmm1bKik0' \
--data '{
"position": 1,
"title": "a",
"bodyV2": {
"markdown": "test4\n\ntest3\n\n# test1\n",
"blocknote": null
},
"createdBy": {
"source": "EMAIL"
}
}'
```
And this does not work:
```
curl http://localhost:3000/rest/notes \
--request POST \
--header 'Content-Type: application/json' \
--header 'Authorization: Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiIyMDIwMjAyMC0xYzI1LTRkMDItYmYyNS02YWVjY2Y3ZWE0MTkiLCJ0eXBlIjoiQVBJX0tFWSIsIndvcmtzcGFjZUlkIjoiMjAyMDIwMjAtMWMyNS00ZDAyLWJmMjUtNmFlY2NmN2VhNDE5IiwiaWF0IjoxNzQxODA1MzQyLCJleHAiOjQ4OTU0MDUzNDEsImp0aSI6ImZlMzU0NTBkLTlhMDMtNGE2ZS04ODVjLTBlNTU3M2Y3YTE0NiJ9.6_g8cwoSE7ZCX1Zzsw44gZIyBdLKNsnDqMOmm1bKik0' \
--data '{
"position": 1,
"title": "",
"body": "",
"bodyV2": {
"markdown": "test4\n\ntest3\n\n# test1\n"
},
"createdBy": {
"source": "EMAIL"
}
}'
```
---
It would be nice not to require the null value, maybe let's make that a
separate PR?
# Introduction
In this PR we've migrated `twenty-shared` from a `vite` app
[libary-mode](https://vite.dev/guide/build#library-mode) to a
[preconstruct](https://preconstruct.tools/) "atomic" application ( in
the future would like to introduce preconstruct to handle of all our
atomic dependencies such as `twenty-emails` `twenty-ui` etc it will be
integrated at the monorepo's root directly, would be to invasive in the
first, starting incremental via `twenty-shared`)
For more information regarding the motivations please refer to nor:
- https://github.com/twentyhq/core-team-issues/issues/587
-
https://github.com/twentyhq/core-team-issues/issues/281#issuecomment-2630949682
close https://github.com/twentyhq/core-team-issues/issues/589
close https://github.com/twentyhq/core-team-issues/issues/590
## How to test
In order to ease the review this PR will ship all the codegen at the
very end, the actual meaning full diff is `+2,411 −114`
In order to migrate existing dependent packages to `twenty-shared` multi
barrel new arch you need to run in local:
```sh
yarn tsx packages/twenty-shared/scripts/migrateFromSingleToMultiBarrelImport.ts && \
npx nx run-many -t lint --fix -p twenty-front twenty-ui twenty-server twenty-emails twenty-shared twenty-zapier
```
Note that `migrateFromSingleToMultiBarrelImport` is idempotent, it's atm
included in the PR but should not be merged. ( such as codegen will be
added before merging this script will be removed )
## Misc
- related opened issue preconstruct
https://github.com/preconstruct/preconstruct/issues/617
## Closed related PR
- https://github.com/twentyhq/twenty/pull/11028
- https://github.com/twentyhq/twenty/pull/10993
- https://github.com/twentyhq/twenty/pull/10960
## Upcoming enhancement: ( in others dedicated PRs )
- 1/ refactor generate barrel to export atomic module instead of `*`
- 2/ generate barrel own package with several files and tests
- 3/ Migration twenty-ui the same way
- 4/ Use `preconstruct` at monorepo global level
## Conclusion
As always any suggestions are welcomed !
Félix Malfait