In this PR, I'm solving several issues:
1) We were not checking if the currentWorkspaceMember was owning the
message in the thread. It kind of worked before because the case of
shared threads (with shared threads visibility restriction) was not
happening that often. It seems that the bug has always been there
2) Re-implement orphan messages and threads deletion on messageChannel
deletion. We used to brutally look for all orphans, we disabled it last
week because it was too heavy on db. I've re-implemented it more
carefully and "surgically"
3) Gmail sync was not handling folder synced correctly. It was
leveraging labelIds which it shouldn't do (this is a AND AND parameter)
in full sync
4) Added a command to clean orphan message threads manually if needed.
Usually this is done when you remove a messageChannel, or change
blocklist rules but it can be useful to have it to debug
## Context
To simplify the way we inject our default datasource, I've recently
removed the token injection that was confusion since we only had once
configured on the module level. Now I'm removing TypeORM service which
allows us to instantiate a new Datasource with the same parameters as
the default one, it was redundant and confusing.
- Removed `IMAPMessageLocator` service which relied on `Message-ID`
header lookups.
We now fetch messages directly by UID, which should eliminate most of
the issues we were hitting.
This had a challenge, `messageIdsToFetch` were flattened and UIDs are
only unique within their own folder.
To preserve context without modifying the core message module, I landed
on using composite keys for IMAP message IDs.
Each UID is now prefixed with its folder name, and we parse it back when
needed to recover the folder.
This feels like a reasonable trade-off since it resolves a lot of the
failures we saw with `Message-ID` lookups.
And Syncs are faster now since there's no double scanning.
- Added `QRESYNC` support for servers that provide it, giving us faster
incremental sync.
- Refactored code to better split concerns.
### Compatibility Test
| Server | Status |
|---------------------|------------|
| Gmail | ✅ Tested |
| Dovecot | ✅ Tested |
| Stalwart | ✅ Tested |
| Titan Email | ✅ Tested |
| FastMail | ✅ Tested |
<img width="804" height="418" alt="image"
src="https://github.com/user-attachments/assets/b414a484-2798-4839-b395-9dd5c153effe"
/>
- Added `ImapFindSentMailboxService` to replace utility function it
fixes an edge case where special flag `Sent` folder may have zero
messages, we fall back to regex based approach to find other candidates
- Fixed edge case where some servers may not return text content for
body it parses the HTML in that case
- Some other enhancements with capability based detection
Tested with Stalwart server and Titan email
/closes #13884
In this PR:
- refactor timelineActivity computation to batch it
- make sure to pass the authContext to insert-query-builder in order to
pass user information to timelineActivity
- refactor message-participant /calendar-participant creation to batch
more
- fix favorite on view race condition (FE)
- deprecate PARTIAL_CALENDAR_EVENT_FETCH_LIST syncStage as we will
deprecate partial vs full notion (we will just leverage cursor emptyness
or not)
- introduce calendar / messging SCHEDULED syncStage that will allow
better performance granularity later
- activate quick message import after message list fetch to speed
performance on small message lists
- Removed caching of IMAP client as it was problematic and didn't really
matter because of our pipeline, also added retry logic to client.
- Some code refactoring and cleanup
- Breaking changes:
- Removed messageChannel.syncCursor
- Uses messageFolder.syncCursor instead
- Removed Date based sync cursor in favor of message UID which is much
more reliable and precise
---------
Co-authored-by: Charles Bochet <charles@twenty.com>
In this PR:
- adding a try / catch around all ORM internal methods save, insert,
upsert, findOne, ...
- leveraging this error to prevent messageChannels to get FAILED
- optimizing messaging BATCH_SIZE and THROTTLE threshold according to
local tests
-
<img width="1510" height="851" alt="image"
src="https://github.com/user-attachments/assets/802fd933-caac-4291-9cde-34a1ddf59c06"
/>
This PR aims to improve performances as we are making A LOT of queries
after having added events emission to the ORM layer. While investigating
issues, I've come across multiple problems I will describe below
## Add index on workspace.activationStatus as we are querying it a lot
As per title
## Add logs on core datasource destroy
It seems that we have postgres connection pool destruction in
production. I cannot reproduce locally but I suspect the Query Timeouts
to be the root cause. I'm fixing most of the Query Timeouts cause in
this PR but I'm adding the logs so we have more information in case in
keeps happening in production.
## GraphQL query runner createMany
It was using a for loop on each record. This is as issue has we will
emit an event separately for each record => we should always try to
batch events.
Replacing by a save on all records. Note that this is not perfect as we
should avoid using save (bad performances), and use insert + updateMany
instead. As I have follow up discussions regarding permissions, I
haven't replaced it by insert and updateMany yet. Using save instead of
a for loop makes the problem less worrying.
## Introduce updateMany in ORM @Weiko @ijreilly FYI
.save(manyRecords) is bad as it's querying the data for no good reason
(and doing a select for each record...).
We already have .insert(), i'm introducing .updateMany()
I think our ORM layer should be simplified a lot but I'm not starting
the refacto yet
## Fixing ORM @Weiko @ijreilly FYI
- Fixing events emission in delete function
- Fixing events emission in insert function
- make sure everything is batched
## Events performance @Weiko @ijreilly FYI
Do not emit timelineActivity db events as this does not seem useful and
is quite heavy
## Messaging and Calendar performance @bosiraphael FYI
Rework many functions to make sure they are batched. This does not touch
the driver layer and I have heavily tested it. (multiple account, with
multiple channels, common thread, common messages, etc...)
## Workflow Trigger relation Fetch performance @martmull FYI
Improved performance by batching
- Renamed `WorkflowActionAdapter` to `ToolExecutorWorkflowAction`
- Renamed `settingPermission` table to `permissionFlag` and `setting`
column to `flag`
- Decoupled the send email logic from workflows to tools
- Add new `Tools Permission` section in FE
---------
Co-authored-by: Félix Malfait <felix.malfait@gmail.com>
Co-authored-by: Félix Malfait <felix@twenty.com>
In this PR, behind a feature flag, we add a permission layer check based
on the read permission.
It is done by computing a map of an object's fields, where keys are the
column names and values the fieldMetadata id, making them comparable to
the restricted fields ids list stored in the permission cache.
For mutations (create, update, delete, destroy), we need to check the
read permission on the returned field, as they may differ from the
updated field. The write field permission will be tackled in a different
PR.
## Problem
Previously, the newly created "message list fetch process" left the
message channel in an "ONGOING" status indefinitely.
## Why
Before only "fullSync" messageChannel were concerned by this method.
What happened it sometimes the providers gave empty messageExternalIds
during the initial fetch. Happens for a newly created email account on
gmail or microsoft (not sure which one)
Now that we gather both of sync methods (partial and full) we cannot use
this anly longer. Because after the first sync, if no new messages
arrived in the last 5 minutes, there will be none, so it was consiedered
as emptyMailbox (which is wrong)
## Solution
Removed this logic from the messageChannel since now we will rely on
each messageFolder (messageList) to handle the logic of going for a full
or not sync.
## Question
We might see some bugs in case some newly created email account without
messageList in case the provider does not give a nextSyncCursor at the
messageList level. Not easy to test
## Bonus
We also cleant a useless service method called getCursor that was not
used anywhere.
---------
Co-authored-by: prastoin <paul@twenty.com>
## Bug description
All old messages were deleted after the first partial sync because the
diff between the existing messages and the messages returned from the
fetch was done considering that it was always a full sync (ie, that we
always retrieve the full list of message ids). But in a partial sync,
only the new messages appear in the list.
This bug was introduced by https://github.com/twentyhq/twenty/pull/13302
when trying to merge the logic between the full sync and the partial
sync.
## Fix
The fix is to adapt the behavior to the type of sync, by looking at the
presence of the sync cursor.
---------
Co-authored-by: Guillim <guillim@users.noreply.github.com>
## Context
We would like to start leveraging more messageFolders during messageSync
which would allow users to select folders they want to sync on their
mailbox. MessageFolder is an abstraction that means folder for
Microsoft, labels for Gmail, not supported for IMAP.
## What
1) We do not aim to build a FE now but we would like to take it into a
consideration if they have been modified through API. So **out of
scope**
2) MessageFolders will be synced regularly from Gmail, Microsoft. This
is currently partially done and improving it is **out of scope**
3) Change: we were having two synchronization mechanism so far:
FULL_SYNC (first time, or when no cursor is present) and PARTIAL_SYNC
(when we have a cursor, we can fetch the diff since the last pull). This
Full vs Partial mode was a high level concept. We now think it's an
driver implementation detail (and can be inferred from the existence of
a cursor). Why making this change now: as we are trying to pull several
folders, a folder could need a full sync if it had no cursor, and
another a partial if it has one. It does not make sense anymore to have
this full vs partial difference at MessageChannel level
4) Once we are sure this work, we can start syncing different folders on
Gmail side (the case for the user it to be able to fetch Promotion
label)
## Implementation strategy
1) Re-use PartialMessageList implementation / API and rename it to
MessageList at it's more complete
2) re-use driver level fullMessageList methods and call them
messageListWithoutCursor
3) make sure that these method are folder specific
## Tests
### Gmail
- Fresh fetch (without cursor): OK
- Message import: OK
- Additional fetch (with messageChannel cursor): OK
### Microsoft
- Fresh fetch (without cursor): OK
- Message import: OK
- Additional fetch (with messageChannel cursor): OK
### Imap
- Fresh fetch (without cursor): OK
- Message import: OK
- Additional fetch (with messageChannel cursor): OK
## Context
Add an eventEmitter instance to twenty datasources so we can emit DB
events.
Add input and output formatting to twenty orm (formatData, formatResult)
Those 2 elements simplified existing logic when we interact with the
ORM, input will be formatted by the ORM so we can directly use
field-like structure instead of column-like. The output will be
formatted, for builder queries it will be in `result.generatedMaps`
where `result.raw` preserves the previous column-like structure.
Important change: We now have an authContext that we can pass when we
get a repository, this will be used for the different events emitted in
the ORM. We also removed the caching for repositories as it was not
scaling well and not necessary imho
Note: An upcoming PR should handle the onDelete: cascade behavior where
we send DESTROY events in cascade when there is an onDelete: CASCADE on
the FK.
---------
Co-authored-by: Charles Bochet <charles@twenty.com>
# Context
We had an error saying "Unknown error importing calendar events for
[...]: Access token is undefined or empty. Please provide a valid token.
For more help -
https://github.com/microsoftgraph/msgraph-sdk-javascript/blob/dev/docs/CustomAuthenticationProvider.md
"
Reason is that the access token method for microsoft is a bit different
than the one from google. And in microsoft case, we want to check the
access token in the authProvider in case it fails. Currently it was not
catched, so it broke services above that counted on the accesstoken to
be valid.
That ended in UNKNOWN failure for our calendar event fetch service.
# Solution
This PR should solve the issue since :
1. forcing the method to break if accesstoken renewal fails
2. logs will help to know what kind of errors will be sent in case we
need to tackle this issue again
3. we now throw TEMPORARY error instead of unknown, allowing 3
getClientConfig failure before it is definitive
Why so many changes while it should have been simple :
The root cause is the `authProvider` from
`'@microsoft/microsoft-graph-client'` npm package. It does not throw a
custom error, and we cannot catch it on calling `Client.init`. Errors
only occurs when the client from
```
const client = this.microsoftOAuth2ClientManagerService.getOAuth2Client(refreshtoken)
```
is used, as in `client.api('/messages').get().catch(err => [...])`
So we need to go in every call using the client and catch errors, and
rethrow whenver we need as a newly created message type
`MessageImportDriverExceptionCode.CLIENT_NOT_AVAILABLE`
We discussed 1. and 2. with @bosiraphael already
I added 3. to make our system more robust without waiting for more
failures
# Related
Fixes : https://github.com/twentyhq/twenty/issues/12880
---------
Co-authored-by: Charles Bochet <charles@twenty.com>
### Added IMAP integration
This PR adds support for connecting email accounts via IMAP protocol,
allowing users to sync their emails without OAuth.
#### DB Changes:
- Added customConnectionParams and connectionType fields to
ConnectedAccountWorkspaceEntity
#### UI:
- Added settings pages for creating and editing IMAP connections with
proper validation and connection testing.
- Implemented reconnection flows for handling permission issues.
#### Backend:
- Built ImapConnectionModule with corresponding resolver and service for
managing IMAP connections.
- Created MessagingIMAPDriverModule to handle IMAP client operations,
message fetching/parsing, and error handling.
#### Dependencies:
Integrated `imapflow` and `mailparser` libraries with their type
definitions to handle the IMAP protocol communication.
---------
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
Co-authored-by: Félix Malfait <felix.malfait@gmail.com>
Co-authored-by: Félix Malfait <felix@twenty.com>
This PR improves error handling in `handleDriverException` by adding a
duck-typing check for `MessageImportDriverException` objects. It ensures
that errors are correctly identified and processed even if they are
received as plain objects rather than class instances. This change
prevents missed exception handling and increases the robustness of the
message import process.
This PR fixes this issue from the connected account refresh token
service that is
This PR fixes error handling in `handleDriverException` by ensuring that
errors resembling `MessageImportDriverException` are correctly detected,
even if they are plain objects and not true class instances. This
prevents missed exception handling due to failed `instanceof` checks.
Was introduced by [this
PR](https://github.com/twentyhq/twenty/pull/12233) that did not know all
provider cases that can occur.
Fixes https://github.com/twentyhq/twenty/issues/12589
# What
Fully deprecate old relations because we have one bug tied to it and it
make the codebase complex
# How I've made this PR:
1. remove metadata datasource (we only keep 'core') => this was causing
extra complexity in the refactor + flaky reset
2. merge dev and demo datasets => as I needed to update the tests which
is very painful, I don't want to do it twice
3. remove all code tied to RELATION_METADATA /
relation-metadata.resolver, or anything tied to the old relation system
4. Remove ONE_TO_ONE and MANY_TO_MANY that are not supported
5. fix impacts on the different areas : see functional testing below
# Functional testing
## Functional testing from the front-end:
1. Database Reset ✅
2. Sign In ✅
3. Workspace sign-up ✅
5. Browsing table / kanban / show ✅
6. Assigning a record in a one to many / in a many to one ✅
7. Deleting a record involved in a relation ✅ => broken but not tied to
this PR
8. "Add new" from relation picker ✅ => broken but not tied to this PR
9. Creating a Task / Note, Updating a Task / Note relations, Deleting a
Task / Note (from table, show page, right drawer) ✅ => broken but not
tied to this PR
10. creating a relation from settings (custom / standard x oneToMany /
manyToOne) ✅
11. updating a relation from settings should not be possible ✅
12. deleting a relation from settings (custom / standard x oneToMany /
manyToOne) ✅
13. Make sure timeline activity still work (relation were involved
there), espacially with Task / Note => to be double checked ✅ => Cannot
convert undefined or null to object
14. Workspace deletion / User deletion ✅
15. CSV Import should keep working ✅
16. Permissions: I have tested without permissions V2 as it's still hard
to test v2 work and it's not in prod yet ✅
17. Workflows global test ✅
## From the API:
1. Review open-api documentation (REST) ✅
2. Make sure REST Api are still able to fetch relations ==> won't do, we
have a coupling Get/Update/Create there, this requires refactoring
3. Make sure REST Api is still able to update / remove relation => won't
do same
## Automated tests
1. lint + typescript ✅
2. front unit tests: ✅
3. server unit tests 2 ✅
4. front stories: ✅
5. server integration: ✅
6. chromatic check : expected 0
7. e2e check : expected no more that current failures
## Remove // Todos
1. All are captured by functional tests above, nothing additional to do
## (Un)related regressions
1. Table loading state is not working anymore, we see the empty state
before table content
2. Filtering by Creator Tim Ap return empty results
3. Not possible to add Tasks / Notes / Files from show page
# Result
## New seeds that can be easily extended
<img width="1920" alt="image"
src="https://github.com/user-attachments/assets/d290d130-2a5f-44e6-b419-7e42a89eec4b"
/>
## -5k lines of code
## No more 'metadata' dataSource (we only have 'core)
## No more relationMetadata (I haven't drop the table yet it's not
referenced in the code anymore)
## We are ready to fix the 6 months lag between current API results and
our mocked tests
## No more bug on relation creation / deletion
---------
Co-authored-by: Weiko <corentin@twenty.com>
Co-authored-by: Félix Malfait <felix@twenty.com>
# Improved participant matching with additional emails support
Closes#8991
This PR extends the participant matching system to support additional
emails in addition to primary emails for both calendar events and
messages. Previously, the system only matched participants based on
primary emails, missing matches with secondary email addresses.
- Contact creation now consider both primary and additional emails when
checking for existing contacts
- Calendar and message participant listeners now handle both primary and
additional email changes
- Added tests
## To test this PR:
Check that:
- Primary emails take precedence over additional emails in matching
- Case-insensitive email comparisons work correctly
- A contact is not created if a person already exists with the email as
its additional email
- Event listeners handle both creation and update scenarios
- Matching and unmatching logic works for complex email change scenarios
- When unmatching after a change in a primary or secondary email, events
and messages should be rematched if another person has this email as its
primary or secondary email.
---------
Co-authored-by: guillim <guigloo@msn.com>
I believe that some emails with invalid characters are breaking the sync
process.
this PR attempts to create a "safeParseAddress" function. Hopefully this
will change current behavior of a single email breaking the entire sync
process to the sync process "skipping" an invalid email address and
continuing on.
I opened this because of issues explained in #12336
---------
Co-authored-by: guillim <guigloo@msn.com>
Charles Bochet