## What
In this PR, we are fixing two issues:
- while deleting an object, the views are not properly refreshed leading
to FE bug. This is due to the fact that we were refetching **views
associated to the object** which has been deleted. As views are properly
deleted in the BE, the FE gets no views from this call and cannot
optimistically react. In this case, I'm triggering a complete view
refetch
- messaging error handling had a hole
## Improvements
- Add logs to all gql operations and rest calls to help debug CPU issues
on the backend. These are temporary and should be removed
- Remove nested relations from workflowVersions load (used to add manual
triggers in the side bar). On some workspaces this call result in a
response of 4MB which is heavy on CPU
- Investigated Redis Usage ==> made a few improvements, we are should
still migrate to the new cache service once available
- investigated db calls in messaging / calendar fetch list + workflow
enqueue run cron jobs. Everything seems to be properly batched
- Removed buffer to string conversion on message and we instead pass the
direct Buffer
- Removed redundant flattening of results
- Cached `JSDOM` and `DOMPurify` Instance
- MailParser added `skipTextToHtml` along with some other options, which
reduced parsing time from 50ms to 35ms
---------
Co-authored-by: Félix Malfait <felix.malfait@gmail.com>
Co-authored-by: Félix Malfait <felix@twenty.com>
## Context
While refactoring messaging, I forgot that workspaces that have been
created before july do not have 'CALENDAR_EVENT_LIST_FETCH_PENDING' and
'MESSAGE_LIST_FETCH_PENDING' enum values (as we don't sync-metadata the
enum values, we would need a migrate command that has not been written
yet)
In the current implementation CALENDAR_EVENT_LIST_FETCH_PENDING =
FULL_CALENDAR_EVENT_LIST_FETCH_PENDING, so I'm putting it back as it was
before
## Context
1. We are debugging the experience on messaging and calendar sync. I'm
adding two scripts to relaunch messaging and calendar channels
2. We are seeing errors in Sentry regarding microsoft drivers errors. I
was about to fix them but I'm refactoring / simplifying a bit the logic
first. It will be a betters starting point
In this PR, I'm solving several issues:
1) We were not checking if the currentWorkspaceMember was owning the
message in the thread. It kind of worked before because the case of
shared threads (with shared threads visibility restriction) was not
happening that often. It seems that the bug has always been there
2) Re-implement orphan messages and threads deletion on messageChannel
deletion. We used to brutally look for all orphans, we disabled it last
week because it was too heavy on db. I've re-implemented it more
carefully and "surgically"
3) Gmail sync was not handling folder synced correctly. It was
leveraging labelIds which it shouldn't do (this is a AND AND parameter)
in full sync
4) Added a command to clean orphan message threads manually if needed.
Usually this is done when you remove a messageChannel, or change
blocklist rules but it can be useful to have it to debug
## Context
To simplify the way we inject our default datasource, I've recently
removed the token injection that was confusion since we only had once
configured on the module level. Now I'm removing TypeORM service which
allows us to instantiate a new Datasource with the same parameters as
the default one, it was redundant and confusing.
- Removed `IMAPMessageLocator` service which relied on `Message-ID`
header lookups.
We now fetch messages directly by UID, which should eliminate most of
the issues we were hitting.
This had a challenge, `messageIdsToFetch` were flattened and UIDs are
only unique within their own folder.
To preserve context without modifying the core message module, I landed
on using composite keys for IMAP message IDs.
Each UID is now prefixed with its folder name, and we parse it back when
needed to recover the folder.
This feels like a reasonable trade-off since it resolves a lot of the
failures we saw with `Message-ID` lookups.
And Syncs are faster now since there's no double scanning.
- Added `QRESYNC` support for servers that provide it, giving us faster
incremental sync.
- Refactored code to better split concerns.
### Compatibility Test
| Server | Status |
|---------------------|------------|
| Gmail | ✅ Tested |
| Dovecot | ✅ Tested |
| Stalwart | ✅ Tested |
| Titan Email | ✅ Tested |
| FastMail | ✅ Tested |
<img width="804" height="418" alt="image"
src="https://github.com/user-attachments/assets/b414a484-2798-4839-b395-9dd5c153effe"
/>
- Added `ImapFindSentMailboxService` to replace utility function it
fixes an edge case where special flag `Sent` folder may have zero
messages, we fall back to regex based approach to find other candidates
- Fixed edge case where some servers may not return text content for
body it parses the HTML in that case
- Some other enhancements with capability based detection
Tested with Stalwart server and Titan email
/closes #13884
In this PR:
- refactor timelineActivity computation to batch it
- make sure to pass the authContext to insert-query-builder in order to
pass user information to timelineActivity
- refactor message-participant /calendar-participant creation to batch
more
- fix favorite on view race condition (FE)
- deprecate PARTIAL_CALENDAR_EVENT_FETCH_LIST syncStage as we will
deprecate partial vs full notion (we will just leverage cursor emptyness
or not)
- introduce calendar / messging SCHEDULED syncStage that will allow
better performance granularity later
- activate quick message import after message list fetch to speed
performance on small message lists
- Removed caching of IMAP client as it was problematic and didn't really
matter because of our pipeline, also added retry logic to client.
- Some code refactoring and cleanup
- Breaking changes:
- Removed messageChannel.syncCursor
- Uses messageFolder.syncCursor instead
- Removed Date based sync cursor in favor of message UID which is much
more reliable and precise
---------
Co-authored-by: Charles Bochet <charles@twenty.com>
In this PR:
- adding a try / catch around all ORM internal methods save, insert,
upsert, findOne, ...
- leveraging this error to prevent messageChannels to get FAILED
- optimizing messaging BATCH_SIZE and THROTTLE threshold according to
local tests
-
<img width="1510" height="851" alt="image"
src="https://github.com/user-attachments/assets/802fd933-caac-4291-9cde-34a1ddf59c06"
/>
This PR aims to improve performances as we are making A LOT of queries
after having added events emission to the ORM layer. While investigating
issues, I've come across multiple problems I will describe below
## Add index on workspace.activationStatus as we are querying it a lot
As per title
## Add logs on core datasource destroy
It seems that we have postgres connection pool destruction in
production. I cannot reproduce locally but I suspect the Query Timeouts
to be the root cause. I'm fixing most of the Query Timeouts cause in
this PR but I'm adding the logs so we have more information in case in
keeps happening in production.
## GraphQL query runner createMany
It was using a for loop on each record. This is as issue has we will
emit an event separately for each record => we should always try to
batch events.
Replacing by a save on all records. Note that this is not perfect as we
should avoid using save (bad performances), and use insert + updateMany
instead. As I have follow up discussions regarding permissions, I
haven't replaced it by insert and updateMany yet. Using save instead of
a for loop makes the problem less worrying.
## Introduce updateMany in ORM @Weiko @ijreilly FYI
.save(manyRecords) is bad as it's querying the data for no good reason
(and doing a select for each record...).
We already have .insert(), i'm introducing .updateMany()
I think our ORM layer should be simplified a lot but I'm not starting
the refacto yet
## Fixing ORM @Weiko @ijreilly FYI
- Fixing events emission in delete function
- Fixing events emission in insert function
- make sure everything is batched
## Events performance @Weiko @ijreilly FYI
Do not emit timelineActivity db events as this does not seem useful and
is quite heavy
## Messaging and Calendar performance @bosiraphael FYI
Rework many functions to make sure they are batched. This does not touch
the driver layer and I have heavily tested it. (multiple account, with
multiple channels, common thread, common messages, etc...)
## Workflow Trigger relation Fetch performance @martmull FYI
Improved performance by batching
neo773