Now that all existing and new workspaces have the following syncStage:
- CALENDAR_EVENT_LIST_FETCH_PENDING
- MESSAGE_LIST_FETCH_PENDING
We can fully deprecate the old FULL_CALENDAR_EVENT_LIST_FETCH_PENDING
and PARTIAL_CALENDAR_EVENT_LIST_FETCH_PENDING (full vs partial is now
directly inferred from the presence of a cursor)
In this PR:
- refactor timelineActivity computation to batch it
- make sure to pass the authContext to insert-query-builder in order to
pass user information to timelineActivity
- refactor message-participant /calendar-participant creation to batch
more
- fix favorite on view race condition (FE)
- deprecate PARTIAL_CALENDAR_EVENT_FETCH_LIST syncStage as we will
deprecate partial vs full notion (we will just leverage cursor emptyness
or not)
- introduce calendar / messging SCHEDULED syncStage that will allow
better performance granularity later
- activate quick message import after message list fetch to speed
performance on small message lists
## Context
We would like to start leveraging more messageFolders during messageSync
which would allow users to select folders they want to sync on their
mailbox. MessageFolder is an abstraction that means folder for
Microsoft, labels for Gmail, not supported for IMAP.
## What
1) We do not aim to build a FE now but we would like to take it into a
consideration if they have been modified through API. So **out of
scope**
2) MessageFolders will be synced regularly from Gmail, Microsoft. This
is currently partially done and improving it is **out of scope**
3) Change: we were having two synchronization mechanism so far:
FULL_SYNC (first time, or when no cursor is present) and PARTIAL_SYNC
(when we have a cursor, we can fetch the diff since the last pull). This
Full vs Partial mode was a high level concept. We now think it's an
driver implementation detail (and can be inferred from the existence of
a cursor). Why making this change now: as we are trying to pull several
folders, a folder could need a full sync if it had no cursor, and
another a partial if it has one. It does not make sense anymore to have
this full vs partial difference at MessageChannel level
4) Once we are sure this work, we can start syncing different folders on
Gmail side (the case for the user it to be able to fetch Promotion
label)
## Implementation strategy
1) Re-use PartialMessageList implementation / API and rename it to
MessageList at it's more complete
2) re-use driver level fullMessageList methods and call them
messageListWithoutCursor
3) make sure that these method are folder specific
## Tests
### Gmail
- Fresh fetch (without cursor): OK
- Message import: OK
- Additional fetch (with messageChannel cursor): OK
### Microsoft
- Fresh fetch (without cursor): OK
- Message import: OK
- Additional fetch (with messageChannel cursor): OK
### Imap
- Fresh fetch (without cursor): OK
- Message import: OK
- Additional fetch (with messageChannel cursor): OK
Closes https://github.com/twentyhq/core-team-issues/issues/748
In the frame of the work on permissions we
- remove all raw queries possible to use repositories instead
- forbid usage workspaceDataSource.executeRawQueries()
- restrict usage of workspaceDataSource.query() to force developers to
pass on shouldBypassPermissionChecks to use it.
---------
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
Context
workspaceMemberId is not always present in AuthContext. Solution
implemented here is to fetch workspaceMemberId via userId.
QRQC coming ...
Tested
- FindManyCalendarEvents / FindOneCalendarEvent
- FindManyMessages / FindOneMessage
closes https://github.com/twentyhq/twenty/issues/12027
Several users have complained about not being able to read their emails
anymore.
This is because the find-messages post query hook is expecting
ObjectRecord[] as an input but is actually getting a graphql Connection
Typing was wrong. This PR fixes the typing and make sure the post query
hook always get an ObjectRecord[]
Possiblity to reconnect you account on Failed Unkown errors.
This PR triggers a similar flow than the Failed errors for unsufficient
permission already existing in twenty.
Allows our users to force the synchro again even though they have a
strange error. It's been asked by some customers since we have had a
couple of issues in messaging lately
Fixes https://github.com/twentyhq/twenty/issues/11411
Done :
- move metrics and health cache services from health module to metrics
module
- refactor metrics counter from specific method to set up from enum keys
- add OpenTelemetry (Otel) instrumentation for metrics
- set up Otel SDK to send metrics to Otel collector
To do later :
- implement Otel instrumentation for traces + plug Sentry on top
# Introduction
In this PR we've migrated `twenty-shared` from a `vite` app
[libary-mode](https://vite.dev/guide/build#library-mode) to a
[preconstruct](https://preconstruct.tools/) "atomic" application ( in
the future would like to introduce preconstruct to handle of all our
atomic dependencies such as `twenty-emails` `twenty-ui` etc it will be
integrated at the monorepo's root directly, would be to invasive in the
first, starting incremental via `twenty-shared`)
For more information regarding the motivations please refer to nor:
- https://github.com/twentyhq/core-team-issues/issues/587
-
https://github.com/twentyhq/core-team-issues/issues/281#issuecomment-2630949682
close https://github.com/twentyhq/core-team-issues/issues/589
close https://github.com/twentyhq/core-team-issues/issues/590
## How to test
In order to ease the review this PR will ship all the codegen at the
very end, the actual meaning full diff is `+2,411 −114`
In order to migrate existing dependent packages to `twenty-shared` multi
barrel new arch you need to run in local:
```sh
yarn tsx packages/twenty-shared/scripts/migrateFromSingleToMultiBarrelImport.ts && \
npx nx run-many -t lint --fix -p twenty-front twenty-ui twenty-server twenty-emails twenty-shared twenty-zapier
```
Note that `migrateFromSingleToMultiBarrelImport` is idempotent, it's atm
included in the PR but should not be merged. ( such as codegen will be
added before merging this script will be removed )
## Misc
- related opened issue preconstruct
https://github.com/preconstruct/preconstruct/issues/617
## Closed related PR
- https://github.com/twentyhq/twenty/pull/11028
- https://github.com/twentyhq/twenty/pull/10993
- https://github.com/twentyhq/twenty/pull/10960
## Upcoming enhancement: ( in others dedicated PRs )
- 1/ refactor generate barrel to export atomic module instead of `*`
- 2/ generate barrel own package with several files and tests
- 3/ Migration twenty-ui the same way
- 4/ Use `preconstruct` at monorepo global level
## Conclusion
As always any suggestions are welcomed !
### Context
For calendar and message sync job health monitoring, we used to
increment a counter in redis cache which could lead to concurrency
issue.
### Solution
- Update to a set structure in place of counter + use sAdd redis method
which is atomic
- Each minute another counter was incremented on a new cache key ->
Update to a 15s window
- Remove ONGOING status not needed. We only need status at job end (or
fail).
### Potential improvements
- Check for cache key existence before fetching data to avoid useless
call to redis ?
closes https://github.com/twentyhq/twenty/issues/10070
Context :
We want to implement some counters to monitor server health. First
counters will track : messageChannel sync status during job execution
and invalid captcha.
How :
Counters are stored in cache and grouped by one-minute windows.
Controllers are created for each metric, aggregating counter over a
five-minutes window.
Endpoints are public and will be queried by Prometheus.
closes https://github.com/twentyhq/core-team-issues/issues/55
Fixes#9827
Also uncovered a conflict with `@objectType('Relation')` and
`@objectType('relation)`
I don't want to address it in this PR so I will create a followup issue
when we close this but I think there's a confusion between
Relation/RelationMetadata, it's unclear what is what
---------
Co-authored-by: Antoine Moreaux <moreaux.antoine@gmail.com>
## Context
Following this https://github.com/twentyhq/twenty/issues/4199
This has not been fully implemented, after 5months of dead code I'm
removing the feature for the time being until we re-prioritise the
feature (unlikely during these next 6 months) to keep the codebase a bit
cleaner (no need to maintain dead features)
Feel free to reopen / revert this PR once feature is ready
## Test
locally after importing emails
## Context
ObjectMetadataService became quite large and handles too many
responsibilities. I'm trying to refactor a bit this part in preparation
of a larger work that will combine object-metadata services and
sync-metadata logic
- Created a STANDARD_OBJECT_ICONS that can be reused in relation
creation to refer to a standard object icon.
- Created a STANDARD_OBJECT_FIELD_IDS that can be used with an object
name to get its standard field ids.
- Moved migration, record and relation creations to dedicated services,
refactored to improve performances and readability
- Refactored some validation logic
---------
Co-authored-by: Charles Bochet <charles@twenty.com>
Fix syncedAt no longer been set on message sync.
Fix calendar data model:
- Add `syncedAt` to `CalendarChannelWorkspaceEntity`
- Move `recurringEventExternalId` from `CalendarEventWorkspaceEntity` to
`CalendarChannelEventAssociationWorkspaceEntity` since the id is
relative to one channel
Fix save queries on calendar sync after regression.
- Add message deletion and thread cleaning during full message list
fetch
- Add thread cleaning during partial message list fetch
- Delete provider from cache key
In this PR:
- removing ugprade-0.24 commands as we are releasing 0.30
- introducing cache:flush command
- refactoring upgrade command and sync-metadata command to use the
ActiveWorkspacesCommand so they consistently run on all workspaces or
selected workspaces
Fixes:
- clear localStorage on sign out
- fix missing workspaceMember in verify resolver
- do not throw on datasource already destroyed exception which can
happen with race condition when several resolvers are resolving in
parallel
Closes#6657
- Fix listeners
- Refactor jobs to take array of events
- Fix calendar events and messages deletion
---------
Co-authored-by: Charles Bochet <charles@twenty.com>
## Context
As we grow, the messaging scripts are experiencing performance issues
forcing us to temporarily disable them on the cloud.
While investigating the performance, I have noticed that generating the
entity schema (for twentyORM) in the repository is taking ~500ms locally
on my Mac M2 so likely more on pods. Caching the entitySchema then!
I'm also clarifying naming around schemaVersion and cacheVersions ==>
both are renamed workspaceMetadataVersion and migrated to the workspace
table (the workspaceCacheVersion table is dropped).
Closes#6382
Create SetUserVarsAccountsToReconnectCommand.
This command loops on all workspaces and:
- deletes all user vars with deprecated key `ACCOUNTS_TO_RECONNECT`
- creates a key value pair of type `USER_VAR` with a key of
`ACCOUNTS_TO_RECONNECT_INSUFFICIENT_PERMISSIONS` for all connect
accounts with a message channel or calendar channel with status
`FAILED_INSUFFICIENT_PERMISSIONS`
neo773