Commit Graph

93 Commits

Author SHA1 Message Date
Guillim 3ec72a2bca logging ids to import (#11115) 2025-03-24 09:17:31 +00:00
Paul Rastoin 9ad8287dbc [REFACTOR] twenty-shared multi barrel and CJS/ESM build with preconstruct (#11083)
# Introduction

In this PR we've migrated `twenty-shared` from a `vite` app
[libary-mode](https://vite.dev/guide/build#library-mode) to a
[preconstruct](https://preconstruct.tools/) "atomic" application ( in
the future would like to introduce preconstruct to handle of all our
atomic dependencies such as `twenty-emails` `twenty-ui` etc it will be
integrated at the monorepo's root directly, would be to invasive in the
first, starting incremental via `twenty-shared`)

For more information regarding the motivations please refer to nor:
- https://github.com/twentyhq/core-team-issues/issues/587
-
https://github.com/twentyhq/core-team-issues/issues/281#issuecomment-2630949682

close https://github.com/twentyhq/core-team-issues/issues/589
close https://github.com/twentyhq/core-team-issues/issues/590

## How to test
In order to ease the review this PR will ship all the codegen at the
very end, the actual meaning full diff is `+2,411 −114`
In order to migrate existing dependent packages to `twenty-shared` multi
barrel new arch you need to run in local:
```sh
yarn tsx packages/twenty-shared/scripts/migrateFromSingleToMultiBarrelImport.ts && \
npx nx run-many -t lint --fix -p twenty-front twenty-ui twenty-server twenty-emails twenty-shared twenty-zapier
```
Note that `migrateFromSingleToMultiBarrelImport` is idempotent, it's atm
included in the PR but should not be merged. ( such as codegen will be
added before merging this script will be removed )

## Misc
- related opened issue preconstruct
https://github.com/preconstruct/preconstruct/issues/617

## Closed related PR
- https://github.com/twentyhq/twenty/pull/11028
- https://github.com/twentyhq/twenty/pull/10993
- https://github.com/twentyhq/twenty/pull/10960

## Upcoming enhancement: ( in others dedicated PRs )
- 1/ refactor generate barrel to export atomic module instead of `*`
- 2/ generate barrel own package with several files and tests
- 3/ Migration twenty-ui the same way
- 4/ Use `preconstruct` at monorepo global level

## Conclusion
As always any suggestions are welcomed !
2025-03-22 19:16:06 +01:00
Charles Bochet b49dc57bbf Microsoft mail sync: make sure that message list fetch is using max batch size (#11095)
Fixes https://github.com/twentyhq/core-team-issues/issues/641
2025-03-21 15:20:39 +01:00
Guillim 8b513a7d3b contact creation was recreating a primaryemail du to casing (#11042)
a first step to fix some of the issues i notices in grafana for netzero

in grafana, we log this error :  
```
error: error: duplicate key value violates unique constraint "IDX_UNIQUE_87914cd3ce963115f8cb943e2ac"

query failed: INSERT INTO "workspace_20fjk7nksxxxxxxxxxxx"."person"("cdrOpportunityId", "siteOpportunityId", "xLinkPrimaryLinkLabel", "xLinkPrimaryLinkUrl", "xLinkSecondaryLinks", "linkedinLinkPrimaryLinkLabel", "linkedinLinkPrimaryLinkUrl", "linkedinLinkSecondaryLinks", "emailsPrimaryEmail", "emailsAdditionalEmails", "companyId", "deletedAt", "position", "phonesPrimaryPhoneNumber", "phonesPrimaryPhoneCountryCode", "phonesPrimaryPhoneCallingCode", "phonesAdditionalPhones", "createdAt", "id", "searchVector", "avatarUrl", "city", "jobTitle", "nameFirstName", "nameLastName", "updatedAt", "createdBySource", "createdByWorkspaceMemberId", "createdByName", "createdByContext") VALUES (DEFAULT, DEFAULT, DEFAULT, DEFAULT, DEFAULT, DEFAULT, DEFAULT, DEFAULT, $1, $2, $3, DEFAULT, $4, DEFAULT, DEFAULT, DEFAULT, DEFAULT, DEFAULT, $5, DEFAULT, DEFAULT, DEFAULT, DEFAULT, $6, $7, DEFAULT, $8, $9, $10, $11), (DEFAULT, DEFAULT, DEFAULT, DEFAULT, DEFAULT, DEFAULT, DEFAULT, DEFAULT, $12, $13, $14, DEFAULT, $15, DEFAULT, DEFAULT, DEFAULT, DEFAULT, DEFAULT, $16, DEFAULT, DEFAULT, DEFAULT, DEFAULT, $17, $18, DEFAULT, $19, $20, $21, $22), (DEFAULT, DEFAULT, DEFAULT, DEFAULT, DEFAULT, DEFAULT, DEFAULT, DEFAULT, $23, $24, $25, DEFAULT, $26, DEFAULT, DEFAULT, DEFAULT, DEFAULT, DEFAULT, $27, DEFAULT, DEFAULT, DEFAULT, DEFAULT, $28, $29, DEFAULT, $30, $31, $32, $33), (DEFAULT, DEFAULT, DEFAULT, DEFAULT, DEFAULT, DEFAULT, DEFAULT, DEFAULT, $34, $35, $36, DEFAULT, $37, DEFAULT, DEFAULT, DEFAULT, DEFAULT, DEFAULT, $38, DEFAULT, DEFAULT, DEFAULT, DEFAULT, $39, $40, DEFAULT, $41, $42, $43, $44) RETURNING "xLinkPrimaryLinkLabel", "xLinkPrimaryLinkUrl", "xLinkSecondaryLinks", "linkedinLinkPrimaryLinkLabel", "linkedinLinkPrimaryLinkUrl", "linkedinLinkSecondaryLinks", "emailsPrimaryEmail", "emailsAdditionalEmails", "companyId", "deletedAt", "position", "phonesPrimaryPhoneNumber", "phonesPrimaryPhoneCountryCode", "phonesPrimaryPhoneCallingCode", "phonesAdditionalPhones", "createdAt", "id", "searchVector", "avatarUrl", "city", "jobTitle", "nameFirstName", "nameLastName", "updatedAt", "createdBySource", "createdByWorkspaceMemberId", "createdByName", "createdByContext" 
```
The index is related to the People table on the netzero workspace
schema. When I looked at the email trying to be inserted, it contains
captial letters. I looked at the DB and there are existing small caps
emails. That's why my guess it that the failure comes from the contact
creation service.

Intersting fact, it comes from the CALENDAR source, not the email one.
That's why i added a little bit of error logging in case it reproduces
itself
2025-03-19 18:24:35 +01:00
Guillim 4bdcf77028 Bettertyping (#10725)
To get better at catching errors on providers
(It will ease also my coming PR to send microsoft messages using
workflow)
2025-03-07 14:11:56 +01:00
Guillim 21c7d2081d send email for outlook (#10703)
driver implementation for sending emails with microsoft

---------

Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
2025-03-07 10:12:49 +01:00
Guillim 341439092a send email refacto (#10691)
working refacto with min module deps of messaging-import-manager-module
2025-03-06 12:56:40 +00:00
Guillim 55a45c50cc microsoft sync failed (#10381)
This PR is supposed to solve an issue with the syncrhonisation of
messages, specifically with microsoft driver. Microsoft calls don't need
access_Token so refreshing toekns was not implemented.

However, microsoft rely on its client which calls its refresfh_token,
and I might have missed some underlying dependency from microsoft
impelemtation so I setup the access token process to refresh it

Needs a talk before to be merged

Fix : https://github.com/twentyhq/twenty/issues/10367

EDIT:
it was a problem with microsoft making refreshtoken expire (contrarily
to google) which needs to be handled.
2025-03-05 15:22:51 +00:00
Guillim 696c510933 Removing empty handles from MessageWithParticipants (#10625)
fixes https://github.com/twentyhq/twenty/issues/10623
2025-03-04 14:53:24 +01:00
Guillim ad9af65898 internal-messages-backend-and-frontend (#10291)
fixes https://github.com/twentyhq/twenty/issues/10263
fixes https://github.com/twentyhq/twenty/issues/10262 also
2025-02-19 15:02:10 +01:00
Guillim bab529b138 Fixing unhandled exception for microsoft driver (#10223)
Fixing a difficult bug: an unhandled exception for microsoft driver when
one of the many Microsoft BATCH request failed.
2025-02-14 18:50:53 +01:00
Weiko 934a97df51 Fix custom exceptions not propagating error code (#10210)
## Context
In some CustomException exceptions, we were instantiating a code without
initializing it which was overriding the parent code and it was then
lost when retrieving it in filters.
Removing them to make sure we don't reproduce this pattern
2025-02-14 10:27:47 +01:00
Guillim a4806b72c7 folders (#10081)
# Folders

Adding the possibility to synchronize messages form more than one
microsoft folder (think "inbox" or "sent items")

It will keep the current way for gmail.

- step 1 : implement a first version of full message & partial message 
- step 2 : implement retro-compatibility which includes the command to
run the migration to backfill microsoft synccursor from messageChannelt
o messageFolders
2025-02-11 16:19:53 +00:00
martmull 736b845c98 335 workflow implement workflow cron triggers backend (#9988)
[Backend side] Add cron triggers to workflow
Closes https://github.com/twentyhq/core-team-issues/issues/335
2025-02-05 11:02:49 +00:00
Paul Rastoin 7fd89678b7 [CHORE] Avoid isDefined duplicated reference, move it to twenty-shared (#9967)
# Introduction
Avoid having multiple `isDefined` definition across our pacakges
Also avoid importing `isDefined` from `twenty-ui` which exposes a huge
barrel for a such little util function

## In a nutshell
Removed own `isDefined.ts` definition from `twenty-ui` `twenty-front`
and `twenty-server` to move it to `twenty-shared`.
Updated imports for each packages, and added explicit dependencies to
`twenty-shared` if not already in place

Related PR https://github.com/twentyhq/twenty/pull/9941
2025-02-01 12:10:10 +01:00
Guillim a5273732b3 Icon microsoft (#9907)
Implementing the Outlook icon for CreatedBy, only for emails.

Not in this PR original scope : The similar feature for calendar created
records. Since it was straightforward, I added it to the scope of this
PR.

Fix https://github.com/twentyhq/core-team-issues/issues/252
2025-01-30 16:09:42 +00:00
Guillim 72d4619cd4 adding the predicate in the UPSERT function (#9891)
Bug fix : we are adding the predicate in the UPSERT function in order to
fix a postgres constraint
2025-01-28 16:51:37 +01:00
Charles Bochet 6f72f1af33 Fix Messaging sync upsert in messageChannel association (#9880)
We have recently changed an .insert into a .upsert but we forgot to add
the unique index on the postgres schema.
2025-01-28 11:48:09 +01:00
Félix Malfait af8d22ee99 Fix ObjectType casing and conflict between Relation and RelationMetadata (#9849)
Fixes #9827 

Also uncovered a conflict with `@objectType('Relation')` and
`@objectType('relation)`

I don't want to address it in this PR so I will create a followup issue
when we close this but I think there's a confusion between
Relation/RelationMetadata, it's unclear what is what

---------

Co-authored-by: Antoine Moreaux <moreaux.antoine@gmail.com>
2025-01-28 10:06:18 +01:00
Félix Malfait 549c3faf71 Add server translation (#9847)
First proof of concept for server-side translation.

The goal was to translate one metadata item:

<img width="939" alt="Screenshot 2025-01-26 at 08 18 41"
src="https://github.com/user-attachments/assets/e42a3f7f-f5e3-4ee7-9be5-272a2adccb23"
/>
2025-01-27 21:07:49 +01:00
Guillim 98671b6cd2 bug fix urgent (#9851)
upsert messaging
2025-01-27 10:38:18 +01:00
Guillim b2a0eb6620 [Microsoft Outlook] fix + error handling (#9696)
- bettter error handling
- small fix in the amount of items Microsoft graph api accepts
2025-01-17 11:04:49 +01:00
Charles Bochet f545bd1c40 Treat suspended workspace as workspaces that need to be synced (#9669)
In this PR:
- migrate WorkspaceActivationStatus to twenty-shared (and update case to
make FE and BE consistent)
- introduce isWorkspaceActiveOrSuspended in twenty-shared
- refactor the code to use it (when we fetch data on the FE, we want to
keep SUSPENDED workspace working + when we sync workspaces we want it
too)
2025-01-16 15:01:04 +01:00
Guillim f077efd171 Outlook integration (#9631)
Get Partial messages
2025-01-16 09:50:01 +01:00
Guillim fc484bde2d [Outlook integration] part 2 : GetMessages (#9612)
### Introducing

- mock files in order to setup unit test on parsing outlook messages
- special spec files for development purposes : dev.spec files. They are
CI skipped with xdescribe but very useful for iterating on new messages
format
- main functionality : getMessages. We use microsoft default client to
do so, using the $batch endpoint to group calls by 20

### documentation
final touch to add troubleshooting tips
2025-01-15 09:48:57 +01:00
Thomas Trompette 9360a6b1de Clean logs from console.time (#9620)
As title. We do not need those anymore.
2025-01-14 18:32:54 +01:00
Guillim 34ee64a36c [Microsoft integration] getFullMessageList (#9544)
Creation of the GmailGetMessageListService
Implementation of the driver to MS Graph API getFullMessageList
2025-01-13 10:13:37 +01:00
Charles Bochet b46bc84dab Fix mail import gmail driver when message has been deleted (#9546)
We forgot to ignore messages that were removed
2025-01-10 16:52:29 +01:00
Raphaël Bosi de56c01206 8733 refactor gmailhandleerrorservice (#8901)
Closes #8733 
- Refactor `GmailHandleErrorService`
- Add tests and mocks for the errors
2024-12-05 17:57:57 +01:00
Guillim 7e4277fbe4 Better errors handling (#8835)
- [ ] Catch this specific `500` error
- [ ] Make sure catched `500` errors are sent to sentry for the Cloud
version
- [ ] Hide the option to sync email with google in this situation where
the according env var is missing
- [x] Add Worskpace information to all catched errors for better
debugging

fix #8607
2024-12-03 12:16:38 +01:00
Raphaël Bosi 08f163592b Gmail error handling fixes (#8732)
Gmail error handling fixes
2024-11-26 11:22:03 +01:00
martmull 39373b4a28 8643 fix sentry error (#8644)
- fixes missing data in event payload when adding a new workspaceMember
- add strong typing to database event emitters
2024-11-21 16:09:36 +00:00
brendanlaschke f9c076df31 o365 calendar sync (#8044)
Implemented:

* Account Connect
* Calendar sync via delta ids then requesting single events


I think I would split the messaging part into a second pr - that's a
step more complex then the calendar :)

---------

Co-authored-by: bosiraphael <raphael.bosi@gmail.com>
2024-11-07 18:13:22 +01:00
martmull 695991881f 6071 return only updated fields of records in zapier update trigger (#8193)
- move webhook triggers into `entity-events-to-db.listener.ts`
- refactor event management
- add a `@OnDatabaseEvent` decorator to manage database events
- add updatedFields in updated events
- update openApi webhooks docs
- update zapier integration
2024-11-04 17:44:36 +01:00
martmull f138a1cf6e 7417 workflows i can send emails using the email account (#7431)
- update `send-email.workflow-action.ts` so it send email via the google
sdk
- remove useless `workflow-action.email.ts`
- add `send` authorization to google api scopes
- update the front workflow email step form to provide a
`connectedAccountId` from the available connected accounts
- update the permissions of connected accounts: ask users to reconnect
when selecting missing send permission


![image](https://github.com/user-attachments/assets/fe3c329d-fd67-4d0d-8450-099c35933645)
2024-10-08 23:29:09 +02:00
Raphaël Bosi 33d5a57ea2 6694 remove raw queries from connected account (#7101)
Closes #6694

---------

Co-authored-by: Charles Bochet <charles@twenty.com>
2024-10-08 13:31:44 +02:00
Raphaël Bosi 23e7958218 Fix syncStageStartedAt not correctly set (#7436)
Fix syncStageStartedAt not correctly set

---------

Co-authored-by: Charles Bochet <charles@twenty.com>
2024-10-05 00:44:52 +02:00
Weiko 2f223f3294 Fix 'name' column wrongly added in standard objects (#7428)
## Context
Name shouldn't be added to all tables, especially standard objects
because they already have their own labelIdentifierFieldMetadata
specified in the workspace-entity schema. This PR removes this column
from the "base" list of columns to add when creating a new object/table
and moves it to the object-metadata service that is, as of today, only
used for custom objects. Also had to modify the migration-runner to
handle column creation in a table creation migration (this was available
in the migration definition already but was not doing anything)

This also fixes an issue in standard objects that already have a "name"
field defined with a different field type, this is even more important
when the said field is a composite field. For example people already has
a FULL_NAME name field which clashes with the default TEXT name field
meaning it was only creating 1 field metadata for 'name' but 3 columns
were created: `name, nameFirstName, nameLastName`. This inconsistency
with metadata (which is our source of truth everywhere) brought some
issues (lately, converting back typeorm response to gql (including
composition) was broken).
2024-10-04 18:31:19 +02:00
Raphaël Bosi c9c2f32922 7154 deleted event is not emitted when calling destroyone (#7159)
Closes #7154
2024-09-27 15:52:04 +02:00
Raphaël Bosi 6a5f9492d3 7142 make messaging full message list fetch idempotent (#7148)
- Add message deletion and thread cleaning during full message list
fetch
- Add thread cleaning  during partial message list fetch
- Delete provider from cache key
2024-09-19 18:32:25 +02:00
Félix Malfait 3025ac346c Upgrade sentry (#7145)
Upgrave Sentry to v8 and add Sentry Cron monitoring

---------

Co-authored-by: Charles Bochet <charles@twenty.com>
2024-09-19 18:09:24 +02:00
Charles Bochet f54eea0227 Optimize sync, reset, seed commands to flush cache and to use less memory (#7034)
In this PR:
- removing ugprade-0.24 commands as we are releasing 0.30
- introducing cache:flush command
- refactoring upgrade command and sync-metadata command to use the
ActiveWorkspacesCommand so they consistently run on all workspaces or
selected workspaces

Fixes:
- clear localStorage on sign out
- fix missing workspaceMember in verify resolver
- do not throw on datasource already destroyed exception which can
happen with race condition when several resolvers are resolving in
parallel
2024-09-15 12:47:45 +02:00
martmull 3190f4a87b 6658 workflows add a first twenty piece email sender (#6965) 2024-09-12 11:00:25 +02:00
Raphaël Bosi cd66ea74a2 6657 Refactor and fix blocklist (#6803)
Closes #6657
- Fix listeners
- Refactor jobs to take array of events
- Fix calendar events and messages deletion

---------

Co-authored-by: Charles Bochet <charles@twenty.com>
2024-08-31 16:38:47 +02:00
Charles Bochet 8dd772aee2 Fix post merge conflicts on messaging services 2024-08-27 19:30:25 +02:00
Raphaël Bosi e6a55f270b 5617 Create CalendarOngoingStaleCron Job (#6748)
Closes #5617

---------

Co-authored-by: Charles Bochet <charles@twenty.com>
2024-08-27 19:13:15 +02:00
Raphaël Bosi e771793626 6655 remove field direction in message and add it in mcma (#6743)
Closes #6655 
- Remove direction from message
- Add direction do mcma
- Create migration command
- Create upgrade 0.24
2024-08-27 19:11:04 +02:00
Raphaël Bosi 81fa3f0c41 6256 refactor messaging module to remove all provider specific code and put it inside the drivers folders (#6721)
Closes #6256 
Closes #6257 
+ Create custom exceptions

---------

Co-authored-by: Charles Bochet <charles@twenty.com>
2024-08-27 18:14:45 +02:00
Weiko f6fd92adcb [POC] add graphql query runner (#6747)
## Context
The goal is to replace pg_graphql with our own ORM wrapper (TwentyORM).
This PR tries to add some parsing logic to convert graphql requests to
send to the ORM to replace pg_graphql implementation.

---------

Co-authored-by: Charles Bochet <charles@twenty.com>
2024-08-27 17:06:39 +02:00
Raphaël Bosi 1eeeae8564 6686 Add try catch on every cron job, and send exception to exceptionHandler (#6705)
Closes #6686

---------

Co-authored-by: Charles Bochet <charles@twenty.com>
2024-08-22 18:23:05 +02:00