Commit Graph

371 Commits

Author SHA1 Message Date
Etienne 61b9d82950 create groupBy resolver + mock groupBy service (#14530)
To test :

```
query : 
query companiesGroupBy($groupBy: CompanyGroupByInput){
  companiesGroupBy(groupBy: $groupBy){
    groupByDimensionValues
    countEmptyId
    sumAnnualRecurringRevenueAmountMicros
    countEmptyAddress
    percentageEmptySearchVector
  }
}

input : 
{"groupBy": {"name": true, "employees": true, "domainName": {"primaryLinkUrl": true}}}

response : 
{
  "data": {
    "companiesGroupBy": [
      {
        "groupByDimensionValues": [
          "Text 1",
          "twenty.com",
          "1"
        ],
        "countEmptyId": 39,
        "sumAnnualRecurringRevenueAmountMicros": 20,
        "countEmptyAddress": 56,
        "percentageEmptySearchVector": 88
      },
      {
        "groupByDimensionValues": [
          "Long long long long and long text 2",
          "github.com/twentyhq/twenty",
          "2"
        ],
        "countEmptyId": 20,
        "sumAnnualRecurringRevenueAmountMicros": 56,
        "countEmptyAddress": 88,
        "percentageEmptySearchVector": 2
      },
      {
        "groupByDimensionValues": [
          "Text 3",
          "react.dev",
          "3"
        ],
        "countEmptyId": 56,
        "sumAnnualRecurringRevenueAmountMicros": 88,
        "countEmptyAddress": 2,
        "percentageEmptySearchVector": 39
      }
    ]
  }
}
```
closes https://github.com/twentyhq/core-team-issues/issues/1439
2025-09-17 10:23:34 +02:00
Abdullah. 2a7a83de81 feat(search): Add unaccent support for accent-insensitive search. (#14464)
##  Add accent-insensitive search functionality

### 🎯 Overview
Implements accent-insensitive search across all searchable fields in
Twenty CRM.
Users can now search for "jose" to find "José", "muller" to find
"Müller", "cafe" to find "café", etc.

### 🔍 Problem
Twenty's search functionality was accent-sensitive, requiring users to
type exact accented characters to find records.
This created a poor user experience, especially for international names
and content.

### 💡 Solution
Added PostgreSQL `unaccent` extension with a custom immutable wrapper
function to enable accent-insensitive full-text search across all
searchable field types.

### 📋 Changes Made
**Modified Files:**  
- `packages/twenty-server/scripts/setup-db.ts`  
-
`packages/twenty-server/src/engine/api/graphql/graphql-query-runner/utils/compute-where-condition-parts.ts`
-
`packages/twenty-server/src/engine/workspace-manager/workspace-sync-metadata/utils/get-ts-vector-column-expression.util.ts`

### 🗄️ Database Setup (`setup-db.ts`)
```sql
-- Added unaccent extension
CREATE EXTENSION IF NOT EXISTS "unaccent";

-- Created immutable wrapper function
CREATE OR REPLACE FUNCTION unaccent_immutable(text) RETURNS text AS $$
  SELECT public.unaccent($1)
$$ LANGUAGE sql IMMUTABLE;
```

### 🔍 Search Vector Generation
(`get-ts-vector-column-expression.util.ts`)
Applied `public.unaccent_immutable()` to all searchable field types:  
- TEXT fields (job titles, names, etc.)  
- FULL_NAME fields (first/last names)  
- EMAILS fields (both email address and domain)  
- ADDRESS fields  
- LINKS fields  
- RICH_TEXT and RICH_TEXT_V2 fields

### 🔎 Query Processing (`compute-where-condition-parts.ts`)
Enhanced search queries to use `public.unaccent_immutable()` for both:  
- Full-text search (`@@` operator with `to_tsquery`)  
- Pattern matching (`ILIKE` operator)

### 🧠 Technical Rationale: Why the Wrapper Function?
**The Challenge:**  
PostgreSQL's built-in `unaccent()` is marked as **STABLE**, but
`GENERATED ALWAYS AS` expressions (used for search vector columns)
require **IMMUTABLE** functions.

**The Solution:**  
Created an IMMUTABLE wrapper function that calls the underlying
`unaccent()` function:
-  Satisfies PostgreSQL's immutability requirements for generated
columns
-  Maintains the exact same functionality as the original `unaccent()`
-  Uses fully qualified `public.unaccent_immutable()` to ensure
function resolution from workspace schemas

**Alternative Approaches Considered:**  
-  Modifying `search_path`: would affect workspace isolation  
-  Computing unaccent at query time: would hurt performance  
-  Using triggers: would complicate data consistency

### 🎯 Impact
For **Person** records, accent-insensitive search now works on:  
- Name (first/last name): `"jose garcia"` finds `"José García"`  
- Email: `"jose@cafe.com"` finds `"josé@café.com"`  
- Job Title: `"manager"` finds `"Managér"` or `"Gerente de Café"`

Applies to all searchable standard objects:  
- Companies, People, Opportunities, Notes, Tasks, etc.  
- Any custom fields of searchable types (TEXT, EMAILS, etc.)

###  Testing
- Database reset completes successfully  
- Workspace seeding works without errors  
- Search vectors generate with unaccent functionality  
- All searchable field types properly handle accented characters

---------

Co-authored-by: Félix Malfait <felix.malfait@gmail.com>
2025-09-17 09:01:35 +02:00
Etienne 06605b2f36 simplify gql schema gen (#14426)
closes https://github.com/twentyhq/core-team-issues/issues/1358
2025-09-15 14:01:53 +02:00
Charles Bochet 1e97ad48c0 Fix on view filters (#14462)
Fixes https://github.com/twentyhq/twenty/issues/14058
2025-09-12 22:45:26 +02:00
Marie 5900a171f1 Enable filtering out of expected 400 errors (#14371)
We decided to send 4xx errors to sentry from the FE, except for
ForbiddenErrors.
In some cases, we also need not to send to sentry some 4xx errors, like
user input error, because the error stemming from the user input is
unpredictable (ex: duplicate entry for a record with unicity
constraints) and thus acceptable.

Let's add a isExpected extension on UserInputErrors to introduce a
specific behaviour for them (do not send to sentry).
2025-09-09 19:30:56 +02:00
nitin 4ba1bd24f0 remove IS_API_KEY_ROLES_ENABLED feature flag (#14366) 2025-09-09 19:29:29 +02:00
Raphaël Bosi 3d2d14eb71 Pre hook for dashboard creation + updates on the entity (#14354)
Closes https://github.com/twentyhq/core-team-issues/issues/1417

- The pre-hook creates a page-layout and links it to the dashboard
- Added `position`, `createdBy`, `attachments`, `searchVector`,
`favorites` to the dashboard entity
- Updated the view seed
- Updated the page layout services to work within a transaction
2025-09-09 12:55:22 +00:00
Etienne 719be52b53 Upsert - fixes (#14358)
- add more explicit error message on upsert conflicts
- increase MAX_RECORDS_QUERY constant
- add soft deleted records when returning upserted records




fixes https://github.com/twentyhq/private-issues/issues/300
2025-09-09 08:44:07 +00:00
neo773 ad944e1d2c merge records composite type (#14005)
/closes #13989

---------

Co-authored-by: Charles Bochet <charles@twenty.com>
2025-09-08 17:50:26 +02:00
Paul Rastoin 9e81618773 Refactor morph field name and morph data loader (#14299)
# Introduction
Storing morph relation field names directly in database, using morphId
to aggregate them
Removing dynamic morph field metadata computation in schemas and data
loader

Will add integration tests on morph data loader entry

closes https://github.com/twentyhq/core-team-issues/issues/1425 
closes https://github.com/twentyhq/core-team-issues/issues/1424 
closes https://github.com/twentyhq/core-team-issues/issues/1423
2025-09-08 17:20:18 +02:00
martmull 1801b5086a Remove subscriptions job (#14250)
Remove subscriptions publish job

---------

Co-authored-by: Charles Bochet <charlesBochet@users.noreply.github.com>
2025-09-02 14:40:30 +02:00
Weiko 9ad476c6bb Fix dataloader service not provided in core graphql config (#14248)
## Context
Views and some other objects "leak" in the core API, this should be
fixed but in the meantime they fail if they use dataloaders because it's
not provided in the core config

```
  [
    TypeError: Cannot read properties of undefined (reading 'objectMetadataLoader')
  ]
  ```
in view.resolver using dataloaders to compute the name field

How to test: query views from /graphql instead of /metadata

Note: we could have a dedicated dataloader service for each gql server in the future
2025-09-02 13:17:59 +02:00
Etienne 04d4c07b69 fix - use createMany service in createOne one (#14208)
closes : https://github.com/twentyhq/twenty/issues/14072
2025-09-02 11:40:43 +02:00
neo773 a7a79090cb fix: QueryRunner overrideDataByFieldMetadata sub fields not being stringified (#14187)
Adding `secondaryLinks` to company is broken, this fixes it

Regression can be traced back to this commit
https://github.com/twentyhq/twenty/pull/10912/files#diff-8a992c6bff80abc8c9075d585d293f6aa98c246df097efb4f904b82dc3ff8114R210-R214



https://github.com/user-attachments/assets/463cee62-8ee3-45dc-9912-fd8fb5244579
2025-09-02 10:05:42 +02:00
Paul Rastoin 7df9094939 RelationFieldMetadataType to MorphOrRelationFieldMetadataType and so on (#14160) 2025-08-29 15:25:12 +00:00
martmull 71d97c5398 14064 extensibility add coretriggereventlistener table (#14138)
This Pr continues the extensibility journey
- adds a `core.databaseEventTrigger` table
- add a oneToMany relation between `core.serverlessFunction` and
`core.databaseEventTrigger`
- add a job `CallDatabaseEventTriggerJobsJob` triggered by
`EntityEventsToDbListener` triggering `serverlessFunction` based on the
`core.databaseEventTrigger.settings.eventName`
- add a new `trigger-queue` to carry this job
- renamed `DatabaseEventTriggerListener` into
`WorkflowDatabaseEventTriggerListener`
2025-08-29 08:11:35 +02:00
Weiko 533d7fe49a Deprecate legacy core datasource token (#14096) 2025-08-27 20:09:54 +02:00
Raphaël Bosi 30c42345f0 Core views frontend (#13932)
Parallel code path to read and write core views when
IS_CORE_VIEW_ENABLED.

Migrated view key to an enum.

---------

Co-authored-by: Charles Bochet <charles@twenty.com>
Co-authored-by: Lucas Bordeau <bordeau.lucas@gmail.com>
2025-08-20 19:04:58 +02:00
Guillim d63ede5aca morphfix (#13978)
quick fix
2025-08-19 18:16:41 +02:00
Etienne f6d0af5fb8 Move duplicate key error handling in ORM (#13893)
Error available in REST API
<img width="1360" height="653" alt="Screenshot 2025-08-13 at 11 48 19"
src="https://github.com/user-attachments/assets/a055859d-ce54-4ff6-9aba-fac48e03a98d"
/>

`curl http://localhost:3000/rest/people \
  --request POST \
  --header 'Content-Type: application/json' \
  --data '{
  "emails": {
    "primaryEmail": "test@test.com"
  }
}'`

closes https://github.com/twentyhq/twenty/issues/13567
2025-08-14 16:38:04 +00:00
Guillim 3f66e81a9f Missing the joincolumnname parsing (#13898)
in graphql-selecvted-fields-parser, we forgot to treat the
joincolumnname case, on top of the relation itself.
2025-08-13 16:49:22 +02:00
Charles Bochet ef42dace30 Fix destroy many resolver broken on custom objects (#13858)
Fixes https://github.com/twentyhq/twenty/issues/13804
2025-08-12 16:04:21 +02:00
Félix Malfait 3c0f3fd2ae Run yarn dedupe and upgrade TS (#13853)
Upgrading TS to the latest version, and cleaning up packages with yarn
dedupe
2025-08-12 15:18:59 +02:00
Guillim 63c8bb9530 morph backend CORE api (#13517)
# Morph Relation needs some work on the core API

Especially on the Morph types

<img width="682" height="394" alt="Screenshot 2025-07-31 at 12 19 40"
src="https://github.com/user-attachments/assets/8b185ae3-3fe7-4886-97b1-ec143bb1b0f4"
/>


## Focus Areas:
- GraphQL schema generation
- Query parsing and field selection
- Morph relation field support
- Input type definition preparation
2025-08-12 11:47:02 +02:00
Charles Bochet b14063fe06 Re-add prettier (#13812)
Re add prettier in eslint.configs
2025-08-11 14:10:04 +02:00
Félix Malfait 464a480043 Continue ESLINT9 Migration (#13795)
Might already fix #13793
2025-08-10 23:25:58 +02:00
Félix Malfait 5fae14377a Eslint migration 4 (#13773)
A new attempt to migrate!
2025-08-08 16:21:57 +02:00
Félix Malfait 0236f79ccf Fix assert unreachable, remove unused variables (#13753)
Small code cleanup
2025-08-08 09:04:45 +02:00
Félix Malfait 8b4b9ef8da Change type import rule (#13751)
Forcing "type" to be explicit, works best will rollup on the frontend to
exclude depdendencies
2025-08-08 01:27:05 +02:00
Félix Malfait b44b46109c Update try/catch syntax (#13732)
Update try/catch syntax to prepare for eslint migration
2025-08-07 22:19:43 +02:00
Etienne 58d25ebd83 CreateMany optim - .save -> .updateMany + position (#13704)
fixes : https://github.com/twentyhq/core-team-issues/issues/1312

---------

Co-authored-by: Charles Bochet <charles@twenty.com>
2025-08-07 00:52:28 +02:00
Etienne 723f1016a6 Timestamp column migration in all workspace schema tables (#13679)
fixes https://github.com/twentyhq/private-issues/issues/288

Test with : 
`npx nx run twenty-server:command
upgrade:1-3:update-timestamp-column-type-in-workspace-schema`
2025-08-06 22:12:52 +02:00
Marie 9957057d1f [permissions] Adapt field permissions to connect + createMany (#13655)
In this PR we adapt field permission to two things

1. Connect: the recent insertion of the connect feature introduces the
possibility to have "connect" objects in typeORM's expressionMap's
valueSet, where we previously only had column names (if i followed
correctly). For instance for a person object that a N - 1 relationship
to company, person will have both companyId and company as possible
valueSet keys for the upsert. We need to reflect that in our
`getColumnNameToFieldMetadataIdMap` util that returns a map containing
every possible value we could encounter in valueSet. In an attempt to
tie this to the schema generation where this is introduced, I created
the shallow util extractGraphQLRelationFieldNames (probably ill-named -
im willing to update the name if @etiennejouan has a better idea?) to
remind us that these two are linked.

2. CreateMany: When calling query builders methods directly on custom
object (like we do in graphql-create-many-resolver), we need to be
careful to call them with a selection of readable fields. We had a
debate on whether we should compute this selection containing all
readable fields by default under the hood when no selected fields are
indicated. I am still not 100% convinced as I think it should remain the
caller's responsibility, but this case reminds us that it could easily
be forgotten by developers - although it is all the more the case as we
don't have seeds yet that help us realize that ([PR on the
way](https://github.com/twentyhq/twenty/pull/13646)).
2025-08-06 08:31:52 +00:00
nitin 676cf838af [CleanUp] Post api keys and webhooks migration cleanup (#13576) 2025-08-05 12:43:09 +05:30
Félix Malfait 7383b0935c Remove unused files (#13612)
Remove a few unused files
2025-08-04 22:35:36 +02:00
Etienne 3c485ccb37 Upsert in CreateMany - fixes (#13598)
fixes https://github.com/twentyhq/twenty/issues/13553
and https://github.com/twentyhq/core-team-issues/issues/1300
2025-08-04 19:06:40 +02:00
martmull b12fad533d Fix workflowRun real time updates (#13605)
as title
2025-08-04 15:28:22 +00:00
Charles Bochet dc3c194f3e Improve messaging sync performances (#13549)
This PR aims to improve performances as we are making A LOT of queries
after having added events emission to the ORM layer. While investigating
issues, I've come across multiple problems I will describe below

## Add index on workspace.activationStatus as we are querying it a lot

As per title

## Add logs on core datasource destroy

It seems that we have postgres connection pool destruction in
production. I cannot reproduce locally but I suspect the Query Timeouts
to be the root cause. I'm fixing most of the Query Timeouts cause in
this PR but I'm adding the logs so we have more information in case in
keeps happening in production.

## GraphQL query runner createMany

It was using a for loop on each record. This is as issue has we will
emit an event separately for each record => we should always try to
batch events.
Replacing by a save on all records. Note that this is not perfect as we
should avoid using save (bad performances), and use insert + updateMany
instead. As I have follow up discussions regarding permissions, I
haven't replaced it by insert and updateMany yet. Using save instead of
a for loop makes the problem less worrying.

## Introduce updateMany in ORM @Weiko @ijreilly FYI

.save(manyRecords) is bad as it's querying the data for no good reason
(and doing a select for each record...).
We already have .insert(), i'm introducing .updateMany()

I think our ORM layer should be simplified a lot but I'm not starting
the refacto yet

## Fixing ORM @Weiko @ijreilly  FYI
- Fixing events emission in delete function
- Fixing events emission in insert function
- make sure everything is batched

## Events performance @Weiko @ijreilly FYI

Do not emit timelineActivity db events as this does not seem useful and
is quite heavy

## Messaging and Calendar performance @bosiraphael FYI

Rework many functions to make sure they are batched. This does not touch
the driver layer and I have heavily tested it. (multiple account, with
multiple channels, common thread, common messages, etc...)

## Workflow Trigger relation Fetch performance @martmull FYI

Improved performance by batching
2025-08-03 23:56:44 +02:00
nitin 4cd2a87833 Enable roles on api keys (#13334) 2025-08-02 23:54:10 +02:00
Antoine Moreaux 23353e31e6 feat(domain-manager): refactor custom domain validation and improve c… (#13388) 2025-08-01 09:01:27 +02:00
Marie b41502a4b8 [permissions] Update permission check layer (#13485)
Fixes https://github.com/twentyhq/core-team-issues/issues/1262

In this PR we add the update permission check layer by 
- for the graphql api: extracting columns to update from the
expressionMap
- for rest api: .save() is used so we need to add the permission layer
to .save directly. We also take advantage of this PR to filter out
non-readable fields from save response (other save returns the whole
entity) - this was planned in
https://github.com/twentyhq/core-team-issues/issues/1216

The current solution does not work with rest api depth 2 queries, but
this seem to already not work on main (for timeout reasons though, so
different). I offer to create a ticket to fix it altogether later.
2025-07-31 18:37:01 +02:00
neo773 06e0814ce8 feat: merge records (#13436)
Co-authored-by: Félix Malfait <felix.malfait@gmail.com>
2025-07-31 10:06:36 +00:00
neo773 66c484fe11 fix: add userFriendlyMessage to handleDuplicateKeyError (#13513)
/closes #13512
2025-07-31 09:02:08 +02:00
Etienne 2a82a33fff remove gql connect input for morph field - temporary (#13475)
related to https://github.com/twentyhq/core-team-issues/issues/1278
2025-07-28 18:43:27 +02:00
Marie 16ba72edab [permissions] Adapt FE to non-readable fields (#13440)
Fixes https://github.com/twentyhq/core-team-issues/issues/1261 and
https://github.com/twentyhq/core-team-issues/issues/1153

In this PR we
- remove fields user do not have read access on from queries (findMany,
findDuplicates, returned fields from create or update etc.)
- remove fields user do not have read access on from views 

This is behind a feature flag
2025-07-28 16:24:08 +00:00
Etienne 13bed8e4d2 fix createMany resolver when updating softdeleted record (#13425)
Context: updateOne/Many works on soft deleted records but the update of
the upsertMany throws error

fixes https://github.com/twentyhq/twenty/issues/13195
2025-07-25 14:35:04 +02:00
Etienne 741924751b Connect - Import Relation (#13419)
re-opened https://github.com/twentyhq/twenty/pull/13213
2025-07-25 09:48:17 +02:00
Etienne 88a6913217 Connect/Disconnect - Add Disconnect logic + Migration to query builders (insert/update) (#13271)
Context : 
Large PR with 600+ test files. Enable connect and disconnect logic in
createMany (upsert true) / updateOne / updateMany resolvers

- Add disconnect logic
- Gather disconnect and connect logic -> called relation nested queries
- Move logic to query builder (insert and update one) with a preparation
step in .set/.values and an execution step in .execute
- Add integration tests

Test : 
- Test API call on updateMany, updateOne, createMany (upsert:true) with
connect/disconnect
2025-07-24 17:04:38 +02:00
Guillim 1ea451c8be Morph Relations : deleteOneField (#13349)
This PR adapts teh deleteOneField to make sure morph relations can be
deleted, either
- from a relation type, 
- or from a morph relation type (which is the most obvious case).

This PR covers 
- the deletion of fieldMetadata 
- and the migrationof workspace schemas, by using the already existing
definition of relation migrations

This PR implements a new test suite: "Delete Object metadata with morph
relation" and completes the other test suites for FieldMetadata and
ObjectMetadata creation/deletion.

Last, we added a nitpick from @paul I forgot on a previous PR on
process-nested-realtion-v2

Fixes https://github.com/twentyhq/core-team-issues/issues/1197
2025-07-24 12:40:14 +00:00
Abdul Rahman e93adde4b8 Decouple Send Email node from workflows (#13322)
- Renamed `WorkflowActionAdapter` to `ToolExecutorWorkflowAction`
- Renamed `settingPermission` table to `permissionFlag` and `setting`
column to `flag`
- Decoupled the send email logic from workflows to tools
- Add new `Tools Permission` section in FE

---------

Co-authored-by: Félix Malfait <felix.malfait@gmail.com>
Co-authored-by: Félix Malfait <felix@twenty.com>
2025-07-24 12:31:33 +02:00