## Introduction
https://github.com/twentyhq/twenty/pull/14363
In this PR we're creating builder and runner for index metadata
Removing its previous integration within object builder
## Next:
- implem index impact on field and object metadata api
- Add integration testing coverage add integration test on index
- Implement uniqueness toggle in field metadata api
- add integration test on uniqueness
related to https://github.com/twentyhq/core-team-issues/issues/1344
## ✨ Add accent-insensitive search functionality
### 🎯 Overview
Implements accent-insensitive search across all searchable fields in
Twenty CRM.
Users can now search for "jose" to find "José", "muller" to find
"Müller", "cafe" to find "café", etc.
### 🔍 Problem
Twenty's search functionality was accent-sensitive, requiring users to
type exact accented characters to find records.
This created a poor user experience, especially for international names
and content.
### 💡 Solution
Added PostgreSQL `unaccent` extension with a custom immutable wrapper
function to enable accent-insensitive full-text search across all
searchable field types.
### 📋 Changes Made
**Modified Files:**
- `packages/twenty-server/scripts/setup-db.ts`
-
`packages/twenty-server/src/engine/api/graphql/graphql-query-runner/utils/compute-where-condition-parts.ts`
-
`packages/twenty-server/src/engine/workspace-manager/workspace-sync-metadata/utils/get-ts-vector-column-expression.util.ts`
### 🗄️ Database Setup (`setup-db.ts`)
```sql
-- Added unaccent extension
CREATE EXTENSION IF NOT EXISTS "unaccent";
-- Created immutable wrapper function
CREATE OR REPLACE FUNCTION unaccent_immutable(text) RETURNS text AS $$
SELECT public.unaccent($1)
$$ LANGUAGE sql IMMUTABLE;
```
### 🔍 Search Vector Generation
(`get-ts-vector-column-expression.util.ts`)
Applied `public.unaccent_immutable()` to all searchable field types:
- TEXT fields (job titles, names, etc.)
- FULL_NAME fields (first/last names)
- EMAILS fields (both email address and domain)
- ADDRESS fields
- LINKS fields
- RICH_TEXT and RICH_TEXT_V2 fields
### 🔎 Query Processing (`compute-where-condition-parts.ts`)
Enhanced search queries to use `public.unaccent_immutable()` for both:
- Full-text search (`@@` operator with `to_tsquery`)
- Pattern matching (`ILIKE` operator)
### 🧠 Technical Rationale: Why the Wrapper Function?
**The Challenge:**
PostgreSQL's built-in `unaccent()` is marked as **STABLE**, but
`GENERATED ALWAYS AS` expressions (used for search vector columns)
require **IMMUTABLE** functions.
**The Solution:**
Created an IMMUTABLE wrapper function that calls the underlying
`unaccent()` function:
- ✅ Satisfies PostgreSQL's immutability requirements for generated
columns
- ✅ Maintains the exact same functionality as the original `unaccent()`
- ✅ Uses fully qualified `public.unaccent_immutable()` to ensure
function resolution from workspace schemas
**Alternative Approaches Considered:**
- ❌ Modifying `search_path`: would affect workspace isolation
- ❌ Computing unaccent at query time: would hurt performance
- ❌ Using triggers: would complicate data consistency
### 🎯 Impact
For **Person** records, accent-insensitive search now works on:
- Name (first/last name): `"jose garcia"` finds `"José García"`
- Email: `"jose@cafe.com"` finds `"josé@café.com"`
- Job Title: `"manager"` finds `"Managér"` or `"Gerente de Café"`
Applies to all searchable standard objects:
- Companies, People, Opportunities, Notes, Tasks, etc.
- Any custom fields of searchable types (TEXT, EMAILS, etc.)
### ✅ Testing
- Database reset completes successfully
- Workspace seeding works without errors
- Search vectors generate with unaccent functionality
- All searchable field types properly handle accented characters
---------
Co-authored-by: Félix Malfait <felix.malfait@gmail.com>
Closes https://github.com/twentyhq/core-team-issues/issues/1417
- The pre-hook creates a page-layout and links it to the dashboard
- Added `position`, `createdBy`, `attachments`, `searchVector`,
`favorites` to the dashboard entity
- Updated the view seed
- Updated the page layout services to work within a transaction
resolves#14190
added refreshCoreViews() call after object creation to immediately
update core views state, ensuring new objects appear in the navigation
drawer without requiring a refresh
---------
Co-authored-by: Charles Bochet <charles@twenty.com>
Co-authored-by: prastoin <paul@twenty.com>
Improve workflow enqueue cron : instead of relying on the cache to know
how many workflows we can enqueue, query the DB. Then set the cache and
process the not started workflows.
Also adding a second cron that will look for workflows enqueued one hour
ago or more and put these back in the not started status. This will
allow the first cron to start these again.
resolvetwentyhq/core-team-issues#1255
- Introduced a new field isWeekStartMonday in the workspaceMember table.
- Added a function updateCalendarStartDay to update the
isWeekStartMonday value when the user toggles the corresponding field in
the settings.
The logic works for me, but I couldn’t find a way to create the column
locally in the database for the workspaceMember tables. Please let me
know if this approach looks good and how to properly create dynamic
columns that are not directly handled via migration.
https://github.com/user-attachments/assets/4eebada5-6a96-4a88-8e96-98f1501859e3
---------
Co-authored-by: Félix Malfait <felix.malfait@gmail.com>
Co-authored-by: Félix Malfait <felix@twenty.com>
This PR adds the ability to save an any field filter to a view.
It adds a new `anyFieldFilterValue` on both core view and workspace view
entities.
It also introduces the necessary utils that mimic the logic that manages
the save of record filters and record sorts on views.
## Problem
After migrating webhooks and API keys from workspace to core level, REST
API endpoints were still creating entities in workspace schema
(`workspace_*`) instead of core schema, causing webhooks to not fire.
## Solution
- Added dedicated REST controllers for webhooks (`/rest/webhooks`) and
API keys (`/rest/apiKeys`)
- Updated dynamic controller to block workspace-gated entities from
being processed
- Fixed OpenAPI documentation to exclude these endpoints from playground
- Ensured return formats match GraphQL resolvers exactly
## Testing
✅ All endpoints tested with provided auth token - webhooks and API keys
now correctly stored in `core` schema
# Introduction
Following https://github.com/twentyhq/twenty/pull/13264, this PR
introduces several `fieldMetadataEntity` typing enhancement suggestions.
Mainly any nullable field metadata entity properties are now either
nullable or defined.
Or never if field is dynamically required or not depending on the field
metadata type
This enhance DevX
## Standards
- field enum ( `MULTI_SELECT`, `SELECT`, `RATING` ) will never have
`options` set to `NULL` in db
- field `RELATION` or `MORH_RELATION` won't ever have its relation
fields set to `NULL` in db
- field of any type `settings`, even if possibly defined, can still be
`NULL` in db
- field of any type `defaultValue`, even if possibly defined, can still
be `NULL` in db
It coud be interesting to guard these standards by adding dedicated pg
constraints on each field
## TypesScript type tests
added coverage for each `settings`, `defaultValue`, and `options`
depending on the current `fieldMetadata`
Honestly I don' know if this typescript assertions test file is not
overkill, but regarding metadata staticness it might be very interesting
to have this guard
## Possible improvements
- We could type as `unknown` instead of "all" on `FieldMetadataType`
inferrance
- We still need to deprecate remaining duplicated entities such as
`Index/Field/MetadataInterface` etc not a huge refactor neither urgent
# Introduction
Following https://github.com/twentyhq/twenty/pull/13310
> After this PR merge will create a new one removing the type and
replacing it to ObjectMetadataEntity.
This is it !
# Introduction
Following `FieldMetadataInterface` deprecation in
https://github.com/twentyhq/twenty/pull/13264
As for the previous PR will rename and remove all the file in a
secondary PR to avoid conflicts and over loading this one
## Improvements
Removed optional properties from the `objectMetadataEntity` model and
added utils to retrieve test data
## Notes
By touching to `ObjectMetadataDTO` I would have expected a twenty-front
codegenerated types mutation, but it does not seem to be granular enough
to null/undefined coercion
# Introduction
Following https://github.com/twentyhq/twenty/pull/13264
> After this PR merge will create a new one removing the type and
replacing it to FieldMetadataEntity.
This is it !
# Introduction
From the moment replaced the FieldMetadataInterface definition to:
```ts
import { FieldMetadataType } from 'twenty-shared/types';
import { FieldMetadataEntity } from 'src/engine/metadata-modules/field-metadata/field-metadata.entity';
export type FieldMetadataInterface<
T extends FieldMetadataType = FieldMetadataType,
> = FieldMetadataEntity<T>;
```
After this PR merge will create a new one removing the type and
replacing it to `FieldMetadataEntity`.
Did not renamed it here to avoid conflicts on naming + type issues fixs
within the same PR
## Field metadata entity RELATION or MORPH
Relations fields cannot be null for those field metadata entity instance
anymore, but are never for the others see
`packages/twenty-server/src/engine/metadata-modules/field-metadata/types/field-metadata-entity-test.type.ts`
( introduced TypeScript tests )
## Concerns
- TS_VECTOR is the most at risk with the `generatedType` and
`asExpression` removal from interface
## What's next
- `FielMetadataInterface` removal and rename ( see introduction )
- Depcrecating `ObjectMetadataInterface`
- Refactor `FieldMetadataEntity` optional fiels to be nullable only
- TO DIG `never` occurences on settings, defaultValue etc
- Some interfaces will be replaced by the `FlatFieldMetadata` when
deprecating the current sync and comparators tools
In this PR
- introduction of fieldPermission entity
- addition of upsertFieldPermission in role resolver
- computing of permissions taking fieldPermission into account. In order
to limit what is stored in Redis we only store fields restrictions. For
instance for objectMetadata with id XXX with a restriction on field with
id YYY we store:
`"XXX":{"canRead":true,"canUpdate":false,"canSoftDelete":false,"canDestroy":false,"restrictedFields":{"YYY":{"canRead":false,"canUpdate":null}}}`
---------
Co-authored-by: Charles Bochet <charlesBochet@users.noreply.github.com>
This PR does not produce any functional change
First step of the workflow branch feature
- add gather `workflowRun.output` and `workflowRun.context` into one
column `workflowRun.runContext`
- add a command to fill `runContext` from `output` and `context` in
existing records
- maintain `runContext` up to date during workflow runs
Modifying the data-model can sometimes fail in the middle of your
operation, due to the way we handle both metadata update and schema
migration separately, a field can be created while the associated column
creation failed (same for object/table and such). This is also an issue
because WorkspaceMigrations are then stored as FAILED can never really
recovered by themselves so the schema is broken and we can't update the
models anymore.
This PR adds a executeMigrationFromPendingMigrationsWithinTransaction
method where we can (and must) pass a queryRunner executing a
transaction, which should come from the metadata services so that if
anything during metadata update OR schema update fails, it rolls back
everything (this also mean a workspaceMigration should never stay in a
failed state now).
This also fixes some issues with migration not running in the correct
order due to having the same timestamp and having to do some weird logic
to fix that.
This is a first step and fix before working on a much more reliable
solution in the upcoming weeks where we will refactor the way we
interact with the data model.
---------
Co-authored-by: Charles Bochet <charlesBochet@users.noreply.github.com>
Fixes: #12722
The problem is that there is no TS_VECTOR field in workflow objects.
Thus, I have added this field to three objects: workflow,
workflowVersions, and workflowRuns.
---------
Co-authored-by: Thomas Trompette <thomas.trompette@sfr.fr>
### Added IMAP integration
This PR adds support for connecting email accounts via IMAP protocol,
allowing users to sync their emails without OAuth.
#### DB Changes:
- Added customConnectionParams and connectionType fields to
ConnectedAccountWorkspaceEntity
#### UI:
- Added settings pages for creating and editing IMAP connections with
proper validation and connection testing.
- Implemented reconnection flows for handling permission issues.
#### Backend:
- Built ImapConnectionModule with corresponding resolver and service for
managing IMAP connections.
- Created MessagingIMAPDriverModule to handle IMAP client operations,
message fetching/parsing, and error handling.
#### Dependencies:
Integrated `imapflow` and `mailparser` libraries with their type
definitions to handle the IMAP protocol communication.
---------
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
Co-authored-by: Félix Malfait <felix.malfait@gmail.com>
Co-authored-by: Félix Malfait <felix@twenty.com>
In this PR:
## Improve recompute metadata cache performance. We are aiming for
~100ms
Deleting relationMetadata table and FKs pointing on it
Fetching indexMetadata and indexFieldMetadata in a separate query as
typeorm is suboptimizing
## Remove caching lock
As recomputing the metadata cache is lighter, we try to stop preventing
multiple concurrent computations. This also simplifies interfaces
## Introduce self recovery mecanisms to recompute cache automatically if
corrupted
Aka getFreshObjectMetadataMaps
## custom object resolver performance improvement: 1sec to 200ms
Double check queries and indexes used while creating a custom object
Remove the queries to db to use the cached objectMetadataMap
## reduce objectMetadataMaps to 500kb
<img width="222" alt="image"
src="https://github.com/user-attachments/assets/2370dc80-49b6-4b63-8d5e-30c5ebdaa062"
/>
We used to stored 3 fieldMetadataMaps (byId, byName, byJoinColumnName).
While this is great for devXP, this is not great for performances.
Using the same mecanisme as for objectMetadataMap: we only keep byIdMap
and introduce two otherMaps to idByName, idByJoinColumnName to make the
bridge
## Add dataloader on IndexMetadata (aka indexMetadataList in the API)
## Improve field resolver performances too
## Deprecate ClientConfig
## Why
After the changes on relations, index on relations were skipped by the
syncmetadata service, so no more migrations were generated for relation
fields.
We wanted to fix this.
## Test
This PR adds unit tests for the `createIndexMigration` utility in the
workspace migration builder. The tests cover:
- Creating index migrations for simple fields (e.g., text fields)
- Creating index migrations for relation fields (ensuring correct column
naming, e.g., `authorId` for the `author` objectmetadataname)
## Excluded
The delete index on relation does not need the column names so i don't
think i needed to work on this method. I might be wrong.
## Checklist
- [x] Added/updated unit tests for index migration creation
- [x] Verified correct handling of simple and relation fields
- [x] Ensured all tests pass
---------
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
Co-authored-by: Charles Bochet <charles@twenty.com>
# What
Fully deprecate old relations because we have one bug tied to it and it
make the codebase complex
# How I've made this PR:
1. remove metadata datasource (we only keep 'core') => this was causing
extra complexity in the refactor + flaky reset
2. merge dev and demo datasets => as I needed to update the tests which
is very painful, I don't want to do it twice
3. remove all code tied to RELATION_METADATA /
relation-metadata.resolver, or anything tied to the old relation system
4. Remove ONE_TO_ONE and MANY_TO_MANY that are not supported
5. fix impacts on the different areas : see functional testing below
# Functional testing
## Functional testing from the front-end:
1. Database Reset ✅
2. Sign In ✅
3. Workspace sign-up ✅
5. Browsing table / kanban / show ✅
6. Assigning a record in a one to many / in a many to one ✅
7. Deleting a record involved in a relation ✅ => broken but not tied to
this PR
8. "Add new" from relation picker ✅ => broken but not tied to this PR
9. Creating a Task / Note, Updating a Task / Note relations, Deleting a
Task / Note (from table, show page, right drawer) ✅ => broken but not
tied to this PR
10. creating a relation from settings (custom / standard x oneToMany /
manyToOne) ✅
11. updating a relation from settings should not be possible ✅
12. deleting a relation from settings (custom / standard x oneToMany /
manyToOne) ✅
13. Make sure timeline activity still work (relation were involved
there), espacially with Task / Note => to be double checked ✅ => Cannot
convert undefined or null to object
14. Workspace deletion / User deletion ✅
15. CSV Import should keep working ✅
16. Permissions: I have tested without permissions V2 as it's still hard
to test v2 work and it's not in prod yet ✅
17. Workflows global test ✅
## From the API:
1. Review open-api documentation (REST) ✅
2. Make sure REST Api are still able to fetch relations ==> won't do, we
have a coupling Get/Update/Create there, this requires refactoring
3. Make sure REST Api is still able to update / remove relation => won't
do same
## Automated tests
1. lint + typescript ✅
2. front unit tests: ✅
3. server unit tests 2 ✅
4. front stories: ✅
5. server integration: ✅
6. chromatic check : expected 0
7. e2e check : expected no more that current failures
## Remove // Todos
1. All are captured by functional tests above, nothing additional to do
## (Un)related regressions
1. Table loading state is not working anymore, we see the empty state
before table content
2. Filtering by Creator Tim Ap return empty results
3. Not possible to add Tasks / Notes / Files from show page
# Result
## New seeds that can be easily extended
<img width="1920" alt="image"
src="https://github.com/user-attachments/assets/d290d130-2a5f-44e6-b419-7e42a89eec4b"
/>
## -5k lines of code
## No more 'metadata' dataSource (we only have 'core)
## No more relationMetadata (I haven't drop the table yet it's not
referenced in the code anymore)
## We are ready to fix the 6 months lag between current API results and
our mocked tests
## No more bug on relation creation / deletion
---------
Co-authored-by: Weiko <corentin@twenty.com>
Co-authored-by: Félix Malfait <felix@twenty.com>
Closes https://github.com/twentyhq/core-team-issues/issues/748
In the frame of the work on permissions we
- remove all raw queries possible to use repositories instead
- forbid usage workspaceDataSource.executeRawQueries()
- restrict usage of workspaceDataSource.query() to force developers to
pass on shouldBypassPermissionChecks to use it.
---------
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
Arik Chakma