feat(search): Add unaccent support for accent-insensitive search. (#14464)

##  Add accent-insensitive search functionality

### 🎯 Overview
Implements accent-insensitive search across all searchable fields in
Twenty CRM.
Users can now search for "jose" to find "José", "muller" to find
"Müller", "cafe" to find "café", etc.

### 🔍 Problem
Twenty's search functionality was accent-sensitive, requiring users to
type exact accented characters to find records.
This created a poor user experience, especially for international names
and content.

### 💡 Solution
Added PostgreSQL `unaccent` extension with a custom immutable wrapper
function to enable accent-insensitive full-text search across all
searchable field types.

### 📋 Changes Made
**Modified Files:**  
- `packages/twenty-server/scripts/setup-db.ts`  
-
`packages/twenty-server/src/engine/api/graphql/graphql-query-runner/utils/compute-where-condition-parts.ts`
-
`packages/twenty-server/src/engine/workspace-manager/workspace-sync-metadata/utils/get-ts-vector-column-expression.util.ts`

### 🗄️ Database Setup (`setup-db.ts`)
```sql
-- Added unaccent extension
CREATE EXTENSION IF NOT EXISTS "unaccent";

-- Created immutable wrapper function
CREATE OR REPLACE FUNCTION unaccent_immutable(text) RETURNS text AS $$
  SELECT public.unaccent($1)
$$ LANGUAGE sql IMMUTABLE;
```

### 🔍 Search Vector Generation
(`get-ts-vector-column-expression.util.ts`)
Applied `public.unaccent_immutable()` to all searchable field types:  
- TEXT fields (job titles, names, etc.)  
- FULL_NAME fields (first/last names)  
- EMAILS fields (both email address and domain)  
- ADDRESS fields  
- LINKS fields  
- RICH_TEXT and RICH_TEXT_V2 fields

### 🔎 Query Processing (`compute-where-condition-parts.ts`)
Enhanced search queries to use `public.unaccent_immutable()` for both:  
- Full-text search (`@@` operator with `to_tsquery`)  
- Pattern matching (`ILIKE` operator)

### 🧠 Technical Rationale: Why the Wrapper Function?
**The Challenge:**  
PostgreSQL's built-in `unaccent()` is marked as **STABLE**, but
`GENERATED ALWAYS AS` expressions (used for search vector columns)
require **IMMUTABLE** functions.

**The Solution:**  
Created an IMMUTABLE wrapper function that calls the underlying
`unaccent()` function:
-  Satisfies PostgreSQL's immutability requirements for generated
columns
-  Maintains the exact same functionality as the original `unaccent()`
-  Uses fully qualified `public.unaccent_immutable()` to ensure
function resolution from workspace schemas

**Alternative Approaches Considered:**  
-  Modifying `search_path`: would affect workspace isolation  
-  Computing unaccent at query time: would hurt performance  
-  Using triggers: would complicate data consistency

### 🎯 Impact
For **Person** records, accent-insensitive search now works on:  
- Name (first/last name): `"jose garcia"` finds `"José García"`  
- Email: `"jose@cafe.com"` finds `"josé@café.com"`  
- Job Title: `"manager"` finds `"Managér"` or `"Gerente de Café"`

Applies to all searchable standard objects:  
- Companies, People, Opportunities, Notes, Tasks, etc.  
- Any custom fields of searchable types (TEXT, EMAILS, etc.)

###  Testing
- Database reset completes successfully  
- Workspace seeding works without errors  
- Search vectors generate with unaccent functionality  
- All searchable field types properly handle accented characters

---------

Co-authored-by: Félix Malfait <felix.malfait@gmail.com>
This commit is contained in:
Abdullah.
2025-09-17 12:01:35 +05:00
committed by GitHub
parent f8a842781d
commit 2a7a83de81
5 changed files with 43 additions and 18 deletions
@@ -18,7 +18,9 @@ describe('getTsVectorColumnExpressionFromFields', () => {
const fields = [nameTextField] as FieldTypeAndNameMetadata[];
const result = getTsVectorColumnExpressionFromFields(fields);
expect(result).toContain("to_tsvector('simple', COALESCE(\"name\", ''))");
expect(result).toContain(
"to_tsvector('simple', COALESCE(public.unaccent_immutable(\"name\"), ''))",
);
});
it('should handle multiple fields', () => {
@@ -28,13 +30,22 @@ describe('getTsVectorColumnExpressionFromFields', () => {
emailsEmailsField,
] as FieldTypeAndNameMetadata[];
const result = getTsVectorColumnExpressionFromFields(fields);
const expected = `
to_tsvector('simple', COALESCE("nameFirstName", '') || ' ' || COALESCE("nameLastName", '') || ' ' || COALESCE("jobTitle", '') || ' ' ||
COALESCE("emailsPrimaryEmail", '') || ' ' ||
COALESCE(SPLIT_PART("emailsPrimaryEmail", '@', 2), ''))
`.trim();
expect(result.trim()).toBe(expected);
expect(result).toContain(
'COALESCE(public.unaccent_immutable("nameFirstName"), \'\')',
);
expect(result).toContain(
'COALESCE(public.unaccent_immutable("nameLastName"), \'\')',
);
expect(result).toContain(
'COALESCE(public.unaccent_immutable("jobTitle"), \'\')',
);
expect(result).toContain(
'COALESCE(public.unaccent_immutable("emailsPrimaryEmail"), \'\')',
);
expect(result).toContain(
"COALESCE(public.unaccent_immutable(SPLIT_PART(\"emailsPrimaryEmail\", '@', 2)), '')",
);
});
it('should handle rich text fields', () => {
@@ -43,7 +54,9 @@ describe('getTsVectorColumnExpressionFromFields', () => {
] as FieldTypeAndNameMetadata[];
const result = getTsVectorColumnExpressionFromFields(fields);
expect(result).toBe("to_tsvector('simple', COALESCE(\"body\", ''))");
expect(result).toBe(
"to_tsvector('simple', COALESCE(public.unaccent_immutable(\"body\"), ''))",
);
});
it('should handle rich text v2 fields', () => {
@@ -53,7 +66,7 @@ describe('getTsVectorColumnExpressionFromFields', () => {
const result = getTsVectorColumnExpressionFromFields(fields);
expect(result).toBe(
"to_tsvector('simple', COALESCE(\"bodyV2Markdown\", ''))",
"to_tsvector('simple', COALESCE(public.unaccent_immutable(\"bodyV2Markdown\"), ''))",
);
});
});
@@ -82,10 +82,10 @@ const getColumnExpression = (
switch (fieldType) {
case FieldMetadataType.EMAILS:
return `
COALESCE(${quotedColumnName}, '') || ' ' ||
COALESCE(SPLIT_PART(${quotedColumnName}, '@', 2), '')`;
COALESCE(public.unaccent_immutable(${quotedColumnName}), '') || ' ' ||
COALESCE(public.unaccent_immutable(SPLIT_PART(${quotedColumnName}, '@', 2)), '')`;
default:
return `COALESCE(${quotedColumnName}, '')`;
return `COALESCE(public.unaccent_immutable(${quotedColumnName}), '')`;
}
};