feat(search): Add unaccent support for accent-insensitive search. (#14464)
## ✨ Add accent-insensitive search functionality ### 🎯 Overview Implements accent-insensitive search across all searchable fields in Twenty CRM. Users can now search for "jose" to find "José", "muller" to find "Müller", "cafe" to find "café", etc. ### 🔍 Problem Twenty's search functionality was accent-sensitive, requiring users to type exact accented characters to find records. This created a poor user experience, especially for international names and content. ### 💡 Solution Added PostgreSQL `unaccent` extension with a custom immutable wrapper function to enable accent-insensitive full-text search across all searchable field types. ### 📋 Changes Made **Modified Files:** - `packages/twenty-server/scripts/setup-db.ts` - `packages/twenty-server/src/engine/api/graphql/graphql-query-runner/utils/compute-where-condition-parts.ts` - `packages/twenty-server/src/engine/workspace-manager/workspace-sync-metadata/utils/get-ts-vector-column-expression.util.ts` ### 🗄️ Database Setup (`setup-db.ts`) ```sql -- Added unaccent extension CREATE EXTENSION IF NOT EXISTS "unaccent"; -- Created immutable wrapper function CREATE OR REPLACE FUNCTION unaccent_immutable(text) RETURNS text AS $$ SELECT public.unaccent($1) $$ LANGUAGE sql IMMUTABLE; ``` ### 🔍 Search Vector Generation (`get-ts-vector-column-expression.util.ts`) Applied `public.unaccent_immutable()` to all searchable field types: - TEXT fields (job titles, names, etc.) - FULL_NAME fields (first/last names) - EMAILS fields (both email address and domain) - ADDRESS fields - LINKS fields - RICH_TEXT and RICH_TEXT_V2 fields ### 🔎 Query Processing (`compute-where-condition-parts.ts`) Enhanced search queries to use `public.unaccent_immutable()` for both: - Full-text search (`@@` operator with `to_tsquery`) - Pattern matching (`ILIKE` operator) ### 🧠 Technical Rationale: Why the Wrapper Function? **The Challenge:** PostgreSQL's built-in `unaccent()` is marked as **STABLE**, but `GENERATED ALWAYS AS` expressions (used for search vector columns) require **IMMUTABLE** functions. **The Solution:** Created an IMMUTABLE wrapper function that calls the underlying `unaccent()` function: - ✅ Satisfies PostgreSQL's immutability requirements for generated columns - ✅ Maintains the exact same functionality as the original `unaccent()` - ✅ Uses fully qualified `public.unaccent_immutable()` to ensure function resolution from workspace schemas **Alternative Approaches Considered:** - ❌ Modifying `search_path`: would affect workspace isolation - ❌ Computing unaccent at query time: would hurt performance - ❌ Using triggers: would complicate data consistency ### 🎯 Impact For **Person** records, accent-insensitive search now works on: - Name (first/last name): `"jose garcia"` finds `"José García"` - Email: `"jose@cafe.com"` finds `"josé@café.com"` - Job Title: `"manager"` finds `"Managér"` or `"Gerente de Café"` Applies to all searchable standard objects: - Companies, People, Opportunities, Notes, Tasks, etc. - Any custom fields of searchable types (TEXT, EMAILS, etc.) ### ✅ Testing - Database reset completes successfully - Workspace seeding works without errors - Search vectors generate with unaccent functionality - All searchable field types properly handle accented characters --------- Co-authored-by: Félix Malfait <felix.malfait@gmail.com>
This commit is contained in:
+22
-9
@@ -18,7 +18,9 @@ describe('getTsVectorColumnExpressionFromFields', () => {
|
||||
const fields = [nameTextField] as FieldTypeAndNameMetadata[];
|
||||
const result = getTsVectorColumnExpressionFromFields(fields);
|
||||
|
||||
expect(result).toContain("to_tsvector('simple', COALESCE(\"name\", ''))");
|
||||
expect(result).toContain(
|
||||
"to_tsvector('simple', COALESCE(public.unaccent_immutable(\"name\"), ''))",
|
||||
);
|
||||
});
|
||||
|
||||
it('should handle multiple fields', () => {
|
||||
@@ -28,13 +30,22 @@ describe('getTsVectorColumnExpressionFromFields', () => {
|
||||
emailsEmailsField,
|
||||
] as FieldTypeAndNameMetadata[];
|
||||
const result = getTsVectorColumnExpressionFromFields(fields);
|
||||
const expected = `
|
||||
to_tsvector('simple', COALESCE("nameFirstName", '') || ' ' || COALESCE("nameLastName", '') || ' ' || COALESCE("jobTitle", '') || ' ' ||
|
||||
COALESCE("emailsPrimaryEmail", '') || ' ' ||
|
||||
COALESCE(SPLIT_PART("emailsPrimaryEmail", '@', 2), ''))
|
||||
`.trim();
|
||||
|
||||
expect(result.trim()).toBe(expected);
|
||||
expect(result).toContain(
|
||||
'COALESCE(public.unaccent_immutable("nameFirstName"), \'\')',
|
||||
);
|
||||
expect(result).toContain(
|
||||
'COALESCE(public.unaccent_immutable("nameLastName"), \'\')',
|
||||
);
|
||||
expect(result).toContain(
|
||||
'COALESCE(public.unaccent_immutable("jobTitle"), \'\')',
|
||||
);
|
||||
expect(result).toContain(
|
||||
'COALESCE(public.unaccent_immutable("emailsPrimaryEmail"), \'\')',
|
||||
);
|
||||
expect(result).toContain(
|
||||
"COALESCE(public.unaccent_immutable(SPLIT_PART(\"emailsPrimaryEmail\", '@', 2)), '')",
|
||||
);
|
||||
});
|
||||
|
||||
it('should handle rich text fields', () => {
|
||||
@@ -43,7 +54,9 @@ describe('getTsVectorColumnExpressionFromFields', () => {
|
||||
] as FieldTypeAndNameMetadata[];
|
||||
const result = getTsVectorColumnExpressionFromFields(fields);
|
||||
|
||||
expect(result).toBe("to_tsvector('simple', COALESCE(\"body\", ''))");
|
||||
expect(result).toBe(
|
||||
"to_tsvector('simple', COALESCE(public.unaccent_immutable(\"body\"), ''))",
|
||||
);
|
||||
});
|
||||
|
||||
it('should handle rich text v2 fields', () => {
|
||||
@@ -53,7 +66,7 @@ describe('getTsVectorColumnExpressionFromFields', () => {
|
||||
const result = getTsVectorColumnExpressionFromFields(fields);
|
||||
|
||||
expect(result).toBe(
|
||||
"to_tsvector('simple', COALESCE(\"bodyV2Markdown\", ''))",
|
||||
"to_tsvector('simple', COALESCE(public.unaccent_immutable(\"bodyV2Markdown\"), ''))",
|
||||
);
|
||||
});
|
||||
});
|
||||
|
||||
+3
-3
@@ -82,10 +82,10 @@ const getColumnExpression = (
|
||||
switch (fieldType) {
|
||||
case FieldMetadataType.EMAILS:
|
||||
return `
|
||||
COALESCE(${quotedColumnName}, '') || ' ' ||
|
||||
COALESCE(SPLIT_PART(${quotedColumnName}, '@', 2), '')`;
|
||||
COALESCE(public.unaccent_immutable(${quotedColumnName}), '') || ' ' ||
|
||||
COALESCE(public.unaccent_immutable(SPLIT_PART(${quotedColumnName}, '@', 2)), '')`;
|
||||
|
||||
default:
|
||||
return `COALESCE(${quotedColumnName}, '')`;
|
||||
return `COALESCE(public.unaccent_immutable(${quotedColumnName}), '')`;
|
||||
}
|
||||
};
|
||||
|
||||
Reference in New Issue
Block a user