## Context
A self-hosted instance upgrading from 2.0.3 to v2.22.0 got stuck with
one workspace failing at `2.16.0_BackfillSearchFieldMetadataCommand`:
```
[QueryFailedError] duplicate key value violates unique constraint "IDX_SEARCH_FIELD_METADATA_OBJECT_FIELD_UNIQUE"
Detail: Key ("objectMetadataId", "fieldMetadataId")=(...) already exists.
```
The failure happened on a retry after a previous partial run, and
reproduced even though the command already recomputes
`flatSearchFieldMetadataMaps` before deriving the create-set (#22884).
## Root cause
The idempotency dedupe compares `(objectMetadataId, fieldMetadataId)`
pairs across two differently-fresh caches:
- The **existing rows** side comes from `flatSearchFieldMetadataMaps`,
which is recomputed from the database (real current ids).
- The **candidate** side resolves ids through `flatObjectMetadataMaps` /
`flatFieldMetadataMaps`, which are **not** invalidated. During a
cross-version upgrade these can be stale, since the migration runner
only invalidates the cache keys a migration touched.
When a stale map resolves a candidate to an outdated id, the dedupe key
doesn't match the existing row and the row is re-emitted. The migration
runner then re-resolves the universal identifiers against fresh maps at
execution time and inserts with the real current ids — exactly the pair
already committed by the earlier partial run (each per-application
migration commits independently) — tripping the unique constraint and
failing the upgrade.
## Fix
Two independent layers, either of which would have prevented the
failure:
1. **Consistent snapshot for the build phase**: the command now
invalidates and recomputes all three maps the dedupe depends on
(`flatObjectMetadataMaps`, `flatFieldMetadataMaps`,
`flatSearchFieldMetadataMaps`), so candidate resolution, existing-row
keys, and the runner all see the same database state.
2. **Id-churn-proof dedupe**: every row this command creates carries a
deterministic universal identifier (`getSearchFieldUniversalIdentifier`,
derived from application + field universal identifiers, no database ids
involved) and `(workspaceId, universalIdentifier)` is unique. The build
util now also skips any candidate whose deterministic universal
identifier already exists, catching leftovers from a previous partial
run even if objects/fields were recreated under new ids in between.
Deliberately **not** done: `ON CONFLICT DO NOTHING` in the create action
handler — it is shared by all runtime `searchFieldMetadata` creation,
and swallowing a conflict would leave the flat-entity cache holding an
entity id that differs from the row actually in the database.
## Test
Added a regression test reproducing the failure shape: an existing row
with the same deterministic universal identifier but stale metadata ids
must not be re-emitted by the backfill.
Note: `ReconcileSearchFieldMetadataCommand` (2.20) has the same
stale-cache exposure; hardening it is left to a follow-up.
<!-- This is an auto-generated description by cubic. -->
<a
href="https://cubic.dev/pr/twentyhq/twenty/pull/23060?utm_source=github"
target="_blank" rel="noopener noreferrer"
data-no-image-dialog="true"><picture><source
media="(prefers-color-scheme: dark)"
srcset="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"><source
media="(prefers-color-scheme: light)"
srcset="https://www.cubic.dev/buttons/review-in-cubic-light.svg"><img
alt="Review in cubic"
src="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"></picture></a>
<!-- End of auto-generated description by cubic. -->
Paul Rastoin
6b55a6b51c
