## Summary
Fixes a performance-correctness bug in the createMany upsert path where
the
existing-record lookup built an overly broad WHERE clause, causing full
table
scans and multi-second latency on bulk upserts.
Reported via Sentry: a 100-record create*(upsert: true) request on
_sdWorkspace
completed with HTTP 200 but took ~14s. The candidate-lookup SELECT alone
took
~6.7s because it fetched a huge superset of rows before matching in
memory.
## Root cause
buildWhereConditions created one IN(...) per conflicting column across
the whole
batch, and findExistingRecords OR-ed them together:
WHERE "cbCustomerId" IN ($1..$100) OR "environment" IN ($101..$200)
For a composite unique index (cbCustomerId, environment), this is
semantically
wrong: it OR's the columns instead of matching them as a tuple. Because
environment is low-cardinality (prod/staging/dev/test), the second IN
alone
matched almost the entire table, forcing a Seq Scan and shipping the
whole
result set to the app for in-memory filtering.
## Fix
buildWhereConditions now generates targeted lookup conditions:
- Single-column unique keys collapse into one `column IN
(distinctValues)`
condition (instead of N OR-ed equalities).
- Composite unique keys produce one `(colA = ? AND colB = ?)` condition
per
input record — the columns are ANDed as a tuple, and separate unique
indexes
are still OR-ed together.
- Conditions are deduplicated (robust JSON-based key, no separator
collisions)
to avoid redundant OR branches.
- Values are now typed as string | number | boolean instead of being
implicitly
coerced to string.
## Benchmark
Reproduced the incident with a table mirroring _sdWorkspace:
high-cardinality
cbCustomerId + low-cardinality environment (4 values), composite unique
index on
(cbCustomerId, environment), 100-record upsert batch. EXPLAIN (ANALYZE,
BUFFERS)
on a warm 500k-row / 392 MB table:
Metric | OLD (colA IN OR colB IN) | NEW (targeted) | Improvement
----------------------|--------------------------|----------------|------------
Scan type | Seq Scan (full table) | Index Scan | index vs full scan
Rows returned to app | 500,000 | 100 | ~5,000x fewer
Buffers touched | ~45,455 (~355 MB) | 400 (~3.2 MB) | ~110x fewer
Execution time | 224 ms | 11.9 ms | ~19x faster
Data shipped to app | ~322 MB | ~64 KB | ~5,000x less
The cache-independent facts are the proof: the old query returned the
entire
table for a 100-record batch (the "overly broad record set" from the
report),
while the new query returns exactly the matching rows via the composite
index.
## Test plan
- [x] Unit tests for buildWhereConditions (single-column IN batching,
nested
paths, composite AND tuples, dedup incl. separator-collision safety,
numeric values, mixed single-column + composite OR) — 22 passing across
build-where-conditions, get-matching-record-id, get-value-from-path.
- [x] Upsert integration suites pass (upsert +
composite-unique-index-upsert,
10 tests).
- [x] EXPLAIN ANALYZE benchmark confirms Index Scan and bounded row
counts.
<!-- This is an auto-generated description by cubic. -->
<a
href="https://cubic.dev/pr/twentyhq/twenty/pull/22721?utm_source=github"
target="_blank" rel="noopener noreferrer"
data-no-image-dialog="true"><picture><source
media="(prefers-color-scheme: dark)"
srcset="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"><source
media="(prefers-color-scheme: light)"
srcset="https://www.cubic.dev/buttons/review-in-cubic-light.svg"><img
alt="Review in cubic"
src="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"></picture></a>
<!-- End of auto-generated description by cubic. -->
Etienne
7475b5f16f
