feat(ai): reliable bulk data import via code-interpreter (#22209)

## Summary Makes AI-assisted bulk data import (CSV/Excel/spreadsheets) reliable and token-efficient by letting an entire import run inside a single code-interpreter call, with a persistent sandbox session and server-side bulk helpers. Also includes supporting improvements to attachment handling, upsert reporting, and field-permission error messages. ## Changes ### Code interpreter - **Persistent per-session kernel** in `LocalDriver`: a long-lived Python process per `sessionId` keeps variables, imports, and files alive across calls (matching E2B behavior). Falls back to the existing ephemeral per-call path when no session is provided. CAN BE REMOVED, INTERESTING FOR DEV X - Idle watchdog that self-terminates the kernel, configurable via the new `CODE_INTERPRETER_IDLE_TIMEOUT_MS` config variable; the process also exits on parent shutdown (EOF on control fd). CAN BE REMOVED, INTERESTING FOR DEV X - New `bulk_upsert` and `lookup_by` helpers on the sandbox `twenty` object for idempotent batched writes (≤200/batch) and bounded relation-ID resolution. ### Records - `upsert_many_*` now reports a `created` / `updated` / `total` split in its result and log line (new `isFreshlyCreatedRecord` util). ### AI chat - `replaceUnsupportedFileParts`: user-attached files whose MIME type the model can't handle natively (and that aren't code-interpreter-supported) are downgraded to a descriptive text note instead of being sent as unsupported file parts. Modality→MIME mapping drives native support detection. - Finalize dangling tool parts before `convertToModelMessages` to avoid malformed model messages. - Extracted shared types/constants for code-interpreter file extraction. ### Permissions - Field permission-denied exceptions now include the field name and entity name for easier debugging. ### Skill docs - Added the bulk-import recipe ## To do in following PR - [ ] Skill command migration ## Test plan - [x] Unit tests for `getNativeMimeTypesForModalities` and `replaceUnsupportedFileParts` pass - [x] Run a bulk import (>50 rows) end-to-end through the code interpreter and verify a single sandbox call handles read → resolve relations → upsert → summary - [x] Verify session persistence: define a variable in one call, use it in the next within the same session - [x] Verify the kernel self-terminates after `CODE_INTERPRETER_IDLE_TIMEOUT_MS` - [x] Verify unsupported attachments are replaced with a text note for models lacking the modality - [x] Verify `upsert_many_*` returns correct created/updated counts - [x] Verify field-restricted role triggers a permission error naming the field and entity - [ ] Test with [hotel_business.xlsx](https://github.com/user-attachments/files/29376307/hotel_business.xlsx) and simple "import record" prompt  <a href="https://cubic.dev/pr/twentyhq/twenty/pull/22209?utm_source=github" target="_blank" rel="noopener noreferrer" data-no-image-dialog="true"><picture><source media="(prefers-color-scheme: dark)" srcset="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"><source media="(prefers-color-scheme: light)" srcset="https://www.cubic.dev/buttons/review-in-cubic-light.svg"><img alt="Review in cubic" src="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"></picture></a>
2026-06-29 11:54:27 +02:00
parent 02d62c4175
commit 56deba351b
21 changed files with 1059 additions and 88 deletions
@@ -24,6 +24,10 @@ class TwentyMCP:
    call_tool(name, args) accepts both — catalog tools are routed via
    execute_tool transparently and the envelope is unwrapped, so callers
    see the inner tool's result directly.
+
+    For bulk imports, prefer the higher-level helpers over hand-rolled loops:
+     - bulk_upsert(plural, records): batched, idempotent write path (max 200/batch).
+     - lookup_by(plural, field, values): bounded { value: id } map for relations.
    """

    _MCP_NATIVE_TOOLS = frozenset({
@@ -85,6 +89,100 @@ class TwentyMCP:
                return wrapped['result']
        return wrapped

+    def bulk_upsert(self, plural: str, records: list, batch_size: int = 200):
+        """
+        Upsert many records in batches, paginating to completion.
+
+        This is the recommended write path for imports: upsert dedupes on the
+        object's unique fields (e.g. email) server-side, so re-running a partial
+        or timed-out import is idempotent. Batches are capped at 200 (the platform
+        maximum); the loop runs entirely server-side so the agent never pays the
+        per-batch context cost.
+
+        Args:
+            plural: Plural object name, e.g. 'companies', 'people'.
+            records: List of record dicts to upsert.
+            batch_size: Records per call (max 200).
+
+        Returns:
+            { 'created': int, 'updated': int, 'upserted': int, 'failed': int,
+              'errors': [ {offset, error}, ... up to 10 ] }
+
+        Example:
+            summary = twenty.bulk_upsert('people', people_rows)
+        """
+        size = min(max(int(batch_size), 1), 200)
+        created, updated, failed, errors = 0, 0, 0, []
+        for offset in range(0, len(records), size):
+            chunk = records[offset:offset + size]
+            try:
+                result = self.call_tool('upsert_many_' + plural, {'records': chunk})
+                if isinstance(result, dict):
+                    created += int(result.get('created', 0))
+                    updated += int(result.get('updated', 0))
+            except Exception as exc:
+                failed += len(chunk)
+                if len(errors) < 10:
+                    errors.append({'offset': offset, 'error': str(exc)})
+        return {'created': created, 'updated': updated,
+                'upserted': created + updated, 'failed': failed, 'errors': errors}
+
+    def lookup_by(self, plural: str, field: str, values: list, select: list = None):
+        """
+        Resolve records by a key field, returning a { value: id } map.
+
+        Use this to build relation lookups (e.g. company domain/name -> id) before
+        an import, since relations link by ID. The query is bounded to the distinct
+        values you pass (batched with an 'in' filter, max 200 per call), so it never
+        reads the whole object into the sandbox.
+
+        Args:
+            plural: Plural object name, e.g. 'companies'.
+            field: Field path to match on, e.g. 'name' or 'domainName.primaryLinkUrl'.
+            values: List of key values referenced by the import.
+            select: Fields to return (defaults to the matched field + id).
+
+        Returns:
+            dict mapping each found value to its record id. Missing values are absent.
+
+        Example:
+            company_ids = twenty.lookup_by('companies', 'name', ['Acme', 'Globex'])
+            # { 'Acme': 'uuid-1', 'Globex': 'uuid-2' }
+        """
+        distinct = [value for value in dict.fromkeys(values) if value is not None]
+        select_fields = select or ['id', field]
+        mapping = {}
+        for offset in range(0, len(distinct), 200):
+            chunk = distinct[offset:offset + 200]
+            result = self.call_tool('find_many_' + plural, {
+                'limit': 200,
+                'select': select_fields,
+                **self._nest_field_path(field, {'in': chunk}),
+            })
+            for record in (result.get('records', []) if isinstance(result, dict) else []):
+                key = self._read_field_path(record, field)
+                if key is not None and key not in mapping:
+                    mapping[key] = record.get('id')
+        return mapping
+
+    @staticmethod
+    def _nest_field_path(field: str, leaf: dict):
+        """Turn 'domainName.primaryLinkUrl' + leaf into a nested filter dict."""
+        parts = field.split('.')
+        nested = leaf
+        for part in reversed(parts):
+            nested = {part: nested}
+        return nested
+
+    @staticmethod
+    def _read_field_path(record: dict, field: str):
+        """Read a possibly nested field path (e.g. 'domainName.primaryLinkUrl')."""
+        current = record
+        for part in field.split('.'):
+            if not isinstance(current, dict):
+                return None
+            current = current.get(part)
+        return current

    def _raw_mcp_call(self, name: str, arguments: dict = None):
        """Low-level: issue a tools/call against the MCP surface verbatim."""