Charles Bochet
a7324252fd
## Context Sentry issue [7438578272](https://twenty-v7.sentry.io/issues/7438578272/) (Logic Function Layer Build Failed, 7.9K events over 3 months): the yarn-install tool Lambda dies with `Runtime.OutOfMemory` / `signal: killed` while building an application's dependency layer. Every layer build for the affected application fails permanently, each database-event trigger re-attempts it, and one workspace produced ~2.5K events in the last week alone. ## Root cause The offending application declares `twenty-ui@1.0.0-alpha.0` (181MB unpacked, dragging in 141MB of `@tabler/icons*`) and dev tooling as production `dependencies` of server-side logic functions. Installing that tree needs just under 3GB during Yarn 4's fetch/link phase, so the 1024MB sandbox is OOM-killed. And even a successful install could never ship: AWS caps a function plus all its layers at 250MB unzipped. The user never sees any of this: the OOM is retried forever, and nothing tells them their dependencies are the problem. ## Fix 1. **Raise the yarn-install Lambda to 4096MB** so legitimate dependency trees install. Tool function names now include the memory/timeout/ephemeral-storage constants in their content hash, so a config change rotates the function name and the ensure path creates a fresh function with the new configuration — without this, the constant change would never reach already-deployed functions (their config is only applied at creation, and the ensure path early-returns when the function exists). 2. **Propagate Lambda's own errors to the user.** The install OOM (`Runtime.OutOfMemory` on the invoke) and the layer size rejection (`InvalidParameterValueException` at `PublishLayerVersion`) map to a new `LOGIC_FUNCTION_DEPENDENCIES_SIZE_EXCEEDED` code telling the user to move packages their logic functions don't import out of `dependencies`. Surfacing per API boundary: - **Sync / install (CLI)**: the workspace migration interceptor formats it into the same metadata validation error shape the SDK already renders, as one `logicFunction` entry carrying the remedy and the underlying AWS detail — no SDK rendering changes needed. - **`executeOneLogicFunction`**: mapped to `UserInputError` in the GraphQL handler. - **Route triggers**: HTTP 422 with the user-facing message, no Sentry capture. - **Background triggers**: skip instead of retrying, since no retry can succeed until the user changes their application. ## Test The error originates in AWS behavior, which CI (local driver, no AWS) cannot reproduce — so the chain is verified link by link: - **Real AWS, manual (not in CI)**: reproduced with the offending application's actual package.json against real Lambdas in the dev account — OOM-killed at 1024MB and 2048MB (exact prod error signature), install succeeds at 4096MB (~4min), and the resulting 292MB layer is rejected by `PublishLayerVersion` with the exact `InvalidParameterValueException` this PR matches. Same matrix reproduced in local cgroups beforehand. - **Server unit specs**: AWS error payload → exception mapping (`build-yarn-install-failure-exception`), exception → validation payload formatting (interceptor handler), `executeOneLogicFunction` GraphQL mapping, route filter 422 mapping, tool-function/layer name hashing. - **SDK integration spec (mocked server)**: runs the real `app dev` orchestrator on the minimal app with `syncApplication` mocked to return the validation-shaped failure, and asserts the CLI report renders the error code and remedy. It covers CLI rendering only — no test installs actual oversized dependencies, by design. - Docs updated (dependency size limits, sync failure taxonomy, route platform error responses).
127 lines
7.8 KiB
Plaintext
127 lines
7.8 KiB
Plaintext
---
|
|
title: Syncing & recovery
|
|
description: Which command to use when, how to read the sync output, and a recovery ladder for when local metadata drifts — before reaching a full reset.
|
|
icon: "compass"
|
|
---
|
|
|
|
Local app development revolves around **syncing**: the CLI rebuilds your manifest and the server applies only the difference between it and the metadata already in your workspace. This page covers which command to reach for, how to read what a sync changed, and what to do — in order — when local state looks inconsistent.
|
|
|
|
## Which command, when
|
|
|
|
<Note>
|
|
For day-to-day local iteration you almost always want `yarn twenty dev`. Deploying and publishing are for shipping releases, **not** for the local loop.
|
|
</Note>
|
|
|
|
| You want to… | Command | Notes |
|
|
| --- | --- | --- |
|
|
| Iterate locally with live sync | `yarn twenty dev` | Watches your files and syncs on every change. |
|
|
| Sync once and exit (CI, scripts, hooks) | `yarn twenty apply` | One build + sync, then exits. Add `--force` to skip the destructive-change confirmation. |
|
|
| Preview changes **without applying them** | `yarn twenty plan` | Computes and prints the diff; writes nothing. |
|
|
| Remove the app from the workspace | `yarn twenty app:uninstall` | Add `--yes` to skip the prompt. |
|
|
| Ship a tarball to a server | `yarn twenty app:publish --private` | Requires a **strictly higher** `package.json` version — see [Publishing](/developers/extend/apps/operations/publishing). |
|
|
| Publish to the marketplace (npm) | `yarn twenty app:publish` | — |
|
|
| Install / upgrade a deployed version | `yarn twenty app:install` | Installs the version currently deployed. |
|
|
| Wipe the local server and start clean | `yarn twenty docker:reset` | Deletes **all** local data — last resort. |
|
|
|
|
<Note>
|
|
`yarn twenty dev --once` and `yarn twenty dev --once --dry-run` still work as deprecated aliases for `yarn twenty apply` and `yarn twenty plan`.
|
|
</Note>
|
|
|
|
### Local sync does not need a version bump
|
|
|
|
The strictly-increasing `version` rule (`VERSION_ALREADY_EXISTS` on deploy, `APP_ALREADY_INSTALLED` / `CANNOT_DOWNGRADE_APPLICATION` on install) applies to **`app:publish` / `app:install`** — the release path. `yarn twenty dev` syncs your manifest in place and never requires a version change, so you don't need to touch `package.json` to iterate. If you find yourself bumping the version to test a local change, you're using the release path when you want the dev loop.
|
|
|
|
## Reading the sync output
|
|
|
|
Every sync prints the metadata changes it applied (or would apply, with `plan`), Terraform-style — one block per entity with its attributes, then a summary line:
|
|
|
|
```text filename="Terminal"
|
|
# objectMetadata "rocket" will be created
|
|
+ icon = "IconRocket"
|
|
+ labelSingular = "Rocket"
|
|
+ ...
|
|
|
|
# fieldMetadata "launchedAt" will be updated
|
|
~ isNullable = false -> true
|
|
|
|
Plan: 2 to add, 1 to change, 1 to destroy.
|
|
|
|
✓ Synced My App (4 files)
|
|
```
|
|
|
|
This is your first diagnostic: it tells you exactly which objects, fields, and layouts changed, so you can confirm a sync did what you expected before checking the UI.
|
|
|
|
Destructive changes (`to destroy`) are listed with what they drop (e.g. `objectMetadata "auditNote" — drops the table and all its rows`) and require interactive confirmation, or `--force` in scripts.
|
|
|
|
When a sync fails on a single entity, the error names the offending entity and its `universalIdentifier`, for example:
|
|
|
|
```text
|
|
Migration action 'create' for 'fieldMetadata' (universalIdentifier: 2020...4337) failed
|
|
```
|
|
|
|
Use that identifier to find the entity in your manifest (and, if needed, in the workspace) instead of guessing which one conflicts.
|
|
|
|
## Previewing changes (plan)
|
|
|
|
`yarn twenty plan` builds your manifest, asks the server for the migration plan, and prints it — **without applying anything**. It's the safe way to answer "what would this sync change?" before committing to it.
|
|
|
|
```bash filename="Terminal"
|
|
yarn twenty plan
|
|
```
|
|
|
|
```text filename="Terminal"
|
|
Building manifest...
|
|
Computing metadata plan (read-only, nothing will be applied)...
|
|
|
|
# fieldMetadata "crewCapacity" will be created
|
|
+ ...
|
|
|
|
Plan: 1 to add, 1 to change, 0 to destroy.
|
|
|
|
✓ Plan complete for My App — no changes were applied
|
|
```
|
|
|
|
A plan:
|
|
|
|
- **Writes nothing** — no metadata migration, no application record update, no default role/tab changes, and no API client generation.
|
|
- Returns the **same diff** a real sync would apply, so you can review created/updated/deleted entities up front.
|
|
- Is useful before a risky change, when reviewing an AI-generated change, or in a script that should fail if an unexpected change is about to land.
|
|
|
|
<Note>
|
|
A plan only previews **metadata** changes, and it requires the app to have been synced at least once (so the workspace knows about it). If you run it against an app that was never synced, the server reports that the app is not installed — run `yarn twenty dev` once first.
|
|
</Note>
|
|
|
|
## Recovery ladder
|
|
|
|
When local metadata looks wrong, escalate in this order and stop as soon as you're unblocked. Each step is more disruptive than the last.
|
|
|
|
1. **Re-sync.** Run `yarn twenty apply` again. Syncs are idempotent — re-running a clean manifest is safe and often resolves a transient hiccup.
|
|
2. **Preview the plan.** Run `yarn twenty plan` to see exactly what the next sync intends to change, without applying it.
|
|
3. **Read the named error.** If a sync fails, note the metadata type and `universalIdentifier` in the message (see above) and locate that entity in your manifest. A conflict usually points to a duplicated or re-used identifier.
|
|
4. **Uninstall and reinstall.** `yarn twenty app:uninstall`, then sync again (`yarn twenty dev`). This rebuilds the app's metadata from a clean slate while keeping the rest of your workspace intact.
|
|
5. **Full reset (last resort).** `yarn twenty docker:reset`, then re-seed and re-sync.
|
|
|
|
<Warning>
|
|
`yarn twenty docker:reset` deletes **all** data in your local instance — every workspace, record, and app. Only use it once the earlier steps have failed.
|
|
</Warning>
|
|
|
|
<Note>
|
|
Hit a metadata error? Please [open an issue](https://github.com/twentyhq/twenty/issues/new/choose) and include the failing migration message (with its metadata type and `universalIdentifier`), the `Metadata changes` output from the sync, and the commands you ran.
|
|
</Note>
|
|
|
|
## Avoid concurrent syncs on one workspace
|
|
|
|
Syncing applies metadata migrations. Running several sync, deploy, or install operations against the **same workspace at the same time** — for example, multiple terminals or AI agents iterating in parallel — can interleave those migrations and leave metadata in a partially-applied state.
|
|
|
|
The server serializes syncs per workspace to prevent this, but you should still funnel sensitive metadata operations through a **single** process rather than firing them concurrently. If you orchestrate development with multiple agents, route their sync/deploy/install calls through one queue so only one runs at a time.
|
|
|
|
## Telling failures apart
|
|
|
|
When something goes wrong, the metadata diff and named errors let you place the failure:
|
|
|
|
- **Manifest build error** — the CLI fails before syncing (`MANIFEST_BUILD_FAILED`, `TYPECHECK_FAILED`); fix your app source.
|
|
- **Sync / migration error** — the build succeeds but applying the diff fails, naming the entity and `universalIdentifier`; fix the conflicting metadata.
|
|
- **Dependencies size error** — the sync or install fails because the app's production `dependencies` are too large to install as a runtime layer (`LOGIC_FUNCTION_DEPENDENCIES_SIZE_EXCEEDED`); move packages your logic functions do not import at runtime (UI libraries, dev tooling) to `devDependencies`.
|
|
- **App code runtime error** — the sync succeeds but your logic functions or components misbehave at runtime; check [function logs](/developers/extend/apps/operations/cli).
|
|
- **Local instance state** — none of the above and the workspace still looks wrong; work down the recovery ladder.
|