chore(twenty-server): temporary instrumentation for app-install 504 (#21365)

## Why

App installs on cloud intermittently fail with a 504, surfacing in
Sentry as `Migration action 'update' for 'logicFunction' failed` +
`Failed to rollback transaction: Query runner already released`. This is
**temporary instrumentation** to pin down where the time goes — to be
reverted once the bottleneck is fixed. Everything is greppable via
`[install-perf]` and marked `// TODO(install-perf)`.

## What the local repro already told us

I instrumented the manifest-sync/migration path and ran a local harness
(new skipped spec) installing **1 / 8 / 30 logic functions**, for both
create and the checksum-bump **update** (the incident path):

| stage (N=30, update) | ms |
|---|---|
| flat-maps recompute | ~1 |
| build migration | ~11 |
| transaction (all actions + commit) | ~79 |
| post-commit cache invalidate | ~6 |
| **full sync** | **~135** |

Nothing approached 1s, let alone 10s; no slow queries logged. So the
migration/cache code is **not** the algorithmic cause. Given the
in-transaction `UPDATE ... WHERE id=?` is intrinsically fast, a >10s in
prod almost certainly means it was **blocked on a lock**, and the 10s
node-pg `query_timeout` (`core.datasource.ts`) then killed the
connection → the observed errors + 504. Local can't reproduce prod lock
contention / table sizes, hence this instrumentation.

## What this adds (all `TODO`-marked)

- **hrtime per-stage timing** — flat-maps recompute, build vs run,
per-action (`>50ms`), transaction summary, post-commit cache
invalidation. Uses `process.hrtime` because the integration harness
enables fake timers (so `Date.now()` is useless there).
- **`maxQueryExecutionTime`** slow-query logging on the core datasource
(logs the offending SQL).
- **Scoped `SET LOCAL lock_timeout = '8s'`** on the migration
transaction (below the 10s `query_timeout`) → a blocked action fails
fast with a clear *"canceling statement due to lock timeout"* instead of
the opaque connection kill.
- **Best-effort `pg_stat_activity` snapshot on failure** (on a fresh
pooled connection) to identify the blocking session, plus a **guarded
rollback** so a released connection stops masking the real error.
- **Skipped local perf harness**
(`logic-function-install-performance.integration-spec.ts`) — run
manually with `nx test:integration:with-db-reset -- --testPathPattern
"logic-function-install-performance"`.

## How we'll use it

Deploy, reproduce the failing install, and read the `[install-perf]`
logs: the per-action timing names the action, the `lock_timeout` message
+ `pg_stat_activity` snapshot name the **blocking** query/PID. Then
revert this PR and fix the actual contention.

Typecheck (`nx typecheck twenty-server`) is clean.
This commit is contained in:
Charles Bochet
2026-06-09 15:57:25 +02:00
committed by GitHub
parent 7606dd75a8
commit fcaf2b4d9b
5 changed files with 309 additions and 5 deletions
@@ -369,6 +369,8 @@ export class WorkspaceMigrationValidateBuildAndRunService {
const { idByUniversalIdentifierByMetadataName, dryRun, ...buildArgs } =
args;
// TODO(install-perf): temporary, remove.
const buildStart = performance.now();
const validateAndBuildResult =
await this.workspaceMigrationBuildOrchestratorService
.buildWorkspaceMigration(buildArgs)
@@ -379,6 +381,11 @@ export class WorkspaceMigrationValidateBuildAndRunService {
WorkspaceMigrationV2ExceptionCode.BUILDER_INTERNAL_SERVER_ERROR,
);
});
const buildMs = performance.now() - buildStart;
this.logger.log(
`[install-perf] buildWorkspaceMigration took ${buildMs.toFixed(1)}ms (status=${validateAndBuildResult.status})`,
);
if (validateAndBuildResult.status === 'fail') {
if (this.isDebugEnabled) {
@@ -402,11 +409,30 @@ export class WorkspaceMigrationValidateBuildAndRunService {
};
}
const actionCountsByTypeAndMetadataName: Record<string, number> = {};
for (const action of workspaceMigration.actions) {
const key = `${action.type}:${action.metadataName}`;
actionCountsByTypeAndMetadataName[key] =
(actionCountsByTypeAndMetadataName[key] ?? 0) + 1;
}
this.logger.log(
`[install-perf] validateBuildAndRunWorkspaceMigrationFromTo running ${workspaceMigration.actions.length} actions: ${JSON.stringify(actionCountsByTypeAndMetadataName)}`,
);
const runStart = performance.now();
const { hasSchemaMetadataChanged, metadataEvents } =
await this.workspaceMigrationRunnerService.run({
workspaceId: args.workspaceId,
workspaceMigration,
});
const runMs = performance.now() - runStart;
this.logger.log(
`[install-perf] workspaceMigrationRunnerService.run took ${runMs.toFixed(1)}ms for ${workspaceMigration.actions.length} actions`,
);
this.metadataEventEmitter.emitMetadataEvents({
metadataEvents: metadataEvents,