chore(twenty-server): temporary instrumentation for app-install 504 (#21365)

## Why

App installs on cloud intermittently fail with a 504, surfacing in
Sentry as `Migration action 'update' for 'logicFunction' failed` +
`Failed to rollback transaction: Query runner already released`. This is
**temporary instrumentation** to pin down where the time goes — to be
reverted once the bottleneck is fixed. Everything is greppable via
`[install-perf]` and marked `// TODO(install-perf)`.

## What the local repro already told us

I instrumented the manifest-sync/migration path and ran a local harness
(new skipped spec) installing **1 / 8 / 30 logic functions**, for both
create and the checksum-bump **update** (the incident path):

| stage (N=30, update) | ms |
|---|---|
| flat-maps recompute | ~1 |
| build migration | ~11 |
| transaction (all actions + commit) | ~79 |
| post-commit cache invalidate | ~6 |
| **full sync** | **~135** |

Nothing approached 1s, let alone 10s; no slow queries logged. So the
migration/cache code is **not** the algorithmic cause. Given the
in-transaction `UPDATE ... WHERE id=?` is intrinsically fast, a >10s in
prod almost certainly means it was **blocked on a lock**, and the 10s
node-pg `query_timeout` (`core.datasource.ts`) then killed the
connection → the observed errors + 504. Local can't reproduce prod lock
contention / table sizes, hence this instrumentation.

## What this adds (all `TODO`-marked)

- **hrtime per-stage timing** — flat-maps recompute, build vs run,
per-action (`>50ms`), transaction summary, post-commit cache
invalidation. Uses `process.hrtime` because the integration harness
enables fake timers (so `Date.now()` is useless there).
- **`maxQueryExecutionTime`** slow-query logging on the core datasource
(logs the offending SQL).
- **Scoped `SET LOCAL lock_timeout = '8s'`** on the migration
transaction (below the 10s `query_timeout`) → a blocked action fails
fast with a clear *"canceling statement due to lock timeout"* instead of
the opaque connection kill.
- **Best-effort `pg_stat_activity` snapshot on failure** (on a fresh
pooled connection) to identify the blocking session, plus a **guarded
rollback** so a released connection stops masking the real error.
- **Skipped local perf harness**
(`logic-function-install-performance.integration-spec.ts`) — run
manually with `nx test:integration:with-db-reset -- --testPathPattern
"logic-function-install-performance"`.

## How we'll use it

Deploy, reproduce the failing install, and read the `[install-perf]`
logs: the per-action timing names the action, the `lock_timeout` message
+ `pg_stat_activity` snapshot name the **blocking** query/PID. Then
revert this PR and fix the actual contention.

Typecheck (`nx typecheck twenty-server`) is clean.
This commit is contained in:
Charles Bochet
2026-06-09 15:57:25 +02:00
committed by GitHub
parent 7606dd75a8
commit fcaf2b4d9b
5 changed files with 309 additions and 5 deletions
@@ -179,6 +179,8 @@ export class ApplicationManifestMigrationService {
{ workspaceId },
);
// TODO(install-perf): temporary, remove.
const recomputeStart = performance.now();
const cacheResult = await this.workspaceCacheService.getOrRecompute(
workspaceId,
[
@@ -186,6 +188,11 @@ export class ApplicationManifestMigrationService {
'featureFlagsMap',
],
);
const recomputeMs = performance.now() - recomputeStart;
this.logger.log(
`[install-perf] syncMetadataFromManifest ALL_METADATA_NAME getOrRecompute flat-maps took ${recomputeMs.toFixed(1)}ms (logicFunctions=${manifest.logicFunctions.length})`,
);
const { featureFlagsMap, ...existingAllFlatEntityMaps } = cacheResult;
@@ -211,6 +218,7 @@ export class ApplicationManifestMigrationService {
fromAllFlatEntityMaps: existingAllFlatEntityMaps,
});
const validateBuildRunStart = performance.now();
const validateAndBuildResult =
await this.workspaceMigrationValidateBuildAndRunService.validateBuildAndRunWorkspaceMigrationFromTo(
{
@@ -230,6 +238,11 @@ export class ApplicationManifestMigrationService {
dryRun,
},
);
const validateBuildRunMs = performance.now() - validateBuildRunStart;
this.logger.log(
`[install-perf] syncMetadataFromManifest validateBuildAndRunWorkspaceMigrationFromTo took ${validateBuildRunMs.toFixed(1)}ms (dryRun=${dryRun}, actions=${validateAndBuildResult.status === 'success' ? validateAndBuildResult.workspaceMigration.actions.length : 'n/a-failed'})`,
);
if (validateAndBuildResult.status === 'fail') {
throw new WorkspaceMigrationBuilderException(