chore(twenty-server): temporary instrumentation for app-install 504 (#21365)
## Why App installs on cloud intermittently fail with a 504, surfacing in Sentry as `Migration action 'update' for 'logicFunction' failed` + `Failed to rollback transaction: Query runner already released`. This is **temporary instrumentation** to pin down where the time goes — to be reverted once the bottleneck is fixed. Everything is greppable via `[install-perf]` and marked `// TODO(install-perf)`. ## What the local repro already told us I instrumented the manifest-sync/migration path and ran a local harness (new skipped spec) installing **1 / 8 / 30 logic functions**, for both create and the checksum-bump **update** (the incident path): | stage (N=30, update) | ms | |---|---| | flat-maps recompute | ~1 | | build migration | ~11 | | transaction (all actions + commit) | ~79 | | post-commit cache invalidate | ~6 | | **full sync** | **~135** | Nothing approached 1s, let alone 10s; no slow queries logged. So the migration/cache code is **not** the algorithmic cause. Given the in-transaction `UPDATE ... WHERE id=?` is intrinsically fast, a >10s in prod almost certainly means it was **blocked on a lock**, and the 10s node-pg `query_timeout` (`core.datasource.ts`) then killed the connection → the observed errors + 504. Local can't reproduce prod lock contention / table sizes, hence this instrumentation. ## What this adds (all `TODO`-marked) - **hrtime per-stage timing** — flat-maps recompute, build vs run, per-action (`>50ms`), transaction summary, post-commit cache invalidation. Uses `process.hrtime` because the integration harness enables fake timers (so `Date.now()` is useless there). - **`maxQueryExecutionTime`** slow-query logging on the core datasource (logs the offending SQL). - **Scoped `SET LOCAL lock_timeout = '8s'`** on the migration transaction (below the 10s `query_timeout`) → a blocked action fails fast with a clear *"canceling statement due to lock timeout"* instead of the opaque connection kill. - **Best-effort `pg_stat_activity` snapshot on failure** (on a fresh pooled connection) to identify the blocking session, plus a **guarded rollback** so a released connection stops masking the real error. - **Skipped local perf harness** (`logic-function-install-performance.integration-spec.ts`) — run manually with `nx test:integration:with-db-reset -- --testPathPattern "logic-function-install-performance"`. ## How we'll use it Deploy, reproduce the failing install, and read the `[install-perf]` logs: the per-action timing names the action, the `lock_timeout` message + `pg_stat_activity` snapshot name the **blocking** query/PID. Then revert this PR and fix the actual contention. Typecheck (`nx typecheck twenty-server`) is clean.
This commit is contained in:
+26
@@ -369,6 +369,8 @@ export class WorkspaceMigrationValidateBuildAndRunService {
|
||||
const { idByUniversalIdentifierByMetadataName, dryRun, ...buildArgs } =
|
||||
args;
|
||||
|
||||
// TODO(install-perf): temporary, remove.
|
||||
const buildStart = performance.now();
|
||||
const validateAndBuildResult =
|
||||
await this.workspaceMigrationBuildOrchestratorService
|
||||
.buildWorkspaceMigration(buildArgs)
|
||||
@@ -379,6 +381,11 @@ export class WorkspaceMigrationValidateBuildAndRunService {
|
||||
WorkspaceMigrationV2ExceptionCode.BUILDER_INTERNAL_SERVER_ERROR,
|
||||
);
|
||||
});
|
||||
const buildMs = performance.now() - buildStart;
|
||||
|
||||
this.logger.log(
|
||||
`[install-perf] buildWorkspaceMigration took ${buildMs.toFixed(1)}ms (status=${validateAndBuildResult.status})`,
|
||||
);
|
||||
|
||||
if (validateAndBuildResult.status === 'fail') {
|
||||
if (this.isDebugEnabled) {
|
||||
@@ -402,11 +409,30 @@ export class WorkspaceMigrationValidateBuildAndRunService {
|
||||
};
|
||||
}
|
||||
|
||||
const actionCountsByTypeAndMetadataName: Record<string, number> = {};
|
||||
|
||||
for (const action of workspaceMigration.actions) {
|
||||
const key = `${action.type}:${action.metadataName}`;
|
||||
|
||||
actionCountsByTypeAndMetadataName[key] =
|
||||
(actionCountsByTypeAndMetadataName[key] ?? 0) + 1;
|
||||
}
|
||||
|
||||
this.logger.log(
|
||||
`[install-perf] validateBuildAndRunWorkspaceMigrationFromTo running ${workspaceMigration.actions.length} actions: ${JSON.stringify(actionCountsByTypeAndMetadataName)}`,
|
||||
);
|
||||
|
||||
const runStart = performance.now();
|
||||
const { hasSchemaMetadataChanged, metadataEvents } =
|
||||
await this.workspaceMigrationRunnerService.run({
|
||||
workspaceId: args.workspaceId,
|
||||
workspaceMigration,
|
||||
});
|
||||
const runMs = performance.now() - runStart;
|
||||
|
||||
this.logger.log(
|
||||
`[install-perf] workspaceMigrationRunnerService.run took ${runMs.toFixed(1)}ms for ${workspaceMigration.actions.length} actions`,
|
||||
);
|
||||
|
||||
this.metadataEventEmitter.emitMetadataEvents({
|
||||
metadataEvents: metadataEvents,
|
||||
|
||||
Reference in New Issue
Block a user