chore(twenty-server): temporary instrumentation for app-install 504 (#21365)
## Why App installs on cloud intermittently fail with a 504, surfacing in Sentry as `Migration action 'update' for 'logicFunction' failed` + `Failed to rollback transaction: Query runner already released`. This is **temporary instrumentation** to pin down where the time goes — to be reverted once the bottleneck is fixed. Everything is greppable via `[install-perf]` and marked `// TODO(install-perf)`. ## What the local repro already told us I instrumented the manifest-sync/migration path and ran a local harness (new skipped spec) installing **1 / 8 / 30 logic functions**, for both create and the checksum-bump **update** (the incident path): | stage (N=30, update) | ms | |---|---| | flat-maps recompute | ~1 | | build migration | ~11 | | transaction (all actions + commit) | ~79 | | post-commit cache invalidate | ~6 | | **full sync** | **~135** | Nothing approached 1s, let alone 10s; no slow queries logged. So the migration/cache code is **not** the algorithmic cause. Given the in-transaction `UPDATE ... WHERE id=?` is intrinsically fast, a >10s in prod almost certainly means it was **blocked on a lock**, and the 10s node-pg `query_timeout` (`core.datasource.ts`) then killed the connection → the observed errors + 504. Local can't reproduce prod lock contention / table sizes, hence this instrumentation. ## What this adds (all `TODO`-marked) - **hrtime per-stage timing** — flat-maps recompute, build vs run, per-action (`>50ms`), transaction summary, post-commit cache invalidation. Uses `process.hrtime` because the integration harness enables fake timers (so `Date.now()` is useless there). - **`maxQueryExecutionTime`** slow-query logging on the core datasource (logs the offending SQL). - **Scoped `SET LOCAL lock_timeout = '8s'`** on the migration transaction (below the 10s `query_timeout`) → a blocked action fails fast with a clear *"canceling statement due to lock timeout"* instead of the opaque connection kill. - **Best-effort `pg_stat_activity` snapshot on failure** (on a fresh pooled connection) to identify the blocking session, plus a **guarded rollback** so a released connection stops masking the real error. - **Skipped local perf harness** (`logic-function-install-performance.integration-spec.ts`) — run manually with `nx test:integration:with-db-reset -- --testPathPattern "logic-function-install-performance"`. ## How we'll use it Deploy, reproduce the failing install, and read the `[install-perf]` logs: the per-action timing names the action, the `lock_timeout` message + `pg_stat_activity` snapshot name the **blocking** query/PID. Then revert this PR and fix the actual contention. Typecheck (`nx typecheck twenty-server`) is clean.
This commit is contained in:
+13
@@ -179,6 +179,8 @@ export class ApplicationManifestMigrationService {
|
||||
{ workspaceId },
|
||||
);
|
||||
|
||||
// TODO(install-perf): temporary, remove.
|
||||
const recomputeStart = performance.now();
|
||||
const cacheResult = await this.workspaceCacheService.getOrRecompute(
|
||||
workspaceId,
|
||||
[
|
||||
@@ -186,6 +188,11 @@ export class ApplicationManifestMigrationService {
|
||||
'featureFlagsMap',
|
||||
],
|
||||
);
|
||||
const recomputeMs = performance.now() - recomputeStart;
|
||||
|
||||
this.logger.log(
|
||||
`[install-perf] syncMetadataFromManifest ALL_METADATA_NAME getOrRecompute flat-maps took ${recomputeMs.toFixed(1)}ms (logicFunctions=${manifest.logicFunctions.length})`,
|
||||
);
|
||||
|
||||
const { featureFlagsMap, ...existingAllFlatEntityMaps } = cacheResult;
|
||||
|
||||
@@ -211,6 +218,7 @@ export class ApplicationManifestMigrationService {
|
||||
fromAllFlatEntityMaps: existingAllFlatEntityMaps,
|
||||
});
|
||||
|
||||
const validateBuildRunStart = performance.now();
|
||||
const validateAndBuildResult =
|
||||
await this.workspaceMigrationValidateBuildAndRunService.validateBuildAndRunWorkspaceMigrationFromTo(
|
||||
{
|
||||
@@ -230,6 +238,11 @@ export class ApplicationManifestMigrationService {
|
||||
dryRun,
|
||||
},
|
||||
);
|
||||
const validateBuildRunMs = performance.now() - validateBuildRunStart;
|
||||
|
||||
this.logger.log(
|
||||
`[install-perf] syncMetadataFromManifest validateBuildAndRunWorkspaceMigrationFromTo took ${validateBuildRunMs.toFixed(1)}ms (dryRun=${dryRun}, actions=${validateAndBuildResult.status === 'success' ? validateAndBuildResult.workspaceMigration.actions.length : 'n/a-failed'})`,
|
||||
);
|
||||
|
||||
if (validateAndBuildResult.status === 'fail') {
|
||||
throw new WorkspaceMigrationBuilderException(
|
||||
|
||||
Reference in New Issue
Block a user