## Context
The suspended-workspace cleanup is a long-running scheduled job. Under
database or cache pressure, BullMQ can consider an execution stalled and
start a replacement on another worker while the original execution is
still running.
Both executions can then enumerate the same suspended workspaces and run
destructive cleanup concurrently. This amplifies the initial slowdown:
1. Multiple cleanup transactions target the same workspace data.
2. Transactions wait on each other's locks.
3. Database connections remain occupied while waiting.
4. Other workers and API requests have fewer connections available.
There is a second source of unnecessary lock duration in workspace
deletion. The deletion transaction currently starts before field
metadata is read from the workspace cache. If that lookup is slow, the
transaction stays open during an unrelated cache wait.
## What changed
### Prevent overlapping scheduled cleanups
- Acquire a non-blocking PostgreSQL advisory lock before listing
suspended workspaces.
- Skip the execution when another worker already holds the lock.
- Keep the lock on one dedicated PostgreSQL session for the full
callback.
- Release the lock in all normal and error paths.
- Discard the database connection if lock acquisition or release has an
ambiguous failure, preventing a session that may still own the lock from
returning to the pool.
- Encapsulate this lifecycle in `PostgresAdvisoryLockService`, exported
by `TypeORMModule`, so other coarse-grained jobs can reuse it without
handling acquisition and release themselves.
### Shorten the workspace deletion transaction
- Read field metadata and build deletion chunks before starting the
transaction.
- Pass the precomputed chunks into the transactional deletion loop.
- Keep the existing deletion order and SQL behavior unchanged.
## Why a PostgreSQL advisory lock
The lock needs to coordinate workers running in different pods. A
PostgreSQL session advisory lock provides the required behavior:
- It is shared across all workers using the same database.
- Acquisition is non-blocking, a duplicate execution can exit
immediately.
- It has no TTL or renewal heartbeat that could expire during the same
event-loop stall that caused BullMQ to recover the job.
- PostgreSQL automatically releases it when the owning session or
process disappears.
This is deliberately scoped to `CleanSuspendedWorkspacesJob`. It
prevents overlapping scheduled executions, but it is not an exactly-once
mechanism or a global mutex around every workspace-deletion entry point.
## Expected impact
- Prevent one slow cleanup execution from becoming several concurrent
cleanup executions.
- Reduce database lock contention and connection-pool pressure during
cleanup.
- Avoid holding deletion transaction locks while waiting for
workspace-cache data.
- Reduce cleanup-related API latency bursts without changing normal
cleanup semantics.
The advisory lock holds one core database connection for the duration of
the scheduled cleanup. This is intentional and bounded to the single
lock owner.
## Validation
- Focused advisory-lock tests cover successful execution, contention,
callback failure, and unsafe connection disposal when unlock fails.
- Cleanup-job tests cover both the lock-owner and skipped-execution
paths.
- Workspace-service coverage verifies that field metadata is loaded
before the deletion transaction starts.
- `yarn nx typecheck twenty-server`
- Oxlint, Prettier, and Oxfmt checks on the changed files
Weiko
e4e1d24731