Fix workflow cron trigger cache stuck without TTL (#20812)
## Problem The cron-trigger cache key (`module:workflow:workflow-cron-triggers`) can get stuck without a TTL, silently halting **all** cron-triggered workflows for a whole tenant until the key is manually deleted from Redis. Repro path: 1. Cache miss → DB-scan branch runs. 2. Inner loop writes triggers via `hashSet` (creates the key, **no TTL yet**). 3. Worker crashes / OOMs / gets killed by a deploy between any `hashSet` and the trailing `expire(1h)` call. 4. Key now exists with TTL = `-1` and a partial set of fields. 5. Next tick: `hashGetValues` returns those fields → `cachedValues.length > 0` → **cache-hit branch** → `expire` is never called. 6. Key has no TTL, so it never auto-expires. The DB-scan branch never runs again. New / missing triggers are never picked up. Workflows go silent. Observed in production: cache key with `TTL: no limit` and 121 fields. Deleting the key restored normal behaviour (next tick rebuilt with TTL ~3600). ## Fix Set the TTL right after first value is added ## Monitoring Added a "Cache miss" log count in workflow dashboard, counted among the last 6 hours. Turns green if >= 5 <img width="1627" height="721" alt="Screenshot 2026-05-21 at 16 46 13" src="https://github.com/user-attachments/assets/8262dd5f-fbbd-43c9-aede-c0ce5d6a0f59" /> --------- Co-authored-by: Cursor <cursoragent@cursor.com>
This commit is contained in:
+25
@@ -342,6 +342,31 @@ end`;
|
||||
}) as Promise<number>;
|
||||
}
|
||||
|
||||
async hashSetWithExpire({
|
||||
key,
|
||||
field,
|
||||
value,
|
||||
ttlMs,
|
||||
}: {
|
||||
key: string;
|
||||
field: string;
|
||||
value: string;
|
||||
ttlMs: Milliseconds;
|
||||
}): Promise<void> {
|
||||
if (!this.isRedisCache()) {
|
||||
throw new Error('hashSetWithExpire is only supported with Redis cache');
|
||||
}
|
||||
|
||||
const redisClient = (this.cache as RedisCache).store.client;
|
||||
const prefixedKey = this.getKey(key);
|
||||
|
||||
await redisClient
|
||||
.multi()
|
||||
.hSet(prefixedKey, field, value)
|
||||
.pExpire(prefixedKey, ttlMs)
|
||||
.exec();
|
||||
}
|
||||
|
||||
async hashDelete({
|
||||
key,
|
||||
field,
|
||||
|
||||
Reference in New Issue
Block a user