Two workers drained the same job queue without claiming rows first — when they raced, both executed the same job. A client received the welcome email twice.
processWorkflowQueue runs from both Inngest workflowQueueDrain and api/billing-cron.ts. When they overlap in time, both see the same queue row.
The old code plain-SELECTed pending rows, executed the action, then marked it completed. Between SELECT and completion, a second drain could read and run the same row.
The fix uses UPDATE workflow_queue SET status='processing', attempts=attempts+1, claimed_at=NOW() … RETURNING * wrapped in a SELECT … FOR UPDATE SKIP LOCKED subquery. The second drain sees different rows instead of re-running ours.
Rows abandoned in processing by a crashed drain (deploy, OOM, timeout) return to pending after 10 minutes via the new claimed_at column.
expect(SRC).not.toMatch(/SELECT \* FROM workflow_queue\s*\n\s*WHERE status = 'pending'/)
UPDATE workflow_queue SET status = 'processing', attempts = attempts + 1, claimed_at = NOW()
status = 'completed' (no attempts increment here)
CASE WHEN attempts >= 3 THEN 'failed' ELSE 'pending' END
Real impact: A newly created client received the welcome email twice, one second apart. Every job in this queue is a side effect on a real person — emails, invoice triggers, SMS. Running one twice is not cosmetic.