We needed a centralized "job" processor for our CI runs, but continuously had issues with transactions across services:
INSERT INTO ci_jobs(...) VALUES (...) RETURNING id
redis-cli LPUSH $id
wasn't good enough, since a temporary inability to connect to redis would kill the run in a strange way.
Instead, I used postgres itself as the job server with PUBLISH / SUBSCRIBE and an atomic claiming mechanism using FOR UPDATE SKIP LOCKED using Postgres.
See the blog post below for more details:
36.5k views36.5k
Comments