Founder at Distributed Containers Inc.·

We needed a centralized "job" processor for our CI runs, but continuously had issues with transactions across services:

INSERT INTO ci_jobs(...) VALUES (...) RETURNING id

redis-cli LPUSH $id

wasn't good enough, since a temporary inability to connect to redis would kill the run in a strange way.

Instead, I used postgres itself as the job server with PUBLISH / SUBSCRIBE and an atomic claiming mechanism using FOR UPDATE SKIP LOCKED using Postgres.

See the blog post below for more details:

READ LESS
System design hack: Postgres is a great pub/sub & job server (layerci.com)
4 upvotes·34.8K views
Avatar of Colin Chartier

Colin Chartier

Founder at Distributed Containers Inc.