Alternatives to Sidekiq logo

Alternatives to Sidekiq

Resque, Celery, RabbitMQ, delayed_job, and Kafka are the most popular alternatives and competitors to Sidekiq.
1.1K
629
+ 1
408

What is Sidekiq and what are its top alternatives?

Sidekiq is a popular open-source job scheduler for Ruby that uses Redis to manage the job queue. It offers features such as worker resilience, scheduling, web interface, and active job support. However, one limitation of Sidekiq is that it requires a Redis server to function properly.

  1. Resque: Resque is a task queuing system built on Redis, known for its simple setup and maintenance. Key features include job prioritization, multiple queues, and failure handling. Pros include high reliability and flexibility, while cons include lack of real-time monitoring.
  2. Delayed::Job: Delayed::Job is a database-backed job scheduler for Ruby applications. It offers a simple and robust way to handle background jobs. Pros include easy setup and tight integration with Rails, while cons include potential database bloat.
  3. Que: Que is a high-performance job queue for Ruby applications that uses PostgreSQL. Key features include job persistence in the database and multi-threaded worker support. Pros include low overhead and transactional job processing, while cons include limited support for other databases.
  4. Sidekiq Pro: Sidekiq Pro is the commercial version of Sidekiq with additional features such as job batch processing, priority queues, and job tagging. Pros include enhanced functionality and professional support, while cons include the need for a paid license.
  5. Sneakers: Sneakers is a fast and simple background processing framework for Ruby applications using RabbitMQ. Key features include worker concurrency control, dead-letter handling, and batch processing. Pros include high performance and RabbitMQ integration, while cons include limited Redis support.
  6. GoodJob: GoodJob is a multithreaded, Postgres-based background job queuing system for Ruby on Rails applications. It offers features such as ActiveJob compatibility, job scheduling, and real-time monitoring. Pros include simplicity and scalability, while cons include limited database support.
  7. Queuery: Queuery is a simple, Redis-based job queue library for Ruby applications. It focuses on ease of use and minimal configuration while providing essential job queuing functionality. Pros include lightweight and easy integration, while cons include limited advanced features.
  8. Shoryuken: Shoryuken is a concurrent Amazon SQS client for Ruby that enables efficient background job processing. Key features include auto-scaling, pre-fetching, and dead-letter handling. Pros include high scalability and Amazon SQS integration, while cons include limited support for other messaging systems.
  9. Sucker Punch: Sucker Punch is a simple, single-threaded background processing library for Ruby applications. It is designed for lightweight jobs that do not require complex queuing systems. Pros include simplicity and zero-configuration setup, while cons include limited scalability and reliability.
  10. Backburner: Backburner is a multi-threaded and single-threaded job processing library for Ruby applications that supports beanstalkd and other queuing systems. Key features include job priorities, delayed jobs, and job scheduling. Pros include flexibility and queuing system support, while cons include potential complexity in setup and maintenance.

Top Alternatives to Sidekiq

  • Resque
    Resque

    Background jobs can be any Ruby class or module that responds to perform. Your existing classes can easily be converted to background jobs or you can create new classes specifically to do work. Or, you can do both. ...

  • Celery
    Celery

    Celery is an asynchronous task queue/job queue based on distributed message passing. It is focused on real-time operation, but supports scheduling as well. ...

  • RabbitMQ
    RabbitMQ

    RabbitMQ gives your applications a common platform to send and receive messages, and your messages a safe place to live until received. ...

  • delayed_job
    delayed_job

    Delayed_job (or DJ) encapsulates the common pattern of asynchronously executing longer tasks in the background. It is a direct extraction from Shopify where the job table is responsible for a multitude of core tasks. ...

  • Kafka
    Kafka

    Kafka is a distributed, partitioned, replicated commit log service. It provides the functionality of a messaging system, but with a unique design. ...

  • Amazon SQS
    Amazon SQS

    Transmit any volume of data, at any level of throughput, without losing messages or requiring other services to be always available. With SQS, you can offload the administrative burden of operating and scaling a highly available messaging cluster, while paying a low price for only what you use. ...

  • Faktory
    Faktory

    Redis -> Sidekiq == Faktory -> Faktory. Faktory is a server daemon which provides a simple API to produce and consume background jobs. Jobs are a small JSON hash with a few mandatory keys. ...

  • Redis
    Redis

    Redis is an open source (BSD licensed), in-memory data structure store, used as a database, cache, and message broker. Redis provides data structures such as strings, hashes, lists, sets, sorted sets with range queries, bitmaps, hyperloglogs, geospatial indexes, and streams. ...

Sidekiq alternatives & related posts

Resque logo

Resque

118
126
9
A Redis-backed Ruby library for creating background jobs, placing them on multiple queues, and processing them later
118
126
+ 1
9
PROS OF RESQUE
  • 5
    Free
  • 3
    Scalable
  • 1
    Easy to use on heroku
CONS OF RESQUE
    Be the first to leave a con

    related Resque posts

    Celery logo

    Celery

    1.6K
    1.6K
    280
    Distributed task queue
    1.6K
    1.6K
    + 1
    280
    PROS OF CELERY
    • 99
      Task queue
    • 63
      Python integration
    • 40
      Django integration
    • 30
      Scheduled Task
    • 19
      Publish/subsribe
    • 8
      Various backend broker
    • 6
      Easy to use
    • 5
      Great community
    • 5
      Workflow
    • 4
      Free
    • 1
      Dynamic
    CONS OF CELERY
    • 4
      Sometimes loses tasks
    • 1
      Depends on broker

    related Celery posts

    James Cunningham
    Operations Engineer at Sentry · | 21 upvotes · 356.6K views

    Sentry started as (and remains) an open-source project, growing out of an error logging tool built in 2008. That original build nine years ago was Django and Celery (Python’s asynchronous task codebase), with PostgreSQL as the database and Redis as the power behind Celery.

    We displayed a truly shrewd notion of branding even then, giving the project a catchy name that companies the world over remain jealous of to this day: django-db-log. For the longest time, Sentry’s subtitle on GitHub was “A simple Django app, built with love.” A slightly more accurate description probably would have included Starcraft and Soylent alongside love; regardless, this captured what Sentry was all about.

    #MessageQueue #InMemoryDatabases

    See more
    James Cunningham
    Operations Engineer at Sentry · | 18 upvotes · 1.7M views
    Shared insights
    on
    CeleryCeleryRabbitMQRabbitMQ
    at

    As Sentry runs throughout the day, there are about 50 different offline tasks that we execute—anything from “process this event, pretty please” to “send all of these cool people some emails.” There are some that we execute once a day and some that execute thousands per second.

    Managing this variety requires a reliably high-throughput message-passing technology. We use Celery's RabbitMQ implementation, and we stumbled upon a great feature called Federation that allows us to partition our task queue across any number of RabbitMQ servers and gives us the confidence that, if any single server gets backlogged, others will pitch in and distribute some of the backlogged tasks to their consumers.

    #MessageQueue

    See more
    RabbitMQ logo

    RabbitMQ

    20.9K
    18.4K
    527
    Open source multiprotocol messaging broker
    20.9K
    18.4K
    + 1
    527
    PROS OF RABBITMQ
    • 234
      It's fast and it works with good metrics/monitoring
    • 79
      Ease of configuration
    • 59
      I like the admin interface
    • 50
      Easy to set-up and start with
    • 21
      Durable
    • 18
      Intuitive work through python
    • 18
      Standard protocols
    • 10
      Written primarily in Erlang
    • 8
      Simply superb
    • 6
      Completeness of messaging patterns
    • 3
      Scales to 1 million messages per second
    • 3
      Reliable
    • 2
      Distributed
    • 2
      Supports MQTT
    • 2
      Better than most traditional queue based message broker
    • 2
      Supports AMQP
    • 1
      Clusterable
    • 1
      Clear documentation with different scripting language
    • 1
      Great ui
    • 1
      Inubit Integration
    • 1
      Better routing system
    • 1
      High performance
    • 1
      Runs on Open Telecom Platform
    • 1
      Delayed messages
    • 1
      Reliability
    • 1
      Open-source
    CONS OF RABBITMQ
    • 9
      Too complicated cluster/HA config and management
    • 6
      Needs Erlang runtime. Need ops good with Erlang runtime
    • 5
      Configuration must be done first, not by your code
    • 4
      Slow

    related RabbitMQ posts

    James Cunningham
    Operations Engineer at Sentry · | 18 upvotes · 1.7M views
    Shared insights
    on
    CeleryCeleryRabbitMQRabbitMQ
    at

    As Sentry runs throughout the day, there are about 50 different offline tasks that we execute—anything from “process this event, pretty please” to “send all of these cool people some emails.” There are some that we execute once a day and some that execute thousands per second.

    Managing this variety requires a reliably high-throughput message-passing technology. We use Celery's RabbitMQ implementation, and we stumbled upon a great feature called Federation that allows us to partition our task queue across any number of RabbitMQ servers and gives us the confidence that, if any single server gets backlogged, others will pitch in and distribute some of the backlogged tasks to their consumers.

    #MessageQueue

    See more

    Around the time of their Series A, Pinterest’s stack included Python and Django, with Tornado and Node.js as web servers. Memcached / Membase and Redis handled caching, with RabbitMQ handling queueing. Nginx, HAproxy and Varnish managed static-delivery and load-balancing, with persistent data storage handled by MySQL.

    See more
    delayed_job logo

    delayed_job

    51
    64
    6
    Database backed asynchronous priority queue -- Extracted from Shopify
    51
    64
    + 1
    6
    PROS OF DELAYED_JOB
    • 3
      Easy to get started
    • 2
      Reliable
    • 1
      Doesn't require Redis
    CONS OF DELAYED_JOB
      Be the first to leave a con

      related delayed_job posts

      John Barton

      Docker Compose might have been a bit of overkill for a dev environment as a solo founder, but I'd found so much with past side projects (though this is no longer a side project) that I would frequently waste time every time I came back to work on the project getting my dev env sorted again.

      Made the conscious choice to make a "prod-ish" docker-compose config up front to make sure that didn't bite me again.

      Structured it so I have the following containers running

      • server - the Rails app in API style
      • client - the Create React App
      • ngrok - ngrok to receive webhooks in dev
      • db - PostgreSQL
      • queues - delayed_job worker
      See more
      Jerome Dalbert
      Principal Backend Software Engineer at StackShare · | 4 upvotes · 86.7K views

      delayed_job is a great Rails background job library for new projects, as it only uses what you already have: a relational database. We happily used it during the company’s first two years.

      But it started to falter as our web and database transactions significantly grew. Our app interacted with users via SMS texts sent inside background jobs. Because the delayed_job daemon ran every couple seconds, this meant that users often waited several long seconds before getting text replies, which was not acceptable. Moreover, job processing was done inside AWS Elastic Beanstalk web instances, which were already under stress and not meant to handle jobs.

      We needed a fast background job system that could process jobs in near real-time and integrate well with AWS. Sidekiq is a fast and popular Ruby background job library, but it does not leverage the Elastic Beanstalk worker architecture, and you have to maintain a Redis instance.

      We ended up choosing active-elastic-job, which seamlessly integrates with worker instances and Amazon SQS. SQS is a fast queue and you don’t need to worry about infrastructure or scaling, as AWS handles it for you.

      We noticed significant performance gains immediately after making the switch.

      #BackgroundProcessing

      See more
      Kafka logo

      Kafka

      23.1K
      21.7K
      607
      Distributed, fault tolerant, high throughput pub-sub messaging system
      23.1K
      21.7K
      + 1
      607
      PROS OF KAFKA
      • 126
        High-throughput
      • 119
        Distributed
      • 92
        Scalable
      • 86
        High-Performance
      • 66
        Durable
      • 38
        Publish-Subscribe
      • 19
        Simple-to-use
      • 18
        Open source
      • 12
        Written in Scala and java. Runs on JVM
      • 9
        Message broker + Streaming system
      • 4
        KSQL
      • 4
        Avro schema integration
      • 4
        Robust
      • 3
        Suport Multiple clients
      • 2
        Extremely good parallelism constructs
      • 2
        Partioned, replayable log
      • 1
        Simple publisher / multi-subscriber model
      • 1
        Fun
      • 1
        Flexible
      CONS OF KAFKA
      • 32
        Non-Java clients are second-class citizens
      • 29
        Needs Zookeeper
      • 9
        Operational difficulties
      • 5
        Terrible Packaging

      related Kafka posts

      Nick Rockwell
      SVP, Engineering at Fastly · | 46 upvotes · 3.4M views

      When I joined NYT there was already broad dissatisfaction with the LAMP (Linux Apache HTTP Server MySQL PHP) Stack and the front end framework, in particular. So, I wasn't passing judgment on it. I mean, LAMP's fine, you can do good work in LAMP. It's a little dated at this point, but it's not ... I didn't want to rip it out for its own sake, but everyone else was like, "We don't like this, it's really inflexible." And I remember from being outside the company when that was called MIT FIVE when it had launched. And been observing it from the outside, and I was like, you guys took so long to do that and you did it so carefully, and yet you're not happy with your decisions. Why is that? That was more the impetus. If we're going to do this again, how are we going to do it in a way that we're gonna get a better result?

      So we're moving quickly away from LAMP, I would say. So, right now, the new front end is React based and using Apollo. And we've been in a long, protracted, gradual rollout of the core experiences.

      React is now talking to GraphQL as a primary API. There's a Node.js back end, to the front end, which is mainly for server-side rendering, as well.

      Behind there, the main repository for the GraphQL server is a big table repository, that we call Bodega because it's a convenience store. And that reads off of a Kafka pipeline.

      See more
      Ashish Singh
      Tech Lead, Big Data Platform at Pinterest · | 38 upvotes · 3M views

      To provide employees with the critical need of interactive querying, we’ve worked with Presto, an open-source distributed SQL query engine, over the years. Operating Presto at Pinterest’s scale has involved resolving quite a few challenges like, supporting deeply nested and huge thrift schemas, slow/ bad worker detection and remediation, auto-scaling cluster, graceful cluster shutdown and impersonation support for ldap authenticator.

      Our infrastructure is built on top of Amazon EC2 and we leverage Amazon S3 for storing our data. This separates compute and storage layers, and allows multiple compute clusters to share the S3 data.

      We have hundreds of petabytes of data and tens of thousands of Apache Hive tables. Our Presto clusters are comprised of a fleet of 450 r4.8xl EC2 instances. Presto clusters together have over 100 TBs of memory and 14K vcpu cores. Within Pinterest, we have close to more than 1,000 monthly active users (out of total 1,600+ Pinterest employees) using Presto, who run about 400K queries on these clusters per month.

      Each query submitted to Presto cluster is logged to a Kafka topic via Singer. Singer is a logging agent built at Pinterest and we talked about it in a previous post. Each query is logged when it is submitted and when it finishes. When a Presto cluster crashes, we will have query submitted events without corresponding query finished events. These events enable us to capture the effect of cluster crashes over time.

      Each Presto cluster at Pinterest has workers on a mix of dedicated AWS EC2 instances and Kubernetes pods. Kubernetes platform provides us with the capability to add and remove workers from a Presto cluster very quickly. The best-case latency on bringing up a new worker on Kubernetes is less than a minute. However, when the Kubernetes cluster itself is out of resources and needs to scale up, it can take up to ten minutes. Some other advantages of deploying on Kubernetes platform is that our Presto deployment becomes agnostic of cloud vendor, instance types, OS, etc.

      #BigData #AWS #DataScience #DataEngineering

      See more
      Amazon SQS logo

      Amazon SQS

      2.2K
      2K
      171
      Fully managed message queuing service
      2.2K
      2K
      + 1
      171
      PROS OF AMAZON SQS
      • 62
        Easy to use, reliable
      • 40
        Low cost
      • 28
        Simple
      • 14
        Doesn't need to maintain it
      • 8
        It is Serverless
      • 4
        Has a max message size (currently 256K)
      • 3
        Triggers Lambda
      • 3
        Easy to configure with Terraform
      • 3
        Delayed delivery upto 15 mins only
      • 3
        Delayed delivery upto 12 hours
      • 1
        JMS compliant
      • 1
        Support for retry and dead letter queue
      • 1
        D
      CONS OF AMAZON SQS
      • 2
        Has a max message size (currently 256K)
      • 2
        Proprietary
      • 2
        Difficult to configure
      • 1
        Has a maximum 15 minutes of delayed messages only

      related Amazon SQS posts

      Jeyabalaji Subramanian

      Recently we were looking at a few robust and cost-effective ways of replicating the data that resides in our production MongoDB to a PostgreSQL database for data warehousing and business intelligence.

      We set ourselves the following criteria for the optimal tool that would do this job: - The data replication must be near real-time, yet it should NOT impact the production database - The data replication must be horizontally scalable (based on the load), asynchronous & crash-resilient

      Based on the above criteria, we selected the following tools to perform the end to end data replication:

      We chose MongoDB Stitch for picking up the changes in the source database. It is the serverless platform from MongoDB. One of the services offered by MongoDB Stitch is Stitch Triggers. Using stitch triggers, you can execute a serverless function (in Node.js) in real time in response to changes in the database. When there are a lot of database changes, Stitch automatically "feeds forward" these changes through an asynchronous queue.

      We chose Amazon SQS as the pipe / message backbone for communicating the changes from MongoDB to our own replication service. Interestingly enough, MongoDB stitch offers integration with AWS services.

      In the Node.js function, we wrote minimal functionality to communicate the database changes (insert / update / delete / replace) to Amazon SQS.

      Next we wrote a minimal micro-service in Python to listen to the message events on SQS, pickup the data payload & mirror the DB changes on to the target Data warehouse. We implemented source data to target data translation by modelling target table structures through SQLAlchemy . We deployed this micro-service as AWS Lambda with Zappa. With Zappa, deploying your services as event-driven & horizontally scalable Lambda service is dumb-easy.

      In the end, we got to implement a highly scalable near realtime Change Data Replication service that "works" and deployed to production in a matter of few days!

      See more
      Praveen Mooli
      Engineering Manager at Taylor and Francis · | 18 upvotes · 3.8M views

      We are in the process of building a modern content platform to deliver our content through various channels. We decided to go with Microservices architecture as we wanted scale. Microservice architecture style is an approach to developing an application as a suite of small independently deployable services built around specific business capabilities. You can gain modularity, extensive parallelism and cost-effective scaling by deploying services across many distributed servers. Microservices modularity facilitates independent updates/deployments, and helps to avoid single point of failure, which can help prevent large-scale outages. We also decided to use Event Driven Architecture pattern which is a popular distributed asynchronous architecture pattern used to produce highly scalable applications. The event-driven architecture is made up of highly decoupled, single-purpose event processing components that asynchronously receive and process events.

      To build our #Backend capabilities we decided to use the following: 1. #Microservices - Java with Spring Boot , Node.js with ExpressJS and Python with Flask 2. #Eventsourcingframework - Amazon Kinesis , Amazon Kinesis Firehose , Amazon SNS , Amazon SQS, AWS Lambda 3. #Data - Amazon RDS , Amazon DynamoDB , Amazon S3 , MongoDB Atlas

      To build #Webapps we decided to use Angular 2 with RxJS

      #Devops - GitHub , Travis CI , Terraform , Docker , Serverless

      See more
      Faktory logo

      Faktory

      6
      27
      3
      Background jobs for any language, by the makers of Sidekiq
      6
      27
      + 1
      3
      PROS OF FAKTORY
      • 2
        Worker language agnostic
      • 1
        Simple service API
      CONS OF FAKTORY
        Be the first to leave a con

        related Faktory posts

        Redis logo

        Redis

        58.4K
        45K
        3.9K
        Open source (BSD licensed), in-memory data structure store
        58.4K
        45K
        + 1
        3.9K
        PROS OF REDIS
        • 886
          Performance
        • 542
          Super fast
        • 513
          Ease of use
        • 444
          In-memory cache
        • 324
          Advanced key-value cache
        • 194
          Open source
        • 182
          Easy to deploy
        • 164
          Stable
        • 155
          Free
        • 121
          Fast
        • 42
          High-Performance
        • 40
          High Availability
        • 35
          Data Structures
        • 32
          Very Scalable
        • 24
          Replication
        • 22
          Great community
        • 22
          Pub/Sub
        • 19
          "NoSQL" key-value data store
        • 16
          Hashes
        • 13
          Sets
        • 11
          Sorted Sets
        • 10
          NoSQL
        • 10
          Lists
        • 9
          Async replication
        • 9
          BSD licensed
        • 8
          Bitmaps
        • 8
          Integrates super easy with Sidekiq for Rails background
        • 7
          Keys with a limited time-to-live
        • 7
          Open Source
        • 6
          Lua scripting
        • 6
          Strings
        • 5
          Awesomeness for Free
        • 5
          Hyperloglogs
        • 4
          Transactions
        • 4
          Outstanding performance
        • 4
          Runs server side LUA
        • 4
          LRU eviction of keys
        • 4
          Feature Rich
        • 4
          Written in ANSI C
        • 4
          Networked
        • 3
          Data structure server
        • 3
          Performance & ease of use
        • 2
          Dont save data if no subscribers are found
        • 2
          Automatic failover
        • 2
          Easy to use
        • 2
          Temporarily kept on disk
        • 2
          Scalable
        • 2
          Existing Laravel Integration
        • 2
          Channels concept
        • 2
          Object [key/value] size each 500 MB
        • 2
          Simple
        CONS OF REDIS
        • 15
          Cannot query objects directly
        • 3
          No secondary indexes for non-numeric data types
        • 1
          No WAL

        related Redis posts

        Russel Werner
        Lead Engineer at StackShare · | 32 upvotes · 2.2M views

        StackShare Feed is built entirely with React, Glamorous, and Apollo. One of our objectives with the public launch of the Feed was to enable a Server-side rendered (SSR) experience for our organic search traffic. When you visit the StackShare Feed, and you aren't logged in, you are delivered the Trending feed experience. We use an in-house Node.js rendering microservice to generate this HTML. This microservice needs to run and serve requests independent of our Rails web app. Up until recently, we had a mono-repo with our Rails and React code living happily together and all served from the same web process. In order to deploy our SSR app into a Heroku environment, we needed to split out our front-end application into a separate repo in GitHub. The driving factor in this decision was mostly due to limitations imposed by Heroku specifically with how processes can't communicate with each other. A new SSR app was created in Heroku and linked directly to the frontend repo so it stays in-sync with changes.

        Related to this, we need a way to "deploy" our frontend changes to various server environments without building & releasing the entire Ruby application. We built a hybrid Amazon S3 Amazon CloudFront solution to host our Webpack bundles. A new CircleCI script builds the bundles and uploads them to S3. The final step in our rollout is to update some keys in Redis so our Rails app knows which bundles to serve. The result of these efforts were significant. Our frontend team now moves independently of our backend team, our build & release process takes only a few minutes, we are now using an edge CDN to serve JS assets, and we have pre-rendered React pages!

        #StackDecisionsLaunch #SSR #Microservices #FrontEndRepoSplit

        See more
        Simon Reymann
        Senior Fullstack Developer at QUANTUSflow Software GmbH · | 30 upvotes · 9.3M views

        Our whole DevOps stack consists of the following tools:

        • GitHub (incl. GitHub Pages/Markdown for Documentation, GettingStarted and HowTo's) for collaborative review and code management tool
        • Respectively Git as revision control system
        • SourceTree as Git GUI
        • Visual Studio Code as IDE
        • CircleCI for continuous integration (automatize development process)
        • Prettier / TSLint / ESLint as code linter
        • SonarQube as quality gate
        • Docker as container management (incl. Docker Compose for multi-container application management)
        • VirtualBox for operating system simulation tests
        • Kubernetes as cluster management for docker containers
        • Heroku for deploying in test environments
        • nginx as web server (preferably used as facade server in production environment)
        • SSLMate (using OpenSSL) for certificate management
        • Amazon EC2 (incl. Amazon S3) for deploying in stage (production-like) and production environments
        • PostgreSQL as preferred database system
        • Redis as preferred in-memory database/store (great for caching)

        The main reason we have chosen Kubernetes over Docker Swarm is related to the following artifacts:

        • Key features: Easy and flexible installation, Clear dashboard, Great scaling operations, Monitoring is an integral part, Great load balancing concepts, Monitors the condition and ensures compensation in the event of failure.
        • Applications: An application can be deployed using a combination of pods, deployments, and services (or micro-services).
        • Functionality: Kubernetes as a complex installation and setup process, but it not as limited as Docker Swarm.
        • Monitoring: It supports multiple versions of logging and monitoring when the services are deployed within the cluster (Elasticsearch/Kibana (ELK), Heapster/Grafana, Sysdig cloud integration).
        • Scalability: All-in-one framework for distributed systems.
        • Other Benefits: Kubernetes is backed by the Cloud Native Computing Foundation (CNCF), huge community among container orchestration tools, it is an open source and modular tool that works with any OS.
        See more