Alternatives to Beanstalkd logo

Alternatives to Beanstalkd

RabbitMQ, Redis, Resque, Kafka, and Gearman are the most popular alternatives and competitors to Beanstalkd.
105
128
+ 1
74

What is Beanstalkd and what are its top alternatives?

Beanstalks's interface is generic, but was originally designed for reducing the latency of page views in high-volume web applications by running time-consuming tasks asynchronously.
Beanstalkd is a tool in the Background Processing category of a tech stack.
Beanstalkd is an open source tool with 5.9K GitHub stars and 826 GitHub forks. Here’s a link to Beanstalkd's open source repository on GitHub

Top Alternatives to Beanstalkd

  • RabbitMQ

    RabbitMQ

    RabbitMQ gives your applications a common platform to send and receive messages, and your messages a safe place to live until received. ...

  • Redis

    Redis

    Redis is an open source, BSD licensed, advanced key-value store. It is often referred to as a data structure server since keys can contain strings, hashes, lists, sets and sorted sets. ...

  • Resque

    Resque

    Background jobs can be any Ruby class or module that responds to perform. Your existing classes can easily be converted to background jobs or you can create new classes specifically to do work. Or, you can do both. ...

  • Kafka

    Kafka

    Kafka is a distributed, partitioned, replicated commit log service. It provides the functionality of a messaging system, but with a unique design. ...

  • Gearman

    Gearman

    Gearman allows you to do work in parallel, to load balance processing, and to call functions between languages. It can be used in a variety of applications, from high-availability web sites to the transport of database replication events. ...

  • Celery

    Celery

    Celery is an asynchronous task queue/job queue based on distributed message passing. It is focused on real-time operation, but supports scheduling as well. ...

  • ZeroMQ

    ZeroMQ

    The 0MQ lightweight messaging kernel is a library which extends the standard socket interfaces with features traditionally provided by specialised messaging middleware products. 0MQ sockets provide an abstraction of asynchronous message queues, multiple messaging patterns, message filtering (subscriptions), seamless access to multiple transport protocols and more. ...

  • Sidekiq

    Sidekiq

    Sidekiq uses threads to handle many jobs at the same time in the same process. It does not require Rails but will integrate tightly with Rails 3/4 to make background processing dead simple. ...

Beanstalkd alternatives & related posts

RabbitMQ logo

RabbitMQ

13.6K
11.7K
511
Open source multiprotocol messaging broker
13.6K
11.7K
+ 1
511
PROS OF RABBITMQ
  • 226
    It's fast and it works with good metrics/monitoring
  • 79
    Ease of configuration
  • 57
    I like the admin interface
  • 49
    Easy to set-up and start with
  • 20
    Durable
  • 18
    Intuitive work through python
  • 18
    Standard protocols
  • 10
    Written primarily in Erlang
  • 7
    Simply superb
  • 6
    Completeness of messaging patterns
  • 3
    Reliable
  • 3
    Scales to 1 million messages per second
  • 2
    Distributed
  • 2
    Supports AMQP
  • 2
    Better than most traditional queue based message broker
  • 1
    High performance
  • 1
    Reliability
  • 1
    Clusterable
  • 1
    Inubit Integration
  • 1
    Clear documentation with different scripting language
  • 1
    Great ui
  • 1
    Runs on Open Telecom Platform
  • 1
    Better routing system
  • 1
    Supports MQTT
CONS OF RABBITMQ
  • 9
    Too complicated cluster/HA config and management
  • 6
    Needs Erlang runtime. Need ops good with Erlang runtime
  • 5
    Configuration must be done first, not by your code
  • 4
    Slow

related RabbitMQ posts

James Cunningham
Operations Engineer at Sentry · | 18 upvotes · 1.2M views
Shared insights
on
Celery
RabbitMQ
at

As Sentry runs throughout the day, there are about 50 different offline tasks that we execute—anything from “process this event, pretty please” to “send all of these cool people some emails.” There are some that we execute once a day and some that execute thousands per second.

Managing this variety requires a reliably high-throughput message-passing technology. We use Celery's RabbitMQ implementation, and we stumbled upon a great feature called Federation that allows us to partition our task queue across any number of RabbitMQ servers and gives us the confidence that, if any single server gets backlogged, others will pitch in and distribute some of the backlogged tasks to their consumers.

#MessageQueue

See more
Tim Abbott
Shared insights
on
RabbitMQ
Python
Redis
at

We've been using RabbitMQ as Zulip's queuing system since we needed a queuing system. What I like about it is that it scales really well and has good libraries for a wide range of platforms, including our own Python. So aside from getting it running, we've had to put basically 0 effort into making it scale for our needs.

However, there's several things that could be better about it: * It's error messages are absolutely terrible; if ever one of our users ends up getting an error with RabbitMQ (even for simple things like a misconfigured hostname), they always end up needing to get help from the Zulip team, because the errors logs are just inscrutable. As an open source project, we've handled this issue by really carefully scripting the installation to be a failure-proof configuration (in this case, setting the RabbitMQ hostname to 127.0.0.1, so that no user-controlled configuration can break it). But it was a real pain to get there and the process of determining we needed to do that caused a significant amount of pain to folks installing Zulip. * The pika library for Python takes a lot of time to startup a RabbitMQ connection; this means that Zulip server restarts are more disruptive than would be ideal. * It's annoying that you need to run the rabbitmqctl management commands as root.

But overall, I like that it has clean, clear semanstics and high scalability, and haven't been tempted to do the work to migrate to something like Redis (which has its own downsides).

See more
Redis logo

Redis

39.2K
28.9K
3.9K
An in-memory database that persists on disk
39.2K
28.9K
+ 1
3.9K
PROS OF REDIS
  • 877
    Performance
  • 535
    Super fast
  • 511
    Ease of use
  • 441
    In-memory cache
  • 321
    Advanced key-value cache
  • 190
    Open source
  • 179
    Easy to deploy
  • 163
    Stable
  • 153
    Free
  • 120
    Fast
  • 40
    High-Performance
  • 39
    High Availability
  • 34
    Data Structures
  • 32
    Very Scalable
  • 23
    Replication
  • 20
    Great community
  • 19
    Pub/Sub
  • 17
    "NoSQL" key-value data store
  • 14
    Hashes
  • 12
    Sets
  • 10
    Sorted Sets
  • 9
    Lists
  • 8
    BSD licensed
  • 8
    NoSQL
  • 7
    Async replication
  • 7
    Integrates super easy with Sidekiq for Rails background
  • 7
    Bitmaps
  • 6
    Open Source
  • 6
    Keys with a limited time-to-live
  • 5
    Strings
  • 5
    Lua scripting
  • 4
    Awesomeness for Free!
  • 4
    Hyperloglogs
  • 3
    outstanding performance
  • 3
    Runs server side LUA
  • 3
    Networked
  • 3
    LRU eviction of keys
  • 3
    Written in ANSI C
  • 3
    Feature Rich
  • 3
    Transactions
  • 2
    Data structure server
  • 2
    Performance & ease of use
  • 1
    Existing Laravel Integration
  • 1
    Automatic failover
  • 1
    Easy to use
  • 1
    Object [key/value] size each 500 MB
  • 1
    Simple
  • 1
    Channels concept
  • 1
    Scalable
  • 1
    Temporarily kept on disk
  • 1
    Dont save data if no subscribers are found
  • 0
    Jk
CONS OF REDIS
  • 12
    Cannot query objects directly
  • 1
    No WAL
  • 1
    No secondary indexes for non-numeric data types

related Redis posts

Robert Zuber

We use MongoDB as our primary #datastore. Mongo's approach to replica sets enables some fantastic patterns for operations like maintenance, backups, and #ETL.

As we pull #microservices from our #monolith, we are taking the opportunity to build them with their own datastores using PostgreSQL. We also use Redis to cache data we’d never store permanently, and to rate-limit our requests to partners’ APIs (like GitHub).

When we’re dealing with large blobs of immutable data (logs, artifacts, and test results), we store them in Amazon S3. We handle any side-effects of S3’s eventual consistency model within our own code. This ensures that we deal with user requests correctly while writes are in process.

See more

I'm working as one of the engineering leads in RunaHR. As our platform is a Saas, we thought It'd be good to have an API (We chose Ruby and Rails for this) and a SPA (built with React and Redux ) connected. We started the SPA with Create React App since It's pretty easy to start.

We use Jest as the testing framework and react-testing-library to test React components. In Rails we make tests using RSpec.

Our main database is PostgreSQL, but we also use MongoDB to store some type of data. We started to use Redis  for cache and other time sensitive operations.

We have a couple of extra projects: One is an Employee app built with React Native and the other is an internal back office dashboard built with Next.js for the client and Python in the backend side.

Since we have different frontend apps we have found useful to have Bit to document visual components and utils in JavaScript.

See more
Resque logo

Resque

109
98
9
A Redis-backed Ruby library for creating background jobs, placing them on multiple queues, and processing them later
109
98
+ 1
9
PROS OF RESQUE
  • 5
    Free
  • 3
    Scalable
  • 1
    Easy to use on heroku
CONS OF RESQUE
    Be the first to leave a con

    related Resque posts

    Kafka logo

    Kafka

    14.3K
    13.4K
    562
    Distributed, fault tolerant, high throughput pub-sub messaging system
    14.3K
    13.4K
    + 1
    562
    PROS OF KAFKA
    • 120
      High-throughput
    • 114
      Distributed
    • 86
      Scalable
    • 79
      High-Performance
    • 64
      Durable
    • 35
      Publish-Subscribe
    • 18
      Simple-to-use
    • 14
      Open source
    • 10
      Written in Scala and java. Runs on JVM
    • 6
      Message broker + Streaming system
    • 4
      Avro schema integration
    • 2
      Suport Multiple clients
    • 2
      Robust
    • 2
      KSQL
    • 2
      Partioned, replayable log
    • 1
      Fun
    • 1
      Extremely good parallelism constructs
    • 1
      Simple publisher / multi-subscriber model
    • 1
      Flexible
    CONS OF KAFKA
    • 27
      Non-Java clients are second-class citizens
    • 26
      Needs Zookeeper
    • 7
      Operational difficulties
    • 2
      Terrible Packaging

    related Kafka posts

    Eric Colson
    Chief Algorithms Officer at Stitch Fix · | 21 upvotes · 1.9M views

    The algorithms and data infrastructure at Stitch Fix is housed in #AWS. Data acquisition is split between events flowing through Kafka, and periodic snapshots of PostgreSQL DBs. We store data in an Amazon S3 based data warehouse. Apache Spark on Yarn is our tool of choice for data movement and #ETL. Because our storage layer (s3) is decoupled from our processing layer, we are able to scale our compute environment very elastically. We have several semi-permanent, autoscaling Yarn clusters running to serve our data processing needs. While the bulk of our compute infrastructure is dedicated to algorithmic processing, we also implemented Presto for adhoc queries and dashboards.

    Beyond data movement and ETL, most #ML centric jobs (e.g. model training and execution) run in a similarly elastic environment as containers running Python and R code on Amazon EC2 Container Service clusters. The execution of batch jobs on top of ECS is managed by Flotilla, a service we built in house and open sourced (see https://github.com/stitchfix/flotilla-os).

    At Stitch Fix, algorithmic integrations are pervasive across the business. We have dozens of data products actively integrated systems. That requires serving layer that is robust, agile, flexible, and allows for self-service. Models produced on Flotilla are packaged for deployment in production using Khan, another framework we've developed internally. Khan provides our data scientists the ability to quickly productionize those models they've developed with open source frameworks in Python 3 (e.g. PyTorch, sklearn), by automatically packaging them as Docker containers and deploying to Amazon ECS. This provides our data scientist a one-click method of getting from their algorithms to production. We then integrate those deployments into a service mesh, which allows us to A/B test various implementations in our product.

    For more info:

    #DataScience #DataStack #Data

    See more
    John Kodumal

    As we've evolved or added additional infrastructure to our stack, we've biased towards managed services. Most new backing stores are Amazon RDS instances now. We do use self-managed PostgreSQL with TimescaleDB for time-series data—this is made HA with the use of Patroni and Consul.

    We also use managed Amazon ElastiCache instances instead of spinning up Amazon EC2 instances to run Redis workloads, as well as shifting to Amazon Kinesis instead of Kafka.

    See more
    Gearman logo

    Gearman

    70
    119
    45
    A generic application framework to farm out work to other machines or processes
    70
    119
    + 1
    45
    PROS OF GEARMAN
    • 11
      Ease of use and very simple APIs
    • 11
      Free
    • 6
      Polyglot
    • 5
      No single point of failure
    • 3
      Scalable
    • 3
      High-throughput
    • 2
      Foreground & background processing
    • 2
      Very fast
    • 1
      Different Programming Languages Channel
    • 1
      Many supported programming languages
    CONS OF GEARMAN
      Be the first to leave a con

      related Gearman posts

      Celery logo

      Celery

      1.3K
      1.2K
      260
      Distributed task queue
      1.3K
      1.2K
      + 1
      260
      PROS OF CELERY
      • 93
        Task queue
      • 59
        Python integration
      • 36
        Django integration
      • 28
        Scheduled Task
      • 18
        Publish/subsribe
      • 6
        Easy to use
      • 6
        Various backend broker
      • 5
        Great community
      • 4
        Free
      • 4
        Workflow
      • 1
        Dynamic
      CONS OF CELERY
      • 4
        Sometimes loses tasks
      • 1
        Depends on broker

      related Celery posts

      James Cunningham
      Operations Engineer at Sentry · | 18 upvotes · 1.2M views
      Shared insights
      on
      Celery
      RabbitMQ
      at

      As Sentry runs throughout the day, there are about 50 different offline tasks that we execute—anything from “process this event, pretty please” to “send all of these cool people some emails.” There are some that we execute once a day and some that execute thousands per second.

      Managing this variety requires a reliably high-throughput message-passing technology. We use Celery's RabbitMQ implementation, and we stumbled upon a great feature called Federation that allows us to partition our task queue across any number of RabbitMQ servers and gives us the confidence that, if any single server gets backlogged, others will pitch in and distribute some of the backlogged tasks to their consumers.

      #MessageQueue

      See more
      Pulkit Sapra

      Hi! I am creating a scraping system in Django, which involves long running tasks between 1 minute & 1 Day. As I am new to Message Brokers and Task Queues, I need advice on which architecture to use for my system. ( Amazon SQS, RabbitMQ, or Celery). The system should be autoscalable using Kubernetes(K8) based on the number of pending tasks in the queue.

      See more
      ZeroMQ logo

      ZeroMQ

      211
      457
      71
      Fast, lightweight messaging library that allows you to design complex communication system without much effort
      211
      457
      + 1
      71
      PROS OF ZEROMQ
      • 24
        Fast
      • 20
        Lightweight
      • 11
        Transport agnostic
      • 7
        No broker required
      • 4
        Low latency
      • 4
        Low level APIs are in C
      • 1
        Open source
      CONS OF ZEROMQ
      • 5
        No message durability
      • 3
        Not a very reliable system - message delivery wise
      • 1
        M x N problem with M producers and N consumers

      related ZeroMQ posts

      Meili Triantafyllidi
      Software engineer at Digital Science · | 5 upvotes · 120.1K views
      Shared insights
      on
      Amazon SQS
      RabbitMQ
      ZeroMQ

      Hi, we are in a ZMQ set up in a push/pull pattern, and we currently start to have more traffic and cases that the service is unavailable or stuck. We want to: * Not loose messages in services outages * Safely restart service without losing messages (ZeroMQ seems to need to close the socket in the receiver before restart manually)

      Do you have experience with this setup with ZeroMQ? Would you suggest RabbitMQ or Amazon SQS (we are in AWS setup) instead? Something else?

      Thank you for your time

      See more
      Sidekiq logo

      Sidekiq

      1K
      532
      407
      Simple, efficient background processing for Ruby
      1K
      532
      + 1
      407
      PROS OF SIDEKIQ
      • 123
        Simple
      • 99
        Efficient background processing
      • 60
        Scalability
      • 37
        Better then resque
      • 26
        Great documentation
      • 15
        Admin tool
      • 14
        Great community
      • 8
        Integrates with redis automatically, with zero config
      • 7
        Great support
      • 7
        Stupidly simple to integrate and run on Rails/Heroku
      • 3
        Freeium
      • 3
        Ruby
      • 2
        Pro version
      • 1
        Dashboard w/live polling
      • 1
        Great ecosystem of addons
      • 1
        Fast
      CONS OF SIDEKIQ
        Be the first to leave a con

        related Sidekiq posts

        Cyril Duchon-Doris

        We decided to use AWS Lambda for several serverless tasks such as

        • Managing AWS backups
        • Processing emails received on Amazon SES and stored to Amazon S3 and notified via Amazon SNS, so as to push a message on our Redis so our Sidekiq Rails workers can process inbound emails
        • Pushing some relevant Amazon CloudWatch metrics and alarms to Slack
        See more

        I'm building a new process management tool. I decided to build with Rails as my backend, using Sidekiq for background jobs. I chose to work with these tools because I've worked with them before and know that they're able to get the job done. They may not be the sexiest tools, but they work and are reliable, which is what I was optimizing for. For data stores, I opted for PostgreSQL and Redis. Because I'm planning on offering dashboards, I wanted a SQL database instead of something like MongoDB that might work early on, but be difficult to use as soon as I want to facilitate aggregate queries.

        On the front-end I'm using Vue.js and vuex in combination with #Turbolinks. In effect, I want to render most pages on the server side without key interactions being managed by Vue.js . This is the first project I'm working on where I've explicitly decided not to include jQuery . I have found React and Redux.js more confusing to setup. I appreciate the opinionated approach from the Vue.js community and that things just work together the way that I'd expect. To manage my javascript dependencies, I'm using Yarn .

        For CSS frameworks, I'm using #Bulma.io. I really appreciate it's minimal nature and that there are no hard javascript dependencies. And to add a little spice, I'm using #font-awesome.

        See more