What is RabbitMQ and what are its top alternatives?
Top Alternatives to RabbitMQ
Kafka is a distributed, partitioned, replicated commit log service. It provides the functionality of a messaging system, but with a unique design. ...
Apache ActiveMQ is fast, supports many Cross Language Clients and Protocols, comes with easy to use Enterprise Integration Patterns and many advanced features while fully supporting JMS 1.1 and J2EE 1.4. Apache ActiveMQ is released under the Apache 2.0 License. ...
The 0MQ lightweight messaging kernel is a library which extends the standard socket interfaces with features traditionally provided by specialised messaging middleware products. 0MQ sockets provide an abstraction of asynchronous message queues, multiple messaging patterns, message filtering (subscriptions), seamless access to multiple transport protocols and more. ...
Amazon Simple Notification Service makes it simple and cost-effective to push to mobile devices such as iPhone, iPad, Android, Kindle Fire, and internet connected smart devices, as well as pushing to other distributed services. Besides pushing cloud notifications directly to mobile devices, SNS can also deliver notifications by SMS text message or email, to Simple Queue Service (SQS) queues, or to any HTTP endpoint. ...
Redis is an open source, BSD licensed, advanced key-value store. It is often referred to as a data structure server since keys can contain strings, hashes, lists, sets and sorted sets. ...
Gearman allows you to do work in parallel, to load balance processing, and to call functions between languages. It can be used in a variety of applications, from high-availability web sites to the transport of database replication events. ...
Beanstalks's interface is generic, but was originally designed for reducing the latency of page views in high-volume web applications by running time-consuming tasks asynchronously. ...
gRPC is a modern open source high performance RPC framework that can run in any environment. It can efficiently connect services in and across data centers with pluggable support for load balancing, tracing, health checking... ...
RabbitMQ alternatives & related posts
- Open source13
- Written in Scala and java. Runs on JVM9
- Message broker + Streaming system6
- Avro schema integration4
- Suport Multiple clients2
- Partioned, replayable log2
- Extremely good parallelism constructs1
- Simple publisher / multi-subscriber model1
- Needs Zookeeper25
- Non-Java clients are second-class citizens24
- Operational difficulties6
- Terrible Packaging1
related Kafka posts
The algorithms and data infrastructure at Stitch Fix is housed in #AWS. Data acquisition is split between events flowing through Kafka, and periodic snapshots of PostgreSQL DBs. We store data in an Amazon S3 based data warehouse. Apache Spark on Yarn is our tool of choice for data movement and #ETL. Because our storage layer (s3) is decoupled from our processing layer, we are able to scale our compute environment very elastically. We have several semi-permanent, autoscaling Yarn clusters running to serve our data processing needs. While the bulk of our compute infrastructure is dedicated to algorithmic processing, we also implemented Presto for adhoc queries and dashboards.
Beyond data movement and ETL, most #ML centric jobs (e.g. model training and execution) run in a similarly elastic environment as containers running Python and R code on Amazon EC2 Container Service clusters. The execution of batch jobs on top of ECS is managed by Flotilla, a service we built in house and open sourced (see https://github.com/stitchfix/flotilla-os).
At Stitch Fix, algorithmic integrations are pervasive across the business. We have dozens of data products actively integrated systems. That requires serving layer that is robust, agile, flexible, and allows for self-service. Models produced on Flotilla are packaged for deployment in production using Khan, another framework we've developed internally. Khan provides our data scientists the ability to quickly productionize those models they've developed with open source frameworks in Python 3 (e.g. PyTorch, sklearn), by automatically packaging them as Docker containers and deploying to Amazon ECS. This provides our data scientist a one-click method of getting from their algorithms to production. We then integrate those deployments into a service mesh, which allows us to A/B test various implementations in our product.
For more info:
- Our Algorithms Tour: https://algorithms-tour.stitchfix.com/
- Our blog: https://multithreaded.stitchfix.com/blog/
- Careers: https://multithreaded.stitchfix.com/careers/
#DataScience #DataStack #Data
As we've evolved or added additional infrastructure to our stack, we've biased towards managed services. Most new backing stores are Amazon RDS instances now. We do use self-managed PostgreSQL with TimescaleDB for time-series data—this is made HA with the use of Patroni and Consul.
We also use managed Amazon ElastiCache instances instead of spinning up Amazon EC2 instances to run Redis workloads, as well as shifting to Amazon Kinesis instead of Kafka.
- Easy to use15
- Open source12
- JMS compliant10
- High Availability6
- Support XA (distributed transactions)3
- Distributed Network of brokers2
- Docker delievery1
related ActiveMQ posts
I want to choose Message Queue with the following features - Highly Available, Distributed, Scalable, Monitoring. I have RabbitMQ, ActiveMQ, Kafka and Apache RocketMQ in mind. But I am confused which one to choose.
I use ActiveMQ because RabbitMQ have stopped giving the support for AMQP 1.0 or above version and the earlier version of AMQP doesn't give the functionality to support OAuth.
If OAuth is not required and we can go with AMQP 0.9 then i still recommend rabbitMq.
- Transport agnostic11
- No broker required6
- Low latency4
- Low level APIs are in C4
- Open source1
- No message durability5
- Not a very reliable system - message delivery wise3
- M x N problem with M producers and N consumers1
related ZeroMQ posts
Hi, we are in a ZMQ set up in a push/pull pattern, and we currently start to have more traffic and cases that the service is unavailable or stuck. We want to: * Not loose messages in services outages * Safely restart service without losing messages (ZeroMQ seems to need to close the socket in the receiver before restart manually)
Do you have experience with this setup with ZeroMQ? Would you suggest RabbitMQ or Amazon SQS (we are in AWS setup) instead? Something else?
Thank you for your time
- Low cost9
- Supports multi subscribers3
related Amazon SNS posts
We decided to use AWS Lambda for several serverless tasks such as
- Managing AWS backups
- Processing emails received on Amazon SES and stored to Amazon S3 and notified via Amazon SNS, so as to push a message on our Redis so our Sidekiq Rails workers can process inbound emails
- Pushing some relevant Amazon CloudWatch metrics and alarms to Slack
Instead of Amazon SNS, which is currently being used to send outbound push notification and including SMS, we want to build the 2 Way SMS using Amazon Pinpoint. Just want to know about Pinpoint and any outstanding issues if we drop SNS since it does not support 2 Way and use Pinpoint for both incoming and outgoing flow.
- Super fast535
- Ease of use510
- In-memory cache442
- Advanced key-value cache321
- Open source189
- Easy to deploy179
- High Availability38
- Data Structures34
- Very Scalable32
- Great community20
- "NoSQL" key-value data store17
- Sorted Sets10
- BSD licensed8
- Async replication7
- Integrates super easy with Sidekiq for Rails background7
- Open Source6
- Keys with a limited time-to-live6
- Lua scripting5
- Awesomeness for Free!4
- outstanding performance3
- Runs server side LUA3
- LRU eviction of keys3
- Written in ANSI C3
- Feature Rich3
- Data structure server2
- Performance & ease of use2
- Existing Laravel Integration1
- Automatic failover1
- Easy to use1
- Object [key/value] size each 500 MB1
- Channels concept1
- Temporarily kept on disk1
- Dont save data if no subscribers are found1
- Cannot query objects directly10
- No WAL1
- No secondary indexes for non-numeric data types1
related Redis posts
We use MongoDB as our primary #datastore. Mongo's approach to replica sets enables some fantastic patterns for operations like maintenance, backups, and #ETL.
As we pull #microservices from our #monolith, we are taking the opportunity to build them with their own datastores using PostgreSQL. We also use Redis to cache data we’d never store permanently, and to rate-limit our requests to partners’ APIs (like GitHub).
When we’re dealing with large blobs of immutable data (logs, artifacts, and test results), we store them in Amazon S3. We handle any side-effects of S3’s eventual consistency model within our own code. This ensures that we deal with user requests correctly while writes are in process.
I'm working as one of the engineering leads in RunaHR. As our platform is a Saas, we thought It'd be good to have an API (We chose Ruby and Rails for this) and a SPA (built with React and Redux ) connected. We started the SPA with Create React App since It's pretty easy to start.
We use Jest as the testing framework and react-testing-library to test React components. In Rails we make tests using RSpec.
Our main database is PostgreSQL, but we also use MongoDB to store some type of data. We started to use Redis for cache and other time sensitive operations.
We have a couple of extra projects: One is an Employee app built with React Native and the other is an internal back office dashboard built with Next.js for the client and Python in the backend side.
- Ease of use and very simple APIs11
- No single point of failure5
- Foreground & background processing2
- Very fast2
- Different Programming Languages Channel1
- Many supported programming languages1
related Gearman posts
- Does one thing well12
- Job delay3
- External admin UI developer friendly3
- Job prioritization2
- External admin UI2
related Beanstalkd posts
I used Kafka originally because it was mandated as part of the top-level IT requirements at a Fortune 500 client. What I found was that it was orders of magnitude more complex ...and powerful than my daily Beanstalkd , and far more flexible, resilient, and manageable than RabbitMQ.
So for any case where utmost flexibility and resilience are part of the deal, I would use Kafka again. But due to the complexities involved, for any time where this level of scalability is not required, I would probably just use Beanstalkd for its simplicity.
I tend to find RabbitMQ to be in an uncomfortable middle place between these two extremities.
- Higth performance13
- Easy setup8
- The future of API7
related gRPC posts
By mid-2015, Uber’s rider growth coupled with its cadence of releasing new services, like Eats and Freight, was pressuring the infrastructure. To allow the decoupling of consumption from production, and to add an abstraction layer between users, developers, and infrastructure, Uber built Catalyst, a serverless internal service mesh.
Uber decided to build their own severless solution, rather that using something like AWS Lambda, speed for its global production environments as well as introspectability.