Kafka聽vs聽RabbitMQ聽vs聽ZeroMQ

Need advice about which tool to choose?Ask the StackShare community!

Kafka

13.9K
13.1K
+ 1
562
RabbitMQ

13.3K
11.4K
+ 1
511
ZeroMQ

210
455
+ 1
68
Advice on Kafka, RabbitMQ, and ZeroMQ
Pulkit Sapra
Needs advice
on
RabbitMQ
Kubernetes
and
Amazon SQS

Hi! I am creating a scraping system in Django, which involves long running tasks between 1 minute & 1 Day. As I am new to Message Brokers and Task Queues, I need advice on which architecture to use for my system. ( Amazon SQS, RabbitMQ, or Celery). The system should be autoscalable using Kubernetes(K8) based on the number of pending tasks in the queue.

See more
Replies (1)
Anis Zehani
Recommends
Kafka

Hello, i highly recommend Apache Kafka, to me it's the best. You can deploy it in cluster mode inside K8S, thus you can have a Highly available system (also auto scalable).

Good luck

See more
Needs advice
on
RabbitMQ
and
Celery

I am just a beginner at these two technologies.

Problem statement: I am getting lakh of users from the sequel server for whom I need to create caches in MongoDB by making different REST API requests.

Here these users can be treated as messages. Each REST API request is a task.

I am confused about whether I should go for RabbitMQ alone or Celery.

If I have to go with RabbitMQ, I prefer to use python with Pika module. But the challenge with Pika is, it is not thread-safe. So I am not finding a way to execute a lakh of API requests in parallel using multiple threads using Pika.

If I have to go with Celery, I don't know how I can achieve better scalability in executing these API requests in parallel.

See more
Replies (1)
Recommends
Redis
rq

For large amounts of small tasks and caches I have had good luck with Redis and RQ. I have not personally used celery but I am fairly sure it would scale well, and I have not used RabbitMQ for anything besides communication between services. If you prefer python my suggestions should feel comfortable.

Sorry I do not have a more information

See more
Meili Triantafyllidi
Software engineer at Digital Science | 5 upvotes 路 103.2K views
Needs advice
on
ZeroMQ
RabbitMQ
and
Amazon SQS

Hi, we are in a ZMQ set up in a push/pull pattern, and we currently start to have more traffic and cases that the service is unavailable or stuck. We want to: * Not loose messages in services outages * Safely restart service without losing messages (ZeroMQ seems to need to close the socket in the receiver before restart manually)

Do you have experience with this setup with ZeroMQ? Would you suggest RabbitMQ or Amazon SQS (we are in AWS setup) instead? Something else?

Thank you for your time

See more
Replies (1)
Shishir Pandey
Recommends
RabbitMQ

ZeroMQ is fast but you need to build build reliability yourself. There are a number of patterns described in the zeromq guide. I have used RabbitMQ before which gives lot of functionality out of the box, you can probably use the worker queues example from the tutorial, it can also persists messages in the queue.

I haven't used Amazon SQS before. Another tool you could use is Kafka.

See more
Decisions about Kafka, RabbitMQ, and ZeroMQ
Kirill Mikhailov

Maybe not an obvious comparison with Kafka, since Kafka is pretty different from rabbitmq. But for small service, Rabbit as a pubsub platform is super easy to use and pretty powerful. Kafka as an alternative was the original choice, but its really a kind of overkill for a small-medium service. Especially if you are not planning to use k8s, since pure docker deployment can be a pain because of networking setup. Google PubSub was another alternative, its actually pretty cheap, but I never tested it since Rabbit was matching really good for mailing/notification services.

See more
Mickael Alliel
DevOps Engineer at Rookout | 4 upvotes 路 172.6K views

In addition to being a lot cheaper, Google Cloud Pub/Sub allowed us to not worry about maintaining any more infrastructure that needed.

We moved from a self-hosted RabbitMQ over to CloudAMQP and decided that since we use GCP anyway, why not try their managed PubSub?

It is one of the better decisions that we made, and we can just focus about building more important stuff!

See more
Get Advice from developers at your company using Private StackShare. Sign up for Private StackShare.
Learn More
Pros of Kafka
Pros of RabbitMQ
Pros of ZeroMQ
  • 120
    High-throughput
  • 114
    Distributed
  • 86
    Scalable
  • 79
    High-Performance
  • 64
    Durable
  • 35
    Publish-Subscribe
  • 18
    Simple-to-use
  • 14
    Open source
  • 10
    Written in Scala and java. Runs on JVM
  • 6
    Message broker + Streaming system
  • 4
    Avro schema integration
  • 2
    Suport Multiple clients
  • 2
    Robust
  • 2
    KSQL
  • 2
    Partioned, replayable log
  • 1
    Fun
  • 1
    Extremely good parallelism constructs
  • 1
    Simple publisher / multi-subscriber model
  • 1
    Flexible
  • 226
    It's fast and it works with good metrics/monitoring
  • 79
    Ease of configuration
  • 57
    I like the admin interface
  • 49
    Easy to set-up and start with
  • 20
    Durable
  • 18
    Intuitive work through python
  • 18
    Standard protocols
  • 10
    Written primarily in Erlang
  • 7
    Simply superb
  • 6
    Completeness of messaging patterns
  • 3
    Reliable
  • 3
    Scales to 1 million messages per second
  • 2
    Distributed
  • 2
    Supports AMQP
  • 2
    Better than most traditional queue based message broker
  • 1
    High performance
  • 1
    Reliability
  • 1
    Clusterable
  • 1
    Inubit Integration
  • 1
    Clear documentation with different scripting language
  • 1
    Great ui
  • 1
    Runs on Open Telecom Platform
  • 1
    Better routing system
  • 1
    Supports MQTT
  • 23
    Fast
  • 19
    Lightweight
  • 11
    Transport agnostic
  • 6
    No broker required
  • 4
    Low latency
  • 4
    Low level APIs are in C
  • 1
    Open source

Sign up to add or upvote prosMake informed product decisions

Cons of Kafka
Cons of RabbitMQ
Cons of ZeroMQ
  • 27
    Non-Java clients are second-class citizens
  • 26
    Needs Zookeeper
  • 7
    Operational difficulties
  • 2
    Terrible Packaging
  • 9
    Too complicated cluster/HA config and management
  • 6
    Needs Erlang runtime. Need ops good with Erlang runtime
  • 5
    Configuration must be done first, not by your code
  • 4
    Slow
  • 5
    No message durability
  • 3
    Not a very reliable system - message delivery wise
  • 1
    M x N problem with M producers and N consumers

Sign up to add or upvote consMake informed product decisions