Google Cloud Pub/Sub vs Kafka vs RabbitMQ

Overview

RabbitMQ

Stacks21.8K

Followers18.9K

Votes558

GitHub Stars13.2K

Forks4.0K

Kafka

Stacks24.2K

Followers22.3K

Votes607

GitHub Stars31.2K

Forks14.8K

Google Cloud Pub/Sub

Stacks550

Followers428

Votes13

Google Cloud Pub/Sub vs Kafka vs RabbitMQ: What are the differences?

Key differences between Google Cloud Pub/Sub and Kafka and RabbitMQ

Google Cloud Pub/Sub, Kafka, and RabbitMQ are all popular messaging systems used for building distributed applications. However, they have some key differences that set them apart.

Scalability: Google Cloud Pub/Sub is a fully managed service, meaning that it automatically scales based on demand. It can handle millions of messages per second without any manual configuration. Kafka and RabbitMQ, on the other hand, require manual configuration for scalability.
Managed vs Self-Hosted: Pub/Sub is a managed service provided by Google Cloud, which means that the infrastructure and maintenance are taken care of by Google. Kafka and RabbitMQ, on the other hand, are self-hosted systems, meaning that you need to manage the infrastructure and maintenance yourself.
Message Order: Kafka guarantees order within a partition, which makes it suitable for applications that require strict ordering. RabbitMQ and Pub/Sub do not provide strict ordering guarantees.
Data Retention: Kafka has built-in data retention and allows you to store messages for a longer duration. RabbitMQ and Pub/Sub do not have native data retention capabilities and require additional setup for storing messages beyond a certain time frame.
Protocol Support: RabbitMQ supports multiple protocols, including AMQP, MQTT, and STOMP. Kafka supports a custom binary protocol. Pub/Sub supports both custom binary and HTTP/REST protocols.
Message Durability: Kafka and RabbitMQ store messages on disk, providing durability even in the event of system failures. Pub/Sub, in its basic tier, does not provide disk storage for messages, which means that messages may be lost in the event of system failures. However, Pub/Sub offers message persistence as an optional feature in its premium tier.

In summary, Google Cloud Pub/Sub is a fully managed service that offers automatic scalability and support for multiple protocols. Kafka and RabbitMQ, while requiring manual management, provide strict message ordering, built-in data retention, and message durability through disk storage.

Share your Stack

Help developers discover the tools you use. Get visibility for your team's tech choices and contribute to the community's knowledge.

View Docs

CLI (Node.js)

Manual

Advice on RabbitMQ, Kafka, Google Cloud Pub/Sub

viradiya

Apr 12, 2020

Needs adviceon

AngularJS

ASP.NET Core

MSSQL

We are going to develop a microservices-based application. It consists of AngularJS, ASP.NET Core, and MSSQL.

We have 3 types of microservices. Emailservice, Filemanagementservice, Filevalidationservice

I am a beginner in microservices. But I have read about RabbitMQ, but come to know that there are Redis and Kafka also in the market. So, I want to know which is best.

933k views933k

Comments

André

Technology Manager at GS1 Portugal - Codipor

Jul 30, 2020

Needs adviceon

.NET Core

Hello dear developers, our company is starting a new project for a new Web App, and we are currently designing the Architecture (we will be using .NET Core). We want to embark on something new, so we are thinking about migrating from a monolithic perspective to a microservices perspective. We wish to containerize those microservices and make them independent from each other. Is it the best way for microservices to communicate with each other via ESB, or is there a new way of doing this? Maybe complementing with an API Gateway? Can you recommend something else different than the two tools I provided?

We want something good for Cost/Benefit; performance should be high too (but not the primary constraint).

Thank you very much in advance :)

461k views461k

Comments

mediafinger

Feb 13, 2019

Reviewon

Kafka

RabbitMQ

The question for which Message Queue to use mentioned "availability, distributed, scalability, and monitoring". I don't think that this excludes many options already. I does not sound like you would take advantage of Kafka's strengths (replayability, based on an even sourcing architecture). You could pick one of the AMQP options.

I would recommend the RabbitMQ message broker, which not only implements the AMQP standard 0.9.1 (it can support 1.x or other protocols as well) but has also several very useful extensions built in. It ticks the boxes you mentioned and on top you will get a very flexible system, that allows you to build the architecture, pick the options and trade-offs that suite your case best.

For more information about RabbitMQ, please have a look at the linked markdown I assembled. The second half explains many configuration options. It also contains links to managed hosting and to libraries (though it is missing Python's - which should be Puka, I assume).

159k views159k

Comments

Detailed Comparison

RabbitMQ	Kafka	Google Cloud Pub/Sub
RabbitMQ gives your applications a common platform to send and receive messages, and your messages a safe place to live until received.	Kafka is a distributed, partitioned, replicated commit log service. It provides the functionality of a messaging system, but with a unique design.	Cloud Pub/Sub is a fully-managed real-time messaging service that allows you to send and receive messages between independent applications. You can leverage Cloud Pub/Sub’s flexibility to decouple systems and components hosted on Google Cloud Platform or elsewhere on the Internet.
Robust messaging for applications;Easy to use;Runs on all major operating systems;Supports a huge number of developer platforms;Open source and commercially supported	Written at LinkedIn in Scala;Used by LinkedIn to offload processing of all page and other views;Defaults to using persistence, uses OS disk cache for hot data (has higher throughput then any of the above having persistence enabled);Supports both on-line as off-line processing	-
Statistics
GitHub Stars 13.2K	GitHub Stars 31.2K	GitHub Stars -
GitHub Forks 4.0K	GitHub Forks 14.8K	GitHub Forks -
Stacks 21.8K	Stacks 24.2K	Stacks 550
Followers 18.9K	Followers 22.3K	Followers 428
Votes 558	Votes 607	Votes 13
Pros & Cons
Pros 235 It's fast and it works with good metrics/monitoring 80 Ease of configuration 60 I like the admin interface 52 Easy to set-up and start with 22 Durable Cons 9 Too complicated cluster/HA config and management 6 Needs Erlang runtime. Need ops good with Erlang runtime 5 Configuration must be done first, not by your code 4 Slow	Pros 126 High-throughput 119 Distributed 92 Scalable 86 High-Performance 66 Durable Cons 32 Non-Java clients are second-class citizens 29 Needs Zookeeper 9 Operational difficulties 5 Terrible Packaging	Pros 9 Easy to set-up and start with 2 A great choice for microservice architecture 2 Efficient and practical for complex systems Cons 2 Need integration with stackdriver for monitoring
Integrations
No integrations available	No integrations available	Google Cloud Functions Cloud Functions for Firebase

What are some alternatives to RabbitMQ, Kafka, Google Cloud Pub/Sub?

Firebase

Firebase is a cloud service designed to power real-time, collaborative applications. Simply add the Firebase library to your application to gain access to a shared data structure; any changes you make to that data are automatically synchronized with the Firebase cloud and with other clients within milliseconds.

Socket.IO

It enables real-time bidirectional event-based communication. It works on every platform, browser or device, focusing equally on reliability and speed.

Celery

Celery is an asynchronous task queue/job queue based on distributed message passing. It is focused on real-time operation, but supports scheduling as well.

PubNub

PubNub makes it easy for you to add real-time capabilities to your apps, without worrying about the infrastructure. Build apps that allow your users to engage in real-time across mobile, browser, desktop and server.

Pusher

Pusher is the category leader in delightful APIs for app developers building communication and collaboration features.

Amazon SQS

Transmit any volume of data, at any level of throughput, without losing messages or requiring other services to be always available. With SQS, you can offload the administrative burden of operating and scaling a highly available messaging cluster, while paying a low price for only what you use.

NSQ

NSQ is a realtime distributed messaging platform designed to operate at scale, handling billions of messages per day. It promotes distributed and decentralized topologies without single points of failure, enabling fault tolerance and high availability coupled with a reliable message delivery guarantee. See features & guarantees.

SignalR

SignalR allows bi-directional communication between server and client. Servers can now push content to connected clients instantly as it becomes available. SignalR supports Web Sockets, and falls back to other compatible techniques for older browsers. SignalR includes APIs for connection management (for instance, connect and disconnect events), grouping connections, and authorization.

Ably

Ably offers WebSockets, stream resume, history, presence, and managed third-party integrations to make it simple to build, extend, and deliver digital realtime experiences at scale.

Syncano

Syncano is a backend platform to build powerful real-time apps more efficiently. Integrate with any API, minimize boilerplate code and control your data - all from one place.

Related Comparisons

Google Cloud Pub/Sub vs Kafka vs RabbitMQ: What are the differences?

Key differences between Google Cloud Pub/Sub and Kafka and RabbitMQ

Google Cloud Pub/Sub, Kafka, and RabbitMQ are all popular messaging systems used for building distributed applications. However, they have some key differences that set them apart.

Scalability: Google Cloud Pub/Sub is a fully managed service, meaning that it automatically scales based on demand. It can handle millions of messages per second without any manual configuration. Kafka and RabbitMQ, on the other hand, require manual configuration for scalability.
Managed vs Self-Hosted: Pub/Sub is a managed service provided by Google Cloud, which means that the infrastructure and maintenance are taken care of by Google. Kafka and RabbitMQ, on the other hand, are self-hosted systems, meaning that you need to manage the infrastructure and maintenance yourself.
Message Order: Kafka guarantees order within a partition, which makes it suitable for applications that require strict ordering. RabbitMQ and Pub/Sub do not provide strict ordering guarantees.
Data Retention: Kafka has built-in data retention and allows you to store messages for a longer duration. RabbitMQ and Pub/Sub do not have native data retention capabilities and require additional setup for storing messages beyond a certain time frame.
Protocol Support: RabbitMQ supports multiple protocols, including AMQP, MQTT, and STOMP. Kafka supports a custom binary protocol. Pub/Sub supports both custom binary and HTTP/REST protocols.
Message Durability: Kafka and RabbitMQ store messages on disk, providing durability even in the event of system failures. Pub/Sub, in its basic tier, does not provide disk storage for messages, which means that messages may be lost in the event of system failures. However, Pub/Sub offers message persistence as an optional feature in its premium tier.

Google Cloud Pub/Sub vs Kafka vs RabbitMQ

Overview

Google Cloud Pub/Sub vs Kafka vs RabbitMQ: What are the differences?