Azure Service Bus vs Kafka

Overview

Kafka

Stacks24.2K

Followers22.3K

Votes607

GitHub Stars31.2K

Forks14.8K

Azure Service Bus

Stacks553

Followers536

Votes7

Azure Service Bus vs Kafka: What are the differences?

Both Kafka and Azure Service Bus are messaging platforms that allow for the exchange of messages between distributed systems. Here are some differences between the two that might help you make a decision:

Architecture: Kafka is a distributed streaming platform that is built to handle large volumes of data with minimal delay. Whereas, Azure Service Bus is a messaging service that offers advanced features like message queuing, publish-subscribe messaging, and dead-lettering, making it an ideal choice for enterprise-level applications.
Protocol Support: Kafka uses a custom binary protocol and supports a wide range of client libraries, while Azure Service Bus uses industry-standard protocols such as AMQP and supports client libraries for several programming languages.
Scaling: Kafka's architecture is built to scale horizontally, which means its distributed design enables smooth scaling of producers and consumers. While Azure Service Bus also allows scaling, its control over scaling is not as extensive as Kafka's, and it necessitates the use of supplementary services like Azure Functions for autoscaling.
Storage: Kafka stores messages on disk and can handle large volumes of data without any issues. Azure Service Bus also supports large volumes of data, but its message size is limited to 256KB by default (although this can be increased with message partitioning).
Cost: Kafka is a platform that is available to use for free as it is open-source. However, it needs considerable configuration and management. On the other hand, Azure Service Bus is a fully managed service that requires less management. But, it comes with a cost.
Ecosystem: Kafka has a large ecosystem with many tools and libraries for building stream processing applications, while Azure Service Bus has a smaller ecosystem.

In summary, Kafka is a highly scalable streaming platform designed for processing large volumes of data in real-time, while Azure Service Bus is a fully-managed messaging service that provides enterprise-level messaging features such as message queuing, publish-subscribe messaging, and dead-lettering.

Share your Stack

Help developers discover the tools you use. Get visibility for your team's tech choices and contribute to the community's knowledge.

View Docs

CLI (Node.js)

Manual

Advice on Kafka, Azure Service Bus

viradiya

Apr 12, 2020

Needs adviceon

AngularJS

ASP.NET Core

MSSQL

We are going to develop a microservices-based application. It consists of AngularJS, ASP.NET Core, and MSSQL.

We have 3 types of microservices. Emailservice, Filemanagementservice, Filevalidationservice

I am a beginner in microservices. But I have read about RabbitMQ, but come to know that there are Redis and Kafka also in the market. So, I want to know which is best.

933k views933k

Comments

André

Technology Manager at GS1 Portugal - Codipor

Jul 30, 2020

Needs adviceon

.NET Core

Hello dear developers, our company is starting a new project for a new Web App, and we are currently designing the Architecture (we will be using .NET Core). We want to embark on something new, so we are thinking about migrating from a monolithic perspective to a microservices perspective. We wish to containerize those microservices and make them independent from each other. Is it the best way for microservices to communicate with each other via ESB, or is there a new way of doing this? Maybe complementing with an API Gateway? Can you recommend something else different than the two tools I provided?

We want something good for Cost/Benefit; performance should be high too (but not the primary constraint).

Thank you very much in advance :)

461k views461k

Comments

Ishfaq

Feb 28, 2020

Needs advice

Our backend application is sending some external messages to a third party application at the end of each backend (CRUD) API call (from UI) and these external messages take too much extra time (message building, processing, then sent to the third party and log success/failure), UI application has no concern to these extra third party messages.

So currently we are sending these third party messages by creating a new child thread at end of each REST API call so UI application doesn't wait for these extra third party API calls.

I want to integrate Apache Kafka for these extra third party API calls, so I can also retry on failover third party API calls in a queue(currently third party messages are sending from multiple threads at the same time which uses too much processing and resources) and logging, etc.

Question 1: Is this a use case of a message broker?

Question 2: If it is then Kafka vs RabitMQ which is the better?

804k views804k

Comments

Detailed Comparison

Kafka	Azure Service Bus
Kafka is a distributed, partitioned, replicated commit log service. It provides the functionality of a messaging system, but with a unique design.	It is a cloud messaging system for connecting apps and devices across public and private clouds. You can depend on it when you need highly-reliable cloud messaging service between applications and services, even when one or more is offline.
Written at LinkedIn in Scala;Used by LinkedIn to offload processing of all page and other views;Defaults to using persistence, uses OS disk cache for hot data (has higher throughput then any of the above having persistence enabled);Supports both on-line as off-line processing	-
Statistics
GitHub Stars 31.2K	GitHub Stars -
GitHub Forks 14.8K	GitHub Forks -
Stacks 24.2K	Stacks 553
Followers 22.3K	Followers 536
Votes 607	Votes 7
Pros & Cons
Pros 126 High-throughput 119 Distributed 92 Scalable 86 High-Performance 66 Durable Cons 32 Non-Java clients are second-class citizens 29 Needs Zookeeper 9 Operational difficulties 5 Terrible Packaging	Pros 4 Easy Integration with .Net 2 Cloud Native 1 Use while high messaging need Cons 1 Observability of messages in the queue is lacking 1 Limited features in Basic tier 1 Skills can only be used in Azure - vendor lock-in 1 Lacking in JMS support

What are some alternatives to Kafka, Azure Service Bus?

RabbitMQ

RabbitMQ gives your applications a common platform to send and receive messages, and your messages a safe place to live until received.

Celery

Celery is an asynchronous task queue/job queue based on distributed message passing. It is focused on real-time operation, but supports scheduling as well.

Amazon SQS

Transmit any volume of data, at any level of throughput, without losing messages or requiring other services to be always available. With SQS, you can offload the administrative burden of operating and scaling a highly available messaging cluster, while paying a low price for only what you use.

NSQ

NSQ is a realtime distributed messaging platform designed to operate at scale, handling billions of messages per day. It promotes distributed and decentralized topologies without single points of failure, enabling fault tolerance and high availability coupled with a reliable message delivery guarantee. See features & guarantees.

ActiveMQ

Apache ActiveMQ is fast, supports many Cross Language Clients and Protocols, comes with easy to use Enterprise Integration Patterns and many advanced features while fully supporting JMS 1.1 and J2EE 1.4. Apache ActiveMQ is released under the Apache 2.0 License.

ZeroMQ

The 0MQ lightweight messaging kernel is a library which extends the standard socket interfaces with features traditionally provided by specialised messaging middleware products. 0MQ sockets provide an abstraction of asynchronous message queues, multiple messaging patterns, message filtering (subscriptions), seamless access to multiple transport protocols and more.

Apache NiFi

An easy to use, powerful, and reliable system to process and distribute data. It supports powerful and scalable directed graphs of data routing, transformation, and system mediation logic.

Gearman

Gearman allows you to do work in parallel, to load balance processing, and to call functions between languages. It can be used in a variety of applications, from high-availability web sites to the transport of database replication events.

Memphis

Highly scalable and effortless data streaming platform. Made to enable developers and data teams to collaborate and build real-time and streaming apps fast.

IronMQ

An easy-to-use highly available message queuing service. Built for distributed cloud applications with critical messaging needs. Provides on-demand message queuing with advanced features and cloud-optimized performance.

Related Comparisons

Architecture: Kafka is a distributed streaming platform that is built to handle large volumes of data with minimal delay. Whereas, Azure Service Bus is a messaging service that offers advanced features like message queuing, publish-subscribe messaging, and dead-lettering, making it an ideal choice for enterprise-level applications.
Protocol Support: Kafka uses a custom binary protocol and supports a wide range of client libraries, while Azure Service Bus uses industry-standard protocols such as AMQP and supports client libraries for several programming languages.
Scaling: Kafka's architecture is built to scale horizontally, which means its distributed design enables smooth scaling of producers and consumers. While Azure Service Bus also allows scaling, its control over scaling is not as extensive as Kafka's, and it necessitates the use of supplementary services like Azure Functions for autoscaling.
Storage: Kafka stores messages on disk and can handle large volumes of data without any issues. Azure Service Bus also supports large volumes of data, but its message size is limited to 256KB by default (although this can be increased with message partitioning).
Cost: Kafka is a platform that is available to use for free as it is open-source. However, it needs considerable configuration and management. On the other hand, Azure Service Bus is a fully managed service that requires less management. But, it comes with a cost.
Ecosystem: Kafka has a large ecosystem with many tools and libraries for building stream processing applications, while Azure Service Bus has a smaller ecosystem.

Azure Service Bus vs Kafka