Hazelcast vs Kafka

Overview

Kafka

Stacks24.2K

Followers22.3K

Votes607

GitHub Stars31.2K

Forks14.8K

Hazelcast

Stacks427

Followers474

Votes59

GitHub Stars6.4K

Forks1.9K

Hazelcast vs Kafka: What are the differences?

Hazelcast and Kafka are distributed systems addressing different aspects of real-time data processing. Let's explore the key differences between them.

Data Processing Model: Hazelcast is an in-memory data grid platform that allows distributed storage and processing of data. It provides distributed caching, distributed computing, and distributed messaging capabilities. On the other hand, Kafka is a distributed streaming platform designed for handling real-time data feeds. It enables data streams to be processed in a fault-tolerant and scalable manner.
Data Storage and Persistence: Hazelcast provides an in-memory data grid, allowing fast data access and processing. It stores data in memory across a cluster of nodes. However, Hazelcast also supports the option to persist data to disk for fault tolerance. In contrast, Kafka does not store data in memory like Hazelcast. Instead, Kafka uses a distributed commit log to store and replicate data on disk.
Messaging vs Event Streaming: Hazelcast supports publish-subscribe messaging, where messages are sent to a channel and multiple subscribers receive those messages. It provides reliable messaging with features like message ordering and durability. Kafka, on the other hand, is an event streaming platform that enables the continuous flow of events between systems or applications. It provides a fault-tolerant event-driven architecture, allowing real-time data processing and analytics.
Data Processing Paradigm: Hazelcast supports distributed computing paradigms, such as MapReduce and parallel computation models. It enables distributed data processing across a cluster of nodes, allowing for faster and more efficient processing of large datasets. Kafka, on the other hand, follows the publish-subscribe messaging model and is specifically tailored for real-time data processing and streaming.
Data Replication and Fault Tolerance: Hazelcast provides built-in data replication and fault tolerance mechanisms. It replicates data across multiple nodes in a cluster, ensuring that data is available even in the event of node failures. Kafka uses a distributed commit log for data replication and fault tolerance. It replicates data across multiple brokers, providing high availability and durability.
Integration Ecosystem: Hazelcast integrates well with various programming languages and frameworks. It provides client libraries for Java, .NET, Python, and other languages, making it easy to develop applications using Hazelcast. Kafka, on the other hand, comes with a well-established ecosystem and has extensive support for integration with various technologies and frameworks, such as Apache Spark, Apache Flink, and Apache Storm.

In summary, Hazelcast is an in-memory data grid platform with distributed caching, computing, and messaging capabilities, while Kafka is a distributed streaming platform designed for handling real-time data feeds. Hazelcast focuses on in-memory data storage and processing, while Kafka excels in event streaming and real-time data processing.

Share your Stack

Help developers discover the tools you use. Get visibility for your team's tech choices and contribute to the community's knowledge.

View Docs

CLI (Node.js)

Manual

Advice on Kafka, Hazelcast

viradiya

Apr 12, 2020

Needs adviceon

AngularJS

ASP.NET Core

MSSQL

We are going to develop a microservices-based application. It consists of AngularJS, ASP.NET Core, and MSSQL.

We have 3 types of microservices. Emailservice, Filemanagementservice, Filevalidationservice

I am a beginner in microservices. But I have read about RabbitMQ, but come to know that there are Redis and Kafka also in the market. So, I want to know which is best.

933k views933k

Comments

Ishfaq

Feb 28, 2020

Needs advice

Our backend application is sending some external messages to a third party application at the end of each backend (CRUD) API call (from UI) and these external messages take too much extra time (message building, processing, then sent to the third party and log success/failure), UI application has no concern to these extra third party messages.

So currently we are sending these third party messages by creating a new child thread at end of each REST API call so UI application doesn't wait for these extra third party API calls.

I want to integrate Apache Kafka for these extra third party API calls, so I can also retry on failover third party API calls in a queue(currently third party messages are sending from multiple threads at the same time which uses too much processing and resources) and logging, etc.

Question 1: Is this a use case of a message broker?

Question 2: If it is then Kafka vs RabitMQ which is the better?

804k views804k

Comments

Roman

Senior Back-End Developer, Software Architect

Feb 12, 2019

Reviewon

Kafka

I use Kafka because it has almost infinite scaleability in terms of processing events (could be scaled to process hundreds of thousands of events), great monitoring (all sorts of metrics are exposed via JMX).

Downsides of using Kafka are:

you have to deal with Zookeeper
you have to implement advanced routing yourself (compared to RabbitMQ it has no advanced routing)

10.8k views10.8k

Comments

Detailed Comparison

Kafka	Hazelcast
Kafka is a distributed, partitioned, replicated commit log service. It provides the functionality of a messaging system, but with a unique design.	With its various distributed data structures, distributed caching capabilities, elastic nature, memcache support, integration with Spring and Hibernate and more importantly with so many happy users, Hazelcast is feature-rich, enterprise-ready and developer-friendly in-memory data grid solution.
Written at LinkedIn in Scala;Used by LinkedIn to offload processing of all page and other views;Defaults to using persistence, uses OS disk cache for hot data (has higher throughput then any of the above having persistence enabled);Supports both on-line as off-line processing	Distributed implementations of java.util.{Queue, Set, List, Map};Distributed implementation of java.util.concurrent.locks.Lock;Distributed implementation of java.util.concurrent.ExecutorService;Distributed MultiMap for one-to-many relationships;Distributed Topic for publish/subscribe messaging;Synchronous (write-through) and asynchronous (write-behind) persistence;Transaction support;Socket level encryption support for secure clusters;Second level cache provider for Hibernate;Monitoring and management of the cluster via JMX;Dynamic HTTP session clustering;Support for cluster info and membership events;Dynamic discovery, scaling, partitioning with backups and fail-over
Statistics
GitHub Stars 31.2K	GitHub Stars 6.4K
GitHub Forks 14.8K	GitHub Forks 1.9K
Stacks 24.2K	Stacks 427
Followers 22.3K	Followers 474
Votes 607	Votes 59
Pros & Cons
Pros 126 High-throughput 119 Distributed 92 Scalable 86 High-Performance 66 Durable Cons 32 Non-Java clients are second-class citizens 29 Needs Zookeeper 9 Operational difficulties 5 Terrible Packaging	Pros 11 High Availibility 6 Distributed compute 6 Distributed Locking 5 Sharding 4 Load balancing Cons 4 License needed for SSL
Integrations
No integrations available	Java Spring

What are some alternatives to Kafka, Hazelcast?

Redis

Redis is an open source (BSD licensed), in-memory data structure store, used as a database, cache, and message broker. Redis provides data structures such as strings, hashes, lists, sets, sorted sets with range queries, bitmaps, hyperloglogs, geospatial indexes, and streams.

RabbitMQ

RabbitMQ gives your applications a common platform to send and receive messages, and your messages a safe place to live until received.

Celery

Celery is an asynchronous task queue/job queue based on distributed message passing. It is focused on real-time operation, but supports scheduling as well.

Amazon SQS

Transmit any volume of data, at any level of throughput, without losing messages or requiring other services to be always available. With SQS, you can offload the administrative burden of operating and scaling a highly available messaging cluster, while paying a low price for only what you use.

NSQ

NSQ is a realtime distributed messaging platform designed to operate at scale, handling billions of messages per day. It promotes distributed and decentralized topologies without single points of failure, enabling fault tolerance and high availability coupled with a reliable message delivery guarantee. See features & guarantees.

ActiveMQ

Apache ActiveMQ is fast, supports many Cross Language Clients and Protocols, comes with easy to use Enterprise Integration Patterns and many advanced features while fully supporting JMS 1.1 and J2EE 1.4. Apache ActiveMQ is released under the Apache 2.0 License.

ZeroMQ

The 0MQ lightweight messaging kernel is a library which extends the standard socket interfaces with features traditionally provided by specialised messaging middleware products. 0MQ sockets provide an abstraction of asynchronous message queues, multiple messaging patterns, message filtering (subscriptions), seamless access to multiple transport protocols and more.

Apache NiFi

An easy to use, powerful, and reliable system to process and distribute data. It supports powerful and scalable directed graphs of data routing, transformation, and system mediation logic.

Aerospike

Aerospike is an open-source, modern database built from the ground up to push the limits of flash storage, processors and networks. It was designed to operate with predictable low latency at high throughput with uncompromising reliability – both high availability and ACID guarantees.

Gearman

Gearman allows you to do work in parallel, to load balance processing, and to call functions between languages. It can be used in a variety of applications, from high-availability web sites to the transport of database replication events.

Related Comparisons

Hazelcast vs Kafka: What are the differences?

Hazelcast and Kafka are distributed systems addressing different aspects of real-time data processing. Let's explore the key differences between them.

Data Processing Model: Hazelcast is an in-memory data grid platform that allows distributed storage and processing of data. It provides distributed caching, distributed computing, and distributed messaging capabilities. On the other hand, Kafka is a distributed streaming platform designed for handling real-time data feeds. It enables data streams to be processed in a fault-tolerant and scalable manner.
Data Storage and Persistence: Hazelcast provides an in-memory data grid, allowing fast data access and processing. It stores data in memory across a cluster of nodes. However, Hazelcast also supports the option to persist data to disk for fault tolerance. In contrast, Kafka does not store data in memory like Hazelcast. Instead, Kafka uses a distributed commit log to store and replicate data on disk.
Messaging vs Event Streaming: Hazelcast supports publish-subscribe messaging, where messages are sent to a channel and multiple subscribers receive those messages. It provides reliable messaging with features like message ordering and durability. Kafka, on the other hand, is an event streaming platform that enables the continuous flow of events between systems or applications. It provides a fault-tolerant event-driven architecture, allowing real-time data processing and analytics.
Data Processing Paradigm: Hazelcast supports distributed computing paradigms, such as MapReduce and parallel computation models. It enables distributed data processing across a cluster of nodes, allowing for faster and more efficient processing of large datasets. Kafka, on the other hand, follows the publish-subscribe messaging model and is specifically tailored for real-time data processing and streaming.
Data Replication and Fault Tolerance: Hazelcast provides built-in data replication and fault tolerance mechanisms. It replicates data across multiple nodes in a cluster, ensuring that data is available even in the event of node failures. Kafka uses a distributed commit log for data replication and fault tolerance. It replicates data across multiple brokers, providing high availability and durability.
Integration Ecosystem: Hazelcast integrates well with various programming languages and frameworks. It provides client libraries for Java, .NET, Python, and other languages, making it easy to develop applications using Hazelcast. Kafka, on the other hand, comes with a well-established ecosystem and has extensive support for integration with various technologies and frameworks, such as Apache Spark, Apache Flink, and Apache Storm.

Hazelcast vs Kafka

Overview