Need advice about which tool to choose?Ask the StackShare community!

Hazelcast

226
332
+ 1
53
Kafka

13.7K
12.7K
+ 1
556
Add tool

Hazelcast vs Kafka: What are the differences?

Developers describe Hazelcast as "Clustering and highly scalable data distribution platform for Java". With its various distributed data structures, distributed caching capabilities, elastic nature, memcache support, integration with Spring and Hibernate and more importantly with so many happy users, Hazelcast is feature-rich, enterprise-ready and developer-friendly in-memory data grid solution. On the other hand, Kafka is detailed as "Distributed, fault tolerant, high throughput pub-sub messaging system". Kafka is a distributed, partitioned, replicated commit log service. It provides the functionality of a messaging system, but with a unique design.

Hazelcast and Kafka are primarily classified as "In-Memory Databases" and "Message Queue" tools respectively.

Some of the features offered by Hazelcast are:

  • Distributed implementations of java.util.{Queue, Set, List, Map}
  • Distributed implementation of java.util.concurrent.locks.Lock
  • Distributed implementation of java.util.concurrent.ExecutorService

On the other hand, Kafka provides the following key features:

  • Written at LinkedIn in Scala
  • Used by LinkedIn to offload processing of all page and other views
  • Defaults to using persistence, uses OS disk cache for hot data (has higher throughput then any of the above having persistence enabled)

"High Availibility" is the primary reason why developers consider Hazelcast over the competitors, whereas "High-throughput" was stated as the key factor in picking Kafka.

Hazelcast and Kafka are both open source tools. Kafka with 12.5K GitHub stars and 6.7K forks on GitHub appears to be more popular than Hazelcast with 3.15K GitHub stars and 1.15K GitHub forks.

Slack, Shopify, and SendGrid are some of the popular companies that use Kafka, whereas Hazelcast is used by Yammer, Seat Pagine Gialle, and Stormpath. Kafka has a broader approval, being mentioned in 500 company stacks & 451 developers stacks; compared to Hazelcast, which is listed in 25 company stacks and 15 developer stacks.

Advice on Hazelcast and Kafka
Needs advice
on
Redis
RabbitMQ
and
Kafka

We are going to develop a microservices-based application. It consists of AngularJS, ASP.NET Core, and MSSQL.

We have 3 types of microservices. Emailservice, Filemanagementservice, Filevalidationservice

I am a beginner in microservices. But I have read about RabbitMQ, but come to know that there are Redis and Kafka also in the market. So, I want to know which is best.

See more
Replies (4)
Maheedhar Aluri
Recommends
Kafka

Kafka is an Enterprise Messaging Framework whereas Redis is an Enterprise Cache Broker, in-memory database and high performance database.Both are having their own advantages, but they are different in usage and implementation. Now if you are creating microservices check the user consumption volumes, its generating logs, scalability, systems to be integrated and so on. I feel for your scenario initially you can go with KAFKA bu as the throughput, consumption and other factors are scaling then gradually you can add Redis accordingly.

See more
Recommends
Angular 2

I first recommend that you choose Angular over AngularJS if you are starting something new. AngularJs is no longer getting enhancements, but perhaps you meant Angular. Regarding microservices, I recommend considering microservices when you have different development teams for each service that may want to use different programming languages and backend data stores. If it is all the same team, same code language, and same data store I would not use microservices. I might use a message queue, in which case RabbitMQ is a good one. But you may also be able to simply write your own in which you write a record in a table in MSSQL and one of your services reads the record from the table and processes it. The most challenging part of doing it yourself is writing a service that does a good job of reading the queue without reading the same message multiple times or missing a message; and that is where RabbitMQ can help.

See more
Recommends
NATS

We found that the CNCF landscape is a good advisor when working going into the cloud / microservices space: https://landscape.cncf.io/fullscreen=yes. When choosing a technology one important criteria to me is if it is cloud native or not. Neither Redis, RabbitMQ nor Kafka is cloud native. The try to adapt but will be replaced eventually with technologies that are cloud native.

We have gone with NATS and have never looked back. We haven't spend a single minute on server maintainance in the last year and the setup of a cluster is way too easy. With the new features NATS incorporates now (and the ones still on the roadmap) it is already and will be sooo much mure than Redis, RabbitMQ and Kafka are. It can replace service discovery, load balancing, global multiclusters and failover, etc, etc.

Your thought might be: But I don't need all of that! Well, at the same time it is much more leightweight than Redis, RabbitMQ and especially Kafka.

See more
Amit Mor
Software Architect at Payoneer · | 3 upvotes · 176K views
Recommends
Kafka

I think something is missing here and you should consider answering it to yourself. You are building a couple of services. Why are you considering event-sourcing architecture using Message Brokers such as the above? Won't a simple REST service based arch suffice? Read about CQRS and the problems it entails (state vs command impedance for example). Do you need Pub/Sub or Push/Pull? Is queuing of messages enough or would you need querying or filtering of messages before consumption? Also, someone would have to manage these brokers (unless using managed, cloud provider based solution), automate their deployment, someone would need to take care of backups, clustering if needed, disaster recovery, etc. I have a good past experience in terms of manageability/devops of the above options with Kafka and Redis, not so much with RabbitMQ. Both are very performant. But also note that Redis is not a pure message broker (at time of writing) but more of a general purpose in-memory key-value store. Kafka nowadays is much more than a distributed message broker. Long story short. In my taste, you should go with a minialistic approach and try to avoid either of them if you can, especially if your architecture does not fall nicely into event sourcing. If not I'd examine Kafka. If you need more capabilities than I'd consider Redis and use it for all sorts of other things such as a cache.

See more
View all (4)
Pramod Nikam
Co Founder at Usability Designs · | 2 upvotes · 118.9K views
Needs advice
on
NSQ
Kafka
and
Apache Thrift

I am looking into IoT World Solution where we have MQTT Broker. This MQTT Broker Sits in one of the Data Center. We are doing a lot of Alert and Alarm related processing on that Data, Currently, we are looking into Solution which can do distributed persistence of log/alert primarily on remote Disk.

Our primary need is to use lightweight where operational complexity and maintenance costs can be significantly reduced. We want to do it on-premise so we are not considering cloud solutions.

We looked into the following alternatives:

Apache Kafka - Great choice but operation and maintenance wise very complex. Rabbit MQ - High availability is the issue, Apache Pulsar - Operational Complexity. NATS - Absence of persistence. Akka Streams - Big learning curve and operational streams.

So we are looking into a lightweight library that can do distributed persistence preferably with publisher and subscriber model. Preferable on JVM stack.

See more
Replies (1)
Naresh Kancharla
Staff Engineer at Nutanix · | 4 upvotes · 116.2K views
Recommends
Kafka

Kafka is best fit here. Below are the advantages with Kafka ACLs (Security), Schema (protobuf), Scale, Consumer driven and No single point of failure.

Operational complexity is manageable with open source monitoring tools.

See more
Needs advice
on
RabbitMQ
and
Kafka

Our backend application is sending some external messages to a third party application at the end of each backend (CRUD) API call (from UI) and these external messages take too much extra time (message building, processing, then sent to the third party and log success/failure), UI application has no concern to these extra third party messages.

So currently we are sending these third party messages by creating a new child thread at end of each REST API call so UI application doesn't wait for these extra third party API calls.

I want to integrate Apache Kafka for these extra third party API calls, so I can also retry on failover third party API calls in a queue(currently third party messages are sending from multiple threads at the same time which uses too much processing and resources) and logging, etc.

Question 1: Is this a use case of a message broker?

Question 2: If it is then Kafka vs RabitMQ which is the better?

See more
Replies (4)
Tarun Batra
Back End Developer at instabox · | 7 upvotes · 140.7K views
Recommends
RabbitMQ

RabbitMQ is great for queuing and retrying. You can send the requests to your backend which will further queue these requests in RabbitMQ (or Kafka, too). The consumer on the other end can take care of processing . For a detailed analysis, check this blog about choosing between Kafka and RabbitMQ.

See more
Trevor Rydalch
Software Engineer at InsideSales.com · | 6 upvotes · 140.7K views
Recommends
RabbitMQ

Well, first off, it's good practice to do as little non-UI work on the foreground thread as possible, regardless of whether the requests take a long time. You don't want the UI thread blocked.

This sounds like a good use case for RabbitMQ. Primarily because you don't need each message processed by more than one consumer. If you wanted to process a single message more than once (say for different purposes), then Apache Kafka would be a much better fit as you can have multiple consumer groups consuming from the same topics independently.

Have your API publish messages containing the data necessary for the third-party request to a Rabbit queue and have consumers reading off there. If it fails, you can either retry immediately, or publish to a deadletter queue where you can reprocess them whenever you want (shovel them back into the regular queue).

See more
Guillaume Maka
Full Stack Web Developer · | 2 upvotes · 140.2K views
Recommends
RabbitMQ

As far as I understand, Kafka is a like a persisted event state manager where you can plugin various source of data and transform/query them as event via a stream API. Regarding your use case I will consider using RabbitMQ if your intent is to implement service inter-communication kind of thing. RabbitMQ is a good choice for one-one publisher/subscriber (or consumer) and I think you can also have multiple consumers by configuring a fanout exchange. RabbitMQ provide also message retries, message cancellation, durable queue, message requeue, message ACK....

See more
Recommends
RabbitMQ

In my opinion RabbitMQ fits better in your case because you don’t have order in queue. You can process your messages in any order. You don’t need to store the data what you sent. Kafka is a persistent storage like the blockchain. RabbitMQ is a message broker. Kafka is not a good solution for the system with confirmations of the messages delivery.

See more
View all (4)
Get Advice from developers at your company using Private StackShare. Sign up for Private StackShare.
Learn More
Pros of Hazelcast
Pros of Kafka
  • 9
    High Availibility
  • 6
    Distributed Locking
  • 5
    Distributed compute
  • 5
    Sharding
  • 4
    Load balancing
  • 3
    Map-reduce functionality
  • 3
    Publish-subscribe
  • 3
    Sql query support in cluster wide
  • 2
    Written in java. runs on jvm
  • 2
    Simple-to-use
  • 2
    Multiple client language support
  • 2
    Rest interface
  • 2
    Optimis locking for map
  • 1
    Easy to use
  • 1
    Performance
  • 1
    Super Fast
  • 1
    Admin Interface (Management Center)
  • 1
    Better Documentation
  • 118
    High-throughput
  • 113
    Distributed
  • 85
    Scalable
  • 78
    High-Performance
  • 64
    Durable
  • 35
    Publish-Subscribe
  • 17
    Simple-to-use
  • 14
    Open source
  • 10
    Written in Scala and java. Runs on JVM
  • 6
    Message broker + Streaming system
  • 4
    Avro schema integration
  • 2
    Suport Multiple clients
  • 2
    Robust
  • 2
    KSQL
  • 2
    Partioned, replayable log
  • 1
    Fun
  • 1
    Extremely good parallelism constructs
  • 1
    Simple publisher / multi-subscriber model
  • 1
    Flexible

Sign up to add or upvote prosMake informed product decisions

Cons of Hazelcast
Cons of Kafka
  • 3
    License needed for SSL
  • 27
    Non-Java clients are second-class citizens
  • 26
    Needs Zookeeper
  • 7
    Operational difficulties
  • 2
    Terrible Packaging

Sign up to add or upvote consMake informed product decisions

What is Hazelcast?

With its various distributed data structures, distributed caching capabilities, elastic nature, memcache support, integration with Spring and Hibernate and more importantly with so many happy users, Hazelcast is feature-rich, enterprise-ready and developer-friendly in-memory data grid solution.

What is Kafka?

Kafka is a distributed, partitioned, replicated commit log service. It provides the functionality of a messaging system, but with a unique design.

Need advice about which tool to choose?Ask the StackShare community!

Jobs that mention Hazelcast and Kafka as a desired skillset
What companies use Hazelcast?
What companies use Kafka?
See which teams inside your own company are using Hazelcast or Kafka.
Sign up for Private StackShareLearn More

Sign up to get full access to all the companiesMake informed product decisions

What tools integrate with Hazelcast?
What tools integrate with Kafka?

Sign up to get full access to all the tool integrationsMake informed product decisions

Blog Posts

Mar 24 2021 at 12:57PM

Pinterest

+7
3
1174
Jun 24 2020 at 4:42PM

Pinterest

+4
4
994
+6
2
1420
Jan 7 2020 at 5:09PM

Ably Realtime

+2
7
1690
What are some alternatives to Hazelcast and Kafka?
Redis
Redis is an open source, BSD licensed, advanced key-value store. It is often referred to as a data structure server since keys can contain strings, hashes, lists, sets and sorted sets.
Apache Spark
Spark is a fast and general processing engine compatible with Hadoop data. It can run in Hadoop clusters through YARN or Spark's standalone mode, and it can process data in HDFS, HBase, Cassandra, Hive, and any Hadoop InputFormat. It is designed to perform both batch processing (similar to MapReduce) and new workloads like streaming, interactive queries, and machine learning.
Cassandra
Partitioning means that Cassandra can distribute your data across multiple machines in an application-transparent matter. Cassandra will automatically repartition as machines are added and removed from the cluster. Row store means that like relational databases, Cassandra organizes data by rows and columns. The Cassandra Query Language (CQL) is a close relative of SQL.
Memcached
Memcached is an in-memory key-value store for small chunks of arbitrary data (strings, objects) from results of database calls, API calls, or page rendering.
Apache Ignite
It is a memory-centric distributed database, caching, and processing platform for transactional, analytical, and streaming workloads delivering in-memory speeds at petabyte scale
See all alternatives
How developers use Hazelcast and Kafka
Pinterest uses
Kafka

http://media.tumblr.com/d319bd2624d20c8a81f77127d3c878d0/tumblr_inline_nanyv6GCKl1s1gqll.png

Front-end messages are logged to Kafka by our API and application servers. We have batch processing (on the middle-left) and real-time processing (on the middle-right) pipelines to process the experiment data. For batch processing, after daily raw log get to s3, we start our nightly experiment workflow to figure out experiment users groups and experiment metrics. We use our in-house workflow management system Pinball to manage the dependencies of all these MapReduce jobs.

Vital Labs, Inc. uses
Hazelcast

HazelCast is the foundation for the distributed system that hosts our APIs and intelligent workflows. We wrap the core HazelCast functions in Clojure protocols to implement micro-services on top of a coherent, single-process instance per virtual node.

Coolfront Technologies uses
Kafka

Building out real-time streaming server to present data insights to Coolfront Mobile customers and internal sales and marketing teams.

ShareThis uses
Kafka

We are using Kafka as a message queue to process our widget logs.

Christopher Davison uses
Kafka

Used for communications and triggering jobs across ETL systems

theskyinflames uses
Kafka

Used as a integration middleware by messaging interchanging.