Kafka vs Redis

Need advice about which tool to choose?Ask the StackShare community!

Kafka

13.9K
13.1K
+ 1
562
Redis

38.3K
28.1K
+ 1
3.9K
Add tool

Kafka vs Redis: What are the differences?

Kafka: Distributed, fault tolerant, high throughput pub-sub messaging system. Kafka is a distributed, partitioned, replicated commit log service. It provides the functionality of a messaging system, but with a unique design; Redis: An in-memory database that persists on disk. Redis is an open source, BSD licensed, advanced key-value store. It is often referred to as a data structure server since keys can contain strings, hashes, lists, sets and sorted sets.

Kafka can be classified as a tool in the "Message Queue" category, while Redis is grouped under "In-Memory Databases".

"High-throughput", "Distributed" and "Scalable" are the key factors why developers consider Kafka; whereas "Performance", "Super fast" and "Ease of use " are the primary reasons why Redis is favored.

Kafka and Redis are both open source tools. Redis with 37.4K GitHub stars and 14.4K forks on GitHub appears to be more popular than Kafka with 12.7K GitHub stars and 6.81K GitHub forks.

Airbnb, Uber Technologies, and Instagram are some of the popular companies that use Redis, whereas Kafka is used by Uber Technologies, Spotify, and Slack. Redis has a broader approval, being mentioned in 3261 company stacks & 1781 developers stacks; compared to Kafka, which is listed in 509 company stacks and 470 developer stacks.

Advice on Kafka and Redis
Needs advice
on
Redis
RabbitMQ
and
Kafka

We are going to develop a microservices-based application. It consists of AngularJS, ASP.NET Core, and MSSQL.

We have 3 types of microservices. Emailservice, Filemanagementservice, Filevalidationservice

I am a beginner in microservices. But I have read about RabbitMQ, but come to know that there are Redis and Kafka also in the market. So, I want to know which is best.

See more
Replies (4)
Maheedhar Aluri
Recommends
Kafka

Kafka is an Enterprise Messaging Framework whereas Redis is an Enterprise Cache Broker, in-memory database and high performance database.Both are having their own advantages, but they are different in usage and implementation. Now if you are creating microservices check the user consumption volumes, its generating logs, scalability, systems to be integrated and so on. I feel for your scenario initially you can go with KAFKA bu as the throughput, consumption and other factors are scaling then gradually you can add Redis accordingly.

See more
Recommends
Angular 2

I first recommend that you choose Angular over AngularJS if you are starting something new. AngularJs is no longer getting enhancements, but perhaps you meant Angular. Regarding microservices, I recommend considering microservices when you have different development teams for each service that may want to use different programming languages and backend data stores. If it is all the same team, same code language, and same data store I would not use microservices. I might use a message queue, in which case RabbitMQ is a good one. But you may also be able to simply write your own in which you write a record in a table in MSSQL and one of your services reads the record from the table and processes it. The most challenging part of doing it yourself is writing a service that does a good job of reading the queue without reading the same message multiple times or missing a message; and that is where RabbitMQ can help.

See more
Amit Mor
Software Architect at Payoneer · | 3 upvotes · 198.9K views
Recommends
Kafka

I think something is missing here and you should consider answering it to yourself. You are building a couple of services. Why are you considering event-sourcing architecture using Message Brokers such as the above? Won't a simple REST service based arch suffice? Read about CQRS and the problems it entails (state vs command impedance for example). Do you need Pub/Sub or Push/Pull? Is queuing of messages enough or would you need querying or filtering of messages before consumption? Also, someone would have to manage these brokers (unless using managed, cloud provider based solution), automate their deployment, someone would need to take care of backups, clustering if needed, disaster recovery, etc. I have a good past experience in terms of manageability/devops of the above options with Kafka and Redis, not so much with RabbitMQ. Both are very performant. But also note that Redis is not a pure message broker (at time of writing) but more of a general purpose in-memory key-value store. Kafka nowadays is much more than a distributed message broker. Long story short. In my taste, you should go with a minialistic approach and try to avoid either of them if you can, especially if your architecture does not fall nicely into event sourcing. If not I'd examine Kafka. If you need more capabilities than I'd consider Redis and use it for all sorts of other things such as a cache.

See more
Recommends
NATS

We found that the CNCF landscape is a good advisor when working going into the cloud / microservices space: https://landscape.cncf.io/fullscreen=yes. When choosing a technology one important criteria to me is if it is cloud native or not. Neither Redis, RabbitMQ nor Kafka is cloud native. The try to adapt but will be replaced eventually with technologies that are cloud native.

We have gone with NATS and have never looked back. We haven't spend a single minute on server maintainance in the last year and the setup of a cluster is way too easy. With the new features NATS incorporates now (and the ones still on the roadmap) it is already and will be sooo much mure than Redis, RabbitMQ and Kafka are. It can replace service discovery, load balancing, global multiclusters and failover, etc, etc.

Your thought might be: But I don't need all of that! Well, at the same time it is much more leightweight than Redis, RabbitMQ and especially Kafka.

See more
View all (4)
Needs advice
on
Redis
Kafka
and
Django

I'm building a website where users can participate, like and dislike any given challenge.

Problem : If 10k or 1 million users join the given challenge at a time it can cause a race condition in my database MySQL and in also Redis.

What I want : Aggregating joined participated users, likes and dislikes.

Solution : I'm thinking about using Kafka as a Queue message broker then users event one by one saving into Redis, database and aggregate them.

One problem is also here saving and doing aggregate takes time now; how can I show users they have successfully joined the challenge?

One solution is that when a user joins the challenge I send a request to the Kafka queue then update the current user UI and show a success message (not updating the other users' joined messages to current user because I am not using Websockets)

Other App example Take the same example of https://stackshare.io posts. On posts users can like, dislike and comments.

Estimated users : 1 million Stack : Django, Mysql, Redis and Kafka

Questions

  • How I can manage these kinds of things?
  • How do big tech companies handle this?
  • Where am I right or wrong?
  • Are there other tools that can help me in this situation?
  • I am using locks in Redis when total like, dislike and joined users increment or decrement. Should I be doing this? Is it the same for transactions in MySQL?

I need the best approach to handle this situation that can also be scalable.

Thanks in advance for reading my post and giving me suggestions on this. ☺️

See more
Replies (1)
Oirere Jr.
Software Engineeer at Serftre Inc · | 4 upvotes · 9K views
Recommends
Kafka

Consider using SQL support on Apache Pinot, which is an online analytic processing datastore which can write complex SQL queries and also join different tables in Pinot with those in other datastores. Pinot enables you to build dashboards for quick analysis and reporting on aggregated data.

See more
Pramod Nikam
Co Founder at Usability Designs · | 2 upvotes · 135.7K views
Needs advice
on
NSQ
Kafka
and
Apache Thrift

I am looking into IoT World Solution where we have MQTT Broker. This MQTT Broker Sits in one of the Data Center. We are doing a lot of Alert and Alarm related processing on that Data, Currently, we are looking into Solution which can do distributed persistence of log/alert primarily on remote Disk.

Our primary need is to use lightweight where operational complexity and maintenance costs can be significantly reduced. We want to do it on-premise so we are not considering cloud solutions.

We looked into the following alternatives:

Apache Kafka - Great choice but operation and maintenance wise very complex. Rabbit MQ - High availability is the issue, Apache Pulsar - Operational Complexity. NATS - Absence of persistence. Akka Streams - Big learning curve and operational streams.

So we are looking into a lightweight library that can do distributed persistence preferably with publisher and subscriber model. Preferable on JVM stack.

See more
Replies (1)
Naresh Kancharla
Staff Engineer at Nutanix · | 4 upvotes · 133K views
Recommends
Kafka

Kafka is best fit here. Below are the advantages with Kafka ACLs (Security), Schema (protobuf), Scale, Consumer driven and No single point of failure.

Operational complexity is manageable with open source monitoring tools.

See more
Decisions about Kafka and Redis
Kirill Mikhailov

Maybe not an obvious comparison with Kafka, since Kafka is pretty different from rabbitmq. But for small service, Rabbit as a pubsub platform is super easy to use and pretty powerful. Kafka as an alternative was the original choice, but its really a kind of overkill for a small-medium service. Especially if you are not planning to use k8s, since pure docker deployment can be a pain because of networking setup. Google PubSub was another alternative, its actually pretty cheap, but I never tested it since Rabbit was matching really good for mailing/notification services.

See more
Get Advice from developers at your company using Private StackShare. Sign up for Private StackShare.
Learn More
Pros of Kafka
Pros of Redis
  • 120
    High-throughput
  • 114
    Distributed
  • 86
    Scalable
  • 79
    High-Performance
  • 64
    Durable
  • 35
    Publish-Subscribe
  • 18
    Simple-to-use
  • 14
    Open source
  • 10
    Written in Scala and java. Runs on JVM
  • 6
    Message broker + Streaming system
  • 4
    Avro schema integration
  • 2
    Suport Multiple clients
  • 2
    Robust
  • 2
    KSQL
  • 2
    Partioned, replayable log
  • 1
    Fun
  • 1
    Extremely good parallelism constructs
  • 1
    Simple publisher / multi-subscriber model
  • 1
    Flexible
  • 877
    Performance
  • 535
    Super fast
  • 511
    Ease of use
  • 442
    In-memory cache
  • 321
    Advanced key-value cache
  • 190
    Open source
  • 179
    Easy to deploy
  • 163
    Stable
  • 153
    Free
  • 120
    Fast
  • 40
    High-Performance
  • 39
    High Availability
  • 34
    Data Structures
  • 32
    Very Scalable
  • 23
    Replication
  • 20
    Great community
  • 19
    Pub/Sub
  • 17
    "NoSQL" key-value data store
  • 14
    Hashes
  • 12
    Sets
  • 10
    Sorted Sets
  • 9
    Lists
  • 8
    BSD licensed
  • 8
    NoSQL
  • 7
    Async replication
  • 7
    Integrates super easy with Sidekiq for Rails background
  • 7
    Bitmaps
  • 6
    Open Source
  • 6
    Keys with a limited time-to-live
  • 5
    Strings
  • 5
    Lua scripting
  • 4
    Awesomeness for Free!
  • 4
    Hyperloglogs
  • 3
    outstanding performance
  • 3
    Runs server side LUA
  • 3
    Networked
  • 3
    LRU eviction of keys
  • 3
    Written in ANSI C
  • 3
    Feature Rich
  • 3
    Transactions
  • 2
    Data structure server
  • 2
    Performance & ease of use
  • 1
    Existing Laravel Integration
  • 1
    Automatic failover
  • 1
    Easy to use
  • 1
    Object [key/value] size each 500 MB
  • 1
    Simple
  • 1
    Channels concept
  • 1
    Scalable
  • 1
    Temporarily kept on disk
  • 1
    Dont save data if no subscribers are found
  • 0
    Jk

Sign up to add or upvote prosMake informed product decisions

Cons of Kafka
Cons of Redis
  • 27
    Non-Java clients are second-class citizens
  • 26
    Needs Zookeeper
  • 7
    Operational difficulties
  • 2
    Terrible Packaging
  • 12
    Cannot query objects directly
  • 1
    No WAL
  • 1
    No secondary indexes for non-numeric data types

Sign up to add or upvote consMake informed product decisions

What is Kafka?

Kafka is a distributed, partitioned, replicated commit log service. It provides the functionality of a messaging system, but with a unique design.

What is Redis?

Redis is an open source, BSD licensed, advanced key-value store. It is often referred to as a data structure server since keys can contain strings, hashes, lists, sets and sorted sets.

Need advice about which tool to choose?Ask the StackShare community!

What companies use Kafka?
What companies use Redis?
See which teams inside your own company are using Kafka or Redis.
Sign up for Private StackShareLearn More

Sign up to get full access to all the companiesMake informed product decisions

What tools integrate with Kafka?
What tools integrate with Redis?

Sign up to get full access to all the tool integrationsMake informed product decisions

Blog Posts

Mar 24 2021 at 12:57PM

Pinterest

+7
3
1433
Jun 24 2020 at 4:42PM

Pinterest

+4
4
1021
+6
2
1469
Jan 7 2020 at 5:09PM

Ably Realtime

+2
7
1694
What are some alternatives to Kafka and Redis?
ActiveMQ
Apache ActiveMQ is fast, supports many Cross Language Clients and Protocols, comes with easy to use Enterprise Integration Patterns and many advanced features while fully supporting JMS 1.1 and J2EE 1.4. Apache ActiveMQ is released under the Apache 2.0 License.
RabbitMQ
RabbitMQ gives your applications a common platform to send and receive messages, and your messages a safe place to live until received.
Amazon Kinesis
Amazon Kinesis can collect and process hundreds of gigabytes of data per second from hundreds of thousands of sources, allowing you to easily write applications that process information in real-time, from sources such as web site click-streams, marketing and financial information, manufacturing instrumentation and social media, and operational logs and metering data.
Apache Spark
Spark is a fast and general processing engine compatible with Hadoop data. It can run in Hadoop clusters through YARN or Spark's standalone mode, and it can process data in HDFS, HBase, Cassandra, Hive, and any Hadoop InputFormat. It is designed to perform both batch processing (similar to MapReduce) and new workloads like streaming, interactive queries, and machine learning.
Akka
Akka is a toolkit and runtime for building highly concurrent, distributed, and resilient message-driven applications on the JVM.
See all alternatives
Reviews of Kafka and Redis
Review of
Redis

Redis is a good caching tool for a cluster, but our application had performance issues while using Aws Elasticache Redis since some page had 3000 cache hits per a page load and Redis just couldn't quickly process them all in once + latency and object deseialization time - page load took 8-9 seconds. We create a custom hybrid caching based on Redis and EhCache which worked great for our goals. Check it out on github, it's called HybriCache - https://github.com/batir-akhmerov/hybricache.

How developers use Kafka and Redis
Pinterest uses
Kafka

http://media.tumblr.com/d319bd2624d20c8a81f77127d3c878d0/tumblr_inline_nanyv6GCKl1s1gqll.png

Front-end messages are logged to Kafka by our API and application servers. We have batch processing (on the middle-left) and real-time processing (on the middle-right) pipelines to process the experiment data. For batch processing, after daily raw log get to s3, we start our nightly experiment workflow to figure out experiment users groups and experiment metrics. We use our in-house workflow management system Pinball to manage the dependencies of all these MapReduce jobs.

Cloudcraft uses
Redis

Redis is used for storing all ephemeral (that's data you don't necessarily want to store permanently) user data, such as mapping of session IDs (stored in cookies) to current session variables at Cloudcraft.co. The many datastructures supported by Redis also makes it an excellent caching and realtime statistics layer. It doesn't hurt that the author, Antirez, is the nicest guy ever! These days, I would be really hard pressed to find any situation where I would pick something like Memcached over Redis.

Trello uses
Redis

Trello uses Redis for ephemeral data that needs to be shared between server processes but not persisted to disk. Things like the activity level of a session or a temporary OpenID key are stored in Redis, and the application is built to recover gracefully if any of these (or all of them) are lost. We run with allkeys-lru enabled and about five times as much space as its actual working set needs, so Redis automatically discards data that hasn’t been accessed lately, and reconstructs it when necessary.

Stack Exchange uses
Redis

The UI has message inbox that is sent a message when you get a new badge, receive a message, significant event, etc. Done using WebSockets and is powered by redis. Redis has 2 slaves, SQL has 2 replicas, tag engine has 3 nodes, elastic has 3 nodes - any other service has high availability as well (and exists in both data centers).

Brandon Adams uses
Redis

Redis makes certain operations very easy. When I need a high-availability store, I typically look elsewhere, but for rapid development with the ability to land on your feet in prod, Redis is great. The available data types make it easy to build non-trivial indexes that would require complex queries in postgres.

Kent Steiner uses
Redis

I use Redis for cacheing, data storage, mining and augmentation, proprietary distributed event system for disparate apps and services to talk to each other, and more. Redis has some very useful native data types for tracking, slicing and dicing information.

Coolfront Technologies uses
Kafka

Building out real-time streaming server to present data insights to Coolfront Mobile customers and internal sales and marketing teams.

ShareThis uses
Kafka

We are using Kafka as a message queue to process our widget logs.

Christopher Davison uses
Kafka

Used for communications and triggering jobs across ETL systems

theskyinflames uses
Kafka

Used as a integration middleware by messaging interchanging.