Hi, First of all, understand the difference. All works as message brokers. All know JSON. But Redis is not a queue or topic. Its an in memory cache. RabbitMq/Kafka persists data on file system but redis holds in a temporary memory. If redis is down and if u cant use the cache dump wisely then ur data gone. Redis is very short lived broker though its fast due to less i/o operations and other commits that kafak does. I dont have much exp in RabbitMq so cant comment on that but RabbitMq is better in administration than open sourcen kafka in case your maanger dont want to pay money ….. 😂

Coming to decision… If You are ready to risk/ compromise the in memory data inside cache then go for redis. If you are not concerned about horizontal scaling and can do ur job by vertical scaling then go for redis. If you want horizontal scaling and want to persist data on disk for fetching later then go for kafka. But u cant achieve speed unless you fine tune your consumers. Need better understanding of consumer threads, partitioning, poll intervals etc. If you use confluent platform the dont even compare kafka with redis and rabbitmq.

Cheers! Sunil.

We have faced the same question some time ago. Before I begin, DO NOT use Redis as a message broker. It is fast and easy to set up in the beginning but it does not scale. It is not made to be reliable in scale and that is mentioned in the official docs. This analysis of our problems with Redis may help you.

We have used Kafka and RabbitMQ both in scale. We concluded that RabbitMQ is a really good general purpose message broker (for our case) and Kafka is really fast but limited in features. That’s the trade off that we understood from using it. In-fact I blogged about the trade offs between Kafka and RabbitMQ to document it. I hope it helps you in choosing the best pub-sub layer for your use case.