We are currently moving to a microservice architecture and are debating about the different options there are to handle communication between services. We are currently considering Kafka, Redis or RabbitMQ as a message broker. As RabbitMQ is a little bit older, we thought that it may be outdated. Is that true? Can RabbitMQ hold up to more modern tools like Redis and Kafka?
We have faced the same question some time ago. Before I begin, DO NOT use Redis as a message broker. It is fast and easy to set up in the beginning but it does not scale. It is not made to be reliable in scale and that is mentioned in the official docs. This analysis of our problems with Redis may help you.
We have used Kafka and RabbitMQ both in scale. We concluded that RabbitMQ is a really good general purpose message broker (for our case) and Kafka is really fast but limited in features. That’s the trade off that we understood from using it. In-fact I blogged about the trade offs between Kafka and RabbitMQ to document it. I hope it helps you in choosing the best pub-sub layer for your use case.
It depends on your requirements like number of messages to be processed per second, real time messages vs delayed, number of servers available for your cluster, whether you need streaming, etc.. Kafka works for most use cases. Not related to answer but would like to add no matter whatever broker you chose, for connecting to the broker always go for the library provided by the broker rather than Spring kafka or Spring AMQP. If you use Spring, then you will be stuck with specific Spring versions. In case you find bugs in spring then difficult because you will have to upgrade entire application to use a later Spring core version. In general, use as minimum libraries as possible to get rid of nuisance of upgrading them when they are outdated or bugs are found with them.
Thanks for the insight! A fast message broker would be important, persistency isn't. We also plan to deploy the message broker as a docker container to our cluster. I read somewhere online that kafka is not meant to be deployed as a container... Is that true? (What also confused me is that there isn't a official docker image for kafka).
Radis is bit different compared to Rabbit MQand Kafka so use Redis only if its for non critical message flow. Between Rabbit MQ and Kafka , our experience as been for large message processing application Rabbit becomes really unstable and even have encountered corrupt data so we switched to Kafka which is more reliable
Hi Makarand, how easy was your journey from RabbitMQ to Kafka? Is it okay to ask if you have had any specific challenges as RabbitMQ sends to consumers while in Kafka Consumer needs to read. + Message structure is a bit different too?
Also, It would be nice to know if you migrated to Managed Kafka service or self hosted? (I am trying to understand how tough would it be to manage our own Kafka as we are almost finalising going with Kafka) :)
Thanks