Frédéric MARAND

The main reasons for using Go as a Kafka consumer at Deliveroo are (1) simplicity and reliabiliy of code (2) speed of execution.

For (1), Python works just as well and indeed, we do have some low-scale applications using Python as it tends to be more productive for smaller and/or shorter life projects. But Go remains upwards compatible since its inception while the 2->3 conversion in Python was no small matter, and even minor version conversions tend to require many dependency upgrades, which tend not to be synchronised. For (2) at high scale (think 1M transactions/5 minutes), Go is simpler to scale on less hardware. We also use Kotlin and Scala for this because, beyond pure manually written consumers, Go misses a real equivalent to Kafka Streams, as that only exists on the JVM.

The Datadog statsd agent is not really a normal statsd client: it implements a large subset of the original (etsy) features, but also some Datadog-specific features (about histograms). It is used to send metrics to Datadog APM, and its big advantage is the developer experience, which is familiar and easy to use, just like any statsd client, making it trivial to replace an existing statsd metrics client in any application with the Datadog version to publish metrics to Datadog.

I use TypeScript because I tried both on a Meteor project, and found the quantity of errors it enabled us to catch and the simplification of code it allowed was higher than Flow (JS).

I used Kafka originally because it was mandated as part of the top-level IT requirements at a Fortune 500 client. What I found was that it was orders of magnitude more complex ...and powerful than my daily Beanstalkd , and far more flexible, resilient, and manageable than RabbitMQ.

So for any case where utmost flexibility and resilience are part of the deal, I would use Kafka again. But due to the complexities involved, for any time where this level of scalability is not required, I would probably just use Beanstalkd for its simplicity.

I tend to find RabbitMQ to be in an uncomfortable middle place between these two extremities.