Alternatives to Kafka logo

Alternatives to Kafka

ActiveMQ, RabbitMQ, Amazon Kinesis, Apache Spark, and Akka are the most popular alternatives and competitors to Kafka.
3.8K
3.1K
+ 1
473

What is Kafka and what are its top alternatives?

Kafka is a distributed, partitioned, replicated commit log service. It provides the functionality of a messaging system, but with a unique design.
Kafka is a tool in the Message Queue category of a tech stack.
Kafka is an open source tool with 14.4K GitHub stars and 7.6K GitHub forks. Here’s a link to Kafka's open source repository on GitHub

Kafka alternatives & related posts

ActiveMQ logo

ActiveMQ

208
241
53
208
241
+ 1
53
A message broker written in Java together with a full JMS client
ActiveMQ logo
ActiveMQ
VS
Kafka logo
Kafka

related ActiveMQ posts

Naushad Warsi
Naushad Warsi
software developer at klingelnberg · | 1 upvotes · 62K views
ActiveMQ
ActiveMQ
RabbitMQ
RabbitMQ

I use ActiveMQ because RabbitMQ have stopped giving the support for AMQP 1.0 or above version and the earlier version of AMQP doesn't give the functionality to support OAuth.

If OAuth is not required and we can go with AMQP 0.9 then i still recommend rabbitMq.

See more

related RabbitMQ posts

James Cunningham
James Cunningham
Operations Engineer at Sentry · | 18 upvotes · 148.1K views
atSentrySentry
Celery
Celery
RabbitMQ
RabbitMQ
#MessageQueue

As Sentry runs throughout the day, there are about 50 different offline tasks that we execute—anything from “process this event, pretty please” to “send all of these cool people some emails.” There are some that we execute once a day and some that execute thousands per second.

Managing this variety requires a reliably high-throughput message-passing technology. We use Celery's RabbitMQ implementation, and we stumbled upon a great feature called Federation that allows us to partition our task queue across any number of RabbitMQ servers and gives us the confidence that, if any single server gets backlogged, others will pitch in and distribute some of the backlogged tasks to their consumers.

#MessageQueue

See more
Tim Abbott
Tim Abbott
Founder at Zulip · | 10 upvotes · 89.2K views
atZulipZulip
RabbitMQ
RabbitMQ
Python
Python
Redis
Redis

We've been using RabbitMQ as Zulip's queuing system since we needed a queuing system. What I like about it is that it scales really well and has good libraries for a wide range of platforms, including our own Python. So aside from getting it running, we've had to put basically 0 effort into making it scale for our needs.

However, there's several things that could be better about it: * It's error messages are absolutely terrible; if ever one of our users ends up getting an error with RabbitMQ (even for simple things like a misconfigured hostname), they always end up needing to get help from the Zulip team, because the errors logs are just inscrutable. As an open source project, we've handled this issue by really carefully scripting the installation to be a failure-proof configuration (in this case, setting the RabbitMQ hostname to 127.0.0.1, so that no user-controlled configuration can break it). But it was a real pain to get there and the process of determining we needed to do that caused a significant amount of pain to folks installing Zulip. * The pika library for Python takes a lot of time to startup a RabbitMQ connection; this means that Zulip server restarts are more disruptive than would be ideal. * It's annoying that you need to run the rabbitmqctl management commands as root.

But overall, I like that it has clean, clear semanstics and high scalability, and haven't been tempted to do the work to migrate to something like Redis (which has its own downsides).

See more
Amazon Kinesis logo

Amazon Kinesis

330
159
0
330
159
+ 1
0
Store and process terabytes of data each hour from hundreds of thousands of sources
    Be the first to leave a pro
    Amazon Kinesis logo
    Amazon Kinesis
    VS
    Kafka logo
    Kafka

    related Amazon Kinesis posts

    John Kodumal
    John Kodumal
    CTO at LaunchDarkly · | 15 upvotes · 181.5K views
    atLaunchDarklyLaunchDarkly
    Amazon RDS
    Amazon RDS
    PostgreSQL
    PostgreSQL
    TimescaleDB
    TimescaleDB
    Patroni
    Patroni
    Consul
    Consul
    Amazon ElastiCache
    Amazon ElastiCache
    Amazon EC2
    Amazon EC2
    Redis
    Redis
    Amazon Kinesis
    Amazon Kinesis
    Kafka
    Kafka

    As we've evolved or added additional infrastructure to our stack, we've biased towards managed services. Most new backing stores are Amazon RDS instances now. We do use self-managed PostgreSQL with TimescaleDB for time-series data—this is made HA with the use of Patroni and Consul.

    We also use managed Amazon ElastiCache instances instead of spinning up Amazon EC2 instances to run Redis workloads, as well as shifting to Amazon Kinesis instead of Kafka.

    See more
    Tim Specht
    Tim Specht
    ‎Co-Founder and CTO at Dubsmash · | 14 upvotes · 89.6K views
    atDubsmashDubsmash
    Google Analytics
    Google Analytics
    Amazon Kinesis
    Amazon Kinesis
    AWS Lambda
    AWS Lambda
    Amazon SQS
    Amazon SQS
    Google BigQuery
    Google BigQuery
    #ServerlessTaskProcessing
    #GeneralAnalytics
    #RealTimeDataProcessing
    #BigDataAsAService

    In order to accurately measure & track user behaviour on our platform we moved over quickly from the initial solution using Google Analytics to a custom-built one due to resource & pricing concerns we had.

    While this does sound complicated, it’s as easy as clients sending JSON blobs of events to Amazon Kinesis from where we use AWS Lambda & Amazon SQS to batch and process incoming events and then ingest them into Google BigQuery. Once events are stored in BigQuery (which usually only takes a second from the time the client sends the data until it’s available), we can use almost-standard-SQL to simply query for data while Google makes sure that, even with terabytes of data being scanned, query times stay in the range of seconds rather than hours. Before ingesting their data into the pipeline, our mobile clients are aggregating events internally and, once a certain threshold is reached or the app is going to the background, sending the events as a JSON blob into the stream.

    In the past we had workers running that continuously read from the stream and would validate and post-process the data and then enqueue them for other workers to write them to BigQuery. We went ahead and implemented the Lambda-based approach in such a way that Lambda functions would automatically be triggered for incoming records, pre-aggregate events, and write them back to SQS, from which we then read them, and persist the events to BigQuery. While this approach had a couple of bumps on the road, like re-triggering functions asynchronously to keep up with the stream and proper batch sizes, we finally managed to get it running in a reliable way and are very happy with this solution today.

    #ServerlessTaskProcessing #GeneralAnalytics #RealTimeDataProcessing #BigDataAsAService

    See more

    related Apache Spark posts

    Eric Colson
    Eric Colson
    Chief Algorithms Officer at Stitch Fix · | 19 upvotes · 341.3K views
    atStitch FixStitch Fix
    Kafka
    Kafka
    PostgreSQL
    PostgreSQL
    Amazon S3
    Amazon S3
    Apache Spark
    Apache Spark
    Presto
    Presto
    Python
    Python
    R
    R
    PyTorch
    PyTorch
    Docker
    Docker
    Amazon EC2 Container Service
    Amazon EC2 Container Service
    #AWS
    #Etl
    #ML
    #DataScience
    #DataStack
    #Data

    The algorithms and data infrastructure at Stitch Fix is housed in #AWS. Data acquisition is split between events flowing through Kafka, and periodic snapshots of PostgreSQL DBs. We store data in an Amazon S3 based data warehouse. Apache Spark on Yarn is our tool of choice for data movement and #ETL. Because our storage layer (s3) is decoupled from our processing layer, we are able to scale our compute environment very elastically. We have several semi-permanent, autoscaling Yarn clusters running to serve our data processing needs. While the bulk of our compute infrastructure is dedicated to algorithmic processing, we also implemented Presto for adhoc queries and dashboards.

    Beyond data movement and ETL, most #ML centric jobs (e.g. model training and execution) run in a similarly elastic environment as containers running Python and R code on Amazon EC2 Container Service clusters. The execution of batch jobs on top of ECS is managed by Flotilla, a service we built in house and open sourced (see https://github.com/stitchfix/flotilla-os).

    At Stitch Fix, algorithmic integrations are pervasive across the business. We have dozens of data products actively integrated systems. That requires serving layer that is robust, agile, flexible, and allows for self-service. Models produced on Flotilla are packaged for deployment in production using Khan, another framework we've developed internally. Khan provides our data scientists the ability to quickly productionize those models they've developed with open source frameworks in Python 3 (e.g. PyTorch, sklearn), by automatically packaging them as Docker containers and deploying to Amazon ECS. This provides our data scientist a one-click method of getting from their algorithms to production. We then integrate those deployments into a service mesh, which allows us to A/B test various implementations in our product.

    For more info:

    #DataScience #DataStack #Data

    See more
    Conor Myhrvold
    Conor Myhrvold
    Tech Brand Mgr, Office of CTO at Uber · | 5 upvotes · 158.4K views
    atUber TechnologiesUber Technologies
    Kafka
    Kafka
    Kafka Manager
    Kafka Manager
    Hadoop
    Hadoop
    Apache Spark
    Apache Spark
    GitHub
    GitHub

    Why we built Marmaray, an open source generic data ingestion and dispersal framework and library for Apache Hadoop :

    Built and designed by our Hadoop Platform team, Marmaray is a plug-in-based framework built on top of the Hadoop ecosystem. Users can add support to ingest data from any source and disperse to any sink leveraging the use of Apache Spark . The name, Marmaray, comes from a tunnel in Turkey connecting Europe and Asia. Similarly, we envisioned Marmaray within Uber as a pipeline connecting data from any source to any sink depending on customer preference:

    https://eng.uber.com/marmaray-hadoop-ingestion-open-source/

    (Direct GitHub repo: https://github.com/uber/marmaray Kafka Kafka Manager )

    See more
    Akka logo

    Akka

    472
    380
    54
    472
    380
    + 1
    54
    Build powerful concurrent & distributed applications more easily
    Akka logo
    Akka
    VS
    Kafka logo
    Kafka

    related Akka posts

    StackShare Editors
    StackShare Editors
    Kafka
    Kafka
    Scala
    Scala
    Cassandra
    Cassandra
    Akka
    Akka

    To solve the problem of scheduling and executing arbitrary tasks in its distributed infrastructure, PagerDuty created an open-source tool called Scheduler. Scheduler is written in Scala and uses Cassandra for task persistence. It also adds Apache Kafka to handle task queuing and partitioning, with Akka to structure the library’s concurrency.

    The service’s logic schedules a task by passing it to the Scheduler’s Scala API, which serializes the task metadata and enqueues it into Kafka. Scheduler then consumes the tasks, and posts them to Cassandra to prevent data loss.

    See more
    Akka
    Akka
    Kafka
    Kafka

    I decided to use Akka instead of Kafka streams because I have personal relationships at @Lightbend.

    See more
    Apache Storm logo

    Apache Storm

    130
    116
    18
    130
    116
    + 1
    18
    Distributed and fault-tolerant realtime computation
    Apache Storm logo
    Apache Storm
    VS
    Kafka logo
    Kafka

    related Apache Storm posts

    Marc Bollinger
    Marc Bollinger
    Infra & Data Eng Manager at Lumosity · | 4 upvotes · 77.8K views
    atLumosityLumosity
    Node.js
    Node.js
    Ruby
    Ruby
    Kafka
    Kafka
    Scala
    Scala
    Apache Storm
    Apache Storm
    Heron
    Heron
    Redis
    Redis
    Pulsar
    Pulsar

    Lumosity is home to the world's largest cognitive training database, a responsibility we take seriously. For most of the company's history, our analysis of user behavior and training data has been powered by an event stream--first a simple Node.js pub/sub app, then a heavyweight Ruby app with stronger durability. Both supported decent throughput and latency, but they lacked some major features supported by existing open-source alternatives: replaying existing messages (also lacking in most message queue-based solutions), scaling out many different readers for the same stream, the ability to leverage existing solutions for reading and writing, and possibly most importantly: the ability to hire someone externally who already had expertise.

    We ultimately migrated to Kafka in early- to mid-2016, citing both industry trends in companies we'd talked to with similar durability and throughput needs, the extremely strong documentation and community. We pored over Kyle Kingsbury's Jepsen post (https://aphyr.com/posts/293-jepsen-Kafka), as well as Jay Kreps' follow-up (http://blog.empathybox.com/post/62279088548/a-few-notes-on-kafka-and-jepsen), talked at length with Confluent folks and community members, and still wound up running parallel systems for quite a long time, but ultimately, we've been very, very happy. Understanding the internals and proper levers takes some commitment, but it's taken very little maintenance once configured. Since then, the Confluent Platform community has grown and grown; we've gone from doing most development using custom Scala consumers and producers to being 60/40 Kafka Streams/Connects.

    We originally looked into Storm / Heron , and we'd moved on from Redis pub/sub. Heron looks great, but we already had a programming model across services that was more akin to consuming a message consumers than required a topology of bolts, etc. Heron also had just come out while we were starting to migrate things, and the community momentum and direction of Kafka felt more substantial than the older Storm. If we were to start the process over again today, we might check out Pulsar , although the ecosystem is much younger.

    To find out more, read our 2017 engineering blog post about the migration!

    See more
    Apache Flink logo

    Apache Flink

    144
    155
    11
    144
    155
    + 1
    11
    Fast and reliable large-scale data processing engine
    Apache Flink logo
    Apache Flink
    VS
    Kafka logo
    Kafka

    related Apache Flink posts

    Surabhi Bhawsar
    Surabhi Bhawsar
    Technical Architect at Pepcus · | 6 upvotes · 43.3K views
    Kafka
    Kafka
    Apache Flink
    Apache Flink

    I need to build the Alert & Notification framework with the use of a scheduled program. We will analyze the events from the database table and filter events that are falling under a day timespan and send these event messages over email. Currently, we are using Kafka Pub/Sub for messaging. The customer wants us to move on Apache Flink, I am trying to understand how Apache Flink could be fit better for us.

    See more
    Redis logo

    Redis

    15K
    10K
    3.8K
    15K
    10K
    + 1
    3.8K
    An in-memory database that persists on disk
    Redis logo
    Redis
    VS
    Kafka logo
    Kafka

    related Redis posts

    Robert Zuber
    Robert Zuber
    CTO at CircleCI · | 22 upvotes · 223.5K views
    atCircleCICircleCI
    MongoDB
    MongoDB
    PostgreSQL
    PostgreSQL
    Redis
    Redis
    GitHub
    GitHub
    Amazon S3
    Amazon S3

    We use MongoDB as our primary #datastore. Mongo's approach to replica sets enables some fantastic patterns for operations like maintenance, backups, and #ETL.

    As we pull #microservices from our #monolith, we are taking the opportunity to build them with their own datastores using PostgreSQL. We also use Redis to cache data we’d never store permanently, and to rate-limit our requests to partners’ APIs (like GitHub).

    When we’re dealing with large blobs of immutable data (logs, artifacts, and test results), we store them in Amazon S3. We handle any side-effects of S3’s eventual consistency model within our own code. This ensures that we deal with user requests correctly while writes are in process.

    See more
    Thierry Schellenbach
    Thierry Schellenbach
    CEO at Stream · | 17 upvotes · 86.9K views
    atStreamStream
    Redis
    Redis
    Cassandra
    Cassandra
    RocksDB
    RocksDB
    #InMemoryDatabases
    #DataStores
    #Databases

    1.0 of Stream leveraged Cassandra for storing the feed. Cassandra is a common choice for building feeds. Instagram, for instance started, out with Redis but eventually switched to Cassandra to handle their rapid usage growth. Cassandra can handle write heavy workloads very efficiently.

    Cassandra is a great tool that allows you to scale write capacity simply by adding more nodes, though it is also very complex. This complexity made it hard to diagnose performance fluctuations. Even though we had years of experience with running Cassandra, it still felt like a bit of a black box. When building Stream 2.0 we decided to go for a different approach and build Keevo. Keevo is our in-house key-value store built upon RocksDB, gRPC and Raft.

    RocksDB is a highly performant embeddable database library developed and maintained by Facebook’s data engineering team. RocksDB started as a fork of Google’s LevelDB that introduced several performance improvements for SSD. Nowadays RocksDB is a project on its own and is under active development. It is written in C++ and it’s fast. Have a look at how this benchmark handles 7 million QPS. In terms of technology it’s much more simple than Cassandra.

    This translates into reduced maintenance overhead, improved performance and, most importantly, more consistent performance. It’s interesting to note that LinkedIn also uses RocksDB for their feed.

    #InMemoryDatabases #DataStores #Databases

    See more
    ZeroMQ logo

    ZeroMQ

    144
    131
    52
    144
    131
    + 1
    52
    Fast, lightweight messaging library that allows you to design complex communication system without much effort
    ZeroMQ logo
    ZeroMQ
    VS
    Kafka logo
    Kafka
    Apache NiFi logo

    Apache NiFi

    32
    19
    4
    32
    19
    + 1
    4
    A reliable system to process and distribute data
    Apache NiFi logo
    Apache NiFi
    VS
    Kafka logo
    Kafka

    related Amazon SQS posts

    Tim Specht
    Tim Specht
    ‎Co-Founder and CTO at Dubsmash · | 14 upvotes · 89.6K views
    atDubsmashDubsmash
    Google Analytics
    Google Analytics
    Amazon Kinesis
    Amazon Kinesis
    AWS Lambda
    AWS Lambda
    Amazon SQS
    Amazon SQS
    Google BigQuery
    Google BigQuery
    #ServerlessTaskProcessing
    #GeneralAnalytics
    #RealTimeDataProcessing
    #BigDataAsAService

    In order to accurately measure & track user behaviour on our platform we moved over quickly from the initial solution using Google Analytics to a custom-built one due to resource & pricing concerns we had.

    While this does sound complicated, it’s as easy as clients sending JSON blobs of events to Amazon Kinesis from where we use AWS Lambda & Amazon SQS to batch and process incoming events and then ingest them into Google BigQuery. Once events are stored in BigQuery (which usually only takes a second from the time the client sends the data until it’s available), we can use almost-standard-SQL to simply query for data while Google makes sure that, even with terabytes of data being scanned, query times stay in the range of seconds rather than hours. Before ingesting their data into the pipeline, our mobile clients are aggregating events internally and, once a certain threshold is reached or the app is going to the background, sending the events as a JSON blob into the stream.

    In the past we had workers running that continuously read from the stream and would validate and post-process the data and then enqueue them for other workers to write them to BigQuery. We went ahead and implemented the Lambda-based approach in such a way that Lambda functions would automatically be triggered for incoming records, pre-aggregate events, and write them back to SQS, from which we then read them, and persist the events to BigQuery. While this approach had a couple of bumps on the road, like re-triggering functions asynchronously to keep up with the stream and proper batch sizes, we finally managed to get it running in a reliable way and are very happy with this solution today.

    #ServerlessTaskProcessing #GeneralAnalytics #RealTimeDataProcessing #BigDataAsAService

    See more
    Praveen Mooli
    Praveen Mooli
    Technical Leader at Taylor and Francis · | 11 upvotes · 228.3K views
    MongoDB Atlas
    MongoDB Atlas
    Java
    Java
    Spring Boot
    Spring Boot
    Node.js
    Node.js
    ExpressJS
    ExpressJS
    Python
    Python
    Flask
    Flask
    Amazon Kinesis
    Amazon Kinesis
    Amazon Kinesis Firehose
    Amazon Kinesis Firehose
    Amazon SNS
    Amazon SNS
    Amazon SQS
    Amazon SQS