Alternatives to Kafka Streams logo

Alternatives to Kafka Streams

Kafka, Apache Spark, Apache Flink, Apache Beam, and Apache Storm are the most popular alternatives and competitors to Kafka Streams.
397
476
+ 1
0

What is Kafka Streams and what are its top alternatives?

Kafka Streams is a client library for building applications and microservices that process and analyze data stored in Apache Kafka. It allows developers to easily transform and manipulate data streams in real-time, while providing fault-tolerance and scalability. However, one of the limitations of Kafka Streams is the learning curve for beginners due to its complexity.

  1. Apache Flink: Apache Flink is a powerful framework for stream processing and batch processing. It provides stateful processing, event-time processing, and exactly-once processing guarantees, making it a strong alternative to Kafka Streams. Pros: Powerful streaming capabilities, versatile processing options. Cons: Considerably more complex than Kafka Streams.
  2. Apache Beam: Apache Beam is a unified programming model for both batch and stream processing. It supports multiple execution engines and has built-in connectors to various data sources. Pros: Support for multiple execution engines, portability across different platforms. Cons: Steeper learning curve for some users.
  3. Spark Streaming: Spark Streaming is part of the Apache Spark project and provides real-time processing capabilities. It offers fault-tolerance, exactly-once semantics, and integration with the Spark ecosystem. Pros: seamless integration with Spark ecosystem, fault-tolerance. Cons: Batch processing and streaming are somewhat decoupled in Spark.
  4. Databricks Delta: Databricks Delta is a unified data management system that combines the reliability of data lakes and the performance of data warehouses. It provides ACID transactions, time travel, and optimized performance for large-scale data processing. Pros: ACID transactions, optimized large-scale data processing. Cons: Tightly coupled with Databricks ecosystem.
  5. Amazon Kinesis: Amazon Kinesis is a managed service for real-time data streaming and processing on AWS. It offers scalable processing, durability, and integration with other AWS services. Pros: Managed service, seamless integration with AWS ecosystem. Cons: Limited to AWS environment.
  6. Google Cloud Dataflow: Google Cloud Dataflow is a fully managed service for stream and batch processing on Google Cloud Platform. It offers auto-scaling, serverless architecture, and simplified pipeline development. Pros: Fully managed service, auto-scaling. Cons: Limited to Google Cloud Platform.
  7. Confluent ksqlDB: ksqlDB is a streaming SQL engine for Apache Kafka designed to build real-time stream processing applications. It provides a familiar SQL interface for querying and processing data streams. Pros: Streaming SQL interface, integration with Kafka ecosystem. Cons: Limited to Kafka ecosystem.
  8. Rockset: Rockset is a real-time indexing database for serving real-time analytics. It provides SQL support, real-time indexing, and scalable data ingestion for building real-time applications. Pros: Real-time indexing, SQL support. Cons: Limited to real-time analytics use cases.
  9. StreamSets: StreamSets is a data integration platform for building data pipelines for batch and stream processing. It offers a visual interface for designing pipelines, monitoring data flow, and handling data drift. Pros: Visual interface, data drift handling. Cons: More focused on data integration rather than stream processing.
  10. Hazelcast Jet: Hazelcast Jet is a distributed stream processing engine for building low-latency, high-throughput applications. It provides fault-tolerance, high availability, and integration with Hazelcast IMDG. Pros: Low-latency processing, high availability. Cons: More suited for complex event processing use cases.

Top Alternatives to Kafka Streams

  • Kafka
    Kafka

    Kafka is a distributed, partitioned, replicated commit log service. It provides the functionality of a messaging system, but with a unique design. ...

  • Apache Spark
    Apache Spark

    Spark is a fast and general processing engine compatible with Hadoop data. It can run in Hadoop clusters through YARN or Spark's standalone mode, and it can process data in HDFS, HBase, Cassandra, Hive, and any Hadoop InputFormat. It is designed to perform both batch processing (similar to MapReduce) and new workloads like streaming, interactive queries, and machine learning. ...

  • Apache Flink
    Apache Flink

    Apache Flink is an open source system for fast and versatile data analytics in clusters. Flink supports batch and streaming analytics, in one system. Analytical programs can be written in concise and elegant APIs in Java and Scala. ...

  • Apache Beam
    Apache Beam

    It implements batch and streaming data processing jobs that run on any execution engine. It executes pipelines on multiple execution environments. ...

  • Apache Storm
    Apache Storm

    Apache Storm is a free and open source distributed realtime computation system. Storm makes it easy to reliably process unbounded streams of data, doing for realtime processing what Hadoop did for batch processing. Storm has many use cases: realtime analytics, online machine learning, continuous computation, distributed RPC, ETL, and more. Storm is fast: a benchmark clocked it at over a million tuples processed per second per node. It is scalable, fault-tolerant, guarantees your data will be processed, and is easy to set up and operate. ...

  • KSQL
    KSQL

    KSQL is an open source streaming SQL engine for Apache Kafka. It provides a simple and completely interactive SQL interface for stream processing on Kafka; no need to write code in a programming language such as Java or Python. KSQL is open-source (Apache 2.0 licensed), distributed, scalable, reliable, and real-time. ...

  • Samza
    Samza

    It allows you to build stateful applications that process data in real-time from multiple sources including Apache Kafka. ...

  • MySQL
    MySQL

    The MySQL software delivers a very fast, multi-threaded, multi-user, and robust SQL (Structured Query Language) database server. MySQL Server is intended for mission-critical, heavy-load production systems as well as for embedding into mass-deployed software. ...

Kafka Streams alternatives & related posts

Kafka logo

Kafka

23.7K
607
Distributed, fault tolerant, high throughput pub-sub messaging system
23.7K
607
PROS OF KAFKA
  • 126
    High-throughput
  • 119
    Distributed
  • 92
    Scalable
  • 86
    High-Performance
  • 66
    Durable
  • 38
    Publish-Subscribe
  • 19
    Simple-to-use
  • 18
    Open source
  • 12
    Written in Scala and java. Runs on JVM
  • 9
    Message broker + Streaming system
  • 4
    KSQL
  • 4
    Avro schema integration
  • 4
    Robust
  • 3
    Suport Multiple clients
  • 2
    Extremely good parallelism constructs
  • 2
    Partioned, replayable log
  • 1
    Simple publisher / multi-subscriber model
  • 1
    Fun
  • 1
    Flexible
CONS OF KAFKA
  • 32
    Non-Java clients are second-class citizens
  • 29
    Needs Zookeeper
  • 9
    Operational difficulties
  • 5
    Terrible Packaging

related Kafka posts

Nick Rockwell
SVP, Engineering at Fastly · | 46 upvotes · 4.3M views

When I joined NYT there was already broad dissatisfaction with the LAMP (Linux Apache HTTP Server MySQL PHP) Stack and the front end framework, in particular. So, I wasn't passing judgment on it. I mean, LAMP's fine, you can do good work in LAMP. It's a little dated at this point, but it's not ... I didn't want to rip it out for its own sake, but everyone else was like, "We don't like this, it's really inflexible." And I remember from being outside the company when that was called MIT FIVE when it had launched. And been observing it from the outside, and I was like, you guys took so long to do that and you did it so carefully, and yet you're not happy with your decisions. Why is that? That was more the impetus. If we're going to do this again, how are we going to do it in a way that we're gonna get a better result?

So we're moving quickly away from LAMP, I would say. So, right now, the new front end is React based and using Apollo. And we've been in a long, protracted, gradual rollout of the core experiences.

React is now talking to GraphQL as a primary API. There's a Node.js back end, to the front end, which is mainly for server-side rendering, as well.

Behind there, the main repository for the GraphQL server is a big table repository, that we call Bodega because it's a convenience store. And that reads off of a Kafka pipeline.

See more
Ashish Singh
Tech Lead, Big Data Platform at Pinterest · | 38 upvotes · 3.4M views

To provide employees with the critical need of interactive querying, we’ve worked with Presto, an open-source distributed SQL query engine, over the years. Operating Presto at Pinterest’s scale has involved resolving quite a few challenges like, supporting deeply nested and huge thrift schemas, slow/ bad worker detection and remediation, auto-scaling cluster, graceful cluster shutdown and impersonation support for ldap authenticator.

Our infrastructure is built on top of Amazon EC2 and we leverage Amazon S3 for storing our data. This separates compute and storage layers, and allows multiple compute clusters to share the S3 data.

We have hundreds of petabytes of data and tens of thousands of Apache Hive tables. Our Presto clusters are comprised of a fleet of 450 r4.8xl EC2 instances. Presto clusters together have over 100 TBs of memory and 14K vcpu cores. Within Pinterest, we have close to more than 1,000 monthly active users (out of total 1,600+ Pinterest employees) using Presto, who run about 400K queries on these clusters per month.

Each query submitted to Presto cluster is logged to a Kafka topic via Singer. Singer is a logging agent built at Pinterest and we talked about it in a previous post. Each query is logged when it is submitted and when it finishes. When a Presto cluster crashes, we will have query submitted events without corresponding query finished events. These events enable us to capture the effect of cluster crashes over time.

Each Presto cluster at Pinterest has workers on a mix of dedicated AWS EC2 instances and Kubernetes pods. Kubernetes platform provides us with the capability to add and remove workers from a Presto cluster very quickly. The best-case latency on bringing up a new worker on Kubernetes is less than a minute. However, when the Kubernetes cluster itself is out of resources and needs to scale up, it can take up to ten minutes. Some other advantages of deploying on Kubernetes platform is that our Presto deployment becomes agnostic of cloud vendor, instance types, OS, etc.

#BigData #AWS #DataScience #DataEngineering

See more
Apache Spark logo

Apache Spark

3K
140
Fast and general engine for large-scale data processing
3K
140
PROS OF APACHE SPARK
  • 61
    Open-source
  • 48
    Fast and Flexible
  • 8
    One platform for every big data problem
  • 8
    Great for distributed SQL like applications
  • 6
    Easy to install and to use
  • 3
    Works well for most Datascience usecases
  • 2
    Interactive Query
  • 2
    Machine learning libratimery, Streaming in real
  • 2
    In memory Computation
CONS OF APACHE SPARK
  • 4
    Speed

related Apache Spark posts

Eric Colson
Chief Algorithms Officer at Stitch Fix · | 21 upvotes · 6.2M views

The algorithms and data infrastructure at Stitch Fix is housed in #AWS. Data acquisition is split between events flowing through Kafka, and periodic snapshots of PostgreSQL DBs. We store data in an Amazon S3 based data warehouse. Apache Spark on Yarn is our tool of choice for data movement and #ETL. Because our storage layer (s3) is decoupled from our processing layer, we are able to scale our compute environment very elastically. We have several semi-permanent, autoscaling Yarn clusters running to serve our data processing needs. While the bulk of our compute infrastructure is dedicated to algorithmic processing, we also implemented Presto for adhoc queries and dashboards.

Beyond data movement and ETL, most #ML centric jobs (e.g. model training and execution) run in a similarly elastic environment as containers running Python and R code on Amazon EC2 Container Service clusters. The execution of batch jobs on top of ECS is managed by Flotilla, a service we built in house and open sourced (see https://github.com/stitchfix/flotilla-os).

At Stitch Fix, algorithmic integrations are pervasive across the business. We have dozens of data products actively integrated systems. That requires serving layer that is robust, agile, flexible, and allows for self-service. Models produced on Flotilla are packaged for deployment in production using Khan, another framework we've developed internally. Khan provides our data scientists the ability to quickly productionize those models they've developed with open source frameworks in Python 3 (e.g. PyTorch, sklearn), by automatically packaging them as Docker containers and deploying to Amazon ECS. This provides our data scientist a one-click method of getting from their algorithms to production. We then integrate those deployments into a service mesh, which allows us to A/B test various implementations in our product.

For more info:

#DataScience #DataStack #Data

See more
Patrick Sun
Software Engineer at Stitch Fix · | 10 upvotes · 70.2K views

As a frontend engineer on the Algorithms & Analytics team at Stitch Fix, I work with data scientists to develop applications and visualizations to help our internal business partners make data-driven decisions. I envisioned a platform that would assist data scientists in the data exploration process, allowing them to visually explore and rapidly iterate through their assumptions, then share their insights with others. This would align with our team's philosophy of having engineers "deploy platforms, services, abstractions, and frameworks that allow the data scientists to conceive of, develop, and deploy their ideas with autonomy", and solve the pain of data exploration.

The final product, code-named Dora, is built with React, Redux.js and Victory, backed by Elasticsearch to enable fast and iterative data exploration, and uses Apache Spark to move data from our Amazon S3 data warehouse into the Elasticsearch cluster.

See more
Apache Flink logo

Apache Flink

531
38
Fast and reliable large-scale data processing engine
531
38
PROS OF APACHE FLINK
  • 16
    Unified batch and stream processing
  • 8
    Easy to use streaming apis
  • 8
    Out-of-the box connector to kinesis,s3,hdfs
  • 4
    Open Source
  • 2
    Low latency
CONS OF APACHE FLINK
    Be the first to leave a con

    related Apache Flink posts

    Surabhi Bhawsar
    Technical Architect at Pepcus · | 7 upvotes · 733K views
    Shared insights
    on
    KafkaKafkaApache FlinkApache Flink

    I need to build the Alert & Notification framework with the use of a scheduled program. We will analyze the events from the database table and filter events that are falling under a day timespan and send these event messages over email. Currently, we are using Kafka Pub/Sub for messaging. The customer wants us to move on Apache Flink, I am trying to understand how Apache Flink could be fit better for us.

    See more

    I have to build a data processing application with an Apache Beam stack and Apache Flink runner on an Amazon EMR cluster. I saw some instability with the process and EMR clusters that keep going down. Here, the Apache Beam application gets inputs from Kafka and sends the accumulative data streams to another Kafka topic. Any advice on how to make the process more stable?

    See more
    Apache Beam logo

    Apache Beam

    180
    14
    A unified programming model
    180
    14
    PROS OF APACHE BEAM
    • 5
      Open-source
    • 5
      Cross-platform
    • 2
      Portable
    • 2
      Unified batch and stream processing
    CONS OF APACHE BEAM
      Be the first to leave a con

      related Apache Beam posts

      I have to build a data processing application with an Apache Beam stack and Apache Flink runner on an Amazon EMR cluster. I saw some instability with the process and EMR clusters that keep going down. Here, the Apache Beam application gets inputs from Kafka and sends the accumulative data streams to another Kafka topic. Any advice on how to make the process more stable?

      See more
      Apache Storm logo

      Apache Storm

      204
      25
      Distributed and fault-tolerant realtime computation
      204
      25
      PROS OF APACHE STORM
      • 10
        Flexible
      • 6
        Easy setup
      • 4
        Event Processing
      • 3
        Clojure
      • 2
        Real Time
      CONS OF APACHE STORM
        Be the first to leave a con

        related Apache Storm posts

        Marc Bollinger
        Infra & Data Eng Manager at Thumbtack · | 5 upvotes · 1.9M views

        Lumosity is home to the world's largest cognitive training database, a responsibility we take seriously. For most of the company's history, our analysis of user behavior and training data has been powered by an event stream--first a simple Node.js pub/sub app, then a heavyweight Ruby app with stronger durability. Both supported decent throughput and latency, but they lacked some major features supported by existing open-source alternatives: replaying existing messages (also lacking in most message queue-based solutions), scaling out many different readers for the same stream, the ability to leverage existing solutions for reading and writing, and possibly most importantly: the ability to hire someone externally who already had expertise.

        We ultimately migrated to Kafka in early- to mid-2016, citing both industry trends in companies we'd talked to with similar durability and throughput needs, the extremely strong documentation and community. We pored over Kyle Kingsbury's Jepsen post (https://aphyr.com/posts/293-jepsen-Kafka), as well as Jay Kreps' follow-up (http://blog.empathybox.com/post/62279088548/a-few-notes-on-kafka-and-jepsen), talked at length with Confluent folks and community members, and still wound up running parallel systems for quite a long time, but ultimately, we've been very, very happy. Understanding the internals and proper levers takes some commitment, but it's taken very little maintenance once configured. Since then, the Confluent Platform community has grown and grown; we've gone from doing most development using custom Scala consumers and producers to being 60/40 Kafka Streams/Connects.

        We originally looked into Storm / Heron , and we'd moved on from Redis pub/sub. Heron looks great, but we already had a programming model across services that was more akin to consuming a message consumers than required a topology of bolts, etc. Heron also had just come out while we were starting to migrate things, and the community momentum and direction of Kafka felt more substantial than the older Storm. If we were to start the process over again today, we might check out Pulsar , although the ecosystem is much younger.

        To find out more, read our 2017 engineering blog post about the migration!

        See more
        KSQL logo

        KSQL

        56
        5
        Open source streaming SQL for Apache Kafka
        56
        5
        PROS OF KSQL
        • 3
          Streamprocessing on Kafka
        • 2
          SQL syntax with windowing functions over streams
        • 0
          Easy transistion for SQL Devs
        CONS OF KSQL
          Be the first to leave a con

          related KSQL posts

          I have recently started using Confluent/Kafka cloud. We want to do some stream processing. As I was going through Kafka I came across Kafka Streams and KSQL. Both seem to be A good fit for stream processing. But I could not understand which one should be used and one has any advantage over another. We will be using Confluent/Kafka Managed Cloud Instance. In near future, our Producers and Consumers are running on premise and we will be interacting with Confluent Cloud.

          Also, Confluent Cloud Kafka has a primitive interface; is there any better UI interface to manage Kafka Cloud Cluster?

          See more
          Samza logo

          Samza

          24
          0
          A distributed stream processing framework
          24
          0
          PROS OF SAMZA
            Be the first to leave a pro
            CONS OF SAMZA
              Be the first to leave a con

              related Samza posts

              MySQL logo

              MySQL

              126.2K
              3.8K
              The world's most popular open source database
              126.2K
              3.8K
              PROS OF MYSQL
              • 800
                Sql
              • 679
                Free
              • 562
                Easy
              • 528
                Widely used
              • 490
                Open source
              • 180
                High availability
              • 160
                Cross-platform support
              • 104
                Great community
              • 79
                Secure
              • 75
                Full-text indexing and searching
              • 26
                Fast, open, available
              • 16
                Reliable
              • 16
                SSL support
              • 15
                Robust
              • 9
                Enterprise Version
              • 7
                Easy to set up on all platforms
              • 3
                NoSQL access to JSON data type
              • 1
                Relational database
              • 1
                Easy, light, scalable
              • 1
                Sequel Pro (best SQL GUI)
              • 1
                Replica Support
              CONS OF MYSQL
              • 16
                Owned by a company with their own agenda
              • 3
                Can't roll back schema changes

              related MySQL posts

              Nick Rockwell
              SVP, Engineering at Fastly · | 46 upvotes · 4.3M views

              When I joined NYT there was already broad dissatisfaction with the LAMP (Linux Apache HTTP Server MySQL PHP) Stack and the front end framework, in particular. So, I wasn't passing judgment on it. I mean, LAMP's fine, you can do good work in LAMP. It's a little dated at this point, but it's not ... I didn't want to rip it out for its own sake, but everyone else was like, "We don't like this, it's really inflexible." And I remember from being outside the company when that was called MIT FIVE when it had launched. And been observing it from the outside, and I was like, you guys took so long to do that and you did it so carefully, and yet you're not happy with your decisions. Why is that? That was more the impetus. If we're going to do this again, how are we going to do it in a way that we're gonna get a better result?

              So we're moving quickly away from LAMP, I would say. So, right now, the new front end is React based and using Apollo. And we've been in a long, protracted, gradual rollout of the core experiences.

              React is now talking to GraphQL as a primary API. There's a Node.js back end, to the front end, which is mainly for server-side rendering, as well.

              Behind there, the main repository for the GraphQL server is a big table repository, that we call Bodega because it's a convenience store. And that reads off of a Kafka pipeline.

              See more
              Tim Abbott

              We've been using PostgreSQL since the very early days of Zulip, but we actually didn't use it from the beginning. Zulip started out as a MySQL project back in 2012, because we'd heard it was a good choice for a startup with a wide community. However, we found that even though we were using the Django ORM for most of our database access, we spent a lot of time fighting with MySQL. Issues ranged from bad collation defaults, to bad query plans which required a lot of manual query tweaks.

              We ended up getting so frustrated that we tried out PostgresQL, and the results were fantastic. We didn't have to do any real customization (just some tuning settings for how big a server we had), and all of our most important queries were faster out of the box. As a result, we were able to delete a bunch of custom queries escaping the ORM that we'd written to make the MySQL query planner happy (because postgres just did the right thing automatically).

              And then after that, we've just gotten a ton of value out of postgres. We use its excellent built-in full-text search, which has helped us avoid needing to bring in a tool like Elasticsearch, and we've really enjoyed features like its partial indexes, which saved us a lot of work adding unnecessary extra tables to get good performance for things like our "unread messages" and "starred messages" indexes.

              I can't recommend it highly enough.

              See more