Need advice about which tool to choose?Ask the StackShare community!
Apache Flink vs Kafka: What are the differences?
Apache Flink: Fast and reliable large-scale data processing engine. Apache Flink is an open source system for fast and versatile data analytics in clusters. Flink supports batch and streaming analytics, in one system. Analytical programs can be written in concise and elegant APIs in Java and Scala; Kafka: Distributed, fault tolerant, high throughput pub-sub messaging system. Kafka is a distributed, partitioned, replicated commit log service. It provides the functionality of a messaging system, but with a unique design.
Apache Flink and Kafka are primarily classified as "Big Data" and "Message Queue" tools respectively.
Some of the features offered by Apache Flink are:
- Hybrid batch/streaming runtime that supports batch processing and data streaming programs.
- Custom memory management to guarantee efficient, adaptive, and highly robust switching between in-memory and data processing out-of-core algorithms.
- Flexible and expressive windowing semantics for data stream programs
On the other hand, Kafka provides the following key features:
- Written at LinkedIn in Scala
- Used by LinkedIn to offload processing of all page and other views
- Defaults to using persistence, uses OS disk cache for hot data (has higher throughput then any of the above having persistence enabled)
"Unified batch and stream processing" is the top reason why over 6 developers like Apache Flink, while over 95 developers mention "High-throughput" as the leading cause for choosing Kafka.
Apache Flink and Kafka are both open source tools. Kafka with 12.7K GitHub stars and 6.81K forks on GitHub appears to be more popular than Apache Flink with 9.35K GitHub stars and 5K GitHub forks.
Uber Technologies, Spotify, and Slack are some of the popular companies that use Kafka, whereas Apache Flink is used by Zalando, sovrn Holdings, and BetterCloud. Kafka has a broader approval, being mentioned in 509 company stacks & 470 developers stacks; compared to Apache Flink, which is listed in 20 company stacks and 22 developer stacks.
We have a Kafka topic having events of type A and type B. We need to perform an inner join on both type of events using some common field (primary-key). The joined events to be inserted in Elasticsearch.
In usual cases, type A and type B events (with same key) observed to be close upto 15 minutes. But in some cases they may be far from each other, lets say 6 hours. Sometimes event of either of the types never come.
In all cases, we should be able to find joined events instantly after they are joined and not-joined events within 15 minutes.
The first solution that came to me is to use upsert to update ElasticSearch:
- Use the primary-key as ES document id
- Upsert the records to ES as soon as you receive them. As you are using upsert, the 2nd record of the same primary-key will not overwrite the 1st one, but will be merged with it.
Cons: The load on ES will be higher, due to upsert.
To use Flink:
- Create a KeyedDataStream by the primary-key
- In the ProcessFunction, save the first record in a State. At the same time, create a Timer for 15 minutes in the future
- When the 2nd record comes, read the 1st record from the State, merge those two, and send out the result, and clear the State and the Timer if it has not fired
- When the Timer fires, read the 1st record from the State and send out as the output record.
- Have a 2nd Timer of 6 hours (or more) if you are not using Windowing to clean up the State
Pro: if you have already having Flink ingesting this stream. Otherwise, I would just go with the 1st solution.
Please refer "Structured Streaming" feature of Spark. Refer "Stream - Stream Join" at https://spark.apache.org/docs/latest/structured-streaming-programming-guide.html#stream-stream-joins . In short you need to specify "Define watermark delays on both inputs" and "Define a constraint on time across the two inputs"
We are going to develop a microservices-based application. It consists of AngularJS, ASP.NET Core, and MSSQL.
We have 3 types of microservices. Emailservice, Filemanagementservice, Filevalidationservice
I am a beginner in microservices. But I have read about RabbitMQ, but come to know that there are Redis and Kafka also in the market. So, I want to know which is best.
Kafka is an Enterprise Messaging Framework whereas Redis is an Enterprise Cache Broker, in-memory database and high performance database.Both are having their own advantages, but they are different in usage and implementation. Now if you are creating microservices check the user consumption volumes, its generating logs, scalability, systems to be integrated and so on. I feel for your scenario initially you can go with KAFKA bu as the throughput, consumption and other factors are scaling then gradually you can add Redis accordingly.
I first recommend that you choose Angular over AngularJS if you are starting something new. AngularJs is no longer getting enhancements, but perhaps you meant Angular. Regarding microservices, I recommend considering microservices when you have different development teams for each service that may want to use different programming languages and backend data stores. If it is all the same team, same code language, and same data store I would not use microservices. I might use a message queue, in which case RabbitMQ is a good one. But you may also be able to simply write your own in which you write a record in a table in MSSQL and one of your services reads the record from the table and processes it. The most challenging part of doing it yourself is writing a service that does a good job of reading the queue without reading the same message multiple times or missing a message; and that is where RabbitMQ can help.
I think something is missing here and you should consider answering it to yourself. You are building a couple of services. Why are you considering event-sourcing architecture using Message Brokers such as the above? Won't a simple REST service based arch suffice? Read about CQRS and the problems it entails (state vs command impedance for example). Do you need Pub/Sub or Push/Pull? Is queuing of messages enough or would you need querying or filtering of messages before consumption? Also, someone would have to manage these brokers (unless using managed, cloud provider based solution), automate their deployment, someone would need to take care of backups, clustering if needed, disaster recovery, etc. I have a good past experience in terms of manageability/devops of the above options with Kafka and Redis, not so much with RabbitMQ. Both are very performant. But also note that Redis is not a pure message broker (at time of writing) but more of a general purpose in-memory key-value store. Kafka nowadays is much more than a distributed message broker. Long story short. In my taste, you should go with a minialistic approach and try to avoid either of them if you can, especially if your architecture does not fall nicely into event sourcing. If not I'd examine Kafka. If you need more capabilities than I'd consider Redis and use it for all sorts of other things such as a cache.
We found that the CNCF landscape is a good advisor when working going into the cloud / microservices space: https://landscape.cncf.io/fullscreen=yes. When choosing a technology one important criteria to me is if it is cloud native or not. Neither Redis, RabbitMQ nor Kafka is cloud native. The try to adapt but will be replaced eventually with technologies that are cloud native.
We have gone with NATS and have never looked back. We haven't spend a single minute on server maintainance in the last year and the setup of a cluster is way too easy. With the new features NATS incorporates now (and the ones still on the roadmap) it is already and will be sooo much mure than Redis, RabbitMQ and Kafka are. It can replace service discovery, load balancing, global multiclusters and failover, etc, etc.
Your thought might be: But I don't need all of that! Well, at the same time it is much more leightweight than Redis, RabbitMQ and especially Kafka.
I am looking into IoT World Solution where we have MQTT Broker. This MQTT Broker Sits in one of the Data Center. We are doing a lot of Alert and Alarm related processing on that Data, Currently, we are looking into Solution which can do distributed persistence of log/alert primarily on remote Disk.
Our primary need is to use lightweight where operational complexity and maintenance costs can be significantly reduced. We want to do it on-premise so we are not considering cloud solutions.
We looked into the following alternatives:
Apache Kafka - Great choice but operation and maintenance wise very complex. Rabbit MQ - High availability is the issue, Apache Pulsar - Operational Complexity. NATS - Absence of persistence. Akka Streams - Big learning curve and operational streams.
So we are looking into a lightweight library that can do distributed persistence preferably with publisher and subscriber model. Preferable on JVM stack.
Kafka is best fit here. Below are the advantages with Kafka ACLs (Security), Schema (protobuf), Scale, Consumer driven and No single point of failure.
Operational complexity is manageable with open source monitoring tools.
Our backend application is sending some external messages to a third party application at the end of each backend (CRUD) API call (from UI) and these external messages take too much extra time (message building, processing, then sent to the third party and log success/failure), UI application has no concern to these extra third party messages.
So currently we are sending these third party messages by creating a new child thread at end of each REST API call so UI application doesn't wait for these extra third party API calls.
I want to integrate Apache Kafka for these extra third party API calls, so I can also retry on failover third party API calls in a queue(currently third party messages are sending from multiple threads at the same time which uses too much processing and resources) and logging, etc.
Question 1: Is this a use case of a message broker?
Question 2: If it is then Kafka vs RabitMQ which is the better?
RabbitMQ is great for queuing and retrying. You can send the requests to your backend which will further queue these requests in RabbitMQ (or Kafka, too). The consumer on the other end can take care of processing . For a detailed analysis, check this blog about choosing between Kafka and RabbitMQ.
Well, first off, it's good practice to do as little non-UI work on the foreground thread as possible, regardless of whether the requests take a long time. You don't want the UI thread blocked.
This sounds like a good use case for RabbitMQ. Primarily because you don't need each message processed by more than one consumer. If you wanted to process a single message more than once (say for different purposes), then Apache Kafka would be a much better fit as you can have multiple consumer groups consuming from the same topics independently.
Have your API publish messages containing the data necessary for the third-party request to a Rabbit queue and have consumers reading off there. If it fails, you can either retry immediately, or publish to a deadletter queue where you can reprocess them whenever you want (shovel them back into the regular queue).
As far as I understand, Kafka is a like a persisted event state manager where you can plugin various source of data and transform/query them as event via a stream API. Regarding your use case I will consider using RabbitMQ if your intent is to implement service inter-communication kind of thing. RabbitMQ is a good choice for one-one publisher/subscriber (or consumer) and I think you can also have multiple consumers by configuring a fanout exchange. RabbitMQ provide also message retries, message cancellation, durable queue, message requeue, message ACK....
In my opinion RabbitMQ fits better in your case because you don’t have order in queue. You can process your messages in any order. You don’t need to store the data what you sent. Kafka is a persistent storage like the blockchain. RabbitMQ is a message broker. Kafka is not a good solution for the system with confirmations of the messages delivery.
Hello! [Client sends live video frames -> Server computes and responds the result] Web clients send video frames from their webcam then on the back we need to run them through some algorithm and send the result back as a response. Since everything will need to work in a live mode, we want something fast and also suitable for our case (as everyone needs). Currently, we are considering RabbitMQ for the purpose, but recently I have noticed that there is Redis and Kafka too. Could you please help us choose among them or anything more suitable beyond these guys. I think something similar to our product would be people using their webcam to get Snapchat masks on their faces, and the calculated face points are responded on from the server, then the client-side draw the mask on the user's face. I hope this helps. Thank you!
For your use case, the tool that fits more is definitely Kafka. RabbitMQ was not invented to handle data streams, but messages. Plenty of them, of course, but individual messages. Redis is an in-memory database, which is what makes it so fast. Redis recently included features to handle data stream, but it cannot best Kafka on this, or at least not yet. Kafka is not also super fast, it also provides lots of features to help create software to handle those streams.
For this kind of use case I would recommend either RabbitMQ or Kafka depending on the needs for scaling, redundancy and how you want to design it.
Kafka's true value comes into play when you need to distribute the streaming load over lot's of resources. If you were passing the video frames directly into the queue then you'd probably want to go with Kafka however if you can just pass a pointer to the frames then RabbitMQ should be fine and will be much simpler to run.
Bear in mind too that Kafka is a persistent log, not just a message bus so any data you feed into it is kept available until it expires (which is configurable). This can be useful if you have multiple clients reading from the queue with their own lifecycle but in your case it doesn't sound like that would be necessary. You could also use a RabbitMQ fanout exchange if you need that in the future.
I've used all of them and Kafka is hard to set up and maintain. Mostly is a Java dinosaur that you can set up and. I've used it with Storm but that is another big dinosaur. Redis is mostly for caching. The queue mechanism is not very scalable for multiple processors. Depending on the speed you need to implement on the reliability I would use RabbitMQ. You can store the frames(if they are too big) somewhere else and just have a link to them. Moving data through any of these will increase cost of transportation. With Rabbit, you can always have multiple consumers and check for redundancy. Hope it clears out your thoughts!
I need to build the Alert & Notification framework with the use of a scheduled program. We will analyze the events from the database table and filter events that are falling under a day timespan and send these event messages over email. Currently, we are using Kafka Pub/Sub for messaging. The customer wants us to move on Apache Flink, I am trying to understand how Apache Flink could be fit better for us.
I recommend Apache Flink because it is the pro tool for everybody who has a serious stream processing use case. Flink is used by huge companies, such as Uber, Alibaba or Netflix. AWS is offering Flink as a hosted service. The reason for these companies to decide for Flink are manyfold: Flink offers great performance, support for very large state, exactly-once processing semantics, different APIs (with SQL growing a lot lately), ... Flink supports a many different deployment models, including Kubernetes, Hadoop YARN or custom deployments.
The drawbacks of Apache Flink are medium steep learning curve, and plenty of options (APIs, deployment models, state backends, ...)
These are my personal views, and I have a bias towards Flink, because I've worked a lot on it:
Flink and Kafka (the message bus) work together very well, and that's also the most popular combination (I'm guessing). There's also Kafka Streams, a stream processing library using Kafka (the message bus) as a data transport layer. Some considerations of Kafka Streams vs Flink:
- KStreams has a hard dependency on Kafka, Flink is independent of the message bus, and can easily read and write to many systems (KStreams requires Kafka connect for that)
- Since KStreams is doing data exchange via kafka topics, there's a lot of load on the Kafka cluster (size it appropriately). Monitoring becomes difficult as processing and data storage are in the same cluster. Do you really want your production data being discarded because your processing is eating up all your IO?
- Flink is the older project, it has been battle tested for many years across a lot of different scenarios. There's more libraries, such as a CEP (Complex Event Processing) library and more and more machine learning integrations.
Pros of Apache Flink
- Unified batch and stream processing16
- Easy to use streaming apis8
- Out-of-the box connector to kinesis,s3,hdfs8
- Open Source4
- Low latency2
Pros of Kafka
- High-throughput126
- Distributed119
- Scalable92
- High-Performance86
- Durable66
- Publish-Subscribe38
- Simple-to-use19
- Open source18
- Written in Scala and java. Runs on JVM11
- Message broker + Streaming system8
- Robust4
- Avro schema integration4
- KSQL4
- Suport Multiple clients3
- Partioned, replayable log2
- Simple publisher / multi-subscriber model1
- Flexible1
- Extremely good parallelism constructs1
- Fun1
Sign up to add or upvote prosMake informed product decisions
Cons of Apache Flink
Cons of Kafka
- Non-Java clients are second-class citizens32
- Needs Zookeeper29
- Operational difficulties9
- Terrible Packaging4