Apache Drill vs Kafka

Overview

Kafka

Stacks24.2K

Followers22.3K

Votes607

GitHub Stars31.2K

Forks14.8K

Apache Drill

Stacks74

Followers171

Votes16

Apache Drill vs Kafka: What are the differences?

What is Apache Drill? Schema-Free SQL Query Engine for Hadoop and NoSQL. Apache Drill is a distributed MPP query layer that supports SQL and alternative query languages against NoSQL and Hadoop data storage systems. It was inspired in part by Google's Dremel.

What is Kafka? Distributed, fault tolerant, high throughput pub-sub messaging system. Kafka is a distributed, partitioned, replicated commit log service. It provides the functionality of a messaging system, but with a unique design.

Apache Drill belongs to "Database Tools" category of the tech stack, while Kafka can be primarily classified under "Message Queue".

Some of the features offered by Apache Drill are:

Low-latency SQL queries
Dynamic queries on self-describing data in files (such as JSON, Parquet, text) and MapR-DB/HBase tables, without requiring metadata definitions in the Hive metastore.
ANSI SQL

On the other hand, Kafka provides the following key features:

Written at LinkedIn in Scala
Used by LinkedIn to offload processing of all page and other views
Defaults to using persistence, uses OS disk cache for hot data (has higher throughput then any of the above having persistence enabled)

"NoSQL and Hadoop" is the primary reason why developers consider Apache Drill over the competitors, whereas "High-throughput" was stated as the key factor in picking Kafka.

Kafka is an open source tool with 12.5K GitHub stars and 6.7K GitHub forks. Here's a link to Kafka's open source repository on GitHub.

Share your Stack

Help developers discover the tools you use. Get visibility for your team's tech choices and contribute to the community's knowledge.

View Docs

CLI (Node.js)

Manual

Advice on Kafka, Apache Drill

viradiya

Apr 12, 2020

Needs adviceon

AngularJS

ASP.NET Core

MSSQL

We are going to develop a microservices-based application. It consists of AngularJS, ASP.NET Core, and MSSQL.

We have 3 types of microservices. Emailservice, Filemanagementservice, Filevalidationservice

I am a beginner in microservices. But I have read about RabbitMQ, but come to know that there are Redis and Kafka also in the market. So, I want to know which is best.

933k views933k

Comments

Ishfaq

Feb 28, 2020

Needs advice

Our backend application is sending some external messages to a third party application at the end of each backend (CRUD) API call (from UI) and these external messages take too much extra time (message building, processing, then sent to the third party and log success/failure), UI application has no concern to these extra third party messages.

So currently we are sending these third party messages by creating a new child thread at end of each REST API call so UI application doesn't wait for these extra third party API calls.

I want to integrate Apache Kafka for these extra third party API calls, so I can also retry on failover third party API calls in a queue(currently third party messages are sending from multiple threads at the same time which uses too much processing and resources) and logging, etc.

Question 1: Is this a use case of a message broker?

Question 2: If it is then Kafka vs RabitMQ which is the better?

804k views804k

Comments

Roman

Senior Back-End Developer, Software Architect

Feb 12, 2019

Reviewon

Kafka

I use Kafka because it has almost infinite scaleability in terms of processing events (could be scaled to process hundreds of thousands of events), great monitoring (all sorts of metrics are exposed via JMX).

Downsides of using Kafka are:

you have to deal with Zookeeper
you have to implement advanced routing yourself (compared to RabbitMQ it has no advanced routing)

10.9k views10.9k

Comments

Detailed Comparison

Kafka	Apache Drill
Kafka is a distributed, partitioned, replicated commit log service. It provides the functionality of a messaging system, but with a unique design.	Apache Drill is a distributed MPP query layer that supports SQL and alternative query languages against NoSQL and Hadoop data storage systems. It was inspired in part by Google's Dremel.
Written at LinkedIn in Scala;Used by LinkedIn to offload processing of all page and other views;Defaults to using persistence, uses OS disk cache for hot data (has higher throughput then any of the above having persistence enabled);Supports both on-line as off-line processing	Low-latency SQL queries;Dynamic queries on self-describing data in files (such as JSON, Parquet, text) and MapR-DB/HBase tables, without requiring metadata definitions in the Hive metastore.;ANSI SQL;Nested data support;Integration with Apache Hive (queries on Hive tables and views, support for all Hive file formats and Hive UDFs);BI/SQL tool integration using standard JDBC/ODBC drivers
Statistics
GitHub Stars 31.2K	GitHub Stars -
GitHub Forks 14.8K	GitHub Forks -
Stacks 24.2K	Stacks 74
Followers 22.3K	Followers 171
Votes 607	Votes 16
Pros & Cons
Pros 126 High-throughput 119 Distributed 92 Scalable 86 High-Performance 66 Durable Cons 32 Non-Java clients are second-class citizens 29 Needs Zookeeper 9 Operational difficulties 5 Terrible Packaging	Pros 4 NoSQL and Hadoop 3 Free 3 Lightning speed and simplicity in face of data jungle 2 Well documented for fast install 1 Read Structured and unstructured data

What are some alternatives to Kafka, Apache Drill?

dbForge Studio for MySQL

It is the universal MySQL and MariaDB client for database management, administration and development. With the help of this intelligent MySQL client the work with data and code has become easier and more convenient. This tool provides utilities to compare, synchronize, and backup MySQL databases with scheduling, and gives possibility to analyze and report MySQL tables data.

RabbitMQ

RabbitMQ gives your applications a common platform to send and receive messages, and your messages a safe place to live until received.

dbForge Studio for Oracle

It is a powerful integrated development environment (IDE) which helps Oracle SQL developers to increase PL/SQL coding speed, provides versatile data editing tools for managing in-database and external data.

dbForge Studio for PostgreSQL

It is a GUI tool for database development and management. The IDE for PostgreSQL allows users to create, develop, and execute queries, edit and adjust the code to their requirements in a convenient and user-friendly interface.

Celery

Celery is an asynchronous task queue/job queue based on distributed message passing. It is focused on real-time operation, but supports scheduling as well.

dbForge Studio for SQL Server

It is a powerful IDE for SQL Server management, administration, development, data reporting and analysis. The tool will help SQL developers to manage databases, version-control database changes in popular source control systems, speed up routine tasks, as well, as to make complex database changes.

Amazon SQS

Transmit any volume of data, at any level of throughput, without losing messages or requiring other services to be always available. With SQS, you can offload the administrative burden of operating and scaling a highly available messaging cluster, while paying a low price for only what you use.

NSQ

NSQ is a realtime distributed messaging platform designed to operate at scale, handling billions of messages per day. It promotes distributed and decentralized topologies without single points of failure, enabling fault tolerance and high availability coupled with a reliable message delivery guarantee. See features & guarantees.

ActiveMQ

Apache ActiveMQ is fast, supports many Cross Language Clients and Protocols, comes with easy to use Enterprise Integration Patterns and many advanced features while fully supporting JMS 1.1 and J2EE 1.4. Apache ActiveMQ is released under the Apache 2.0 License.

ZeroMQ

The 0MQ lightweight messaging kernel is a library which extends the standard socket interfaces with features traditionally provided by specialised messaging middleware products. 0MQ sockets provide an abstraction of asynchronous message queues, multiple messaging patterns, message filtering (subscriptions), seamless access to multiple transport protocols and more.

Related Comparisons

Bootstrap vs Materialize

Django vs Laravel vs Node.js

Bootstrap vs Foundation vs Material UI

Node.js vs Spring-Boot

Flyway vs Liquibase

Apache Drill vs Kafka: What are the differences?

Apache Drill belongs to "Database Tools" category of the tech stack, while Kafka can be primarily classified under "Message Queue".

Some of the features offered by Apache Drill are:

Low-latency SQL queries
Dynamic queries on self-describing data in files (such as JSON, Parquet, text) and MapR-DB/HBase tables, without requiring metadata definitions in the Hive metastore.
ANSI SQL

On the other hand, Kafka provides the following key features:

Written at LinkedIn in Scala
Used by LinkedIn to offload processing of all page and other views
Defaults to using persistence, uses OS disk cache for hot data (has higher throughput then any of the above having persistence enabled)

"NoSQL and Hadoop" is the primary reason why developers consider Apache Drill over the competitors, whereas "High-throughput" was stated as the key factor in picking Kafka.

Kafka is an open source tool with 12.5K GitHub stars and 6.7K GitHub forks. Here's a link to Kafka's open source repository on GitHub.