Druid vs Hadoop

Overview

Hadoop

Stacks2.7K

Followers2.3K

Votes56

GitHub Stars15.3K

Forks9.1K

Druid

Stacks376

Followers867

Votes32

Druid vs Hadoop: What are the differences?

Performance and Scalability: Druid is designed to provide real-time query capabilities on large datasets, while Hadoop focuses on distributed storage and batch processing. Druid's architecture allows for efficient ingestion of data and provides fast query responses, making it suitable for interactive analytics. On the other hand, Hadoop's MapReduce framework enables parallel processing of data but may have slower query response times due to its batch processing nature.
Data Model: Druid stores data in a columnar format optimized for low-latency queries, where each column is stored separately and can be compressed individually. Hadoop, on the other hand, stores data in a distributed file system using the Hadoop Distributed File System (HDFS), where files are stored in a block format. This difference in data storage and compression techniques affects the speed and efficiency of data querying.
Data Ingestion: Druid is designed to support real-time data ingestion by providing mechanisms to ingest data streams and continuously update its indexes. In contrast, Hadoop is primarily used for batch processing and requires manual loading of data into the HDFS before it can be processed.
Querying Capabilities: Druid provides sub-second query response times even on large datasets due to its column-based storage and indexing structures. It supports complex aggregations, filtering, and grouping operations, making it suitable for interactive, ad-hoc queries. Hadoop, on the other hand, may have longer query response times as it processes data in batches and requires the execution of MapReduce jobs for querying.
Ecosystem and Tooling: Hadoop has a rich ecosystem of tools and frameworks such as Apache Spark, Apache Hive, and Apache Pig that can be used for various data processing tasks. It also has extensive support for data integration and processing. Druid, while it has a smaller ecosystem compared to Hadoop, provides built-in support for real-time dashboards and OLAP-style analytics.
Data Updates: Druid allows for real-time updates and incremental ingestion of data streams while maintaining query performance. Hadoop typically requires complete reprocessing of data for updates, making it more suitable for batch data processing rather than real-time data analysis.

In Summary, Druid and Hadoop differ in their focus on real-time query capabilities, data storage and compression techniques, data ingestion mechanisms, querying capabilities, ecosystem and tooling support, and handling of data updates.

Share your Stack

Help developers discover the tools you use. Get visibility for your team's tech choices and contribute to the community's knowledge.

View Docs

CLI (Node.js)

Manual

Advice on Hadoop, Druid

pionell

Sep 16, 2020

Needs adviceon

MariaDB

I have a lot of data that's currently sitting in a MariaDB database, a lot of tables that weigh 200gb with indexes. Most of the large tables have a date column which is always filtered, but there are usually 4-6 additional columns that are filtered and used for statistics. I'm trying to figure out the best tool for storing and analyzing large amounts of data. Preferably self-hosted or a cheap solution. The current problem I'm running into is speed. Even with pretty good indexes, if I'm trying to load a large dataset, it's pretty slow.

159k views159k

Comments

Detailed Comparison

Hadoop	Druid
The Apache Hadoop software library is a framework that allows for the distributed processing of large data sets across clusters of computers using simple programming models. It is designed to scale up from single servers to thousands of machines, each offering local computation and storage.	Druid is a distributed, column-oriented, real-time analytics data store that is commonly used to power exploratory dashboards in multi-tenant environments. Druid excels as a data warehousing solution for fast aggregate queries on petabyte sized data sets. Druid supports a variety of flexible filters, exact calculations, approximate algorithms, and other useful calculations.
Statistics
GitHub Stars 15.3K	GitHub Stars -
GitHub Forks 9.1K	GitHub Forks -
Stacks 2.7K	Stacks 376
Followers 2.3K	Followers 867
Votes 56	Votes 32
Pros & Cons
Pros 39 Great ecosystem 11 One stack to rule them all 4 Great load balancer 1 Amazon aws 1 Java syntax	Pros 15 Real Time Aggregations 6 Batch and Real-Time Ingestion 5 OLAP 3 OLAP + OLTP 2 Combining stream and historical analytics Cons 3 Limited sql support 2 Joins are not supported well 1 Complexity
Integrations
No integrations available	Zookeeper

What are some alternatives to Hadoop, Druid?

MongoDB

MongoDB stores data in JSON-like documents that can vary in structure, offering a dynamic, flexible schema. MongoDB was also designed for high availability and scalability, with built-in replication and auto-sharding.

MySQL

The MySQL software delivers a very fast, multi-threaded, multi-user, and robust SQL (Structured Query Language) database server. MySQL Server is intended for mission-critical, heavy-load production systems as well as for embedding into mass-deployed software.

PostgreSQL

PostgreSQL is an advanced object-relational database management system that supports an extended subset of the SQL standard, including transactions, foreign keys, subqueries, triggers, user-defined types and functions.

Microsoft SQL Server

Microsoft® SQL Server is a database management and analysis system for e-commerce, line-of-business, and data warehousing solutions.

SQLite

SQLite is an embedded SQL database engine. Unlike most other SQL databases, SQLite does not have a separate server process. SQLite reads and writes directly to ordinary disk files. A complete SQL database with multiple tables, indices, triggers, and views, is contained in a single disk file.

Cassandra

Partitioning means that Cassandra can distribute your data across multiple machines in an application-transparent matter. Cassandra will automatically repartition as machines are added and removed from the cluster. Row store means that like relational databases, Cassandra organizes data by rows and columns. The Cassandra Query Language (CQL) is a close relative of SQL.

Memcached

Memcached is an in-memory key-value store for small chunks of arbitrary data (strings, objects) from results of database calls, API calls, or page rendering.

MariaDB

Started by core members of the original MySQL team, MariaDB actively works with outside developers to deliver the most featureful, stable, and sanely licensed open SQL server in the industry. MariaDB is designed as a drop-in replacement of MySQL(R) with more features, new storage engines, fewer bugs, and better performance.

RethinkDB

RethinkDB is built to store JSON documents, and scale to multiple machines with very little effort. It has a pleasant query language that supports really useful queries like table joins and group by, and is easy to setup and learn.

ArangoDB

A distributed free and open-source database with a flexible data model for documents, graphs, and key-values. Build high performance applications using a convenient SQL-like query language or JavaScript extensions.

Related Comparisons

Druid vs Hadoop: What are the differences?

Performance and Scalability: Druid is designed to provide real-time query capabilities on large datasets, while Hadoop focuses on distributed storage and batch processing. Druid's architecture allows for efficient ingestion of data and provides fast query responses, making it suitable for interactive analytics. On the other hand, Hadoop's MapReduce framework enables parallel processing of data but may have slower query response times due to its batch processing nature.
Data Model: Druid stores data in a columnar format optimized for low-latency queries, where each column is stored separately and can be compressed individually. Hadoop, on the other hand, stores data in a distributed file system using the Hadoop Distributed File System (HDFS), where files are stored in a block format. This difference in data storage and compression techniques affects the speed and efficiency of data querying.
Data Ingestion: Druid is designed to support real-time data ingestion by providing mechanisms to ingest data streams and continuously update its indexes. In contrast, Hadoop is primarily used for batch processing and requires manual loading of data into the HDFS before it can be processed.
Querying Capabilities: Druid provides sub-second query response times even on large datasets due to its column-based storage and indexing structures. It supports complex aggregations, filtering, and grouping operations, making it suitable for interactive, ad-hoc queries. Hadoop, on the other hand, may have longer query response times as it processes data in batches and requires the execution of MapReduce jobs for querying.
Ecosystem and Tooling: Hadoop has a rich ecosystem of tools and frameworks such as Apache Spark, Apache Hive, and Apache Pig that can be used for various data processing tasks. It also has extensive support for data integration and processing. Druid, while it has a smaller ecosystem compared to Hadoop, provides built-in support for real-time dashboards and OLAP-style analytics.
Data Updates: Druid allows for real-time updates and incremental ingestion of data streams while maintaining query performance. Hadoop typically requires complete reprocessing of data for updates, making it more suitable for batch data processing rather than real-time data analysis.

Druid vs Hadoop

Overview