Amazon Redshift vs Druid

Overview

Amazon Redshift

Stacks1.5K

Followers1.4K

Votes108

Druid

Stacks376

Followers867

Votes32

Amazon Redshift vs Druid: What are the differences?

Introduction:

When comparing Amazon Redshift and Druid, it's essential to understand the key differences between the two data storage and processing solutions. Both are popular choices for handling large volumes of data and providing analytical capabilities, but they have distinct features that make them suitable for different use cases.

Architecture: Amazon Redshift is a fully managed data warehouse service that uses a columnar storage architecture optimized for complex queries and high-performance analytics. In contrast, Druid is a distributed, column-oriented, real-time data store designed to handle high data ingestion rates and provide sub-second queries for time-series data.
Query Processing: Amazon Redshift uses traditional SQL queries and can handle complex joins, aggregations, and window functions efficiently. On the other hand, Druid supports SQL-like queries along with Apache Druid Query Language (DQL) for real-time and interactive analytics. It provides faster query response times for time-series data by utilizing a specialized query engine.
Data Ingestion: Amazon Redshift allows data to be loaded from various sources using tools like AWS Glue, Amazon Kinesis, and Amazon S3. Druid is designed for real-time data ingestion and can directly ingest streaming data from sources like Kafka and Apache Storm. It supports continuous data ingestion and enables interactive analytics on fresh data.
Scalability: Amazon Redshift offers on-demand scalability by automatically managing storage expansion, compute resources, and query optimization. Druid is horizontally scalable and can be easily scaled out by adding more nodes to the cluster, providing the ability to handle massive data sets and an increasing number of queries.
Use Cases: Amazon Redshift is well-suited for traditional data warehousing and business intelligence workloads where ad-hoc queries, reporting, and dashboarding are required. Druid is ideal for use cases that demand real-time analytics, event-driven architectures, time-series data analysis, and interactive dashboarding with near real-time insights.

In Summary, Amazon Redshift and Druid cater to different data processing and analytics requirements, with Redshift excelling in traditional data warehousing tasks and Druid providing real-time analytics capabilities for time-series data and event-driven applications.

Share your Stack

Help developers discover the tools you use. Get visibility for your team's tech choices and contribute to the community's knowledge.

View Docs

CLI (Node.js)

Manual

Advice on Amazon Redshift, Druid

datocrats-org

Jul 29, 2020

Needs adviceon

Amazon EC2

Tableau

PowerBI

We need to perform ETL from several databases into a data warehouse or data lake. We want to

keep raw and transformed data available to users to draft their own queries efficiently
give users the ability to give custom permissions and SSO
move between open-source on-premises development and cloud-based production environments

We want to use inexpensive Amazon EC2 instances only on medium-sized data set 16GB to 32GB feeding into Tableau Server or PowerBI for reporting and data analysis purposes.

319k views319k

Comments

Detailed Comparison

Amazon Redshift	Druid
It is optimized for data sets ranging from a few hundred gigabytes to a petabyte or more and costs less than $1,000 per terabyte per year, a tenth the cost of most traditional data warehousing solutions.	Druid is a distributed, column-oriented, real-time analytics data store that is commonly used to power exploratory dashboards in multi-tenant environments. Druid excels as a data warehousing solution for fast aggregate queries on petabyte sized data sets. Druid supports a variety of flexible filters, exact calculations, approximate algorithms, and other useful calculations.
Optimized for Data Warehousing- It uses columnar storage, data compression, and zone maps to reduce the amount of IO needed to perform queries. Redshift has a massively parallel processing (MPP) architecture, parallelizing and distributing SQL operations to take advantage of all available resources.;Scalable- With a few clicks of the AWS Management Console or a simple API call, you can easily scale the number of nodes in your data warehouse up or down as your performance or capacity needs change.;No Up-Front Costs- You pay only for the resources you provision. You can choose On-Demand pricing with no up-front costs or long-term commitments, or obtain significantly discounted rates with Reserved Instance pricing.;Fault Tolerant- Amazon Redshift has multiple features that enhance the reliability of your data warehouse cluster. All data written to a node in your cluster is automatically replicated to other nodes within the cluster and all data is continuously backed up to Amazon S3.;SQL - Amazon Redshift is a SQL data warehouse and uses industry standard ODBC and JDBC connections and Postgres drivers.;Isolation - Amazon Redshift enables you to configure firewall rules to control network access to your data warehouse cluster.;Encryption – With just a couple of parameter settings, you can set up Amazon Redshift to use SSL to secure data in transit and hardware-acccelerated AES-256 encryption for data at rest.<br>	-
Statistics
Stacks 1.5K	Stacks 376
Followers 1.4K	Followers 867
Votes 108	Votes 32
Pros & Cons
Pros 41 Data Warehousing 27 Scalable 17 SQL 14 Backed by Amazon 5 Encryption	Pros 15 Real Time Aggregations 6 Batch and Real-Time Ingestion 5 OLAP 3 OLAP + OLTP 2 Combining stream and historical analytics Cons 3 Limited sql support 2 Joins are not supported well 1 Complexity
Integrations
SQLite MySQL Oracle PL/SQL	Zookeeper

What are some alternatives to Amazon Redshift, Druid?

Google BigQuery

Run super-fast, SQL-like queries against terabytes of data in seconds, using the processing power of Google's infrastructure. Load data with ease. Bulk load your data using Google Cloud Storage or stream it in. Easy access. Access BigQuery by using a browser tool, a command-line tool, or by making calls to the BigQuery REST API with client libraries such as Java, PHP or Python.

Apache Spark

Spark is a fast and general processing engine compatible with Hadoop data. It can run in Hadoop clusters through YARN or Spark's standalone mode, and it can process data in HDFS, HBase, Cassandra, Hive, and any Hadoop InputFormat. It is designed to perform both batch processing (similar to MapReduce) and new workloads like streaming, interactive queries, and machine learning.

Qubole

Qubole is a cloud based service that makes big data easy for analysts and data engineers.

Presto

Distributed SQL Query Engine for Big Data

Amazon EMR

It is used in a variety of applications, including log analysis, data warehousing, machine learning, financial analysis, scientific simulation, and bioinformatics.

Amazon Athena

Amazon Athena is an interactive query service that makes it easy to analyze data in Amazon S3 using standard SQL. Athena is serverless, so there is no infrastructure to manage, and you pay only for the queries that you run.

Apache Flink

Apache Flink is an open source system for fast and versatile data analytics in clusters. Flink supports batch and streaming analytics, in one system. Analytical programs can be written in concise and elegant APIs in Java and Scala.

lakeFS

It is an open-source data version control system for data lakes. It provides a “Git for data” platform enabling you to implement best practices from software engineering on your data lake, including branching and merging, CI/CD, and production-like dev/test environments.

Altiscale

we run Apache Hadoop for you. We not only deploy Hadoop, we monitor, manage, fix, and update it for you. Then we take it a step further: We monitor your jobs, notify you when something’s wrong with them, and can help with tuning.

Snowflake

Snowflake eliminates the administration and management demands of traditional data warehouses and big data platforms. Snowflake is a true data warehouse as a service running on Amazon Web Services (AWS)—no infrastructure to manage and no knobs to turn.

Related Comparisons

Amazon Redshift vs Druid: What are the differences?

Introduction:

Architecture: Amazon Redshift is a fully managed data warehouse service that uses a columnar storage architecture optimized for complex queries and high-performance analytics. In contrast, Druid is a distributed, column-oriented, real-time data store designed to handle high data ingestion rates and provide sub-second queries for time-series data.
Query Processing: Amazon Redshift uses traditional SQL queries and can handle complex joins, aggregations, and window functions efficiently. On the other hand, Druid supports SQL-like queries along with Apache Druid Query Language (DQL) for real-time and interactive analytics. It provides faster query response times for time-series data by utilizing a specialized query engine.
Data Ingestion: Amazon Redshift allows data to be loaded from various sources using tools like AWS Glue, Amazon Kinesis, and Amazon S3. Druid is designed for real-time data ingestion and can directly ingest streaming data from sources like Kafka and Apache Storm. It supports continuous data ingestion and enables interactive analytics on fresh data.
Scalability: Amazon Redshift offers on-demand scalability by automatically managing storage expansion, compute resources, and query optimization. Druid is horizontally scalable and can be easily scaled out by adding more nodes to the cluster, providing the ability to handle massive data sets and an increasing number of queries.
Use Cases: Amazon Redshift is well-suited for traditional data warehousing and business intelligence workloads where ad-hoc queries, reporting, and dashboarding are required. Druid is ideal for use cases that demand real-time analytics, event-driven architectures, time-series data analysis, and interactive dashboarding with near real-time insights.

Amazon Redshift vs Druid

Overview