Need advice about which tool to choose?Ask the StackShare community!
Dremio vs Google BigQuery: What are the differences?
Introduction
Dremio and Google BigQuery are both powerful data analytics tools that offer various features and functionalities. However, there are key differences between the two platforms that set them apart. In this analysis, we will examine these differences and highlight the unique aspects of each tool.
Scalability: One of the key differences between Dremio and Google BigQuery is their scalability. Dremio offers horizontal scalability, meaning it can distribute query execution across multiple nodes for faster processing. On the other hand, BigQuery provides automatic vertical and horizontal scaling, allowing it to handle large data volumes and scale up or down as needed.
Pricing Model: Another difference between Dremio and Google BigQuery is their pricing models. Dremio follows a subscription-based pricing model, where users pay a fixed fee based on the number of users and the data volume processed. In contrast, BigQuery uses a pay-as-you-go pricing model, where users pay only for the amount of data processed and storage used.
Storage Options: Dremio and BigQuery offer different storage options. Dremio allows users to connect and query data stored in a variety of sources, including Hadoop Distributed File System (HDFS), Amazon S3, and Azure Data Lake Storage. BigQuery, on the other hand, is more suited for cloud-native applications and works best with data stored in Google Cloud Storage.
Data Transformation and Integration: Dremio provides a comprehensive set of data transformation and integration capabilities. It allows users to transform and enrich data using SQL, and also supports data virtualization, where users can create virtual datasets that combine data from multiple sources. BigQuery, on the other hand, focuses more on data warehousing and analytics, providing powerful query and analysis features.
Query Performance: Both Dremio and BigQuery offer fast query performance, but they achieve it in different ways. Dremio uses an in-memory engine and leverages Apache Arrow to accelerate query execution. BigQuery, on the other hand, uses a distributed columnar storage and execution engine that parallelizes queries across multiple nodes for faster processing.
Data Collaboration: Dremio offers robust data collaboration features, allowing users to share, discuss, and collaborate on datasets and queries through a built-in collaboration hub. BigQuery also provides collaboration capabilities, but they are more focused on sharing and granting access to datasets rather than direct collaboration on data analysis.
In summary, Dremio and Google BigQuery have key differences in terms of scalability, pricing model, storage options, data transformation and integration capabilities, query performance, and data collaboration features. Each tool caters to different use cases and requirements, providing unique strengths in the world of data analytics.
We need to perform ETL from several databases into a data warehouse or data lake. We want to
- keep raw and transformed data available to users to draft their own queries efficiently
- give users the ability to give custom permissions and SSO
- move between open-source on-premises development and cloud-based production environments
We want to use inexpensive Amazon EC2 instances only on medium-sized data set 16GB to 32GB feeding into Tableau Server or PowerBI for reporting and data analysis purposes.
You could also use AWS Lambda and use Cloudwatch event schedule if you know when the function should be triggered. The benefit is that you could use any language and use the respective database client.
But if you orchestrate ETLs then it makes sense to use Apache Airflow. This requires Python knowledge.
Though we have always built something custom, Apache airflow (https://airflow.apache.org/) stood out as a key contender/alternative when it comes to open sources. On the commercial offering, Amazon Redshift combined with Amazon Kinesis (for complex manipulations) is great for BI, though Redshift as such is expensive.
You may want to look into a Data Virtualization product called Conduit. It connects to disparate data sources in AWS, on prem, Azure, GCP, and exposes them as a single unified Spark SQL view to PowerBI (direct query) or Tableau. Allows auto query and caching policies to enhance query speeds and experience. Has a GPU query engine and optimized Spark for fallback. Can be deployed on your AWS VM or on prem, scales up and out. Sounds like the ideal solution to your needs.
I am trying to build a data lake by pulling data from multiple data sources ( custom-built tools, excel files, CSV files, etc) and use the data lake to generate dashboards.
My question is which is the best tool to do the following:
- Create pipelines to ingest the data from multiple sources into the data lake
- Help me in aggregating and filtering data available in the data lake.
- Create new reports by combining different data elements from the data lake.
I need to use only open-source tools for this activity.
I appreciate your valuable inputs and suggestions. Thanks in Advance.
Hi Karunakaran. I obviously have an interest here, as I work for the company, but the problem you are describing is one that Zetaris can solve. Talend is a good ETL product, and Dremio is a good data virtualization product, but the problem you are describing best fits a tool that can combine the five styles of data integration (bulk/batch data movement, data replication/data synchronization, message-oriented movement of data, data virtualization, and stream data integration). I may be wrong, but Zetaris is, to the best of my knowledge, the only product in the world that can do this. Zetaris is not a dashboarding tool - you would need to combine us with Tableau or Qlik or PowerBI (or whatever) - but Zetaris can consolidate data from any source and any location (structured, unstructured, on-prem or in the cloud) in real time to allow clients a consolidated view of whatever they want whenever they want it. Please take a look at www.zetaris.com for more information. I don't want to do a "hard sell", here, so I'll say no more! Warmest regards, Rod Beecham.
Pros of Dremio
- Nice GUI to enable more people to work with Data3
- Connect NoSQL databases with RDBMS2
- Easier to Deploy2
- Free1
Pros of Google BigQuery
- High Performance28
- Easy to use25
- Fully managed service22
- Cheap Pricing19
- Process hundreds of GB in seconds16
- Big Data12
- Full table scans in seconds, no indexes needed11
- Always on, no per-hour costs8
- Good combination with fluentd6
- Machine learning4
- Easy to manage1
- Easy to learn0
Sign up to add or upvote prosMake informed product decisions
Cons of Dremio
- Works only on Iceberg structured data1
Cons of Google BigQuery
- You can't unit test changes in BQ data1
Sign up to add or upvote consMake informed product decisions
What is Dremio?
What is Google BigQuery?
Need advice about which tool to choose?Ask the StackShare community!
What companies use Dremio?
What companies use Google BigQuery?
Sign up to get full access to all the companiesMake informed product decisions
What tools integrate with Dremio?
Sign up to get full access to all the tool integrationsMake informed product decisions
Blog Posts
+16
+25
+47
+31
+39+26