Apache Parquet vs Dremio

Overview

Apache Parquet

Stacks97

Followers190

Votes0

Dremio

Stacks116

Followers348

Votes8

Dremio vs Apache Parquet: What are the differences?

What is Dremio? Self-service data for everyone. It is a data-as-a-service platform that empowers users to discover, curate, accelerate, and share any data at any time, regardless of location, volume, or structure. Modern data is managed by a wide range of technologies, including relational databases, NoSQL datastores, file systems, Hadoop, and others.

What is Apache Parquet? *A free and open-source column-oriented data storage format *. It is a columnar storage format available to any project in the Hadoop ecosystem, regardless of the choice of data processing framework, data model or programming language.

Dremio and Apache Parquet are primarily classified as "Big Data as a Service" and "Big Data" tools respectively.

Some of the features offered by Dremio are:

Democratize all your data
Make your data engineers more productive
Accelerate your favorite tools

On the other hand, Apache Parquet provides the following key features:

Columnar storage format
Type-specific encoding
Pig integration

Apache Parquet is an open source tool with 918 GitHub stars and 805 GitHub forks. Here's a link to Apache Parquet's open source repository on GitHub.

🔥 Trending in Databases on StackShare

PostgreSQL Databases

A powerful, open source object-relational database system

Try it View Docs Alternatives

Try

431

83.9k

Advice on Apache Parquet, Dremio

karunakaran

Consultant

Jun 26, 2020

Needs advice

I am trying to build a data lake by pulling data from multiple data sources ( custom-built tools, excel files, CSV files, etc) and use the data lake to generate dashboards.

My question is which is the best tool to do the following:

Create pipelines to ingest the data from multiple sources into the data lake
Help me in aggregating and filtering data available in the data lake.
Create new reports by combining different data elements from the data lake.

I need to use only open-source tools for this activity.

I appreciate your valuable inputs and suggestions. Thanks in Advance.

80.4k views80.4k

Comments

datocrats-org

Jul 29, 2020

Needs adviceon

Amazon EC2

Tableau

PowerBI

We need to perform ETL from several databases into a data warehouse or data lake. We want to

keep raw and transformed data available to users to draft their own queries efficiently
give users the ability to give custom permissions and SSO
move between open-source on-premises development and cloud-based production environments

We want to use inexpensive Amazon EC2 instances only on medium-sized data set 16GB to 32GB feeding into Tableau Server or PowerBI for reporting and data analysis purposes.

319k views319k

Comments

Detailed Comparison

Apache Parquet	Dremio
It is a columnar storage format available to any project in the Hadoop ecosystem, regardless of the choice of data processing framework, data model or programming language.	Dremio—the data lake engine, operationalizes your data lake storage and speeds your analytics processes with a high-performance and high-efficiency query engine while also democratizing data access for data scientists and analysts.
Columnar storage format;Type-specific encoding; Pig integration; Cascading integration; Crunch integration; Apache Arrow integration; Apache Scrooge integration;Adaptive dictionary encoding; Predicate pushdown; Column stats	Democratize all your data; Make your data engineers more productive; Accelerate your favorite tools; Self service, for everybody
Statistics
Stacks 97	Stacks 116
Followers 190	Followers 348
Votes 0	Votes 8
Pros & Cons
No community feedback yet	Pros 3 Nice GUI to enable more people to work with Data 2 Easier to Deploy 2 Connect NoSQL databases with RDBMS 1 Free Cons 1 Works only on Iceberg structured data
Integrations
Hadoop Java Apache Impala Apache Thrift Apache Hive Pig	Amazon S3 Python Tableau Azure Database for PostgreSQL Qlik Sense PowerBI

What are some alternatives to Apache Parquet, Dremio?

MongoDB

MongoDB stores data in JSON-like documents that can vary in structure, offering a dynamic, flexible schema. MongoDB was also designed for high availability and scalability, with built-in replication and auto-sharding.

MySQL

The MySQL software delivers a very fast, multi-threaded, multi-user, and robust SQL (Structured Query Language) database server. MySQL Server is intended for mission-critical, heavy-load production systems as well as for embedding into mass-deployed software.

PostgreSQL

PostgreSQL is an advanced object-relational database management system that supports an extended subset of the SQL standard, including transactions, foreign keys, subqueries, triggers, user-defined types and functions.

Microsoft SQL Server

Microsoft® SQL Server is a database management and analysis system for e-commerce, line-of-business, and data warehousing solutions.

SQLite

SQLite is an embedded SQL database engine. Unlike most other SQL databases, SQLite does not have a separate server process. SQLite reads and writes directly to ordinary disk files. A complete SQL database with multiple tables, indices, triggers, and views, is contained in a single disk file.

Cassandra

Partitioning means that Cassandra can distribute your data across multiple machines in an application-transparent matter. Cassandra will automatically repartition as machines are added and removed from the cluster. Row store means that like relational databases, Cassandra organizes data by rows and columns. The Cassandra Query Language (CQL) is a close relative of SQL.

Memcached

Memcached is an in-memory key-value store for small chunks of arbitrary data (strings, objects) from results of database calls, API calls, or page rendering.

MariaDB

Started by core members of the original MySQL team, MariaDB actively works with outside developers to deliver the most featureful, stable, and sanely licensed open SQL server in the industry. MariaDB is designed as a drop-in replacement of MySQL(R) with more features, new storage engines, fewer bugs, and better performance.

RethinkDB

RethinkDB is built to store JSON documents, and scale to multiple machines with very little effort. It has a pleasant query language that supports really useful queries like table joins and group by, and is easy to setup and learn.

ArangoDB

A distributed free and open-source database with a flexible data model for documents, graphs, and key-values. Build high performance applications using a convenient SQL-like query language or JavaScript extensions.