Overview

Apache Parquet

Stacks97

Followers190

Votes0

ScyllaDB

Stacks143

Followers197

Votes8

Apache Parquet vs Scylla: What are the differences?

Introduction

Apache Parquet is an open-source columnar storage format designed for big data processing frameworks, while Scylla is an open-source distributed NoSQL database. Although both technologies are used in big data processing, they have key differences that set them apart.

Storage Format: Apache Parquet is a storage format that organizes data in a columnar format, which makes it more efficient for large-scale analytics workloads. On the other hand, Scylla is a NoSQL database that stores data in a distributed manner using sharding and replication.
Data Modeling: Apache Parquet is schema-on-read, meaning it does not enforce a strict schema for the stored data. This flexibility allows for easy evolution of the schema over time. In contrast, Scylla follows a schema-on-write approach, where a predefined schema is enforced for the stored data. This ensures data consistency and integrity.
Data Consistency: Apache Parquet does not provide any built-in mechanisms for ensuring data consistency. It primarily focuses on efficient data storage and retrieval. In contrast, Scylla ensures data consistency using techniques like consensus algorithms and distributed transactions. This ensures that data is always in a consistent state across the distributed database.
Querying Language: Apache Parquet does not have a built-in querying language. It is typically used in conjunction with query engines like Apache Hive or Apache Impala. These query engines provide SQL-like interfaces to perform queries on Parquet data. On the other hand, Scylla has its own query language called CQL (Cassandra Query Language), which is similar to SQL and allows for powerful data querying and manipulation.
Data Scalability: Apache Parquet is designed to handle large-scale datasets by organizing data in a columnar format and utilizing compression techniques. It can efficiently process and analyze massive amounts of data. However, it is not primarily focused on distributed storage and scalability. In contrast, Scylla is specifically designed to handle distributed storage and scalability. It can seamlessly scale horizontally by adding more nodes to the cluster, ensuring high availability and fault tolerance.
Data Durability: Apache Parquet is not designed for high data durability as it assumes that the data is stored in a reliable storage system. On the other hand, Scylla ensures high data durability by replicating data across multiple nodes in a distributed manner. Even in the event of node failures, Scylla can maintain data availability and recover the lost data using replication techniques.

In summary, Apache Parquet is a columnar storage format focused on efficient big data processing, while Scylla is a distributed NoSQL database focused on high scalability, data consistency, and durability.

🔥 Trending in Databases on StackShare

PostgreSQL Databases

A powerful, open source object-relational database system

Try it View Docs Alternatives

Try

431

83.9k

Advice on Apache Parquet, ScyllaDB

Tom

CEO at Gentlent

Jun 9, 2020

Decided

The Gentlent Tech Team made lots of updates within the past year. The biggest one being our database:

We decided to migrate our #PostgreSQL -based database systems to a custom implementation of #Cassandra . This allows us to integrate our product data perfectly in a system that just makes sense. High availability and scalability are supported out of the box.

387k views387k

Comments

Vinay

Head of Engineering

Sep 19, 2019

Needs advice

The problem I have is - we need to process & change(update/insert) 55M Data every 2 min and this updated data to be available for Rest API for Filtering / Selection. Response time for Rest API should be less than 1 sec.

The most important factors for me are processing and storing time of 2 min. There need to be 2 views of Data One is for Selection & 2. Changed data.

174k views174k

Comments

Detailed Comparison

Apache Parquet	ScyllaDB
It is a columnar storage format available to any project in the Hadoop ecosystem, regardless of the choice of data processing framework, data model or programming language.	ScyllaDB is the database for data-intensive apps that require high performance and low latency. It enables teams to harness the ever-increasing computing power of modern infrastructures – eliminating barriers to scale as data grows.
Columnar storage format;Type-specific encoding; Pig integration; Cascading integration; Crunch integration; Apache Arrow integration; Apache Scrooge integration;Adaptive dictionary encoding; Predicate pushdown; Column stats	High availability; horizontal scalability; vertical scalability; Cassandra compatible; DynamoDB compatible; wide column; NoSQL; lightweight transactions; change data capture; workload prioritization; shard-per-core; IO scheduler; self-tuning
Statistics
Stacks 97	Stacks 143
Followers 190	Followers 197
Votes 0	Votes 8
Pros & Cons
No community feedback yet	Pros 2 Replication 1 Written in C++ 1 High availability 1 High performance 1 Distributed
Integrations
Hadoop Java Apache Impala Apache Thrift Apache Hive Pig	KairosDB Wireshark JanusGraph Grafana Hackolade Prometheus Kubernetes Datadog Kafka Apache Spark

What are some alternatives to Apache Parquet, ScyllaDB?

MongoDB

MongoDB stores data in JSON-like documents that can vary in structure, offering a dynamic, flexible schema. MongoDB was also designed for high availability and scalability, with built-in replication and auto-sharding.

MySQL

The MySQL software delivers a very fast, multi-threaded, multi-user, and robust SQL (Structured Query Language) database server. MySQL Server is intended for mission-critical, heavy-load production systems as well as for embedding into mass-deployed software.

PostgreSQL

PostgreSQL is an advanced object-relational database management system that supports an extended subset of the SQL standard, including transactions, foreign keys, subqueries, triggers, user-defined types and functions.

Microsoft SQL Server

Microsoft® SQL Server is a database management and analysis system for e-commerce, line-of-business, and data warehousing solutions.

SQLite

SQLite is an embedded SQL database engine. Unlike most other SQL databases, SQLite does not have a separate server process. SQLite reads and writes directly to ordinary disk files. A complete SQL database with multiple tables, indices, triggers, and views, is contained in a single disk file.

Cassandra

Partitioning means that Cassandra can distribute your data across multiple machines in an application-transparent matter. Cassandra will automatically repartition as machines are added and removed from the cluster. Row store means that like relational databases, Cassandra organizes data by rows and columns. The Cassandra Query Language (CQL) is a close relative of SQL.

Memcached

Memcached is an in-memory key-value store for small chunks of arbitrary data (strings, objects) from results of database calls, API calls, or page rendering.

MariaDB

Started by core members of the original MySQL team, MariaDB actively works with outside developers to deliver the most featureful, stable, and sanely licensed open SQL server in the industry. MariaDB is designed as a drop-in replacement of MySQL(R) with more features, new storage engines, fewer bugs, and better performance.

RethinkDB

RethinkDB is built to store JSON documents, and scale to multiple machines with very little effort. It has a pleasant query language that supports really useful queries like table joins and group by, and is easy to setup and learn.

ArangoDB

A distributed free and open-source database with a flexible data model for documents, graphs, and key-values. Build high performance applications using a convenient SQL-like query language or JavaScript extensions.