Apache Parquet vs Microsoft SQL Server

Need advice about which tool to choose?Ask the StackShare community!

Apache Parquet

91
186
+ 1
0
Microsoft SQL Server

19.4K
15K
+ 1
540
Add tool

Apache Parquet vs Microsoft SQL Server: What are the differences?

Apache Parquet vs Microsoft SQL Server

Apache Parquet and Microsoft SQL Server are both data storage solutions commonly used in the industry. While they serve the same purpose of storing and managing data, there are key differences between them. Here are the main differences:

  1. Storage Format: Apache Parquet is a columnar storage file format, while Microsoft SQL Server uses a relational database management system (RDBMS) to store data. In Parquet, data is grouped by columns, which allows for efficient compression and faster query execution. SQL Server, on the other hand, organizes data in tables with rows and columns, following a relational model.

  2. Compression Techniques: Parquet offers various compression techniques such as Snappy, Gzip, and LZO. These compression techniques significantly reduce storage space and improve query performance. In contrast, SQL Server has its own compression algorithms optimized for relational data, but may not offer the same level of compression as Parquet.

  3. Data Types: Parquet supports a wide range of complex data types, including nested structures and lists, making it suitable for handling complex data. SQL Server, being a relational database, primarily supports basic data types such as integers, strings, and dates. Complex data types in SQL Server are often represented using normalization techniques.

  4. Query Performance: Due to its columnar storage format and advanced compression techniques, Parquet excels in analytical workloads. It can efficiently skip irrelevant data during queries, resulting in faster query performance. SQL Server, being a fully-fledged RDBMS, is optimized for transactional workloads and offers features like indexing and caching to improve query performance.

  5. Scalability: Parquet is designed to be highly scalable and distributed, making it suitable for big data processing frameworks like Apache Hadoop and Apache Spark. It can handle large volumes of data across multiple nodes, allowing for parallel processing. SQL Server, on the other hand, is more suitable for traditional, scale-up scenarios where a single server or a cluster of servers handle the workload.

  6. Cost: Parquet is an open-source file format that can be used free of charge. It can be integrated with various data processing frameworks, making it a cost-efficient solution. SQL Server, on the other hand, is a licensed product with associated costs for licensing, maintenance, and support.

In summary, Apache Parquet offers efficient columnar storage, advanced compression, support for complex data types, and excellent query performance for analytical workloads. It is highly scalable and cost-efficient. Microsoft SQL Server, on the other hand, follows a relational model, offers transactional workloads optimization, and is more suitable for traditional scale-up scenarios.

Advice on Apache Parquet and Microsoft SQL Server

I am a Microsoft SQL Server programmer who is a bit out of practice. I have been asked to assist on a new project. The overall purpose is to organize a large number of recordings so that they can be searched. I have an enormous music library but my songs are several hours long. I need to include things like time, date and location of the recording. I don't have a problem with the general database design. I have two primary questions:

  1. I need to use either MySQL or PostgreSQL on a Linux based OS. Which would be better for this application?
  2. I have not dealt with a sound based data type before. How do I store that and put it in a table? Thank you.
See more
Replies (6)

Hi Erin,

Honestly both databases will do the job just fine. I personally prefer Postgres.

Much more important is how you store the audio. While you could technically use a blob type column, it's really not ideal to be storing audio files which are "several hours long" in a database row. Instead consider storing the audio files in an object store (hosted options include backblaze b2 or aws s3) and persisting the key (which references that object) in your database column.

See more
Aaron Westley
Recommends
on
PostgreSQLPostgreSQL

Hi Erin, Chances are you would want to store the files in a blob type. Both MySQL and Postgres support this. Can you explain a little more about your need to store the files in the database? I may be more effective to store the files on a file system or something like S3. To answer your qustion based on what you are descibing I would slighly lean towards PostgreSQL since it tends to be a little better on the data warehousing side.

See more
Christopher Wray
Web Developer at Soltech LLC · | 3 upvotes · 424.9K views
Recommends
on
DirectusDirectus
at

Hey Erin! I would recommend checking out Directus before you start work on building your own app for them. I just stumbled upon it, and so far extremely happy with the functionalities. If your client is just looking for a simple web app for their own data, then Directus may be a great option. It offers "database mirroring", so that you can connect it to any database and set up functionality around it!

See more
Julien DeFrance
Principal Software Engineer at Tophatter · | 3 upvotes · 424.5K views
Recommends
on
Amazon AuroraAmazon Aurora

Hi Erin! First of all, you'd probably want to go with a managed service. Don't spin up your own MySQL installation on your own Linux box. If you are on AWS, thet have different offerings for database services. Standard RDS vs. Aurora. Aurora would be my preferred choice given the benefits it offers, storage optimizations it comes with... etc. Such managed services easily allow you to apply new security patches and upgrades, set up backups, replication... etc. Doing this on your own would either be risky, inefficient, or you might just give up. As far as which database to chose, you'll have the choice between Postgresql, MySQL, Maria DB, SQL Server... etc. I personally would recommend MySQL (latest version available), as the official tooling for it (MySQL Workbench) is great, stable, and moreover free. Other database services exist, I'd recommend you also explore Dynamo DB.

Regardless, you'd certainly only keep high-level records, meta data in Database, and the actual files, most-likely in S3, so that you can keep all options open in terms of what you'll do with them.

See more
Recommends
on
PostgreSQLPostgreSQL

Hi Erin,

  • Coming from "Big" DB engines, such as Oracle or MSSQL, go for PostgreSQL. You'll get all the features you need with PostgreSQL.
  • Your case seems to point to a "NoSQL" or Document Database use case. Since you get covered on this with PostgreSQL which achieves excellent performances on JSON based objects, this is a second reason to choose PostgreSQL. MongoDB might be an excellent option as well if you need "sharding" and excellent map-reduce mechanisms for very massive data sets. You really should investigate the NoSQL option for your use case.
  • Starting with AWS Aurora is an excellent advise. since "vendor lock-in" is limited, but I did not check for JSON based object / NoSQL features.
  • If you stick to Linux server, the PostgreSQL or MySQL provided with your distribution are straightforward to install (i.e. apt install postgresql). For PostgreSQL, make sure you're comfortable with the pg_hba.conf, especially for IP restrictions & accesses.

Regards,

See more
Klaus Nji
Staff Software Engineer at SailPoint Technologies · | 1 upvotes · 424.6K views
Recommends
on
PostgreSQLPostgreSQL

I recommend Postgres as well. Superior performance overall and a more robust architecture.

See more
Get Advice from developers at your company using StackShare Enterprise. Sign up for StackShare Enterprise.
Learn More
Pros of Apache Parquet
Pros of Microsoft SQL Server
    Be the first to leave a pro
    • 139
      Reliable and easy to use
    • 102
      High performance
    • 95
      Great with .net
    • 65
      Works well with .net
    • 56
      Easy to maintain
    • 21
      Azure support
    • 17
      Full Index Support
    • 17
      Always on
    • 10
      Enterprise manager is fantastic
    • 9
      In-Memory OLTP Engine
    • 2
      Easy to setup and configure
    • 2
      Security is forefront
    • 1
      Faster Than Oracle
    • 1
      Decent management tools
    • 1
      Great documentation
    • 1
      Docker Delivery
    • 1
      Columnstore indexes

    Sign up to add or upvote prosMake informed product decisions

    Cons of Apache Parquet
    Cons of Microsoft SQL Server
      Be the first to leave a con
      • 4
        Expensive Licensing
      • 2
        Microsoft

      Sign up to add or upvote consMake informed product decisions

      No Stats
      - No public GitHub repository available -

      What is Apache Parquet?

      It is a columnar storage format available to any project in the Hadoop ecosystem, regardless of the choice of data processing framework, data model or programming language.

      What is Microsoft SQL Server?

      Microsoft® SQL Server is a database management and analysis system for e-commerce, line-of-business, and data warehousing solutions.

      Need advice about which tool to choose?Ask the StackShare community!

      What companies use Apache Parquet?
      What companies use Microsoft SQL Server?
      See which teams inside your own company are using Apache Parquet or Microsoft SQL Server.
      Sign up for StackShare EnterpriseLearn More

      Sign up to get full access to all the companiesMake informed product decisions

      What tools integrate with Apache Parquet?
      What tools integrate with Microsoft SQL Server?

      Sign up to get full access to all the tool integrationsMake informed product decisions

      Blog Posts

      Aug 28 2019 at 3:10AM

      Segment

      PythonJavaAmazon S3+16
      7
      2555
      What are some alternatives to Apache Parquet and Microsoft SQL Server?
      Avro
      It is a row-oriented remote procedure call and data serialization framework developed within Apache's Hadoop project. It uses JSON for defining data types and protocols, and serializes data in a compact binary format.
      Apache Kudu
      A new addition to the open source Apache Hadoop ecosystem, Kudu completes Hadoop's storage layer to enable fast analytics on fast data.
      JSON
      JavaScript Object Notation is a lightweight data-interchange format. It is easy for humans to read and write. It is easy for machines to parse and generate. It is based on a subset of the JavaScript Programming Language.
      Cassandra
      Partitioning means that Cassandra can distribute your data across multiple machines in an application-transparent matter. Cassandra will automatically repartition as machines are added and removed from the cluster. Row store means that like relational databases, Cassandra organizes data by rows and columns. The Cassandra Query Language (CQL) is a close relative of SQL.
      HBase
      Apache HBase is an open-source, distributed, versioned, column-oriented store modeled after Google' Bigtable: A Distributed Storage System for Structured Data by Chang et al. Just as Bigtable leverages the distributed data storage provided by the Google File System, HBase provides Bigtable-like capabilities on top of Apache Hadoop.
      See all alternatives