Overview

Amazon Athena

Stacks505

Followers840

Votes49

AWS Glue

Stacks465

Followers819

Votes9

AWS Glue vs Amazon Athena: What are the differences?

AWS Glue and Amazon Athena are two powerful data analysis and processing services provided by Amazon Web Services (AWS). Let's explore the key differences between them.

Data Processing: AWS Glue is primarily used for Extract, Transform, Load (ETL) processes, where it provides a fully managed and serverless environment for data transformation and integration. It can handle large volumes of data and is highly scalable. On the other hand, Amazon Athena is a query service that allows you to run SQL queries directly on data stored in Amazon S3, without the need for any setup or infrastructure management. Athena is best suited for ad-hoc querying and analysis of data.
Data Catalog: AWS Glue provides a centralized metadata repository called the AWS Glue Data Catalog. This catalog can store and manage metadata information about various data sources, tables, and their schemas. It also supports automatic schema discovery and data cataloging. In contrast, Amazon Athena does not have its own data catalog. It relies on the AWS Glue Data Catalog or an external Hive Metastore for storing and managing metadata.
Data Formats and Compression: AWS Glue supports a wide range of data formats for ingestion, transformation, and processing, including CSV, JSON, Avro, etc. It also allows you to perform compression and decompression of data using codecs like Gzip, Snappy, etc. On the other hand, Amazon Athena supports a limited set of data formats, primarily CSV, JSON, Parquet, and ORC. It also supports columnar compression using Snappy and Zlib.
Data Partitioning: AWS Glue provides built-in support for data partitioning, which allows you to organize your data in multiple directories based on certain columns. This can significantly improve query performance, especially when dealing with large datasets. Amazon Athena, on the other hand, does not have native support for data partitioning. However, you can still use the underlying directory structure in Amazon S3 to mimic the partitioning behavior.
Pricing Model: AWS Glue has a pay-as-you-go pricing model, where you are charged based on the number of Data Processing Units (DPUs) consumed during ETL jobs. DPUs represent the processing power and memory allocated to a job. On the other hand, Amazon Athena follows a pay-per-query pricing model, where you are charged based on the amount of data scanned by each query. This can be cost-effective for sporadic or ad-hoc queries.
Integration with Other AWS Services: AWS Glue seamlessly integrates with other AWS services like Amazon S3, Amazon Redshift, Amazon RDS, etc., allowing you to easily move and process data between these services. It also provides built-in connections to popular data sources like Oracle, MySQL, etc. Amazon Athena, on the other hand, is tightly integrated with Amazon S3 and is primarily used for querying and analyzing data stored in S3. It does not have direct integrations with other AWS services.

In summary, AWS Glue is a comprehensive ETL service that offers managed data transformation, integration, and cataloging capabilities. It is designed for large-scale data processing and provides extensive integration options. On the other hand, Amazon Athena is a powerful querying and analysis service that allows you to run SQL queries directly on data stored in Amazon S3. It is ideal for ad-hoc analysis and does not require any setup or infrastructure management.

🔥 Trending in Databases on StackShare

pgvector Databases

Open-source vector similarity search for Postgres

Try it View Docs Alternatives

Try

189

7

Advice on Amazon Athena, AWS Glue

Aditya

Mar 13, 2021

Review

you can use aws glue service to convert you pipe format data to parquet format , and thus you can achieve data compression . Now you should choose Redshift to copy your data as it is very huge. To manage your data, you should partition your data in S3 bucket and also divide your data across the redshift cluster

220k views220k

Comments

Vamshi

Data Engineer at Tata Consultancy Services

May 29, 2020

Needs adviceon

PySpark

Azure Data Factory

Databricks

I have to collect different data from multiple sources and store them in a single cloud location. Then perform cleaning and transforming using PySpark, and push the end results to other applications like reporting tools, etc. What would be the best solution? I can only think of Azure Data Factory + Databricks. Are there any alternatives to #AWS services + Databricks?

269k views269k

Comments

datocrats-org

Jul 29, 2020

Needs adviceon

Amazon EC2

Tableau

PowerBI

We need to perform ETL from several databases into a data warehouse or data lake. We want to

keep raw and transformed data available to users to draft their own queries efficiently
give users the ability to give custom permissions and SSO
move between open-source on-premises development and cloud-based production environments

We want to use inexpensive Amazon EC2 instances only on medium-sized data set 16GB to 32GB feeding into Tableau Server or PowerBI for reporting and data analysis purposes.

319k views319k

Comments

Detailed Comparison

Amazon Athena	AWS Glue
Amazon Athena is an interactive query service that makes it easy to analyze data in Amazon S3 using standard SQL. Athena is serverless, so there is no infrastructure to manage, and you pay only for the queries that you run.	A fully managed extract, transform, and load (ETL) service that makes it easy for customers to prepare and load their data for analytics.
-	Easy - AWS Glue automates much of the effort in building, maintaining, and running ETL jobs. AWS Glue crawls your data sources, identifies data formats, and suggests schemas and transformations. AWS Glue automatically generates the code to execute your data transformations and loading processes.; Integrated - AWS Glue is integrated across a wide range of AWS services.; Serverless - AWS Glue is serverless. There is no infrastructure to provision or manage. AWS Glue handles provisioning, configuration, and scaling of the resources required to run your ETL jobs on a fully managed, scale-out Apache Spark environment. You pay only for the resources used while your jobs are running.; Developer Friendly - AWS Glue generates ETL code that is customizable, reusable, and portable, using familiar technology - Scala, Python, and Apache Spark. You can also import custom readers, writers and transformations into your Glue ETL code. Since the code AWS Glue generates is based on open frameworks, there is no lock-in. You can use it anywhere.
Statistics
Stacks 505	Stacks 465
Followers 840	Followers 819
Votes 49	Votes 9
Pros & Cons
Pros 16 Use SQL to analyze CSV files 8 Glue crawlers gives easy Data catalogue 7 Cheap 6 Query all my data without running servers 24x7 4 No data base servers yay	Pros 9 Managed Hive Metastore
Integrations
Amazon S3 Presto	Amazon Redshift Amazon S3 Amazon RDS MySQL Microsoft SQL Server Amazon EMR Amazon Aurora Oracle Amazon RDS for PostgreSQL Amazon Redshift Spectrum

What are some alternatives to Amazon Athena, AWS Glue?

Apache Spark

Spark is a fast and general processing engine compatible with Hadoop data. It can run in Hadoop clusters through YARN or Spark's standalone mode, and it can process data in HDFS, HBase, Cassandra, Hive, and any Hadoop InputFormat. It is designed to perform both batch processing (similar to MapReduce) and new workloads like streaming, interactive queries, and machine learning.

Presto

Distributed SQL Query Engine for Big Data

Apache Flink

Apache Flink is an open source system for fast and versatile data analytics in clusters. Flink supports batch and streaming analytics, in one system. Analytical programs can be written in concise and elegant APIs in Java and Scala.

lakeFS

It is an open-source data version control system for data lakes. It provides a “Git for data” platform enabling you to implement best practices from software engineering on your data lake, including branching and merging, CI/CD, and production-like dev/test environments.

Druid

Druid is a distributed, column-oriented, real-time analytics data store that is commonly used to power exploratory dashboards in multi-tenant environments. Druid excels as a data warehousing solution for fast aggregate queries on petabyte sized data sets. Druid supports a variety of flexible filters, exact calculations, approximate algorithms, and other useful calculations.

Apache Kylin

Apache Kylin™ is an open source Distributed Analytics Engine designed to provide SQL interface and multi-dimensional analysis (OLAP) on Hadoop/Spark supporting extremely large datasets, originally contributed from eBay Inc.

Splunk

It provides the leading platform for Operational Intelligence. Customers use it to search, monitor, analyze and visualize machine data.

Apache Impala

Impala is a modern, open source, MPP SQL query engine for Apache Hadoop. Impala is shipped by Cloudera, MapR, and Amazon. With Impala, you can query data, whether stored in HDFS or Apache HBase – including SELECT, JOIN, and aggregate functions – in real time.

Vertica

It provides a best-in-class, unified analytics platform that will forever be independent from underlying infrastructure.

Azure Synapse

It is an analytics service that brings together enterprise data warehousing and Big Data analytics. It gives you the freedom to query data on your terms, using either serverless on-demand or provisioned resources—at scale. It brings these two worlds together with a unified experience to ingest, prepare, manage, and serve data for immediate BI and machine learning needs.