AWS Glue vs Delta Lake vs Apache Spark

Get Advice Icon

Need advice about which tool to choose?Ask the StackShare community!

AWS Glue

463
818
+ 1
9
Delta Lake

99
314
+ 1
0
Apache Spark

3K
3.5K
+ 1
140
Manage your open source components, licenses, and vulnerabilities
Learn More
Pros of AWS Glue
Pros of Delta Lake
Pros of Apache Spark
  • 9
    Managed Hive Metastore
    Be the first to leave a pro
    • 61
      Open-source
    • 48
      Fast and Flexible
    • 8
      One platform for every big data problem
    • 8
      Great for distributed SQL like applications
    • 6
      Easy to install and to use
    • 3
      Works well for most Datascience usecases
    • 2
      Interactive Query
    • 2
      Machine learning libratimery, Streaming in real
    • 2
      In memory Computation

    Sign up to add or upvote prosMake informed product decisions

    Cons of AWS Glue
    Cons of Delta Lake
    Cons of Apache Spark
      Be the first to leave a con
        Be the first to leave a con
        • 4
          Speed

        Sign up to add or upvote consMake informed product decisions

        270
        4.1K
        114
        1.3K
        982
        132
        - No public GitHub repository available -

        What is AWS Glue?

        A fully managed extract, transform, and load (ETL) service that makes it easy for customers to prepare and load their data for analytics.

        What is Delta Lake?

        An open-source storage layer that brings ACID transactions to Apache Spark™ and big data workloads.

        What is Apache Spark?

        Spark is a fast and general processing engine compatible with Hadoop data. It can run in Hadoop clusters through YARN or Spark's standalone mode, and it can process data in HDFS, HBase, Cassandra, Hive, and any Hadoop InputFormat. It is designed to perform both batch processing (similar to MapReduce) and new workloads like streaming, interactive queries, and machine learning.

        Need advice about which tool to choose?Ask the StackShare community!

        What companies use AWS Glue?
        What companies use Delta Lake?
        What companies use Apache Spark?

        Sign up to get full access to all the companiesMake informed product decisions

        What tools integrate with AWS Glue?
        What tools integrate with Delta Lake?
        What tools integrate with Apache Spark?

        Sign up to get full access to all the tool integrationsMake informed product decisions

        Blog Posts

        Mar 24 2021 at 12:57PM

        Pinterest

        GitJenkinsKafka+7
        3
        2233
        MySQLKafkaApache Spark+6
        2
        2085
        Aug 28 2019 at 3:10AM

        Segment

        PythonJavaAmazon S3+16
        7
        2657
        What are some alternatives to AWS Glue, Delta Lake, and Apache Spark?
        AWS Data Pipeline
        AWS Data Pipeline is a web service that provides a simple management system for data-driven workflows. Using AWS Data Pipeline, you define a pipeline composed of the “data sources” that contain your data, the “activities” or business logic such as EMR jobs or SQL queries, and the “schedule” on which your business logic executes. For example, you could define a job that, every hour, runs an Amazon Elastic MapReduce (Amazon EMR)–based analysis on that hour’s Amazon Simple Storage Service (Amazon S3) log data, loads the results into a relational database for future lookup, and then automatically sends you a daily summary email.
        Airflow
        Use Airflow to author workflows as directed acyclic graphs (DAGs) of tasks. The Airflow scheduler executes your tasks on an array of workers while following the specified dependencies. Rich command lines utilities makes performing complex surgeries on DAGs a snap. The rich user interface makes it easy to visualize pipelines running in production, monitor progress and troubleshoot issues when needed.
        Talend
        It is an open source software integration platform helps you in effortlessly turning data into business insights. It uses native code generation that lets you run your data pipelines seamlessly across all cloud providers and get optimized performance on all platforms.
        Alooma
        Get the power of big data in minutes with Alooma and Amazon Redshift. Simply build your pipelines and map your events using Alooma’s friendly mapping interface. Query, analyze, visualize, and predict now.
        Databricks
        Databricks Unified Analytics Platform, from the original creators of Apache Spark™, unifies data science and engineering across the Machine Learning lifecycle from data preparation to experimentation and deployment of ML applications.
        See all alternatives