Overview

Azure Data Factory

Stacks253

Followers484

Votes0

GitHub Stars516

Forks610

Amundsen

Stacks17

Followers42

Votes0

Amundsen vs Azure Data Factory: What are the differences?

Introduction:
Amundsen and Azure Data Factory are both data management platforms with unique features and capabilities. Understanding the key differences between the two can help organizations make informed decisions on which platform best suits their needs.

1. **Data Source Compatibility**: Amundsen is primarily focused on metadata management and discovery, while Azure Data Factory is designed for data integration and orchestration. Amundsen supports a wide range of data sources for metadata ingestion, such as databases, data lakes, and data warehouses, whereas Azure Data Factory is specifically optimized for Microsoft Azure data sources.

2. **User Interface**: Amundsen provides a user-friendly search interface for discovering data assets and lineage relationships, making it easier for users to explore and understand the data landscape. On the other hand, Azure Data Factory offers a visual drag-and-drop interface for building data pipelines and workflows, catering to data engineers and developers.

3. **Functionality**: Amundsen excels in providing data discovery features, allowing users to easily find relevant datasets and understand their lineage. In comparison, Azure Data Factory is more focused on data movement and transformation capabilities, enabling users to create complex data pipelines for ETL processes.

4. **Scalability**: Azure Data Factory is built on the cloud-native architecture of Microsoft Azure, offering scalability and flexibility to manage large-scale data workflows and processing requirements. While Amundsen can scale to accommodate metadata for diverse data sources, it may require additional configuration for handling large volumes of data and user queries.

5. **Integration**: Azure Data Factory seamlessly integrates with various Azure services such as Azure Data Lake Storage, Azure Synapse Analytics, and Azure SQL Database, providing a comprehensive data integration solution within the Azure ecosystem. In contrast, Amundsen can be integrated with external tools and platforms for enhanced metadata management capabilities, making it a versatile option for organizations with diverse data sources and systems.

In Summary, understanding the key differences between Amundsen and Azure Data Factory is crucial for organizations seeking to streamline their data management processes and maximize the value of their data assets.

🔥 Trending in Databases on StackShare

PostgreSQL Databases

A powerful, open source object-relational database system

Try it View Docs Alternatives

Try

431

83.9k

Advice on Azure Data Factory, Amundsen

Vamshi

Data Engineer at Tata Consultancy Services

May 29, 2020

Needs adviceon

PySpark

Azure Data Factory

Databricks

I have to collect different data from multiple sources and store them in a single cloud location. Then perform cleaning and transforming using PySpark, and push the end results to other applications like reporting tools, etc. What would be the best solution? I can only think of Azure Data Factory + Databricks. Are there any alternatives to #AWS services + Databricks?

269k views269k

Comments

Detailed Comparison

Azure Data Factory	Amundsen
It is a service designed to allow developers to integrate disparate data sources. It is a platform somewhat like SSIS in the cloud to manage the data you have both on-prem and in the cloud.	It is a metadata driven application for improving the productivity of data analysts, data scientists and engineers when interacting with data.
Real-Time Integration; Parallel Processing; Data Chunker; Data Masking; Proactive Monitoring; Big Data Processing	Datasets (Tables) schema and usage frequency/popularity; Users bookmark, owner, frequent user; Dashboard popularity, lineage to datasets
Statistics
GitHub Stars 516	GitHub Stars -
GitHub Forks 610	GitHub Forks -
Stacks 253	Stacks 17
Followers 484	Followers 42
Votes 0	Votes 0
Integrations
Octotree Java .NET	Google BigQuery Snowflake AWS Glue Superset Apache Hive

What are some alternatives to Azure Data Factory, Amundsen?

Apache Spark

Spark is a fast and general processing engine compatible with Hadoop data. It can run in Hadoop clusters through YARN or Spark's standalone mode, and it can process data in HDFS, HBase, Cassandra, Hive, and any Hadoop InputFormat. It is designed to perform both batch processing (similar to MapReduce) and new workloads like streaming, interactive queries, and machine learning.

Presto

Distributed SQL Query Engine for Big Data

Amazon Athena

Amazon Athena is an interactive query service that makes it easy to analyze data in Amazon S3 using standard SQL. Athena is serverless, so there is no infrastructure to manage, and you pay only for the queries that you run.

Apache Flink

Apache Flink is an open source system for fast and versatile data analytics in clusters. Flink supports batch and streaming analytics, in one system. Analytical programs can be written in concise and elegant APIs in Java and Scala.

lakeFS

It is an open-source data version control system for data lakes. It provides a “Git for data” platform enabling you to implement best practices from software engineering on your data lake, including branching and merging, CI/CD, and production-like dev/test environments.

Druid

Druid is a distributed, column-oriented, real-time analytics data store that is commonly used to power exploratory dashboards in multi-tenant environments. Druid excels as a data warehousing solution for fast aggregate queries on petabyte sized data sets. Druid supports a variety of flexible filters, exact calculations, approximate algorithms, and other useful calculations.

Apache Kylin

Apache Kylin™ is an open source Distributed Analytics Engine designed to provide SQL interface and multi-dimensional analysis (OLAP) on Hadoop/Spark supporting extremely large datasets, originally contributed from eBay Inc.

Apache Camel

An open source Java framework that focuses on making integration easier and more accessible to developers.

Splunk

It provides the leading platform for Operational Intelligence. Customers use it to search, monitor, analyze and visualize machine data.

Apache Impala

Impala is a modern, open source, MPP SQL query engine for Apache Hadoop. Impala is shipped by Cloudera, MapR, and Amazon. With Impala, you can query data, whether stored in HDFS or Apache HBase – including SELECT, JOIN, and aggregate functions – in real time.