Azure Data Factory vs Trifacta

Overview

Trifacta

Stacks19

Followers41

Votes0

Azure Data Factory

Stacks253

Followers484

Votes0

GitHub Stars516

Forks610

Azure Data Factory vs Trifacta: What are the differences?

Azure Data Factory and Trifacta are both data integration platforms that offer a wide range of features for managing and transforming data. However, there are some key differences between the two platforms that differentiate them in terms of functionality and capabilities.

Data Integration vs Data Transformation: Azure Data Factory primarily focuses on data integration, allowing users to develop, schedule, and orchestrate data integration workflows. It provides capabilities for accessing various data sources, transforming data, and moving it to target destinations. On the other hand, Trifacta specializes in data transformation, providing a user-friendly interface for visually designing data transformation processes and applying advanced data wrangling techniques.
Cloud-Based vs On-Premises: Azure Data Factory is a cloud-based platform that fully operates in the Microsoft Azure environment. It leverages Azure services for data storage, processing, and computation, offering seamless integration with other Azure services. In contrast, Trifacta can be deployed on-premises or in a cloud environment, giving users the flexibility to choose where their data transformation processes are executed.
Code-Free vs Code-Driven: Trifacta emphasizes a code-free approach to data transformation, allowing users to visually design data transformation workflows through its intuitive interface. It offers smart suggestions and automates repetitive tasks, making it easier for non-technical users to work with data. Azure Data Factory, on the other hand, relies on code-driven configurations using JSON-based control flow and data movement definitions, requiring users to have programming knowledge to create and manage workflows.
Data Volume and Scalability: Azure Data Factory is specifically designed to handle large volumes of data and scales seamlessly based on the user's requirements. It can efficiently process and move data across various Azure services, making it suitable for big data scenarios. Trifacta, although capable of handling large datasets, may experience scalability limitations compared to Azure Data Factory when dealing with extremely large volumes of data.
Advanced Analytics and Machine Learning: Azure Data Factory integrates tightly with Azure Machine Learning and Azure Data Lake Analytics, allowing users to incorporate advanced analytics and machine learning capabilities into their data integration workflows. Trifacta, on the other hand, offers limited support for advanced analytics and machine learning, focusing primarily on data profiling, cleansing, and normalizing.
Ecosystem and Integration: Azure Data Factory provides extensive integration with other Azure services, such as Azure Blob Storage, Azure SQL Database, Azure Data Lake Store, and more. It also offers connectors for popular third-party services like Salesforce, Dropbox, and Google Analytics. Trifacta, while supporting various data sources and formats, may have fewer out-of-the-box integrations and connectors compared to Azure Data Factory.

In summary, Azure Data Factory is a cloud-based data integration platform that focuses on data integration workflows, scalability, and advanced analytics integration. Trifacta, on the other hand, is a data transformation platform that offers a code-free interface, flexibility in deployment, and powerful data wrangling capabilities.

Share your Stack

Help developers discover the tools you use. Get visibility for your team's tech choices and contribute to the community's knowledge.

View Docs

CLI (Node.js)

Manual

Advice on Trifacta, Azure Data Factory

Vamshi

Data Engineer at Tata Consultancy Services

May 29, 2020

Needs adviceon

PySpark

Azure Data Factory

Databricks

I have to collect different data from multiple sources and store them in a single cloud location. Then perform cleaning and transforming using PySpark, and push the end results to other applications like reporting tools, etc. What would be the best solution? I can only think of Azure Data Factory + Databricks. Are there any alternatives to #AWS services + Databricks?

269k views269k

Comments

Sarah

Jun 25, 2020

Needs adviceon

OpenRefine

I'm looking for an open-source/free/cheap tool to clean messy data coming from various travel APIs. We use many different APIs and save the info in our DB. However, many duplicates cannot be easily recognized as such.

We would either write an algorithm or use smart technology/tools with ML to help with product management.

While there are many things to be considered, this is one feature that it should have:

"To avoid confusion, we need to merge the suppliers & products accordingly. Products and suppliers must be able to be merged and assigned separately.

Reason: It may happen that one supplier offers different products. E.g., 1 tour operator offers 3 products via 1 API, but only 1 product with 3 (or a different amount of) variations via a different API. Also, the commission may differ for products, which we need to consider. Very often, products that are live (are bookable in real-time) on via 1 API, but are not live on the other. E.g., Supplier product 1 & 2 of API1 are live, product 3 not. For the same supplier, API2 provides live availability for products 1, 2, and 3.

Summing up, when merging the suppliers (tour operators) we need to consider:

Are the products the same for all APIs?
Which booking system API gives a better commission? Note: Some APIs charge us 1-5% depending on the monthly sale, which needs to be considered
Which booking system provides live availability
Is it the same supplier, or is the name only similar?

Most of the time, the supplier names differ even if they are the same (e.g., API1 often names them XX Pty Ltd, while API2 leaves "Pty Ltd" out). Additionally, the product title, description, etc. differ.

We need to write logic and create an algorithm to find the duplicates & to merge, assign, or (de)activate the respective supplier or product. My previous developer started a module to merge the suppliers, which does not seem to work correctly. Also, it is way too time taking considering the high amount of products that we have.

I would recommend merging, assigning etc. products and suppliers only if our algorithm says it's 90- 100% the matching supplier/product. Otherwise, admins need to be able to check & modify this. E.g. everything with a lower possibility of matching will be matched automatically, but can be undone or modified.

The next time the cron job runs, this needs to be considered to avoid recreating duplicates & creating a mess."

I am not sure in what way OpenRefine can help to achieve this and what ML tool can be connected to learn from the decisions the product management team makes. Maybe you have an idea of how other travel portals deal with messy data, duplicates, etc.?

I'm looking for the cheapest solution for a start-up, but it should do the work properly.

19.2k views19.2k

Comments

Detailed Comparison

Trifacta	Azure Data Factory
It is an Intelligent Platform that Interoperates with Your Data Investments. It sits between the data storage and processing environments and the visualization, statistical or machine learning tools used downstream	It is a service designed to allow developers to integrate disparate data sources. It is a platform somewhat like SSIS in the cloud to manage the data you have both on-prem and in the cloud.
Interactive Exploration; Automated visual representations of data based upon its content in the most compelling visual profile; Predictive Transformation; Intelligent Execution; Collaborative Data Governance.	Real-Time Integration; Parallel Processing; Data Chunker; Data Masking; Proactive Monitoring; Big Data Processing
Statistics
GitHub Stars -	GitHub Stars 516
GitHub Forks -	GitHub Forks 610
Stacks 19	Stacks 253
Followers 41	Followers 484
Votes 0	Votes 0
Integrations
Microsoft Azure Google Cloud Storage Snowflake AWS Data Pipeline Tableau	Octotree Java .NET

What are some alternatives to Trifacta, Azure Data Factory?

Apache Spark

Spark is a fast and general processing engine compatible with Hadoop data. It can run in Hadoop clusters through YARN or Spark's standalone mode, and it can process data in HDFS, HBase, Cassandra, Hive, and any Hadoop InputFormat. It is designed to perform both batch processing (similar to MapReduce) and new workloads like streaming, interactive queries, and machine learning.

Presto

Distributed SQL Query Engine for Big Data

Amazon Athena

Amazon Athena is an interactive query service that makes it easy to analyze data in Amazon S3 using standard SQL. Athena is serverless, so there is no infrastructure to manage, and you pay only for the queries that you run.

Apache Flink

Apache Flink is an open source system for fast and versatile data analytics in clusters. Flink supports batch and streaming analytics, in one system. Analytical programs can be written in concise and elegant APIs in Java and Scala.

lakeFS

It is an open-source data version control system for data lakes. It provides a “Git for data” platform enabling you to implement best practices from software engineering on your data lake, including branching and merging, CI/CD, and production-like dev/test environments.

Druid

Druid is a distributed, column-oriented, real-time analytics data store that is commonly used to power exploratory dashboards in multi-tenant environments. Druid excels as a data warehousing solution for fast aggregate queries on petabyte sized data sets. Druid supports a variety of flexible filters, exact calculations, approximate algorithms, and other useful calculations.

Apache Kylin

Apache Kylin™ is an open source Distributed Analytics Engine designed to provide SQL interface and multi-dimensional analysis (OLAP) on Hadoop/Spark supporting extremely large datasets, originally contributed from eBay Inc.

Apache Camel

An open source Java framework that focuses on making integration easier and more accessible to developers.

Splunk

It provides the leading platform for Operational Intelligence. Customers use it to search, monitor, analyze and visualize machine data.

Apache Impala

Impala is a modern, open source, MPP SQL query engine for Apache Hadoop. Impala is shipped by Cloudera, MapR, and Amazon. With Impala, you can query data, whether stored in HDFS or Apache HBase – including SELECT, JOIN, and aggregate functions – in real time.

Related Comparisons

Azure Data Factory vs Trifacta: What are the differences?

Data Integration vs Data Transformation: Azure Data Factory primarily focuses on data integration, allowing users to develop, schedule, and orchestrate data integration workflows. It provides capabilities for accessing various data sources, transforming data, and moving it to target destinations. On the other hand, Trifacta specializes in data transformation, providing a user-friendly interface for visually designing data transformation processes and applying advanced data wrangling techniques.
Cloud-Based vs On-Premises: Azure Data Factory is a cloud-based platform that fully operates in the Microsoft Azure environment. It leverages Azure services for data storage, processing, and computation, offering seamless integration with other Azure services. In contrast, Trifacta can be deployed on-premises or in a cloud environment, giving users the flexibility to choose where their data transformation processes are executed.
Code-Free vs Code-Driven: Trifacta emphasizes a code-free approach to data transformation, allowing users to visually design data transformation workflows through its intuitive interface. It offers smart suggestions and automates repetitive tasks, making it easier for non-technical users to work with data. Azure Data Factory, on the other hand, relies on code-driven configurations using JSON-based control flow and data movement definitions, requiring users to have programming knowledge to create and manage workflows.
Data Volume and Scalability: Azure Data Factory is specifically designed to handle large volumes of data and scales seamlessly based on the user's requirements. It can efficiently process and move data across various Azure services, making it suitable for big data scenarios. Trifacta, although capable of handling large datasets, may experience scalability limitations compared to Azure Data Factory when dealing with extremely large volumes of data.
Advanced Analytics and Machine Learning: Azure Data Factory integrates tightly with Azure Machine Learning and Azure Data Lake Analytics, allowing users to incorporate advanced analytics and machine learning capabilities into their data integration workflows. Trifacta, on the other hand, offers limited support for advanced analytics and machine learning, focusing primarily on data profiling, cleansing, and normalizing.
Ecosystem and Integration: Azure Data Factory provides extensive integration with other Azure services, such as Azure Blob Storage, Azure SQL Database, Azure Data Lake Store, and more. It also offers connectors for popular third-party services like Salesforce, Dropbox, and Google Analytics. Trifacta, while supporting various data sources and formats, may have fewer out-of-the-box integrations and connectors compared to Azure Data Factory.

Azure Data Factory vs Trifacta

Overview