Need advice about which tool to choose?Ask the StackShare community!
Pig vs Talend: What are the differences?
Developers describe Pig as "Platform for analyzing large data sets". Pig is a dataflow programming environment for processing very large files. Pig's language is called Pig Latin. A Pig Latin program consists of a directed acyclic graph where each node represents an operation that transforms data Operations are of two flavors: (1) relational-algebra style operations such as join, filter, project; (2) functional-programming style operators such as map, reduce. . On the other hand, Talend is detailed as "A single, unified suite for all integration needs". It is an open source software integration platform helps you in effortlessly turning data into business insights. It uses native code generation that lets you run your data pipelines seamlessly across all cloud providers and get optimized performance on all platforms.
Pig and Talend can be primarily classified as "Big Data" tools.
Pig is an open source tool with 583 GitHub stars and 449 GitHub forks. Here's a link to Pig's open source repository on GitHub.
I am trying to build a data lake by pulling data from multiple data sources ( custom-built tools, excel files, CSV files, etc) and use the data lake to generate dashboards.
My question is which is the best tool to do the following:
- Create pipelines to ingest the data from multiple sources into the data lake
- Help me in aggregating and filtering data available in the data lake.
- Create new reports by combining different data elements from the data lake.
I need to use only open-source tools for this activity.
I appreciate your valuable inputs and suggestions. Thanks in Advance.
DremioHi Karunakaran. I obviously have an interest here, as I work for the company, but the problem you are describing is one that Zetaris can solve. Talend is a good ETL product, and Dremio is a good data virtualization product, but the problem you are describing best fits a tool that can combine the five styles of data integration (bulk/batch data movement, data replication/data synchronization, message-oriented movement of data, data virtualization, and stream data integration). I may be wrong, but Zetaris is, to the best of my knowledge, the only product in the world that can do this. Zetaris is not a dashboarding tool - you would need to combine us with Tableau or Qlik or PowerBI (or whatever) - but Zetaris can consolidate data from any source and any location (structured, unstructured, on-prem or in the cloud) in real time to allow clients a consolidated view of whatever they want whenever they want it. Please take a look at www.zetaris.com for more information. I don't want to do a "hard sell", here, so I'll say no more! Warmest regards, Rod Beecham.
Pros of Pig
- Finer-grained control on parallelization2
- Proven at Petabyte scale1
- Open-source1
- Join optimizations for highly skewed data1
