Airflow vs Pachyderm: What are the differences?
Airflow: A platform to programmaticaly author, schedule and monitor data pipelines, by Airbnb. Use Airflow to author workflows as directed acyclic graphs (DAGs) of tasks. The Airflow scheduler executes your tasks on an array of workers while following the specified dependencies. Rich command lines utilities makes performing complex surgeries on DAGs a snap. The rich user interface makes it easy to visualize pipelines running in production, monitor progress and troubleshoot issues when needed; Pachyderm: MapReduce without Hadoop. Analyze massive datasets with Docker. Pachyderm is an open source MapReduce engine that uses Docker containers for distributed computations.
Airflow and Pachyderm are primarily classified as "Workflow Manager" and "Big Data" tools respectively.
Some of the features offered by Airflow are:
- Dynamic: Airflow pipelines are configuration as code (Python), allowing for dynamic pipeline generation. This allows for writting code that instantiate pipelines dynamically.
- Extensible: Easily define your own operators, executors and extend the library so that it fits the level of abstraction that suits your environment.
- Elegant: Airflow pipelines are lean and explicit. Parameterizing your scripts is built in the core of Airflow using powerful Jinja templating engine.
On the other hand, Pachyderm provides the following key features:
- Git-like File System
- Dockerized MapReduce
- Microservice Architecture
Airflow and Pachyderm are both open source tools. Airflow with 13K GitHub stars and 4.72K forks on GitHub appears to be more popular than Pachyderm with 3.81K GitHub stars and 369 GitHub forks.