Needs advice
on
JenkinsJenkins
and
AirflowAirflow

I am looking for an open-source scheduler tool with cross-functional application dependencies. Some of the tasks I am looking to schedule are as follows:

  1. Trigger Matillion ETL loads
  2. Trigger Attunity Replication tasks that have downstream ETL loads
  3. Trigger Golden gate Replication Tasks
  4. Shell scripts, wrappers, file watchers
  5. Event-driven schedules

I have used Airflow in the past, and I know we need to create DAGs for each pipeline. I am not familiar with Jenkins, but I know it works with configuration without much underlying code. I want to evaluate both and appreciate any advise

READ LESS
5 upvotes·197.5K views
Replies (2)
Recommends
Airflow

Hi Teja, Jenkins is more a CI/CD tool for triggering build/test and other CD tasks, from what you're describing you may be able to get along with #Jenkins and add lot's of plugins and create pipelines. But, eventually you're going to need to know #Groovy language to orchestrate all those tasks which is going to be similar to what you do with #Airflow, So, IMHO , Airflow is more for production scheduled tasks and Jenkins is more for CI/CD non-production tasks.

READ MORE
6 upvotes·9.1K views
Recommends
Airflow

For 1 - 3, Airflow is a better solution.

Cron and OSQuery may be better solutions for 4 and 5, depending on what you're actually trying to do.

In either case, Jenkins is more trouble than it's worth for these types of workloads.

READ MORE
2 upvotes·9K views
Avatar of Teja Mantrala