Alternatives to Airflow logo

Alternatives to Airflow

Luigi, Apache NiFi, Jenkins, AWS Step Functions, and Apache Beam are the most popular alternatives and competitors to Airflow.
396
369
+ 1
20

What is Airflow and what are its top alternatives?

Use Airflow to author workflows as directed acyclic graphs (DAGs) of tasks. The Airflow scheduler executes your tasks on an array of workers while following the specified dependencies. Rich command lines utilities makes performing complex surgeries on DAGs a snap. The rich user interface makes it easy to visualize pipelines running in production, monitor progress and troubleshoot issues when needed.
Airflow is a tool in the Workflow Manager category of a tech stack.
Airflow is an open source tool with 15.4K GitHub stars and 5.8K GitHub forks. Here’s a link to Airflow's open source repository on GitHub

Airflow alternatives & related posts

Luigi logo

Luigi

31
29
0
31
29
+ 1
0
ETL and data flow management library
    Be the first to leave a pro
    Luigi logo
    Luigi
    VS
    Airflow logo
    Airflow
    Apache NiFi logo

    Apache NiFi

    53
    55
    5
    53
    55
    + 1
    5
    A reliable system to process and distribute data
    Apache NiFi logo
    Apache NiFi
    VS
    Airflow logo
    Airflow

    related Jenkins posts

    Thierry Schellenbach
    Thierry Schellenbach
    CEO at Stream · | 23 upvotes · 83.5K views
    atStreamStream
    Go
    Go
    Travis CI
    Travis CI
    GitHub
    GitHub
    Jenkins
    Jenkins
    #ContinuousIntegration
    #CodeCollaborationVersionControl

    Releasing new versions of our services is done by Travis CI. Travis first runs our test suite. Once it passes, it publishes a new release binary to GitHub.

    Common tasks such as installing dependencies for the Go project, or building a binary are automated using plain old Makefiles. (We know, crazy old school, right?) Our binaries are compressed using UPX.

    Travis has come a long way over the past years. I used to prefer Jenkins in some cases since it was easier to debug broken builds. With the addition of the aptly named “debug build” button, Travis is now the clear winner. It’s easy to use and free for open source, with no need to maintain anything.

    #ContinuousIntegration #CodeCollaborationVersionControl

    See more
    Tymoteusz Paul
    Tymoteusz Paul
    Devops guy at X20X Development LTD · | 17 upvotes · 628.3K views
    Vagrant
    Vagrant
    VirtualBox
    VirtualBox
    Ansible
    Ansible
    Elasticsearch
    Elasticsearch
    Kibana
    Kibana
    Logstash
    Logstash
    TeamCity
    TeamCity
    Jenkins
    Jenkins
    Slack
    Slack
    Apache Maven
    Apache Maven
    Vault
    Vault
    Git
    Git
    Docker
    Docker
    CircleCI
    CircleCI
    LXC
    LXC
    Amazon EC2
    Amazon EC2

    Often enough I have to explain my way of going about setting up a CI/CD pipeline with multiple deployment platforms. Since I am a bit tired of yapping the same every single time, I've decided to write it up and share with the world this way, and send people to read it instead ;). I will explain it on "live-example" of how the Rome got built, basing that current methodology exists only of readme.md and wishes of good luck (as it usually is ;)).

    It always starts with an app, whatever it may be and reading the readmes available while Vagrant and VirtualBox is installing and updating. Following that is the first hurdle to go over - convert all the instruction/scripts into Ansible playbook(s), and only stopping when doing a clear vagrant up or vagrant reload we will have a fully working environment. As our Vagrant environment is now functional, it's time to break it! This is the moment to look for how things can be done better (too rigid/too lose versioning? Sloppy environment setup?) and replace them with the right way to do stuff, one that won't bite us in the backside. This is the point, and the best opportunity, to upcycle the existing way of doing dev environment to produce a proper, production-grade product.

    I should probably digress here for a moment and explain why. I firmly believe that the way you deploy production is the same way you should deploy develop, shy of few debugging-friendly setting. This way you avoid the discrepancy between how production work vs how development works, which almost always causes major pains in the back of the neck, and with use of proper tools should mean no more work for the developers. That's why we start with Vagrant as developer boxes should be as easy as vagrant up, but the meat of our product lies in Ansible which will do meat of the work and can be applied to almost anything: AWS, bare metal, docker, LXC, in open net, behind vpn - you name it.

    We must also give proper consideration to monitoring and logging hoovering at this point. My generic answer here is to grab Elasticsearch, Kibana, and Logstash. While for different use cases there may be better solutions, this one is well battle-tested, performs reasonably and is very easy to scale both vertically (within some limits) and horizontally. Logstash rules are easy to write and are well supported in maintenance through Ansible, which as I've mentioned earlier, are at the very core of things, and creating triggers/reports and alerts based on Elastic and Kibana is generally a breeze, including some quite complex aggregations.

    If we are happy with the state of the Ansible it's time to move on and put all those roles and playbooks to work. Namely, we need something to manage our CI/CD pipelines. For me, the choice is obvious: TeamCity. It's modern, robust and unlike most of the light-weight alternatives, it's transparent. What I mean by that is that it doesn't tell you how to do things, doesn't limit your ways to deploy, or test, or package for that matter. Instead, it provides a developer-friendly and rich playground for your pipelines. You can do most the same with Jenkins, but it has a quite dated look and feel to it, while also missing some key functionality that must be brought in via plugins (like quality REST API which comes built-in with TeamCity). It also comes with all the common-handy plugins like Slack or Apache Maven integration.

    The exact flow between CI and CD varies too greatly from one application to another to describe, so I will outline a few rules that guide me in it: 1. Make build steps as small as possible. This way when something breaks, we know exactly where, without needing to dig and root around. 2. All security credentials besides development environment must be sources from individual Vault instances. Keys to those containers should exist only on the CI/CD box and accessible by a few people (the less the better). This is pretty self-explanatory, as anything besides dev may contain sensitive data and, at times, be public-facing. Because of that appropriate security must be present. TeamCity shines in this department with excellent secrets-management. 3. Every part of the build chain shall consume and produce artifacts. If it creates nothing, it likely shouldn't be its own build. This way if any issue shows up with any environment or version, all developer has to do it is grab appropriate artifacts to reproduce the issue locally. 4. Deployment builds should be directly tied to specific Git branches/tags. This enables much easier tracking of what caused an issue, including automated identifying and tagging the author (nothing like automated regression testing!).

    Speaking of deployments, I generally try to keep it simple but also with a close eye on the wallet. Because of that, I am more than happy with AWS or another cloud provider, but also constantly peeking at the loads and do we get the value of what we are paying for. Often enough the pattern of use is not constantly erratic, but rather has a firm baseline which could be migrated away from the cloud and into bare metal boxes. That is another part where this approach strongly triumphs over the common Docker and CircleCI setup, where you are very much tied in to use cloud providers and getting out is expensive. Here to embrace bare-metal hosting all you need is a help of some container-based self-hosting software, my personal preference is with Proxmox and LXC. Following that all you must write are ansible scripts to manage hardware of Proxmox, similar way as you do for Amazon EC2 (ansible supports both greatly) and you are good to go. One does not exclude another, quite the opposite, as they can live in great synergy and cut your costs dramatically (the heavier your base load, the bigger the savings) while providing production-grade resiliency.

    See more
    AWS Step Functions logo

    AWS Step Functions

    71
    56
    0
    71
    56
    + 1
    0
    Build Distributed Applications Using Visual Workflows
      Be the first to leave a pro
      AWS Step Functions logo
      AWS Step Functions
      VS
      Airflow logo
      Airflow
      Apache Beam logo

      Apache Beam

      27
      26
      0
      27
      26
      + 1
      0
      A unified programming model
        Be the first to leave a pro
        Apache Beam logo
        Apache Beam
        VS
        Airflow logo
        Airflow
        Camunda logo

        Camunda

        24
        19
        0
        24
        19
        + 1
        0
        A Workflow and Decision Automation Platform
          Be the first to leave a pro
          Camunda logo
          Camunda
          VS
          Airflow logo
          Airflow
          Apache Oozie logo

          Apache Oozie

          19
          8
          0
          19
          8
          + 1
          0
          An open-source workflow scheduling system
            Be the first to leave a pro
            Apache Oozie logo
            Apache Oozie
            VS
            Airflow logo
            Airflow
            Workflowy logo

            Workflowy

            14
            7
            0
            14
            7
            + 1
            0
            A web-based app to stay organized
              Be the first to leave a pro
              Workflowy logo
              Workflowy
              VS
              Airflow logo
              Airflow
              Process Street logo

              Process Street

              13
              8
              0
              13
              8
              + 1
              0
              Simple Process and Workflow Management
                Be the first to leave a pro
                Process Street logo
                Process Street
                VS
                Airflow logo
                Airflow
                OTRS logo

                OTRS

                7
                3
                0
                7
                3
                + 1
                0
                A customizable support desk software that manages workflows and structures communication
                  Be the first to leave a pro
                  OTRS logo
                  OTRS
                  VS
                  Airflow logo
                  Airflow
                  Zenaton logo

                  Zenaton

                  7
                  5
                  12
                  7
                  5
                  + 1
                  12
                  A Workflow Builder for Developers. Build event-driven processes in days instead of months.
                  Zenaton logo
                  Zenaton
                  VS
                  Airflow logo
                  Airflow
                  Digdag logo

                  Digdag

                  3
                  3
                  0
                  3
                  3
                  + 1
                  0
                  An open source and multi-cloud workflow engine
                    Be the first to leave a pro
                    Digdag logo
                    Digdag
                    VS
                    Airflow logo
                    Airflow
                    Workfront logo

                    Workfront

                    3
                    1
                    0
                    3
                    1
                    + 1
                    0
                    A platform for enterprise work management
                      Be the first to leave a pro
                      Workfront logo
                      Workfront
                      VS
                      Airflow logo
                      Airflow
                      Kissflow logo

                      Kissflow

                      3
                      0
                      0
                      3
                      0
                      + 1
                      0
                      Digital Workplace is designed for minimal disruption of work
                        Be the first to leave a pro
                        Kissflow logo
                        Kissflow
                        VS
                        Airflow logo
                        Airflow
                        Tallyfy logo

                        Tallyfy

                        1
                        2
                        0
                        1
                        2
                        + 1
                        0
                        Workflow software that turns your daily tasks and approvals into automated, repeatable processes
                          Be the first to leave a pro
                          Tallyfy logo
                          Tallyfy
                          VS
                          Airflow logo
                          Airflow
                          Freeter logo

                          Freeter

                          1
                          0
                          0
                          1
                          0
                          + 1
                          0
                          Gather all the things you need for work in one place and have quick access to them
                            Be the first to leave a pro
                            Freeter logo
                            Freeter
                            VS
                            Airflow logo
                            Airflow