Alternatives to Amazon Managed Workflows for Apache Airflow logo

Alternatives to Amazon Managed Workflows for Apache Airflow

GitHub Actions, Airflow, Apache Beam, Camunda, and Microsoft Power Automate are the most popular alternatives and competitors to Amazon Managed Workflows for Apache Airflow.
14
10
+ 1
0

What is Amazon Managed Workflows for Apache Airflow and what are its top alternatives?

It is a managed orchestration service for Apache Airflow1 that makes it easier to set up and operate end-to-end data pipelines in the cloud at scale. With Managed Workflows, you can use Airflow and Python to create workflows without having to manage the underlying infrastructure for scalability, availability, and security. Managed Workflows automatically scales its workflow execution capacity to meet your needs, and is integrated with AWS security services to help provide you with fast and secure access to data.
Amazon Managed Workflows for Apache Airflow is a tool in the Workflow Manager category of a tech stack.

Top Alternatives to Amazon Managed Workflows for Apache Airflow

  • GitHub Actions
    GitHub Actions

    It makes it easy to automate all your software workflows, now with world-class CI/CD. Build, test, and deploy your code right from GitHub. Make code reviews, branch management, and issue triaging work the way you want. ...

  • Airflow
    Airflow

    Use Airflow to author workflows as directed acyclic graphs (DAGs) of tasks. The Airflow scheduler executes your tasks on an array of workers while following the specified dependencies. Rich command lines utilities makes performing complex surgeries on DAGs a snap. The rich user interface makes it easy to visualize pipelines running in production, monitor progress and troubleshoot issues when needed. ...

  • Apache Beam
    Apache Beam

    It implements batch and streaming data processing jobs that run on any execution engine. It executes pipelines on multiple execution environments. ...

  • Camunda
    Camunda

    It is an open source platform for workflow and decision automation that brings business users and software developers together. ...

  • Microsoft Power Automate
    Microsoft Power Automate

    It is a service that helps you create automated workflows between your favorite apps and services to synchronize files, get notifications, collect data and more. ...

  • Luigi
    Luigi

    It is a Python module that helps you build complex pipelines of batch jobs. It handles dependency resolution, workflow management, visualization etc. It also comes with Hadoop support built in. ...

  • Workflowy
    Workflowy

    It is an organizational tool that makes life easier. It's a surprisingly powerful way to take notes, make lists, collaborate, brainstorm, plan and generally organize your brain. ...

  • Apache Oozie
    Apache Oozie

    It is a server-based workflow scheduling system to manage Hadoop jobs. Workflows in it are defined as a collection of control flow and action nodes in a directed acyclic graph. Control flow nodes define the beginning and the end of a workflow as well as a mechanism to control the workflow execution path. ...

Amazon Managed Workflows for Apache Airflow alternatives & related posts

GitHub Actions logo

GitHub Actions

1.8K
571
20
Automate your workflow from idea to production
1.8K
571
+ 1
20
PROS OF GITHUB ACTIONS
  • 4
    Integration with GitHub
  • 4
    Free
  • 3
    Easy to duplicate a workflow
  • 2
    Docker Support
  • 2
    Read actions in Marketplace
  • 2
    Ready actions in Marketplace
  • 1
    Fast
  • 1
    Configs stored in .github
  • 1
    Active Development Roadmap
CONS OF GITHUB ACTIONS
  • 5
    Lacking [skip ci]
  • 4
    Lacking allow failure
  • 3
    Lacking job specific badges
  • 2
    No ssh login to servers
  • 1
    No Deployment Projects
  • 1
    No manual launch

related GitHub Actions posts

Somnath Mahale
Engineering Leader at Altimetrik Corp. · | 8 upvotes · 119.4K views

I am in the process of evaluating CircleCI, Drone.io, and Github Actions to cover my #CI/ CD needs. I would appreciate your advice on comparative study w.r.t. attributes like language-Inclusive support, code-base integration, performance, cost, maintenance, support, ease of use, ability to deal with big projects, etc. based on actual industry experience.

Thanks in advance!

See more
Omkar Kulkarni
DevOps Engineer at LTI · | 3 upvotes · 36.9K views
Shared insights
on
GitLabGitLabGitHub ActionsGitHub Actions

Hello Everyone, Can some please help me to understand the difference between GitHub Actions And GitLab I have been trying to understand them, but still did not get how exactly they are different.

See more
Airflow logo

Airflow

1.4K
2.3K
123
A platform to programmaticaly author, schedule and monitor data pipelines, by Airbnb
1.4K
2.3K
+ 1
123
PROS OF AIRFLOW
  • 49
    Features
  • 14
    Task Dependency Management
  • 12
    Beautiful UI
  • 12
    Cluster of workers
  • 10
    Extensibility
  • 5
    Open source
  • 5
    Python
  • 4
    Complex workflows
  • 3
    K
  • 3
    Good api
  • 2
    Custom operators
  • 2
    Apache project
  • 2
    Dashboard
CONS OF AIRFLOW
  • 2
    Running it on kubernetes cluster relatively complex
  • 2
    Open source - provides minimum or no support
  • 1
    Logical separation of DAGs is not straight forward
  • 1
    Observability is not great when the DAGs exceed 250

related Airflow posts

Shared insights
on
JenkinsJenkinsAirflowAirflow

I am looking for an open-source scheduler tool with cross-functional application dependencies. Some of the tasks I am looking to schedule are as follows:

  1. Trigger Matillion ETL loads
  2. Trigger Attunity Replication tasks that have downstream ETL loads
  3. Trigger Golden gate Replication Tasks
  4. Shell scripts, wrappers, file watchers
  5. Event-driven schedules

I have used Airflow in the past, and I know we need to create DAGs for each pipeline. I am not familiar with Jenkins, but I know it works with configuration without much underlying code. I want to evaluate both and appreciate any advise

See more
Shared insights
on
AWS Step FunctionsAWS Step FunctionsAirflowAirflow

I am working on a project that grabs a set of input data from AWS S3, pre-processes and divvies it up, spins up 10K batch containers to process the divvied data in parallel on AWS Batch, post-aggregates the data, and pushes it to S3.

I already have software patterns from other projects for Airflow + Batch but have not dealt with the scaling factors of 10k parallel tasks. Airflow is nice since I can look at which tasks failed and retry a task after debugging. But dealing with that many tasks on one Airflow EC2 instance seems like a barrier. Another option would be to have one task that kicks off the 10k containers and monitors it from there.

I have no experience with AWS Step Functions but have heard it's AWS's Airflow. There looks to be plenty of patterns online for Step Functions + Batch. Do Step Functions seem like a good path to check out for my use case? Do you get the same insights on failing jobs / ability to retry tasks as you do with Airflow?

See more
Apache Beam logo

Apache Beam

163
314
14
A unified programming model
163
314
+ 1
14
PROS OF APACHE BEAM
  • 5
    Open-source
  • 5
    Cross-platform
  • 2
    Portable
  • 2
    Unified batch and stream processing
CONS OF APACHE BEAM
    Be the first to leave a con

    related Apache Beam posts

    I have to build a data processing application with an Apache Beam stack and Apache Flink runner on an Amazon EMR cluster. I saw some instability with the process and EMR clusters that keep going down. Here, the Apache Beam application gets inputs from Kafka and sends the accumulative data streams to another Kafka topic. Any advice on how to make the process more stable?

    See more
    Camunda logo

    Camunda

    138
    157
    0
    A Workflow and Decision Automation Platform
    138
    157
    + 1
    0
    PROS OF CAMUNDA
      Be the first to leave a pro
      CONS OF CAMUNDA
        Be the first to leave a con

        related Camunda posts

        Microsoft Power Automate logo

        Microsoft Power Automate

        74
        57
        0
        Streamline repetitive tasks and paperless processes
        74
        57
        + 1
        0
        PROS OF MICROSOFT POWER AUTOMATE
          Be the first to leave a pro
          CONS OF MICROSOFT POWER AUTOMATE
            Be the first to leave a con

            related Microsoft Power Automate posts

            David Rodriguez
            Analytics Implementation Manag at PSN Affiliates · | 6 upvotes · 12.1K views

            We are looking for a process automation tool for form approvals, purchase order approvals, payment agreements, etc. Senior leadership brought up Kissflow but we have the Microsoft suite and Microsoft Power Automate. Rather than paying for another tool, would we get the same functionality out of PowerAutomate that we would with Kissflow?

            See more
            Luigi logo

            Luigi

            68
            173
            8
            ETL and data flow management library
            68
            173
            + 1
            8
            PROS OF LUIGI
            • 5
              Hadoop Support
            • 2
              Python
            • 1
              Open soure
            CONS OF LUIGI
              Be the first to leave a con

              related Luigi posts

              Workflowy logo

              Workflowy

              50
              55
              0
              A web-based app to stay organized
              50
              55
              + 1
              0
              PROS OF WORKFLOWY
                Be the first to leave a pro
                CONS OF WORKFLOWY
                  Be the first to leave a con

                  related Workflowy posts

                  Apache Oozie logo

                  Apache Oozie

                  40
                  69
                  0
                  An open-source workflow scheduling system
                  40
                  69
                  + 1
                  0
                  PROS OF APACHE OOZIE
                    Be the first to leave a pro
                    CONS OF APACHE OOZIE
                      Be the first to leave a con

                      related Apache Oozie posts