Alternatives to Luigi logo

Alternatives to Luigi

Airflow, GitHub Actions, Apache Beam, Camunda, and Microsoft Power Automate are the most popular alternatives and competitors to Luigi.
61
151
+ 1
8

What is Luigi and what are its top alternatives?

It is a Python module that helps you build complex pipelines of batch jobs. It handles dependency resolution, workflow management, visualization etc. It also comes with Hadoop support built in.
Luigi is a tool in the Workflow Manager category of a tech stack.
Luigi is an open source tool with 15.2K GitHub stars and 2.3K GitHub forks. Here’s a link to Luigi's open source repository on GitHub

Top Alternatives to Luigi

  • Airflow

    Airflow

    Use Airflow to author workflows as directed acyclic graphs (DAGs) of tasks. The Airflow scheduler executes your tasks on an array of workers while following the specified dependencies. Rich command lines utilities makes performing complex surgeries on DAGs a snap. The rich user interface makes it easy to visualize pipelines running in production, monitor progress and troubleshoot issues when needed. ...

  • GitHub Actions

    GitHub Actions

    It makes it easy to automate all your software workflows, now with world-class CI/CD. Build, test, and deploy your code right from GitHub. Make code reviews, branch management, and issue triaging work the way you want. ...

  • Apache Beam

    Apache Beam

    It implements batch and streaming data processing jobs that run on any execution engine. It executes pipelines on multiple execution environments. ...

  • Camunda

    Camunda

    It is an open source platform for workflow and decision automation that brings business users and software developers together. ...

  • Microsoft Power Automate

    Microsoft Power Automate

    It is a service that helps you create automated workflows between your favorite apps and services to synchronize files, get notifications, collect data and more. ...

  • Workflowy

    Workflowy

    It is an organizational tool that makes life easier. It's a surprisingly powerful way to take notes, make lists, collaborate, brainstorm, plan and generally organize your brain. ...

  • Apache Oozie

    Apache Oozie

    It is a server-based workflow scheduling system to manage Hadoop jobs. Workflows in it are defined as a collection of control flow and action nodes in a directed acyclic graph. Control flow nodes define the beginning and the end of a workflow as well as a mechanism to control the workflow execution path. ...

  • K2

    K2

    Drive process excellence across your organization by connecting people, systems, and data to orchestrate how and when work gets done. ...

Luigi alternatives & related posts

Airflow logo

Airflow

1.2K
2K
116
A platform to programmaticaly author, schedule and monitor data pipelines, by Airbnb
1.2K
2K
+ 1
116
PROS OF AIRFLOW
  • 45
    Features
  • 14
    Task Dependency Management
  • 12
    Beautiful UI
  • 11
    Cluster of workers
  • 10
    Extensibility
  • 5
    Open source
  • 4
    Python
  • 4
    Complex workflows
  • 3
    K
  • 2
    Custom operators
  • 2
    Good api
  • 2
    Dashboard
  • 2
    Apache project
CONS OF AIRFLOW
  • 2
    Open source - provides minimum or no support
  • 1
    Logical separation of DAGs is not straight forward
  • 1
    Running it on kubernetes cluster relatively complex
  • 1
    Observability is not great when the DAGs exceed 250

related Airflow posts

Shared insights
on
JenkinsJenkinsAirflowAirflow

I am looking for an open-source scheduler tool with cross-functional application dependencies. Some of the tasks I am looking to schedule are as follows:

  1. Trigger Matillion ETL loads
  2. Trigger Attunity Replication tasks that have downstream ETL loads
  3. Trigger Golden gate Replication Tasks
  4. Shell scripts, wrappers, file watchers
  5. Event-driven schedules

I have used Airflow in the past, and I know we need to create DAGs for each pipeline. I am not familiar with Jenkins, but I know it works with configuration without much underlying code. I want to evaluate both and appreciate any advise

See more
Shared insights
on
AWS Step FunctionsAWS Step FunctionsAirflowAirflow

I am working on a project that grabs a set of input data from AWS S3, pre-processes and divvies it up, spins up 10K batch containers to process the divvied data in parallel on AWS Batch, post-aggregates the data, and pushes it to S3.

I already have software patterns from other projects for Airflow + Batch but have not dealt with the scaling factors of 10k parallel tasks. Airflow is nice since I can look at which tasks failed and retry a task after debugging. But dealing with that many tasks on one Airflow EC2 instance seems like a barrier. Another option would be to have one task that kicks off the 10k containers and monitors it from there.

I have no experience with AWS Step Functions but have heard it's AWS's Airflow. There looks to be plenty of patterns online for Step Functions + Batch. Do Step Functions seem like a good path to check out for my use case? Do you get the same insights on failing jobs / ability to retry tasks as you do with Airflow?

See more
GitHub Actions logo

GitHub Actions

1K
411
18
Automate your workflow from idea to production
1K
411
+ 1
18
PROS OF GITHUB ACTIONS
  • 4
    Free
  • 3
    Integration with GitHub
  • 3
    Easy to duplicate a workflow
  • 2
    Read actions in Marketplace
  • 2
    Ready actions in Marketplace
  • 1
    Active Development Roadmap
  • 1
    Fast
  • 1
    Configs stored in .github
  • 1
    Docker Support
CONS OF GITHUB ACTIONS
  • 5
    Lacking [skip ci]
  • 4
    Lacking allow failure
  • 2
    Lacking job specific badges
  • 1
    No ssh login to servers
  • 1
    No Deployment Projects
  • 1
    No manual launch

related GitHub Actions posts

Somnath Mahale
Engineering Leader at Altimetrik Corp. · | 8 upvotes · 97.5K views

I am in the process of evaluating CircleCI, Drone.io, and Github Actions to cover my #CI/ CD needs. I would appreciate your advice on comparative study w.r.t. attributes like language-Inclusive support, code-base integration, performance, cost, maintenance, support, ease of use, ability to deal with big projects, etc. based on actual industry experience.

Thanks in advance!

See more
Omkar Kulkarni
DevOps Engineer at LTI · | 3 upvotes · 18.6K views
Shared insights
on
GitLabGitLabGitHub ActionsGitHub Actions

Hello Everyone, Can some please help me to understand the difference between GitHub Actions And GitLab I have been trying to understand them, but still did not get how exactly they are different.

See more
Apache Beam logo

Apache Beam

148
283
14
A unified programming model
148
283
+ 1
14
PROS OF APACHE BEAM
  • 5
    Open-source
  • 5
    Cross-platform
  • 2
    Portable
  • 2
    Unified batch and stream processing
CONS OF APACHE BEAM
    Be the first to leave a con

    related Apache Beam posts

    I have to build a data processing application with an Apache Beam stack and Apache Flink runner on an Amazon EMR cluster. I saw some instability with the process and EMR clusters that keep going down. Here, the Apache Beam application gets inputs from Kafka and sends the accumulative data streams to another Kafka topic. Any advice on how to make the process more stable?

    See more
    Camunda logo

    Camunda

    118
    135
    0
    A Workflow and Decision Automation Platform
    118
    135
    + 1
    0
    PROS OF CAMUNDA
      Be the first to leave a pro
      CONS OF CAMUNDA
        Be the first to leave a con

        related Camunda posts

        Microsoft Power Automate logo

        Microsoft Power Automate

        55
        42
        0
        Streamline repetitive tasks and paperless processes
        55
        42
        + 1
        0
        PROS OF MICROSOFT POWER AUTOMATE
          Be the first to leave a pro
          CONS OF MICROSOFT POWER AUTOMATE
            Be the first to leave a con

            related Microsoft Power Automate posts

            David Rodriguez
            Analytics Implementation Manag at PSN Affiliates · | 5 upvotes · 7.2K views

            We are looking for a process automation tool for form approvals, purchase order approvals, payment agreements, etc. Senior leadership brought up Kissflow but we have the Microsoft suite and Microsoft Power Automate. Rather than paying for another tool, would we get the same functionality out of PowerAutomate that we would with Kissflow?

            See more
            Workflowy logo

            Workflowy

            44
            48
            0
            A web-based app to stay organized
            44
            48
            + 1
            0
            PROS OF WORKFLOWY
              Be the first to leave a pro
              CONS OF WORKFLOWY
                Be the first to leave a con

                related Workflowy posts

                Apache Oozie logo

                Apache Oozie

                39
                64
                0
                An open-source workflow scheduling system
                39
                64
                + 1
                0
                PROS OF APACHE OOZIE
                  Be the first to leave a pro
                  CONS OF APACHE OOZIE
                    Be the first to leave a con

                    related Apache Oozie posts

                    K2 logo

                    K2

                    36
                    0
                    0
                    Powerful low-code process automation
                    36
                    0
                    + 1
                    0
                    PROS OF K2
                      Be the first to leave a pro
                      CONS OF K2
                        Be the first to leave a con

                        related K2 posts