Alternatives to Celery logo

Alternatives to Celery

RabbitMQ, Kafka, Airflow, Cucumber, and Amazon SQS are the most popular alternatives and competitors to Celery.
1.3K
1.2K
+ 1
260

What is Celery and what are its top alternatives?

Celery is an asynchronous task queue/job queue based on distributed message passing. It is focused on real-time operation, but supports scheduling as well.
Celery is a tool in the Message Queue category of a tech stack.
Celery is an open source tool with 17.4K GitHub stars and 4K GitHub forks. Here’s a link to Celery's open source repository on GitHub

Top Alternatives to Celery

  • RabbitMQ

    RabbitMQ

    RabbitMQ gives your applications a common platform to send and receive messages, and your messages a safe place to live until received. ...

  • Kafka

    Kafka

    Kafka is a distributed, partitioned, replicated commit log service. It provides the functionality of a messaging system, but with a unique design. ...

  • Airflow

    Airflow

    Use Airflow to author workflows as directed acyclic graphs (DAGs) of tasks. The Airflow scheduler executes your tasks on an array of workers while following the specified dependencies. Rich command lines utilities makes performing complex surgeries on DAGs a snap. The rich user interface makes it easy to visualize pipelines running in production, monitor progress and troubleshoot issues when needed. ...

  • Cucumber

    Cucumber

    Cucumber is a tool that supports Behaviour-Driven Development (BDD) - a software development process that aims to enhance software quality and reduce maintenance costs. ...

  • Amazon SQS

    Amazon SQS

    Transmit any volume of data, at any level of throughput, without losing messages or requiring other services to be always available. With SQS, you can offload the administrative burden of operating and scaling a highly available messaging cluster, while paying a low price for only what you use. ...

  • ActiveMQ

    ActiveMQ

    Apache ActiveMQ is fast, supports many Cross Language Clients and Protocols, comes with easy to use Enterprise Integration Patterns and many advanced features while fully supporting JMS 1.1 and J2EE 1.4. Apache ActiveMQ is released under the Apache 2.0 License. ...

  • MQTT

    MQTT

    It was designed as an extremely lightweight publish/subscribe messaging transport. It is useful for connections with remote locations where a small code footprint is required and/or network bandwidth is at a premium. ...

  • Apache NiFi

    Apache NiFi

    An easy to use, powerful, and reliable system to process and distribute data. It supports powerful and scalable directed graphs of data routing, transformation, and system mediation logic. ...

Celery alternatives & related posts

RabbitMQ logo

RabbitMQ

13.3K
11.4K
511
Open source multiprotocol messaging broker
13.3K
11.4K
+ 1
511
PROS OF RABBITMQ
  • 226
    It's fast and it works with good metrics/monitoring
  • 79
    Ease of configuration
  • 57
    I like the admin interface
  • 49
    Easy to set-up and start with
  • 20
    Durable
  • 18
    Intuitive work through python
  • 18
    Standard protocols
  • 10
    Written primarily in Erlang
  • 7
    Simply superb
  • 6
    Completeness of messaging patterns
  • 3
    Reliable
  • 3
    Scales to 1 million messages per second
  • 2
    Distributed
  • 2
    Supports AMQP
  • 2
    Better than most traditional queue based message broker
  • 1
    High performance
  • 1
    Reliability
  • 1
    Clusterable
  • 1
    Inubit Integration
  • 1
    Clear documentation with different scripting language
  • 1
    Great ui
  • 1
    Runs on Open Telecom Platform
  • 1
    Better routing system
  • 1
    Supports MQTT
CONS OF RABBITMQ
  • 9
    Too complicated cluster/HA config and management
  • 6
    Needs Erlang runtime. Need ops good with Erlang runtime
  • 5
    Configuration must be done first, not by your code
  • 4
    Slow

related RabbitMQ posts

James Cunningham
Operations Engineer at Sentry · | 18 upvotes · 1.2M views
Shared insights
on
Celery
RabbitMQ
at

As Sentry runs throughout the day, there are about 50 different offline tasks that we execute—anything from “process this event, pretty please” to “send all of these cool people some emails.” There are some that we execute once a day and some that execute thousands per second.

Managing this variety requires a reliably high-throughput message-passing technology. We use Celery's RabbitMQ implementation, and we stumbled upon a great feature called Federation that allows us to partition our task queue across any number of RabbitMQ servers and gives us the confidence that, if any single server gets backlogged, others will pitch in and distribute some of the backlogged tasks to their consumers.

#MessageQueue

See more
Tim Abbott
Shared insights
on
RabbitMQ
Python
Redis
at

We've been using RabbitMQ as Zulip's queuing system since we needed a queuing system. What I like about it is that it scales really well and has good libraries for a wide range of platforms, including our own Python. So aside from getting it running, we've had to put basically 0 effort into making it scale for our needs.

However, there's several things that could be better about it: * It's error messages are absolutely terrible; if ever one of our users ends up getting an error with RabbitMQ (even for simple things like a misconfigured hostname), they always end up needing to get help from the Zulip team, because the errors logs are just inscrutable. As an open source project, we've handled this issue by really carefully scripting the installation to be a failure-proof configuration (in this case, setting the RabbitMQ hostname to 127.0.0.1, so that no user-controlled configuration can break it). But it was a real pain to get there and the process of determining we needed to do that caused a significant amount of pain to folks installing Zulip. * The pika library for Python takes a lot of time to startup a RabbitMQ connection; this means that Zulip server restarts are more disruptive than would be ideal. * It's annoying that you need to run the rabbitmqctl management commands as root.

But overall, I like that it has clean, clear semanstics and high scalability, and haven't been tempted to do the work to migrate to something like Redis (which has its own downsides).

See more
Kafka logo

Kafka

13.9K
13.1K
562
Distributed, fault tolerant, high throughput pub-sub messaging system
13.9K
13.1K
+ 1
562
PROS OF KAFKA
  • 120
    High-throughput
  • 114
    Distributed
  • 86
    Scalable
  • 79
    High-Performance
  • 64
    Durable
  • 35
    Publish-Subscribe
  • 18
    Simple-to-use
  • 14
    Open source
  • 10
    Written in Scala and java. Runs on JVM
  • 6
    Message broker + Streaming system
  • 4
    Avro schema integration
  • 2
    Suport Multiple clients
  • 2
    Robust
  • 2
    KSQL
  • 2
    Partioned, replayable log
  • 1
    Fun
  • 1
    Extremely good parallelism constructs
  • 1
    Simple publisher / multi-subscriber model
  • 1
    Flexible
CONS OF KAFKA
  • 27
    Non-Java clients are second-class citizens
  • 26
    Needs Zookeeper
  • 7
    Operational difficulties
  • 2
    Terrible Packaging

related Kafka posts

Eric Colson
Chief Algorithms Officer at Stitch Fix · | 21 upvotes · 1.9M views

The algorithms and data infrastructure at Stitch Fix is housed in #AWS. Data acquisition is split between events flowing through Kafka, and periodic snapshots of PostgreSQL DBs. We store data in an Amazon S3 based data warehouse. Apache Spark on Yarn is our tool of choice for data movement and #ETL. Because our storage layer (s3) is decoupled from our processing layer, we are able to scale our compute environment very elastically. We have several semi-permanent, autoscaling Yarn clusters running to serve our data processing needs. While the bulk of our compute infrastructure is dedicated to algorithmic processing, we also implemented Presto for adhoc queries and dashboards.

Beyond data movement and ETL, most #ML centric jobs (e.g. model training and execution) run in a similarly elastic environment as containers running Python and R code on Amazon EC2 Container Service clusters. The execution of batch jobs on top of ECS is managed by Flotilla, a service we built in house and open sourced (see https://github.com/stitchfix/flotilla-os).

At Stitch Fix, algorithmic integrations are pervasive across the business. We have dozens of data products actively integrated systems. That requires serving layer that is robust, agile, flexible, and allows for self-service. Models produced on Flotilla are packaged for deployment in production using Khan, another framework we've developed internally. Khan provides our data scientists the ability to quickly productionize those models they've developed with open source frameworks in Python 3 (e.g. PyTorch, sklearn), by automatically packaging them as Docker containers and deploying to Amazon ECS. This provides our data scientist a one-click method of getting from their algorithms to production. We then integrate those deployments into a service mesh, which allows us to A/B test various implementations in our product.

For more info:

#DataScience #DataStack #Data

See more
John Kodumal

As we've evolved or added additional infrastructure to our stack, we've biased towards managed services. Most new backing stores are Amazon RDS instances now. We do use self-managed PostgreSQL with TimescaleDB for time-series data—this is made HA with the use of Patroni and Consul.

We also use managed Amazon ElastiCache instances instead of spinning up Amazon EC2 instances to run Redis workloads, as well as shifting to Amazon Kinesis instead of Kafka.

See more
Airflow logo

Airflow

1.1K
1.8K
107
A platform to programmaticaly author, schedule and monitor data pipelines, by Airbnb
1.1K
1.8K
+ 1
107
PROS OF AIRFLOW
  • 42
    Features
  • 13
    Task Dependency Management
  • 12
    Beautiful UI
  • 10
    Cluster of workers
  • 9
    Extensibility
  • 5
    Open source
  • 4
    Complex workflows
  • 4
    Python
  • 2
    Custom operators
  • 2
    K
  • 2
    Dashboard
  • 1
    Good api
  • 1
    Apache project
CONS OF AIRFLOW
    Be the first to leave a con

    related Airflow posts

    Shared insights
    on
    Jenkins
    Airflow

    I am looking for an open-source scheduler tool with cross-functional application dependencies. Some of the tasks I am looking to schedule are as follows:

    1. Trigger Matillion ETL loads
    2. Trigger Attunity Replication tasks that have downstream ETL loads
    3. Trigger Golden gate Replication Tasks
    4. Shell scripts, wrappers, file watchers
    5. Event-driven schedules

    I have used Airflow in the past, and I know we need to create DAGs for each pipeline. I am not familiar with Jenkins, but I know it works with configuration without much underlying code. I want to evaluate both and appreciate any advise

    See more
    Shared insights
    on
    AWS Step Functions
    Airflow

    I am working on a project that grabs a set of input data from AWS S3, pre-processes and divvies it up, spins up 10K batch containers to process the divvied data in parallel on AWS Batch, post-aggregates the data, and pushes it to S3.

    I already have software patterns from other projects for Airflow + Batch but have not dealt with the scaling factors of 10k parallel tasks. Airflow is nice since I can look at which tasks failed and retry a task after debugging. But dealing with that many tasks on one Airflow EC2 instance seems like a barrier. Another option would be to have one task that kicks off the 10k containers and monitors it from there.

    I have no experience with AWS Step Functions but have heard it's AWS's Airflow. There looks to be plenty of patterns online for Step Functions + Batch. Do Step Functions seem like a good path to check out for my use case? Do you get the same insights on failing jobs / ability to retry tasks as you do with Airflow?

    See more
    Cucumber logo

    Cucumber

    680
    631
    22
    Simple, human collaboration.
    680
    631
    + 1
    22
    PROS OF CUCUMBER
    • 16
      Simple Syntax
    • 2
      Nice report
    • 2
      Simple usage
    • 2
      Huge community
    CONS OF CUCUMBER
      Be the first to leave a con

      related Cucumber posts

      Benjamin Poon
      QA Manager - Engineering at HBC Digital · | 8 upvotes · 672.5K views

      For our digital QA organization to support a complex hybrid monolith/microservice architecture, our team took on the lofty goal of building out a commonized UI test automation framework. One of the primary requisites included a technical minimalist threshold such that an engineer or analyst with fundamental knowledge of JavaScript could automate their tests with greater ease. Just to list a few: - Nightwatchjs - Selenium - Cucumber - GitHub - Go.CD - Docker - ExpressJS - React - PostgreSQL

      With this structure, we're able to combine the automation efforts of each team member into a centralized repository while also providing new relevant metrics to business owners.

      See more
      Sarah Elson
      Product Growth at LambdaTest · | 4 upvotes · 283.4K views

      @producthunt LambdaTest Selenium JavaScript Java Python PHP Cucumber TeamCity CircleCI With this new release of LambdaTest automation, you can run tests across an Online Selenium Grid of 2000+ browsers and OS combinations to perform cross browser testing. This saves you from the pain of maintaining the infrastructure and also saves you the licensing costs for browsers and operating systems. #testing #Seleniumgrid #Selenium #testautomation #automation #webdriver #producthunt hunted

      See more
      Amazon SQS logo

      Amazon SQS

      1.8K
      1.6K
      163
      Fully managed message queuing service
      1.8K
      1.6K
      + 1
      163
      PROS OF AMAZON SQS
      • 58
        Easy to use, reliable
      • 39
        Low cost
      • 26
        Simple
      • 13
        Doesn't need to maintain it
      • 8
        It is Serverless
      • 4
        Has a max message size (currently 256K)
      • 3
        Delayed delivery upto 15 mins only
      • 3
        Triggers Lambda
      • 3
        Easy to configure with Terraform
      • 3
        Delayed delivery upto 12 hours
      • 1
        JMS compliant
      • 1
        Support for retry and dead letter queue
      • 1
        D
      CONS OF AMAZON SQS
      • 2
        Has a max message size (currently 256K)
      • 2
        Proprietary
      • 2
        Difficult to configure
      • 1
        Has a maximum 15 minutes of delayed messages only

      related Amazon SQS posts

      Praveen Mooli
      Engineering Manager at Taylor and Francis · | 14 upvotes · 1.8M views

      We are in the process of building a modern content platform to deliver our content through various channels. We decided to go with Microservices architecture as we wanted scale. Microservice architecture style is an approach to developing an application as a suite of small independently deployable services built around specific business capabilities. You can gain modularity, extensive parallelism and cost-effective scaling by deploying services across many distributed servers. Microservices modularity facilitates independent updates/deployments, and helps to avoid single point of failure, which can help prevent large-scale outages. We also decided to use Event Driven Architecture pattern which is a popular distributed asynchronous architecture pattern used to produce highly scalable applications. The event-driven architecture is made up of highly decoupled, single-purpose event processing components that asynchronously receive and process events.

      To build our #Backend capabilities we decided to use the following: 1. #Microservices - Java with Spring Boot , Node.js with ExpressJS and Python with Flask 2. #Eventsourcingframework - Amazon Kinesis , Amazon Kinesis Firehose , Amazon SNS , Amazon SQS, AWS Lambda 3. #Data - Amazon RDS , Amazon DynamoDB , Amazon S3 , MongoDB Atlas

      To build #Webapps we decided to use Angular 2 with RxJS

      #Devops - GitHub , Travis CI , Terraform , Docker , Serverless

      See more
      Tim Specht
      ‎Co-Founder and CTO at Dubsmash · | 14 upvotes · 596.5K views

      In order to accurately measure & track user behaviour on our platform we moved over quickly from the initial solution using Google Analytics to a custom-built one due to resource & pricing concerns we had.

      While this does sound complicated, it’s as easy as clients sending JSON blobs of events to Amazon Kinesis from where we use AWS Lambda & Amazon SQS to batch and process incoming events and then ingest them into Google BigQuery. Once events are stored in BigQuery (which usually only takes a second from the time the client sends the data until it’s available), we can use almost-standard-SQL to simply query for data while Google makes sure that, even with terabytes of data being scanned, query times stay in the range of seconds rather than hours. Before ingesting their data into the pipeline, our mobile clients are aggregating events internally and, once a certain threshold is reached or the app is going to the background, sending the events as a JSON blob into the stream.

      In the past we had workers running that continuously read from the stream and would validate and post-process the data and then enqueue them for other workers to write them to BigQuery. We went ahead and implemented the Lambda-based approach in such a way that Lambda functions would automatically be triggered for incoming records, pre-aggregate events, and write them back to SQS, from which we then read them, and persist the events to BigQuery. While this approach had a couple of bumps on the road, like re-triggering functions asynchronously to keep up with the stream and proper batch sizes, we finally managed to get it running in a reliable way and are very happy with this solution today.

      #ServerlessTaskProcessing #GeneralAnalytics #RealTimeDataProcessing #BigDataAsAService

      See more
      ActiveMQ logo

      ActiveMQ

      414
      1K
      72
      A message broker written in Java together with a full JMS client
      414
      1K
      + 1
      72
      PROS OF ACTIVEMQ
      • 16
        Easy to use
      • 13
        Open source
      • 12
        Efficient
      • 10
        JMS compliant
      • 6
        High Availability
      • 5
        Scalable
      • 3
        Support XA (distributed transactions)
      • 3
        Persistence
      • 2
        Distributed Network of brokers
      • 1
        Highly configurable
      • 1
        Docker delievery
      • 0
        RabbitMQ
      CONS OF ACTIVEMQ
      • 1
        Low resilience to exceptions and interruptions
      • 1
        Difficult to scale
      • 1
        Support

      related ActiveMQ posts

      I want to choose Message Queue with the following features - Highly Available, Distributed, Scalable, Monitoring. I have RabbitMQ, ActiveMQ, Kafka and Apache RocketMQ in mind. But I am confused which one to choose.

      See more
      Naushad Warsi
      software developer at klingelnberg · | 1 upvote · 573.5K views
      Shared insights
      on
      ActiveMQ
      RabbitMQ

      I use ActiveMQ because RabbitMQ have stopped giving the support for AMQP 1.0 or above version and the earlier version of AMQP doesn't give the functionality to support OAuth.

      If OAuth is not required and we can go with AMQP 0.9 then i still recommend rabbitMq.

      See more
      MQTT logo

      MQTT

      303
      327
      4
      A machine-to-machine Internet of Things connectivity protocol
      303
      327
      + 1
      4
      PROS OF MQTT
      • 2
        Varying levels of Quality of Service to fit a range of
      • 1
        Very easy to configure and use with open source tools
      • 1
        Lightweight with a relatively small data footprint
      CONS OF MQTT
      • 1
        Easy to configure in an unsecure manner

      related MQTT posts

      Apache NiFi logo

      Apache NiFi

      235
      480
      55
      A reliable system to process and distribute data
      235
      480
      + 1
      55
      PROS OF APACHE NIFI
      • 14
        Visual Data Flows using Directed Acyclic Graphs (DAGs)
      • 7
        Free (Open Source)
      • 5
        Simple-to-use
      • 4
        Reactive with back-pressure
      • 4
        Scalable horizontally as well as vertically
      • 3
        Bi-directional channels
      • 3
        Fast prototyping
      • 2
        Data provenance
      • 2
        Built-in graphical user interface
      • 2
        End-to-end security between all nodes
      • 2
        Can handle messages up to gigabytes in size
      • 1
        Hbase support
      • 1
        Kudu support
      • 1
        Hive support
      • 1
        Slack integration
      • 1
        Support for custom Processor in Java
      • 1
        Lot of articles
      • 1
        Lots of documentation
      CONS OF APACHE NIFI
      • 2
        Memory-intensive
      • 1
        HA support is not full fledge

      related Apache NiFi posts

      I am looking for the best tool to orchestrate #ETL workflows in non-Hadoop environments, mainly for regression testing use cases. Would Airflow or Apache NiFi be a good fit for this purpose?

      For example, I want to run an Informatica ETL job and then run an SQL task as a dependency, followed by another task from Jira. What tool is best suited to set up such a pipeline?

      See more