906
549
+ 1
239

What is Celery?

Celery is an asynchronous task queue/job queue based on distributed message passing. It is focused on real-time operation, but supports scheduling as well.
Celery is a tool in the Message Queue category of a tech stack.
Celery is an open source tool with 14K GitHub stars and 3.5K GitHub forks. Here’s a link to Celery's open source repository on GitHub

Who uses Celery?

Companies
363 companies reportedly use Celery in their tech stacks, including Udemy, Sentry, and Bitbucket.

Developers
508 developers on StackShare have stated that they use Celery.

Why developers like Celery?

Here’s a list of reasons why companies and developers use Celery
Celery Reviews

Here are some stack decisions, common use cases and reviews by companies and developers who chose Celery in their tech stack.

James Cunningham
James Cunningham
Operations Engineer at Sentry · | 22 upvotes · 154.8K views
atSentrySentry
Django
Django
Celery
Celery
PostgreSQL
PostgreSQL
Redis
Redis
#MessageQueue
#InMemoryDatabases

Sentry started as (and remains) an open-source project, growing out of an error logging tool built in 2008. That original build nine years ago was Django and Celery (Python’s asynchronous task codebase), with PostgreSQL as the database and Redis as the power behind Celery.

We displayed a truly shrewd notion of branding even then, giving the project a catchy name that companies the world over remain jealous of to this day: django-db-log. For the longest time, Sentry’s subtitle on GitHub was “A simple Django app, built with love.” A slightly more accurate description probably would have included Starcraft and Soylent alongside love; regardless, this captured what Sentry was all about.

#MessageQueue #InMemoryDatabases

See more
James Cunningham
James Cunningham
Operations Engineer at Sentry · | 18 upvotes · 150.3K views
atSentrySentry
Celery
Celery
RabbitMQ
RabbitMQ
#MessageQueue

As Sentry runs throughout the day, there are about 50 different offline tasks that we execute—anything from “process this event, pretty please” to “send all of these cool people some emails.” There are some that we execute once a day and some that execute thousands per second.

Managing this variety requires a reliably high-throughput message-passing technology. We use Celery's RabbitMQ implementation, and we stumbled upon a great feature called Federation that allows us to partition our task queue across any number of RabbitMQ servers and gives us the confidence that, if any single server gets backlogged, others will pitch in and distribute some of the backlogged tasks to their consumers.

#MessageQueue

See more
Eli Hooten
Eli Hooten
CTO at Codecov · | 6 upvotes · 13.7K views
atCodecovCodecov
Python
Python
Celery
Celery
Redis
Redis

A major aspect of Codecov is the use of long running asynchronous tasks to process large amounts of test coverage data uploaded by our users. Being a Python stack, Celery felt like a natural fit to manage codecov's long running tasks. We rely on Celery to manage all our background queues and asyncronous scheduling. Celery enables us to set timeouts for different tasks which has been instrumental in maintaining our queue in production. Celery also interfaces easily with Redis as a backend store, which allowed it to slot neatly into our existing infrastructure.

See more
StackShare Editors
StackShare Editors
Grafana
Grafana
StatsD
StatsD
Airflow
Airflow
PagerDuty
PagerDuty
Datadog
Datadog
Celery
Celery
AWS EC2
AWS EC2
Flask
Flask

Data science and engineering teams at Lyft maintain several big data pipelines that serve as the foundation for various types of analysis throughout the business.

Apache Airflow sits at the center of this big data infrastructure, allowing users to “programmatically author, schedule, and monitor data pipelines.” Airflow is an open source tool, and “Lyft is the very first Airflow adopter in production since the project was open sourced around three years ago.”

There are several key components of the architecture. A web UI allows users to view the status of their queries, along with an audit trail of any modifications the query. A metadata database stores things like job status and task instance status. A multi-process scheduler handles job requests, and triggers the executor to execute those tasks.

Airflow supports several executors, though Lyft uses CeleryExecutor to scale task execution in production. Airflow is deployed to three Amazon Auto Scaling Groups, with each associated with a celery queue.

Audit logs supplied to the web UI are powered by the existing Airflow audit logs as well as Flask signal.

Datadog, Statsd, Grafana, and PagerDuty are all used to monitor the Airflow system.

See more
Michael Mota
Michael Mota
CEO & Founder at AlterEstate · | 4 upvotes · 26.4K views
atAlterEstateAlterEstate
Celery
Celery
RabbitMQ
RabbitMQ
Django
Django

Automations are what makes a CRM powerful. With Celery and RabbitMQ we've been able to make powerful automations that truly works for our clients. Such as for example, automatic daily reports, reminders for their activities, important notifications regarding their client activities and actions on the website and more.

We use Celery basically for everything that needs to be scheduled for the future, and using RabbitMQ as our Queue-broker is amazing since it fully integrates with Django and Celery storing on our database results of the tasks done so we can see if anything fails immediately.

See more
Sharone Zitzman
Sharone Zitzman
Head of Developer Relations at AppsFlyer · | 2 upvotes · 9.2K views
atCloudifyCloudify
Celery
Celery

For orchestrating the creation of the correct number of instances, managing errors and retries, and finally managing the deallocation of resources we use RabbitMQ in conjunction with the Celery Project framework, along with a self-developed workflow engine. Celery

See more

Celery Alternatives & Comparisons

What are some alternatives to Celery?
RabbitMQ
RabbitMQ gives your applications a common platform to send and receive messages, and your messages a safe place to live until received.
Kafka
Kafka is a distributed, partitioned, replicated commit log service. It provides the functionality of a messaging system, but with a unique design.
Amazon SQS
Transmit any volume of data, at any level of throughput, without losing messages or requiring other services to be always available. With SQS, you can offload the administrative burden of operating and scaling a highly available messaging cluster, while paying a low price for only what you use.
ActiveMQ
Apache ActiveMQ is fast, supports many Cross Language Clients and Protocols, comes with easy to use Enterprise Integration Patterns and many advanced features while fully supporting JMS 1.1 and J2EE 1.4. Apache ActiveMQ is released under the Apache 2.0 License.
ZeroMQ
The 0MQ lightweight messaging kernel is a library which extends the standard socket interfaces with features traditionally provided by specialised messaging middleware products. 0MQ sockets provide an abstraction of asynchronous message queues, multiple messaging patterns, message filtering (subscriptions), seamless access to multiple transport protocols and more.
See all alternatives

Celery's Followers
549 developers follow Celery to keep up with related blogs and decisions.
Rajib Bahar
태화 정
Max Saykov
Dmitry Kovalenko
Ryan McCall
Vitaliy Ivanov
shnela
Sun Hao
Sincere Ye
Rimvydas Zilinskas