813
477
239

What is Celery?

Celery is an asynchronous task queue/job queue based on distributed message passing. It is focused on real-time operation, but supports scheduling as well.
Celery is a tool in the Message Queue category of a tech stack.
Celery is an open source tool with 13K GitHub stars and 3.3K GitHub forks. Here’s a link to Celery's open source repository on GitHub

Who uses Celery?

Companies
345 companies reportedly use Celery in their tech stacks, including Udemy, Sentry, and MOCI.

Developers
435 developers on StackShare have stated that they use Celery.

Why developers like Celery?

Here’s a list of reasons why companies and developers use Celery
Celery Reviews

Here are some stack decisions, common use cases and reviews by companies and developers who chose Celery in their tech stack.

James Cunningham
James Cunningham
Operations Engineer at Sentry · | 22 upvotes · 69.3K views
atSentrySentry
Redis
PostgreSQL
Celery
Django
#InMemoryDatabases
#MessageQueue

Sentry started as (and remains) an open-source project, growing out of an error logging tool built in 2008. That original build nine years ago was Django and Celery (Python’s asynchronous task codebase), with PostgreSQL as the database and Redis as the power behind Celery.

We displayed a truly shrewd notion of branding even then, giving the project a catchy name that companies the world over remain jealous of to this day: django-db-log. For the longest time, Sentry’s subtitle on GitHub was “A simple Django app, built with love.” A slightly more accurate description probably would have included Starcraft and Soylent alongside love; regardless, this captured what Sentry was all about.

#MessageQueue #InMemoryDatabases

See more
James Cunningham
James Cunningham
Operations Engineer at Sentry · | 18 upvotes · 46.8K views
atSentrySentry
RabbitMQ
Celery
#MessageQueue

As Sentry runs throughout the day, there are about 50 different offline tasks that we execute—anything from “process this event, pretty please” to “send all of these cool people some emails.” There are some that we execute once a day and some that execute thousands per second.

Managing this variety requires a reliably high-throughput message-passing technology. We use Celery's RabbitMQ implementation, and we stumbled upon a great feature called Federation that allows us to partition our task queue across any number of RabbitMQ servers and gives us the confidence that, if any single server gets backlogged, others will pitch in and distribute some of the backlogged tasks to their consumers.

#MessageQueue

See more
Eli Hooten
Eli Hooten
CTO at Codecov · | 5 upvotes · 5.6K views
atCodecovCodecov
Redis
Celery
Python

A major aspect of Codecov is the use of long running asynchronous tasks to process large amounts of test coverage data uploaded by our users. Being a Python stack, Celery felt like a natural fit to manage codecov's long running tasks. We rely on Celery to manage all our background queues and asyncronous scheduling. Celery enables us to set timeouts for different tasks which has been instrumental in maintaining our queue in production. Celery also interfaces easily with Redis as a backend store, which allowed it to slot neatly into our existing infrastructure.

See more
StackShare Editors
StackShare Editors
Flask
AWS EC2
Celery
Datadog
PagerDuty
Airflow
StatsD
Grafana

Data science and engineering teams at Lyft maintain several big data pipelines that serve as the foundation for various types of analysis throughout the business.

Apache Airflow sits at the center of this big data infrastructure, allowing users to “programmatically author, schedule, and monitor data pipelines.” Airflow is an open source tool, and “Lyft is the very first Airflow adopter in production since the project was open sourced around three years ago.”

There are several key components of the architecture. A web UI allows users to view the status of their queries, along with an audit trail of any modifications the query. A metadata database stores things like job status and task instance status. A multi-process scheduler handles job requests, and triggers the executor to execute those tasks.

Airflow supports several executors, though Lyft uses CeleryExecutor to scale task execution in production. Airflow is deployed to three Amazon Auto Scaling Groups, with each associated with a celery queue.

Audit logs supplied to the web UI are powered by the existing Airflow audit logs as well as Flask signal.

Datadog, Statsd, Grafana, and PagerDuty are all used to monitor the Airflow system.

See more
Sharone Zitzman
Sharone Zitzman
Head of Developer Relations at Appsflyer · | 2 upvotes · 3K views
atCloudifyCloudify
Celery

For orchestrating the creation of the correct number of instances, managing errors and retries, and finally managing the deallocation of resources we use RabbitMQ in conjunction with the Celery Project framework, along with a self-developed workflow engine. Celery

See more
MoonSoo Kim
MoonSoo Kim
Chief Technology Officer at N R I S E - INTERACTIVE / FACTORY · | 1 upvotes · 3.6K views
atnrisenrise
Celery

백엔드, 어플리케이션 서버 모두 시간이 걸리는 작업들 및 주기적으로 수행해야 하는 작업들은 celery/celerybeat 을 통해 동작됩니다. 신뢰도가 아주 높으며, 유연합니다. gevent 로 동작시키면 더욱 깔끔한 관리가 가능합니다. Celery

See more

Celery Alternatives & Comparisons

What are some alternatives to Celery?
RabbitMQ
RabbitMQ gives your applications a common platform to send and receive messages, and your messages a safe place to live until received.
Kafka
Kafka is a distributed, partitioned, replicated commit log service. It provides the functionality of a messaging system, but with a unique design.
Amazon SQS
Transmit any volume of data, at any level of throughput, without losing messages or requiring other services to be always available. With SQS, you can offload the administrative burden of operating and scaling a highly available messaging cluster, while paying a low price for only what you use.
ActiveMQ
Apache ActiveMQ is fast, supports many Cross Language Clients and Protocols, comes with easy to use Enterprise Integration Patterns and many advanced features while fully supporting JMS 1.1 and J2EE 1.4. Apache ActiveMQ is released under the Apache 2.0 License.
ZeroMQ
The 0MQ lightweight messaging kernel is a library which extends the standard socket interfaces with features traditionally provided by specialised messaging middleware products. 0MQ sockets provide an abstraction of asynchronous message queues, multiple messaging patterns, message filtering (subscriptions), seamless access to multiple transport protocols and more.
See all alternatives

Celery's Stats

Celery's Followers
477 developers follow Celery to keep up with related blogs and decisions.
Gerald Ephraim
Panayiotis Tzagkarakis
Antonio Rodrigues Neto
Alvin Savoy
Wayne Sun
Alex Winkler
Miha Sedej
Dick Koekemoer
Albert King
Himansu Sekhar