1.1K
892
+ 1
247

What is Celery?

Celery is an asynchronous task queue/job queue based on distributed message passing. It is focused on real-time operation, but supports scheduling as well.
Celery is a tool in the Message Queue category of a tech stack.
Celery is an open source tool with 15K GitHub stars and 3.7K GitHub forks. Here’s a link to Celery's open source repository on GitHub

Who uses Celery?

Companies
387 companies reportedly use Celery in their tech stacks, including Udemy, Robinhood, and Accenture.

Developers
684 developers on StackShare have stated that they use Celery.
Private Decisions at about Celery

Here are some stack decisions, common use cases and reviews by members of with Celery in their tech stack.

Dieter Adriaenssens
Dieter Adriaenssens
Shared insights
on
CeleryCelery

Using Celery, the web service creates tasks that are executed by a background worker. Celery uses a RabbitMQ instance as a task queue. Celery

See more
Yaakov Gesher
Yaakov Gesher
Data Pipeline Engineer at Planet Watchers · | 1 upvotes · 33 views
Shared insights
on
CeleryCelery

We used celery, in combination with RabbitMQ and celery-beat, to run periodic tasks, as well as some user-initiated long-running tasks on the server. Celery

See more
Mark Lee
Mark Lee
Head of Growth - SendBird at SendBird · | 1 upvotes · 0 views
Shared insights
on
CeleryCelery

Distributed Task Queue Celery

See more
Seungkwon Park
Seungkwon Park
Shared insights
on
CeleryCelery

http request 이후 바로 response가 즉시 오직 않는 많은 양의 무거운 작업은 celery에 task로 등록합니다.

그래서 유저에겐 빠른 response를 주고 celery에서 해당 업무를 수행합니다. Celery

See more
papaver
papaver
captain of a starship at electronic dreams · | 1 upvotes · 0 views
Shared insights
on
CeleryCelery

python remote workers. simple and easy to use. Celery

See more
Kovid Rathee
Kovid Rathee
Senior Data Analyst at Onedirect · | 1 upvotes · 0 views
Shared insights
on
CeleryCelery

Task queues Celery

See more
Public Decisions about Celery

Here are some stack decisions, common use cases and reviews by companies and developers who chose Celery in their tech stack.

James Cunningham
James Cunningham
Operations Engineer at Sentry · | 21 upvotes · 276K views

Sentry started as (and remains) an open-source project, growing out of an error logging tool built in 2008. That original build nine years ago was Django and Celery (Python’s asynchronous task codebase), with PostgreSQL as the database and Redis as the power behind Celery.

We displayed a truly shrewd notion of branding even then, giving the project a catchy name that companies the world over remain jealous of to this day: django-db-log. For the longest time, Sentry’s subtitle on GitHub was “A simple Django app, built with love.” A slightly more accurate description probably would have included Starcraft and Soylent alongside love; regardless, this captured what Sentry was all about.

#MessageQueue #InMemoryDatabases

See more
James Cunningham
James Cunningham
Operations Engineer at Sentry · | 18 upvotes · 671.1K views
Shared insights
on
CeleryCeleryRabbitMQRabbitMQ
at

As Sentry runs throughout the day, there are about 50 different offline tasks that we execute—anything from “process this event, pretty please” to “send all of these cool people some emails.” There are some that we execute once a day and some that execute thousands per second.

Managing this variety requires a reliably high-throughput message-passing technology. We use Celery's RabbitMQ implementation, and we stumbled upon a great feature called Federation that allows us to partition our task queue across any number of RabbitMQ servers and gives us the confidence that, if any single server gets backlogged, others will pitch in and distribute some of the backlogged tasks to their consumers.

#MessageQueue

See more
Eli Hooten
Eli Hooten
CTO at Codecov · | 7 upvotes · 24.6K views
Shared insights
on
PythonPythonCeleryCeleryRedisRedis
at

A major aspect of Codecov is the use of long running asynchronous tasks to process large amounts of test coverage data uploaded by our users. Being a Python stack, Celery felt like a natural fit to manage codecov's long running tasks. We rely on Celery to manage all our background queues and asyncronous scheduling. Celery enables us to set timeouts for different tasks which has been instrumental in maintaining our queue in production. Celery also interfaces easily with Redis as a backend store, which allowed it to slot neatly into our existing infrastructure.

See more
Michael Mota
Michael Mota
Founder at AlterEstate · | 6 upvotes · 182.2K views

Automations are what makes a CRM powerful. With Celery and RabbitMQ we've been able to make powerful automations that truly works for our clients. Such as for example, automatic daily reports, reminders for their activities, important notifications regarding their client activities and actions on the website and more.

We use Celery basically for everything that needs to be scheduled for the future, and using RabbitMQ as our Queue-broker is amazing since it fully integrates with Django and Celery storing on our database results of the tasks done so we can see if anything fails immediately.

See more
StackShare Editors
StackShare Editors

Data science and engineering teams at Lyft maintain several big data pipelines that serve as the foundation for various types of analysis throughout the business.

Apache Airflow sits at the center of this big data infrastructure, allowing users to “programmatically author, schedule, and monitor data pipelines.” Airflow is an open source tool, and “Lyft is the very first Airflow adopter in production since the project was open sourced around three years ago.”

There are several key components of the architecture. A web UI allows users to view the status of their queries, along with an audit trail of any modifications the query. A metadata database stores things like job status and task instance status. A multi-process scheduler handles job requests, and triggers the executor to execute those tasks.

Airflow supports several executors, though Lyft uses CeleryExecutor to scale task execution in production. Airflow is deployed to three Amazon Auto Scaling Groups, with each associated with a celery queue.

Audit logs supplied to the web UI are powered by the existing Airflow audit logs as well as Flask signal.

Datadog, Statsd, Grafana, and PagerDuty are all used to monitor the Airflow system.

See more
Sharone Zitzman
Sharone Zitzman
Head of Developer Relations at AppsFlyer · | 2 upvotes · 17.6K views
Shared insights
on
CeleryCelery
at

For orchestrating the creation of the correct number of instances, managing errors and retries, and finally managing the deallocation of resources we use RabbitMQ in conjunction with the Celery Project framework, along with a self-developed workflow engine. Celery

See more

Celery Alternatives & Comparisons

What are some alternatives to Celery?
RabbitMQ
RabbitMQ gives your applications a common platform to send and receive messages, and your messages a safe place to live until received.
Kafka
Kafka is a distributed, partitioned, replicated commit log service. It provides the functionality of a messaging system, but with a unique design.
Airflow
Use Airflow to author workflows as directed acyclic graphs (DAGs) of tasks. The Airflow scheduler executes your tasks on an array of workers while following the specified dependencies. Rich command lines utilities makes performing complex surgeries on DAGs a snap. The rich user interface makes it easy to visualize pipelines running in production, monitor progress and troubleshoot issues when needed.
Cucumber
Cucumber is a tool that supports Behaviour-Driven Development (BDD) - a software development process that aims to enhance software quality and reduce maintenance costs.
Amazon SQS
Transmit any volume of data, at any level of throughput, without losing messages or requiring other services to be always available. With SQS, you can offload the administrative burden of operating and scaling a highly available messaging cluster, while paying a low price for only what you use.
See all alternatives

Celery's Followers
892 developers follow Celery to keep up with related blogs and decisions.
Abdulrahman Azmy
Sunil Chandu
逸 杨
Christopher Espinal
Juan Diego Cardona Marin
Thomas Antonio
João Victor Machado
Bruno Brito Semedo
Subham Sarangi
Kashish Bakshi