Need advice about which tool to choose?Ask the StackShare community!

Celery

1.3K
1.3K
+ 1
265
Serverless

1.1K
977
+ 1
23
Add tool

Celery vs Serverless: What are the differences?

What is Celery? Distributed task queue. Celery is an asynchronous task queue/job queue based on distributed message passing. It is focused on real-time operation, but supports scheduling as well.

What is Serverless? The most widely-adopted toolkit for building serverless applications. Build applications comprised of microservices that run in response to events, auto-scale for you, and only charge you when they run. This lowers the total cost of maintaining your apps, enabling you to build more logic, faster. The Framework uses new event-driven compute services, like AWS Lambda, Google CloudFunctions, and more.

Celery and Serverless are primarily classified as "Message Queue" and "Serverless / Task Processing" tools respectively.

"Task queue" is the top reason why over 84 developers like Celery, while over 10 developers mention "API integration " as the leading cause for choosing Serverless.

Celery and Serverless are both open source tools. It seems that Serverless with 30.5K GitHub stars and 3.38K forks on GitHub has more adoption than Celery with 12.7K GitHub stars and 3.3K GitHub forks.

Sentry, Ansible, and OpenLabel are some of the popular companies that use Celery, whereas Serverless is used by Droplr, Plista GmbH, and Hammerhead. Celery has a broader approval, being mentioned in 271 company stacks & 77 developers stacks; compared to Serverless, which is listed in 112 company stacks and 43 developer stacks.

Advice on Celery and Serverless
Needs advice
on
RabbitMQ
and
Celery

I am just a beginner at these two technologies.

Problem statement: I am getting lakh of users from the sequel server for whom I need to create caches in MongoDB by making different REST API requests.

Here these users can be treated as messages. Each REST API request is a task.

I am confused about whether I should go for RabbitMQ alone or Celery.

If I have to go with RabbitMQ, I prefer to use python with Pika module. But the challenge with Pika is, it is not thread-safe. So I am not finding a way to execute a lakh of API requests in parallel using multiple threads using Pika.

If I have to go with Celery, I don't know how I can achieve better scalability in executing these API requests in parallel.

See more
Replies (1)
Recommends
Redis
rq

For large amounts of small tasks and caches I have had good luck with Redis and RQ. I have not personally used celery but I am fairly sure it would scale well, and I have not used RabbitMQ for anything besides communication between services. If you prefer python my suggestions should feel comfortable.

Sorry I do not have a more information

See more
Decisions about Celery and Serverless
Tim Nolet

When adding a new feature to Checkly rearchitecting some older piece, I tend to pick Heroku for rolling it out. But not always, because sometimes I pick AWS Lambda . The short story:

  • Developer Experience trumps everything.
  • AWS Lambda is cheap. Up to a limit though. This impact not only your wallet.
  • If you need geographic spread, AWS is lonely at the top.
The setup

Recently, I was doing a brainstorm at a startup here in Berlin on the future of their infrastructure. They were ready to move on from their initial, almost 100% Ec2 + Chef based setup. Everything was on the table. But we crossed out a lot quite quickly:

  • Pure, uncut, self hosted Kubernetes — way too much complexity
  • Managed Kubernetes in various flavors — still too much complexity
  • Zeit — Maybe, but no Docker support
  • Elastic Beanstalk — Maybe, bit old but does the job
  • Heroku
  • Lambda

It became clear a mix of PaaS and FaaS was the way to go. What a surprise! That is exactly what I use for Checkly! But when do you pick which model?

I chopped that question up into the following categories:

  • Developer Experience / DX 🤓
  • Ops Experience / OX 🐂 (?)
  • Cost 💵
  • Lock in 🔐

Read the full post linked below for all details

See more
Get Advice from developers at your company using Private StackShare. Sign up for Private StackShare.
Learn More
Pros of Celery
Pros of Serverless
  • 94
    Task queue
  • 61
    Python integration
  • 37
    Django integration
  • 29
    Scheduled Task
  • 18
    Publish/subsribe
  • 6
    Easy to use
  • 6
    Various backend broker
  • 5
    Great community
  • 4
    Workflow
  • 4
    Free
  • 1
    Dynamic
  • 12
    API integration
  • 7
    Supports cloud functions for Google, Azure, and IBM
  • 2
    Lower cost
  • 1
    Openwhisk
  • 1
    Auto scale

Sign up to add or upvote prosMake informed product decisions

Cons of Celery
Cons of Serverless
  • 4
    Sometimes loses tasks
  • 1
    Depends on broker
    Be the first to leave a con

    Sign up to add or upvote consMake informed product decisions

    What is Celery?

    Celery is an asynchronous task queue/job queue based on distributed message passing. It is focused on real-time operation, but supports scheduling as well.

    What is Serverless?

    Build applications comprised of microservices that run in response to events, auto-scale for you, and only charge you when they run. This lowers the total cost of maintaining your apps, enabling you to build more logic, faster. The Framework uses new event-driven compute services, like AWS Lambda, Google CloudFunctions, and more.

    Need advice about which tool to choose?Ask the StackShare community!

    What companies use Celery?
    What companies use Serverless?
    See which teams inside your own company are using Celery or Serverless.
    Sign up for Private StackShareLearn More

    Sign up to get full access to all the companiesMake informed product decisions

    What tools integrate with Celery?
    What tools integrate with Serverless?

    Sign up to get full access to all the tool integrationsMake informed product decisions

    Blog Posts

    +47
    49
    69270
    +42
    52
    19833
    +25
    7
    2828
    +24
    13
    16731
    What are some alternatives to Celery and Serverless?
    RabbitMQ
    RabbitMQ gives your applications a common platform to send and receive messages, and your messages a safe place to live until received.
    Kafka
    Kafka is a distributed, partitioned, replicated commit log service. It provides the functionality of a messaging system, but with a unique design.
    Airflow
    Use Airflow to author workflows as directed acyclic graphs (DAGs) of tasks. The Airflow scheduler executes your tasks on an array of workers while following the specified dependencies. Rich command lines utilities makes performing complex surgeries on DAGs a snap. The rich user interface makes it easy to visualize pipelines running in production, monitor progress and troubleshoot issues when needed.
    Cucumber
    Cucumber is a tool that supports Behaviour-Driven Development (BDD) - a software development process that aims to enhance software quality and reduce maintenance costs.
    Amazon SQS
    Transmit any volume of data, at any level of throughput, without losing messages or requiring other services to be always available. With SQS, you can offload the administrative burden of operating and scaling a highly available messaging cluster, while paying a low price for only what you use.
    See all alternatives