Need advice about which tool to choose?Ask the StackShare community!

Celery

Stacks1.6K

Followers1.6K

+ 1

Votes280

DoctorKafka

Stacks3

Followers17

+ 1

Votes0

Add tool

Celery vs DoctorKafka: What are the differences?

Developers describe Celery as "Distributed task queue". Celery is an asynchronous task queue/job queue based on distributed message passing. It is focused on real-time operation, but supports scheduling as well. On the other hand, DoctorKafka is detailed as "A service for Kafka cluster auto healing and workload balancing". DoctorKafka can automatically detect broker failure and reassign the workload on the failed nodes to other nodes. DoctorKafka can also perform load balancing based on topic partitions's network usage, and makes sure that broker network usage does not exceed the defined settings.

Celery and DoctorKafka can be categorized as "Message Queue" tools.

Celery and DoctorKafka are both open source tools. It seems that Celery with 12.9K GitHub stars and 3.33K forks on GitHub has more adoption than DoctorKafka with 488 GitHub stars and 68 GitHub forks.

Advice on Celery and DoctorKafka

Shantha Dodmane

Sep 30, 2020 | 3 upvotes · 323.7K views

Needs advice

Celery

and

RabbitMQ

I am just a beginner at these two technologies.

Problem statement: I am getting lakh of users from the sequel server for whom I need to create caches in MongoDB by making different REST API requests.

Here these users can be treated as messages. Each REST API request is a task.

I am confused about whether I should go for RabbitMQ alone or Celery.

If I have to go with RabbitMQ, I prefer to use python with Pika module. But the challenge with Pika is, it is not thread-safe. So I am not finding a way to execute a lakh of API requests in parallel using multiple threads using Pika.

If I have to go with Celery, I don't know how I can achieve better scalability in executing these API requests in parallel.

Replies (1)

Travis Travelstead

Oct 5, 2020 | 5 upvotes · 265.1K views

Recommends

Redis

For large amounts of small tasks and caches I have had good luck with Redis and RQ. I have not personally used celery but I am fairly sure it would scale well, and I have not used RabbitMQ for anything besides communication between services. If you prefer python my suggestions should feel comfortable.

Sorry I do not have a more information

Manage your open source components, licenses, and vulnerabilities

Learn More

Pros of Celery

Pros of DoctorKafka

99
Task queue
63
Python integration
40
Django integration
30
Scheduled Task
19
Publish/subsribe
8
Various backend broker
6
Easy to use
5
Great community
5
Workflow
4
Free
1
Dynamic

Be the first to leave a pro

Sign up to add or upvote prosMake informed product decisions

Cons of Celery

Cons of DoctorKafka

4
Sometimes loses tasks
1
Depends on broker

Be the first to leave a con

Sign up to add or upvote consMake informed product decisions

- No public GitHub repository available -

What is Celery?

Celery is an asynchronous task queue/job queue based on distributed message passing. It is focused on real-time operation, but supports scheduling as well.

What is DoctorKafka?

DoctorKafka can automatically detect broker failure and reassign the workload on the failed nodes to other nodes. DoctorKafka can also perform load balancing based on topic partitions's network usage, and makes sure that broker network usage does not exceed the defined settings.

Need advice about which tool to choose?Ask the StackShare community!

What companies use Celery?

What companies use DoctorKafka?

No companies found

Manage your open source components, licenses, and vulnerabilities

Learn More

Sign up to get full access to all the companiesMake informed product decisions

What tools integrate with Celery?

What tools integrate with DoctorKafka?

Kafka

Sign up to get full access to all the tool integrationsMake informed product decisions

What are some alternatives to Celery and DoctorKafka?

RabbitMQ

RabbitMQ gives your applications a common platform to send and receive messages, and your messages a safe place to live until received.

Kafka

Kafka is a distributed, partitioned, replicated commit log service. It provides the functionality of a messaging system, but with a unique design.

Airflow

Use Airflow to author workflows as directed acyclic graphs (DAGs) of tasks. The Airflow scheduler executes your tasks on an array of workers while following the specified dependencies. Rich command lines utilities makes performing complex surgeries on DAGs a snap. The rich user interface makes it easy to visualize pipelines running in production, monitor progress and troubleshoot issues when needed.

Cucumber

Cucumber is a tool that supports Behaviour-Driven Development (BDD) - a software development process that aims to enhance software quality and reduce maintenance costs.

MySQL

The MySQL software delivers a very fast, multi-threaded, multi-user, and robust SQL (Structured Query Language) database server. MySQL Server is intended for mission-critical, heavy-load production systems as well as for embedding into mass-deployed software.

See all alternatives

Celery vs DoctorKafka

Need advice about which tool to choose?Ask the StackShare community!

Pros of Celery

Pros of DoctorKafka

Sign up to add or upvote prosMake informed product decisions

Cons of Celery

Cons of DoctorKafka

Sign up to add or upvote consMake informed product decisions

What is Celery?

What is DoctorKafka?

Need advice about which tool to choose?Ask the StackShare community!

What companies use Celery?

What companies use DoctorKafka?

Sign up to get full access to all the companiesMake informed product decisions

What tools integrate with Celery?

What tools integrate with DoctorKafka?

Sign up to get full access to all the tool integrationsMake informed product decisions

Related Comparisons

Trending Comparisons

Top Comparisons