Celery vs NSQ vs RabbitMQ

Overview

RabbitMQ

Stacks21.8K

Followers18.9K

Votes558

GitHub Stars13.2K

Forks4.0K

Celery

Stacks1.7K

Followers1.6K

Votes280

GitHub Stars27.5K

Forks4.9K

NSQ

Stacks142

Followers356

Votes148

Celery vs NSQ vs RabbitMQ: What are the differences?

Key Differences between Celery, NSQ, and RabbitMQ

Messaging Protocol: Celery primarily uses AMQP (Advanced Message Queuing Protocol) whereas NSQ relies on its own proprietary protocol. RabbitMQ also supports AMQP but can be configured to work with other messaging protocols as well.
Scalability and Performance: NSQ is known for its high scalability and performance, designed to handle large message volumes efficiently. Celery and RabbitMQ are also scalable but may require additional configurations for high performance in comparison to NSQ.
Message Delivery Guarantees: NSQ provides at-least-once message delivery guarantees, ensuring that messages are delivered at least once to consumers. Celery, by default, offers at-most-once delivery, which can result in message loss if not handled carefully. RabbitMQ offers different delivery modes for guaranteed message delivery, giving more flexibility in choosing the desired level of guarantee.
Ease of Use and Configuration: Celery is known for its ease of use and simple configuration, making it suitable for smaller projects or rapid deployment. RabbitMQ, while versatile, may require more complex configurations for specific use cases. NSQ falls in between, offering a balance of ease of use and performance-oriented configuration options.
Community and Ecosystem: RabbitMQ has a mature and extensive community with a wide range of plugins and integrations available. Celery also has a strong community support but may not have as many diverse integration options as RabbitMQ. NSQ, being relatively newer, has a growing community and a focus on simplicity and performance.
Fault Tolerance and Monitoring: RabbitMQ provides robust fault tolerance features like clustering and HA (High Availability) queues for reliability. NSQ also offers fault tolerance mechanisms but may require manual intervention for certain scenarios. Celery's fault tolerance capabilities depend on the underlying message broker being used, such as RabbitMQ or Redis for persistence.

In Summary, the key differences between Celery, NSQ, and RabbitMQ lie in their messaging protocols, scalability, message delivery guarantees, ease of use, community support, and fault tolerance mechanisms.

Share your Stack

Help developers discover the tools you use. Get visibility for your team's tech choices and contribute to the community's knowledge.

View Docs

CLI (Node.js)

Manual

Advice on RabbitMQ, Celery, NSQ

viradiya

Apr 12, 2020

Needs adviceon

AngularJS

ASP.NET Core

MSSQL

We are going to develop a microservices-based application. It consists of AngularJS, ASP.NET Core, and MSSQL.

We have 3 types of microservices. Emailservice, Filemanagementservice, Filevalidationservice

I am a beginner in microservices. But I have read about RabbitMQ, but come to know that there are Redis and Kafka also in the market. So, I want to know which is best.

933k views933k

Comments

Pulkit

Software Engineer

Oct 30, 2020

Needs adviceon

Django

Amazon SQS

RabbitMQ

Hi! I am creating a scraping system in Django, which involves long running tasks between 1 minute & 1 Day. As I am new to Message Brokers and Task Queues, I need advice on which architecture to use for my system. ( Amazon SQS, RabbitMQ, or Celery). The system should be autoscalable using Kubernetes(K8) based on the number of pending tasks in the queue.

474k views474k

Comments

Kirill

GO/C developer at Duckling Sales

Feb 16, 2021

Decided

Maybe not an obvious comparison with Kafka, since Kafka is pretty different from rabbitmq. But for small service, Rabbit as a pubsub platform is super easy to use and pretty powerful. Kafka as an alternative was the original choice, but its really a kind of overkill for a small-medium service. Especially if you are not planning to use k8s, since pure docker deployment can be a pain because of networking setup. Google PubSub was another alternative, its actually pretty cheap, but I never tested it since Rabbit was matching really good for mailing/notification services.

266k views266k

Comments

Detailed Comparison

RabbitMQ	Celery	NSQ
RabbitMQ gives your applications a common platform to send and receive messages, and your messages a safe place to live until received.	Celery is an asynchronous task queue/job queue based on distributed message passing. It is focused on real-time operation, but supports scheduling as well.	NSQ is a realtime distributed messaging platform designed to operate at scale, handling billions of messages per day. It promotes distributed and decentralized topologies without single points of failure, enabling fault tolerance and high availability coupled with a reliable message delivery guarantee. See features & guarantees.
Robust messaging for applications;Easy to use;Runs on all major operating systems;Supports a huge number of developer platforms;Open source and commercially supported	-	support distributed topologies with no SPOF;horizontally scalable (no brokers, seamlessly add more nodes to the cluster);low-latency push based message delivery (performance);combination load-balanced and multicast style message routing;excel at both streaming (high-throughput) and job oriented (low-throughput) workloads;primarily in-memory (beyond a high-water mark messages are transparently kept on disk);runtime discovery service for consumers to find producers (nsqlookupd);transport layer security (TLS);data format agnostic;few dependencies (easy to deploy) and a sane, bounded, default configuration;simple TCP protocol supporting client libraries in any language;HTTP interface for stats, admin actions, and producers (no client library needed to publish);integrates with statsd for realtime instrumentation;robust cluster administration interface (nsqadmin)
Statistics
GitHub Stars 13.2K	GitHub Stars 27.5K	GitHub Stars -
GitHub Forks 4.0K	GitHub Forks 4.9K	GitHub Forks -
Stacks 21.8K	Stacks 1.7K	Stacks 142
Followers 18.9K	Followers 1.6K	Followers 356
Votes 558	Votes 280	Votes 148
Pros & Cons
Pros 235 It's fast and it works with good metrics/monitoring 80 Ease of configuration 60 I like the admin interface 52 Easy to set-up and start with 22 Durable Cons 9 Too complicated cluster/HA config and management 6 Needs Erlang runtime. Need ops good with Erlang runtime 5 Configuration must be done first, not by your code 4 Slow	Pros 99 Task queue 63 Python integration 40 Django integration 30 Scheduled Task 19 Publish/subsribe Cons 4 Sometimes loses tasks 1 Depends on broker	Pros 29 It's in golang 20 Lightweight 20 Distributed 18 Easy setup 17 High throughput Cons 1 Get NSQ behavior out of Kafka but not inverse 1 Long term persistence 1 HA

What are some alternatives to RabbitMQ, Celery, NSQ?

Kafka

Kafka is a distributed, partitioned, replicated commit log service. It provides the functionality of a messaging system, but with a unique design.

Amazon SQS

Transmit any volume of data, at any level of throughput, without losing messages or requiring other services to be always available. With SQS, you can offload the administrative burden of operating and scaling a highly available messaging cluster, while paying a low price for only what you use.

ActiveMQ

Apache ActiveMQ is fast, supports many Cross Language Clients and Protocols, comes with easy to use Enterprise Integration Patterns and many advanced features while fully supporting JMS 1.1 and J2EE 1.4. Apache ActiveMQ is released under the Apache 2.0 License.

ZeroMQ

The 0MQ lightweight messaging kernel is a library which extends the standard socket interfaces with features traditionally provided by specialised messaging middleware products. 0MQ sockets provide an abstraction of asynchronous message queues, multiple messaging patterns, message filtering (subscriptions), seamless access to multiple transport protocols and more.

Apache NiFi

An easy to use, powerful, and reliable system to process and distribute data. It supports powerful and scalable directed graphs of data routing, transformation, and system mediation logic.

Gearman

Gearman allows you to do work in parallel, to load balance processing, and to call functions between languages. It can be used in a variety of applications, from high-availability web sites to the transport of database replication events.

Memphis

Highly scalable and effortless data streaming platform. Made to enable developers and data teams to collaborate and build real-time and streaming apps fast.

IronMQ

An easy-to-use highly available message queuing service. Built for distributed cloud applications with critical messaging needs. Provides on-demand message queuing with advanced features and cloud-optimized performance.

Apache Pulsar

Apache Pulsar is a distributed messaging solution developed and released to open source at Yahoo. Pulsar supports both pub-sub messaging and queuing in a platform designed for performance, scalability, and ease of development and operation.

Confluent

It is a data streaming platform based on Apache Kafka: a full-scale streaming platform, capable of not only publish-and-subscribe, but also the storage and processing of data within the stream

Related Comparisons

Celery vs NSQ vs RabbitMQ: What are the differences?

Key Differences between Celery, NSQ, and RabbitMQ

Messaging Protocol: Celery primarily uses AMQP (Advanced Message Queuing Protocol) whereas NSQ relies on its own proprietary protocol. RabbitMQ also supports AMQP but can be configured to work with other messaging protocols as well.
Scalability and Performance: NSQ is known for its high scalability and performance, designed to handle large message volumes efficiently. Celery and RabbitMQ are also scalable but may require additional configurations for high performance in comparison to NSQ.
Message Delivery Guarantees: NSQ provides at-least-once message delivery guarantees, ensuring that messages are delivered at least once to consumers. Celery, by default, offers at-most-once delivery, which can result in message loss if not handled carefully. RabbitMQ offers different delivery modes for guaranteed message delivery, giving more flexibility in choosing the desired level of guarantee.
Ease of Use and Configuration: Celery is known for its ease of use and simple configuration, making it suitable for smaller projects or rapid deployment. RabbitMQ, while versatile, may require more complex configurations for specific use cases. NSQ falls in between, offering a balance of ease of use and performance-oriented configuration options.
Community and Ecosystem: RabbitMQ has a mature and extensive community with a wide range of plugins and integrations available. Celery also has a strong community support but may not have as many diverse integration options as RabbitMQ. NSQ, being relatively newer, has a growing community and a focus on simplicity and performance.
Fault Tolerance and Monitoring: RabbitMQ provides robust fault tolerance features like clustering and HA (High Availability) queues for reliability. NSQ also offers fault tolerance mechanisms but may require manual intervention for certain scenarios. Celery's fault tolerance capabilities depend on the underlying message broker being used, such as RabbitMQ or Redis for persistence.

Celery vs NSQ vs RabbitMQ

Overview