Amazon MQ vs Celery

Need advice about which tool to choose?Ask the StackShare community!

Amazon MQ

42
248
+ 1
11
Celery

1.3K
1.3K
+ 1
265
Add tool

Amazon MQ vs Celery: What are the differences?

Amazon MQ: Managed Message Broker Service for ActiveMQ. Amazon MQ is a managed message broker service for Apache ActiveMQ that makes it easy to set up and operate message brokers in the cloud; Celery: Distributed task queue. Celery is an asynchronous task queue/job queue based on distributed message passing. It is focused on real-time operation, but supports scheduling as well.

Amazon MQ and Celery can be primarily classified as "Message Queue" tools.

Celery is an open source tool with 12.9K GitHub stars and 3.33K GitHub forks. Here's a link to Celery's open source repository on GitHub.

Advice on Amazon MQ and Celery
Needs advice
on
RabbitMQ
and
Celery

I am just a beginner at these two technologies.

Problem statement: I am getting lakh of users from the sequel server for whom I need to create caches in MongoDB by making different REST API requests.

Here these users can be treated as messages. Each REST API request is a task.

I am confused about whether I should go for RabbitMQ alone or Celery.

If I have to go with RabbitMQ, I prefer to use python with Pika module. But the challenge with Pika is, it is not thread-safe. So I am not finding a way to execute a lakh of API requests in parallel using multiple threads using Pika.

If I have to go with Celery, I don't know how I can achieve better scalability in executing these API requests in parallel.

See more
Replies (1)
Recommends
Redis
rq

For large amounts of small tasks and caches I have had good luck with Redis and RQ. I have not personally used celery but I am fairly sure it would scale well, and I have not used RabbitMQ for anything besides communication between services. If you prefer python my suggestions should feel comfortable.

Sorry I do not have a more information

See more
MITHIRIDI PRASANTH
Software Engineer at LightMetrics · | 4 upvotes · 95.7K views
Needs advice
on
Amazon SQS
and
Amazon MQ
in

I want to schedule a message. Amazon SQS provides a delay of 15 minutes, but I want it in some hours.

Example: Let's say a Message1 is consumed by a consumer A but somehow it failed inside the consumer. I would want to put it in a queue and retry after 4hrs. Can I do this in Amazon MQ? I have seen in some Amazon MQ videos saying scheduling messages can be done. But, I'm not sure how.

See more
Replies (1)
Andres Paredes
Lead Senior Software Engineer at InTouch Technology · | 1 upvotes · 72.1K views
Recommends
Amazon SQS

Mithiridi, I believe you are talking about two different things. 1. If you need to process messages with delays of more 15m or at specific times, it's not a good idea to use queues, independently of tool SQM, Rabbit or Amazon MQ. you should considerer another approach using a scheduled job. 2. For dead queues and policy retries RabbitMQ, for example, doesn't support your use case. https://medium.com/@kiennguyen88/rabbitmq-delay-retry-schedule-with-dead-letter-exchange-31fb25a440fc I'm not sure if that is possible SNS/SQS support, they have a maximum delay for delivery (maxDelayTarget) in seconds but it's not clear the number. You can check this out: https://docs.aws.amazon.com/sns/latest/dg/sns-message-delivery-retries.html

See more
Get Advice from developers at your company using Private StackShare. Sign up for Private StackShare.
Learn More
Pros of Amazon MQ
Pros of Celery
  • 7
    Supports low IQ developers
  • 2
    Supports existing protocols (JMS, NMS, AMQP, STOMP, …)
  • 2
    Easy to migrate existing messaging service
  • 94
    Task queue
  • 61
    Python integration
  • 37
    Django integration
  • 29
    Scheduled Task
  • 18
    Publish/subsribe
  • 6
    Easy to use
  • 6
    Various backend broker
  • 5
    Great community
  • 4
    Workflow
  • 4
    Free
  • 1
    Dynamic

Sign up to add or upvote prosMake informed product decisions

Cons of Amazon MQ
Cons of Celery
  • 3
    Slow AF
  • 4
    Sometimes loses tasks
  • 1
    Depends on broker

Sign up to add or upvote consMake informed product decisions

- No public GitHub repository available -

What is Amazon MQ?

Amazon MQ is a managed message broker service for Apache ActiveMQ that makes it easy to set up and operate message brokers in the cloud.

What is Celery?

Celery is an asynchronous task queue/job queue based on distributed message passing. It is focused on real-time operation, but supports scheduling as well.

Need advice about which tool to choose?Ask the StackShare community!

What companies use Amazon MQ?
What companies use Celery?
See which teams inside your own company are using Amazon MQ or Celery.
Sign up for Private StackShareLearn More

Sign up to get full access to all the companiesMake informed product decisions

What tools integrate with Amazon MQ?
What tools integrate with Celery?

Blog Posts

+47
49
69236
+42
52
19818
+25
7
2823
+24
13
16726
What are some alternatives to Amazon MQ and Celery?
Amazon SQS
Transmit any volume of data, at any level of throughput, without losing messages or requiring other services to be always available. With SQS, you can offload the administrative burden of operating and scaling a highly available messaging cluster, while paying a low price for only what you use.
RabbitMQ
RabbitMQ gives your applications a common platform to send and receive messages, and your messages a safe place to live until received.
IBM MQ
It is a messaging middleware that simplifies and accelerates the integration of diverse applications and business data across multiple platforms. It offers proven, enterprise-grade messaging capabilities that skillfully and safely move information.
ActiveMQ
Apache ActiveMQ is fast, supports many Cross Language Clients and Protocols, comes with easy to use Enterprise Integration Patterns and many advanced features while fully supporting JMS 1.1 and J2EE 1.4. Apache ActiveMQ is released under the Apache 2.0 License.
Kafka
Kafka is a distributed, partitioned, replicated commit log service. It provides the functionality of a messaging system, but with a unique design.
See all alternatives