Need advice about which tool to choose?Ask the StackShare community!
Amazon MQ vs Celery: What are the differences?
Message Brokering Mechanism: Amazon MQ is a managed message broker service that uses Apache ActiveMQ, an open-source message broker. Celery, on the other hand, is a distributed task queue that uses message brokers such as RabbitMQ, Redis, or even Amazon SQS with additional configuration. Amazon MQ provides a fully managed solution, while Celery allows more flexibility in choosing the underlying message broker.
Scalability and Performance: Amazon MQ can scale horizontally to handle high throughput and can be configured with multiple brokers and message instances for improved performance. Celery offers scalability by distributing tasks across multiple worker nodes and message brokers, allowing for parallel processing of tasks. Amazon MQ may provide a more robust and reliable performance due to being a managed service.
Pricing and Costs: Amazon MQ follows a usage-based pricing model where customers pay based on the number of broker instances and data transfer. Celery is an open-source framework, so there are no direct costs associated with using Celery itself; however, users may have to incur costs for managing and scaling the underlying message brokers like RabbitMQ or Redis.
Programming Language Support: Amazon MQ supports messaging protocols like AMQP, STOMP, MQTT, and OpenWire, making it versatile for different programming languages and application integrations. Celery is designed for Python applications and integrates seamlessly with Django, Flask, and other Python frameworks, limiting its compatibility with other programming languages.
Monitoring and Management: Amazon MQ provides built-in monitoring and management tools through the AWS Management Console, enabling users to monitor message queues, set access policies, and configure alarms for metrics. Celery requires additional monitoring tools and configurations for tracking task states, monitoring worker nodes, and managing the overall task queue system.
Ecosystem and Community Support: Amazon MQ being part of the AWS ecosystem enjoys strong support and integration with other AWS services, making it preferable for users already using AWS infrastructure. Celery, being popular in the Python community, has a vast ecosystem of plugins, extensions, and community support for integrating with various frameworks and technologies beyond just messaging.
In Summary, Amazon MQ and Celery differ in their underlying technologies, scalability options, pricing models, language support, monitoring capabilities, and ecosystem integrations.
I am just a beginner at these two technologies.
Problem statement: I am getting lakh of users from the sequel server for whom I need to create caches in MongoDB by making different REST API requests.
Here these users can be treated as messages. Each REST API request is a task.
I am confused about whether I should go for RabbitMQ alone or Celery.
If I have to go with RabbitMQ, I prefer to use python with Pika module. But the challenge with Pika is, it is not thread-safe. So I am not finding a way to execute a lakh of API requests in parallel using multiple threads using Pika.
If I have to go with Celery, I don't know how I can achieve better scalability in executing these API requests in parallel.
For large amounts of small tasks and caches I have had good luck with Redis and RQ. I have not personally used celery but I am fairly sure it would scale well, and I have not used RabbitMQ for anything besides communication between services. If you prefer python my suggestions should feel comfortable.
Sorry I do not have a more information
I want to schedule a message. Amazon SQS provides a delay of 15 minutes, but I want it in some hours.
Example: Let's say a Message1 is consumed by a consumer A but somehow it failed inside the consumer. I would want to put it in a queue and retry after 4hrs. Can I do this in Amazon MQ? I have seen in some Amazon MQ videos saying scheduling messages can be done. But, I'm not sure how.
Mithiridi, I believe you are talking about two different things. 1. If you need to process messages with delays of more 15m or at specific times, it's not a good idea to use queues, independently of tool SQM, Rabbit or Amazon MQ. you should considerer another approach using a scheduled job. 2. For dead queues and policy retries RabbitMQ, for example, doesn't support your use case. https://medium.com/@kiennguyen88/rabbitmq-delay-retry-schedule-with-dead-letter-exchange-31fb25a440fc I'm not sure if that is possible SNS/SQS support, they have a maximum delay for delivery (maxDelayTarget) in seconds but it's not clear the number. You can check this out: https://docs.aws.amazon.com/sns/latest/dg/sns-message-delivery-retries.html
Pros of Amazon MQ
- Supports low IQ developers7
- Supports existing protocols (JMS, NMS, AMQP, STOMP, …)3
- Easy to migrate existing messaging service2
Pros of Celery
- Task queue99
- Python integration63
- Django integration40
- Scheduled Task30
- Publish/subsribe19
- Various backend broker8
- Easy to use6
- Great community5
- Workflow5
- Free4
- Dynamic1
Sign up to add or upvote prosMake informed product decisions
Cons of Amazon MQ
- Slow AF4
Cons of Celery
- Sometimes loses tasks4
- Depends on broker1
Sign up to add or upvote consMake informed product decisions
What is Amazon MQ?
What is Celery?
Need advice about which tool to choose?Ask the StackShare community!
What companies use Amazon MQ?
Sign up to get full access to all the companiesMake informed product decisions
What tools integrate with Amazon MQ?
Sign up to get full access to all the tool integrationsMake informed product decisions
Blog Posts
+47
+31+46+42
+25
+24