Need advice about which tool to choose?Ask the StackShare community!

Amazon MQ

55
321
+ 1
12
Celery

1.7K
1.6K
+ 1
280
Add tool

Amazon MQ vs Celery: What are the differences?

  1. Message Brokering Mechanism: Amazon MQ is a managed message broker service that uses Apache ActiveMQ, an open-source message broker. Celery, on the other hand, is a distributed task queue that uses message brokers such as RabbitMQ, Redis, or even Amazon SQS with additional configuration. Amazon MQ provides a fully managed solution, while Celery allows more flexibility in choosing the underlying message broker.

  2. Scalability and Performance: Amazon MQ can scale horizontally to handle high throughput and can be configured with multiple brokers and message instances for improved performance. Celery offers scalability by distributing tasks across multiple worker nodes and message brokers, allowing for parallel processing of tasks. Amazon MQ may provide a more robust and reliable performance due to being a managed service.

  3. Pricing and Costs: Amazon MQ follows a usage-based pricing model where customers pay based on the number of broker instances and data transfer. Celery is an open-source framework, so there are no direct costs associated with using Celery itself; however, users may have to incur costs for managing and scaling the underlying message brokers like RabbitMQ or Redis.

  4. Programming Language Support: Amazon MQ supports messaging protocols like AMQP, STOMP, MQTT, and OpenWire, making it versatile for different programming languages and application integrations. Celery is designed for Python applications and integrates seamlessly with Django, Flask, and other Python frameworks, limiting its compatibility with other programming languages.

  5. Monitoring and Management: Amazon MQ provides built-in monitoring and management tools through the AWS Management Console, enabling users to monitor message queues, set access policies, and configure alarms for metrics. Celery requires additional monitoring tools and configurations for tracking task states, monitoring worker nodes, and managing the overall task queue system.

  6. Ecosystem and Community Support: Amazon MQ being part of the AWS ecosystem enjoys strong support and integration with other AWS services, making it preferable for users already using AWS infrastructure. Celery, being popular in the Python community, has a vast ecosystem of plugins, extensions, and community support for integrating with various frameworks and technologies beyond just messaging.

In Summary, Amazon MQ and Celery differ in their underlying technologies, scalability options, pricing models, language support, monitoring capabilities, and ecosystem integrations.

Advice on Amazon MQ and Celery
Needs advice
on
CeleryCelery
and
RabbitMQRabbitMQ

I am just a beginner at these two technologies.

Problem statement: I am getting lakh of users from the sequel server for whom I need to create caches in MongoDB by making different REST API requests.

Here these users can be treated as messages. Each REST API request is a task.

I am confused about whether I should go for RabbitMQ alone or Celery.

If I have to go with RabbitMQ, I prefer to use python with Pika module. But the challenge with Pika is, it is not thread-safe. So I am not finding a way to execute a lakh of API requests in parallel using multiple threads using Pika.

If I have to go with Celery, I don't know how I can achieve better scalability in executing these API requests in parallel.

See more
Replies (1)
Recommends
on
rqrqRedisRedis

For large amounts of small tasks and caches I have had good luck with Redis and RQ. I have not personally used celery but I am fairly sure it would scale well, and I have not used RabbitMQ for anything besides communication between services. If you prefer python my suggestions should feel comfortable.

Sorry I do not have a more information

See more
MITHIRIDI PRASANTH
Software Engineer at LightMetrics · | 4 upvotes · 269.9K views
Needs advice
on
Amazon MQAmazon MQ
and
Amazon SQSAmazon SQS
in

I want to schedule a message. Amazon SQS provides a delay of 15 minutes, but I want it in some hours.

Example: Let's say a Message1 is consumed by a consumer A but somehow it failed inside the consumer. I would want to put it in a queue and retry after 4hrs. Can I do this in Amazon MQ? I have seen in some Amazon MQ videos saying scheduling messages can be done. But, I'm not sure how.

See more
Replies (1)
Andres Paredes
Lead Senior Software Engineer at InTouch Technology · | 1 upvotes · 206.4K views
Recommends
on
Amazon SQSAmazon SQS

Mithiridi, I believe you are talking about two different things. 1. If you need to process messages with delays of more 15m or at specific times, it's not a good idea to use queues, independently of tool SQM, Rabbit or Amazon MQ. you should considerer another approach using a scheduled job. 2. For dead queues and policy retries RabbitMQ, for example, doesn't support your use case. https://medium.com/@kiennguyen88/rabbitmq-delay-retry-schedule-with-dead-letter-exchange-31fb25a440fc I'm not sure if that is possible SNS/SQS support, they have a maximum delay for delivery (maxDelayTarget) in seconds but it's not clear the number. You can check this out: https://docs.aws.amazon.com/sns/latest/dg/sns-message-delivery-retries.html

See more
Get Advice from developers at your company using StackShare Enterprise. Sign up for StackShare Enterprise.
Learn More
Pros of Amazon MQ
Pros of Celery
  • 7
    Supports low IQ developers
  • 3
    Supports existing protocols (JMS, NMS, AMQP, STOMP, …)
  • 2
    Easy to migrate existing messaging service
  • 99
    Task queue
  • 63
    Python integration
  • 40
    Django integration
  • 30
    Scheduled Task
  • 19
    Publish/subsribe
  • 8
    Various backend broker
  • 6
    Easy to use
  • 5
    Great community
  • 5
    Workflow
  • 4
    Free
  • 1
    Dynamic

Sign up to add or upvote prosMake informed product decisions

Cons of Amazon MQ
Cons of Celery
  • 4
    Slow AF
  • 4
    Sometimes loses tasks
  • 1
    Depends on broker

Sign up to add or upvote consMake informed product decisions

- No public GitHub repository available -

What is Amazon MQ?

Amazon MQ is a managed message broker service for Apache ActiveMQ that makes it easy to set up and operate message brokers in the cloud.

What is Celery?

Celery is an asynchronous task queue/job queue based on distributed message passing. It is focused on real-time operation, but supports scheduling as well.

Need advice about which tool to choose?Ask the StackShare community!

What companies use Amazon MQ?
What companies use Celery?
See which teams inside your own company are using Amazon MQ or Celery.
Sign up for StackShare EnterpriseLearn More

Sign up to get full access to all the companiesMake informed product decisions

What tools integrate with Amazon MQ?
What tools integrate with Celery?

Sign up to get full access to all the tool integrationsMake informed product decisions

Blog Posts

GitHubPythonNode.js+47
54
72279
JavaScriptGitHubPython+42
53
21800
GitHubPythonSlack+25
7
3148
GitHubPythonDocker+24
13
17001
What are some alternatives to Amazon MQ and Celery?
Amazon SQS
Transmit any volume of data, at any level of throughput, without losing messages or requiring other services to be always available. With SQS, you can offload the administrative burden of operating and scaling a highly available messaging cluster, while paying a low price for only what you use.
RabbitMQ
RabbitMQ gives your applications a common platform to send and receive messages, and your messages a safe place to live until received.
IBM MQ
It is a messaging middleware that simplifies and accelerates the integration of diverse applications and business data across multiple platforms. It offers proven, enterprise-grade messaging capabilities that skillfully and safely move information.
ActiveMQ
Apache ActiveMQ is fast, supports many Cross Language Clients and Protocols, comes with easy to use Enterprise Integration Patterns and many advanced features while fully supporting JMS 1.1 and J2EE 1.4. Apache ActiveMQ is released under the Apache 2.0 License.
Kafka
Kafka is a distributed, partitioned, replicated commit log service. It provides the functionality of a messaging system, but with a unique design.
See all alternatives