Need advice about which tool to choose?Ask the StackShare community!
Amazon SQS vs NSQ: What are the differences?
Introduction
In this article, we will discuss the key differences between Amazon Simple Queue Service (SQS) and NSQ. Both Amazon SQS and NSQ are messaging systems that facilitate communication between distributed systems and microservices. However, they have some distinct features and functionalities that set them apart.
Message Persistence: One key difference between Amazon SQS and NSQ is how they handle message persistence. Amazon SQS stores messages in a durable and highly available manner, ensuring that messages are not lost even if a system or network failure occurs. On the other hand, NSQ does not provide built-in message persistence. It relies on the message consumers to handle the persistence of messages if required.
Scaling and Autoscaling: Amazon SQS has built-in capabilities for scaling and autoscaling. It can automatically scale based on the incoming message traffic and the number of concurrent consumers. NSQ, on the other hand, does not have built-in scaling and autoscaling capabilities. Scaling and load balancing in NSQ need to be managed manually.
Delivery Guarantee: Another important difference between Amazon SQS and NSQ is the delivery guarantee they provide. Amazon SQS guarantees at-least-once delivery of messages. It ensures that a message will be delivered to a consumer at least once, but duplicates may occur. NSQ, on the other hand, provides at-most-once delivery guarantee. It does not guarantee that a message will be delivered, and it may be lost if a consumer fails to acknowledge the delivery.
Message Ordering: Amazon SQS guarantees the ordering of messages within a single message group, allowing you to maintain a strict message order. NSQ does not provide any out-of-the-box mechanism to enforce message ordering. If message ordering is crucial for your application, you need to implement a custom solution on top of NSQ.
Visibility Timeout: In Amazon SQS, when a message is being processed by a consumer, the message is temporarily made invisible to other consumers. This is controlled by a configurable visibility timeout. NSQ does not have built-in support for visibility timeout. The visibility of a message in NSQ is controlled solely by the consumer, which needs to handle the visibility of the message explicitly.
Managed Service: Amazon SQS is a managed service provided by Amazon Web Services (AWS). It handles the operational aspects of running a message queue, such as provisioning resources, scaling, and monitoring. NSQ, on the other hand, is a self-hosted solution where you have to manage the infrastructure and operational aspects yourself.
In summary, the key differences between Amazon SQS and NSQ can be summarized as follows: Amazon SQS provides message persistence, scaling and autoscaling capabilities, at-least-once delivery guarantee, message ordering support, visibility timeout feature, and is a managed service. NSQ, on the other hand, does not have built-in message persistence, scaling and autoscaling capabilities, provides at-most-once delivery guarantee, lacks message ordering support, does not have a visibility timeout feature, and is a self-hosted solution.
Hi! I am creating a scraping system in Django, which involves long running tasks between 1 minute & 1 Day. As I am new to Message Brokers and Task Queues, I need advice on which architecture to use for my system. ( Amazon SQS, RabbitMQ, or Celery). The system should be autoscalable using Kubernetes(K8) based on the number of pending tasks in the queue.
Hello, i highly recommend Apache Kafka, to me it's the best. You can deploy it in cluster mode inside K8S, thus you can have a Highly available system (also auto scalable).
Good luck
Hi, we are in a ZMQ set up in a push/pull pattern, and we currently start to have more traffic and cases that the service is unavailable or stuck. We want to: * Not loose messages in services outages * Safely restart service without losing messages (ZeroMQ seems to need to close the socket in the receiver before restart manually)
Do you have experience with this setup with ZeroMQ? Would you suggest RabbitMQ or Amazon SQS (we are in AWS setup) instead? Something else?
Thank you for your time
ZeroMQ is fast but you need to build build reliability yourself. There are a number of patterns described in the zeromq guide. I have used RabbitMQ before which gives lot of functionality out of the box, you can probably use the worker queues example from the tutorial, it can also persists messages in the queue.
I haven't used Amazon SQS before. Another tool you could use is Kafka.
Both would do the trick, but there are some nuances. We work with both.
From the sound of it, your main focus is "not losing messages". In that case, I would go with RabbitMQ with a high availability policy (ha-mode=all) and a main/retry/error queue pattern.
Push messages to an exchange, which sends them to the main queue. If an error occurs, push the errored out message to the retry exchange, which forwards it to the retry queue. Give the retry queue a x-message-ttl and set the main exchange as a dead-letter-exchange. If your message has been retried several times, push it to the error exchange, where the message can remain until someone has time to look at it.
This is a very useful and resilient pattern that allows you to never lose messages. With the high availability policy, you make sure that if one of your rabbitmq nodes dies, another can take over and messages are already mirrored to it.
This is not really possible with SQS, because SQS is a lot more focused on throughput and scaling. Combined with SNS it can do interesting things like deduplication of messages and such. That said, one thing core to its design is that messages have a maximum retention time. The idea is that a message that has stayed in an SQS queue for a while serves no more purpose after a while, so it gets removed - so as to not block up any listener resources for a long time. You can also set up a DLQ here, but these similarly do not hold onto messages forever. Since you seem to depend on messages surviving at all cost, I would suggest that the scaling/throughput benefit of SQS does not outweigh the difference in approach to messages there.
I want to schedule a message. Amazon SQS provides a delay of 15 minutes, but I want it in some hours.
Example: Let's say a Message1 is consumed by a consumer A but somehow it failed inside the consumer. I would want to put it in a queue and retry after 4hrs. Can I do this in Amazon MQ? I have seen in some Amazon MQ videos saying scheduling messages can be done. But, I'm not sure how.
Mithiridi, I believe you are talking about two different things. 1. If you need to process messages with delays of more 15m or at specific times, it's not a good idea to use queues, independently of tool SQM, Rabbit or Amazon MQ. you should considerer another approach using a scheduled job. 2. For dead queues and policy retries RabbitMQ, for example, doesn't support your use case. https://medium.com/@kiennguyen88/rabbitmq-delay-retry-schedule-with-dead-letter-exchange-31fb25a440fc I'm not sure if that is possible SNS/SQS support, they have a maximum delay for delivery (maxDelayTarget) in seconds but it's not clear the number. You can check this out: https://docs.aws.amazon.com/sns/latest/dg/sns-message-delivery-retries.html
I am looking into IoT World Solution where we have MQTT Broker. This MQTT Broker Sits in one of the Data Center. We are doing a lot of Alert and Alarm related processing on that Data, Currently, we are looking into Solution which can do distributed persistence of log/alert primarily on remote Disk.
Our primary need is to use lightweight where operational complexity and maintenance costs can be significantly reduced. We want to do it on-premise so we are not considering cloud solutions.
We looked into the following alternatives:
Apache Kafka - Great choice but operation and maintenance wise very complex. Rabbit MQ - High availability is the issue, Apache Pulsar - Operational Complexity. NATS - Absence of persistence. Akka Streams - Big learning curve and operational streams.
So we are looking into a lightweight library that can do distributed persistence preferably with publisher and subscriber model. Preferable on JVM stack.
Kafka is best fit here. Below are the advantages with Kafka ACLs (Security), Schema (protobuf), Scale, Consumer driven and No single point of failure.
Operational complexity is manageable with open source monitoring tools.