What is MSMQ and what are its top alternatives?
MSMQ, also known as Microsoft Message Queuing, is a messaging protocol developed by Microsoft that allows applications running on separate servers to communicate asynchronously. Its key features include reliable message delivery, support for transactional messaging, and easy integration with Windows-based applications. However, MSMQ is limited to Windows operating systems, lacks support for modern communication protocols like RESTful APIs, and can be complex to set up and manage.
- RabbitMQ: RabbitMQ is an open-source message broker software that supports multiple messaging protocols such as AMQP, MQTT, and STOMP. Key features include message queuing, routing, clustering, and high availability. Pros: Easy to set up and use, strong community support. Cons: Requires knowledge of Erlang programming language.
- Apache Kafka: Apache Kafka is a distributed streaming platform that is used for building real-time data pipelines and streaming applications. It provides high-throughput, fault-tolerant messaging capabilities for large-scale data processing. Pros: Scalable and fault-tolerant, performs well in high-throughput environments. Cons: Complex setup and configuration.
- ActiveMQ: ActiveMQ is an open-source message broker that supports multiple messaging protocols and provides features like JMS, MQTT, and STOMP support. It offers message queuing, publish-subscribe messaging, and integration with Spring Framework. Pros: Easy integration with Java applications, robust message delivery. Cons: Limited community support.
- ZeroMQ: ZeroMQ is a high-performance messaging library that provides sockets for building distributed applications. It supports various messaging patterns like request-reply, publish-subscribe, and pipeline. Pros: Lightweight and fast, supports multiple languages. Cons: Lack of built-in message persistence.
- NATS: NATS is a high-performance messaging system designed for simplicity and performance. It offers support for publish-subscribe messaging, request-reply communication, and distributed queuing. Pros: Lightweight and easy to deploy, good performance for simple messaging scenarios. Cons: Limited features compared to other platforms.
- Redis: Redis is an in-memory data structure store that can be used as a message broker through its pub/sub capabilities. It provides high performance, reliability, and support for multiple data types. Pros: Fast message delivery, easy to set up and use. Cons: Limited to in-memory storage.
- KubeMQ: KubeMQ is a Kubernetes-native message broker that provides messaging, event streaming, and data persistence in a single platform. It offers support for various messaging patterns and can be easily deployed on Kubernetes clusters. Pros: Built for Kubernetes environments, scalable and reliable messaging. Cons: Relatively new in the market, limited community adoption.
- NSQ: NSQ is a real-time distributed messaging platform designed to operate at scale. It features a decentralized architecture, message queuing, and native support for load-balancing and message routing. Pros: Scalable and fault-tolerant, simple configuration and monitoring. Cons: Limited features compared to other platforms.
- RocketMQ: RocketMQ is a distributed messaging and streaming platform developed by Alibaba Group. It supports millions of messages per second with low latency and high availability. Pros: High performance and reliability, built for cloud-native environments. Cons: Complex to set up and manage.
- Amazon SQS: Amazon Simple Queue Service (SQS) is a fully managed message queuing service provided by AWS. It offers reliable message delivery, scalability, and integration with other AWS services. Pros: Fully managed service, scalable and reliable messaging. Cons: Limited flexibility compared to self-hosted solutions.
Top Alternatives to MSMQ
- Kafka
Kafka is a distributed, partitioned, replicated commit log service. It provides the functionality of a messaging system, but with a unique design. ...
- RabbitMQ
RabbitMQ gives your applications a common platform to send and receive messages, and your messages a safe place to live until received. ...
- IBM MQ
It is a messaging middleware that simplifies and accelerates the integration of diverse applications and business data across multiple platforms. It offers proven, enterprise-grade messaging capabilities that skillfully and safely move information. ...
- Azure Service Bus
It is a cloud messaging system for connecting apps and devices across public and private clouds. You can depend on it when you need highly-reliable cloud messaging service between applications and services, even when one or more is offline. ...
- ActiveMQ
Apache ActiveMQ is fast, supports many Cross Language Clients and Protocols, comes with easy to use Enterprise Integration Patterns and many advanced features while fully supporting JMS 1.1 and J2EE 1.4. Apache ActiveMQ is released under the Apache 2.0 License. ...
- Redis
Redis is an open source (BSD licensed), in-memory data structure store, used as a database, cache, and message broker. Redis provides data structures such as strings, hashes, lists, sets, sorted sets with range queries, bitmaps, hyperloglogs, geospatial indexes, and streams. ...
- MQTT
It was designed as an extremely lightweight publish/subscribe messaging transport. It is useful for connections with remote locations where a small code footprint is required and/or network bandwidth is at a premium. ...
- WCF
It is a framework for building service-oriented applications. Using this, you can send data as asynchronous messages from one service endpoint to another. A service endpoint can be part of a continuously available service hosted by IIS, or it can be a service hosted in an application. ...
MSMQ alternatives & related posts
- High-throughput126
- Distributed119
- Scalable92
- High-Performance86
- Durable66
- Publish-Subscribe38
- Simple-to-use19
- Open source18
- Written in Scala and java. Runs on JVM12
- Message broker + Streaming system9
- KSQL4
- Avro schema integration4
- Robust4
- Suport Multiple clients3
- Extremely good parallelism constructs2
- Partioned, replayable log2
- Simple publisher / multi-subscriber model1
- Fun1
- Flexible1
- Non-Java clients are second-class citizens32
- Needs Zookeeper29
- Operational difficulties9
- Terrible Packaging5
related Kafka posts
The algorithms and data infrastructure at Stitch Fix is housed in #AWS. Data acquisition is split between events flowing through Kafka, and periodic snapshots of PostgreSQL DBs. We store data in an Amazon S3 based data warehouse. Apache Spark on Yarn is our tool of choice for data movement and #ETL. Because our storage layer (s3) is decoupled from our processing layer, we are able to scale our compute environment very elastically. We have several semi-permanent, autoscaling Yarn clusters running to serve our data processing needs. While the bulk of our compute infrastructure is dedicated to algorithmic processing, we also implemented Presto for adhoc queries and dashboards.
Beyond data movement and ETL, most #ML centric jobs (e.g. model training and execution) run in a similarly elastic environment as containers running Python and R code on Amazon EC2 Container Service clusters. The execution of batch jobs on top of ECS is managed by Flotilla, a service we built in house and open sourced (see https://github.com/stitchfix/flotilla-os).
At Stitch Fix, algorithmic integrations are pervasive across the business. We have dozens of data products actively integrated systems. That requires serving layer that is robust, agile, flexible, and allows for self-service. Models produced on Flotilla are packaged for deployment in production using Khan, another framework we've developed internally. Khan provides our data scientists the ability to quickly productionize those models they've developed with open source frameworks in Python 3 (e.g. PyTorch, sklearn), by automatically packaging them as Docker containers and deploying to Amazon ECS. This provides our data scientist a one-click method of getting from their algorithms to production. We then integrate those deployments into a service mesh, which allows us to A/B test various implementations in our product.
For more info:
- Our Algorithms Tour: https://algorithms-tour.stitchfix.com/
- Our blog: https://multithreaded.stitchfix.com/blog/
- Careers: https://multithreaded.stitchfix.com/careers/
#DataScience #DataStack #Data
As we've evolved or added additional infrastructure to our stack, we've biased towards managed services. Most new backing stores are Amazon RDS instances now. We do use self-managed PostgreSQL with TimescaleDB for time-series data—this is made HA with the use of Patroni and Consul.
We also use managed Amazon ElastiCache instances instead of spinning up Amazon EC2 instances to run Redis workloads, as well as shifting to Amazon Kinesis instead of Kafka.
- It's fast and it works with good metrics/monitoring234
- Ease of configuration79
- I like the admin interface59
- Easy to set-up and start with50
- Durable21
- Intuitive work through python18
- Standard protocols18
- Written primarily in Erlang10
- Simply superb8
- Completeness of messaging patterns6
- Scales to 1 million messages per second3
- Reliable3
- Distributed2
- Supports MQTT2
- Better than most traditional queue based message broker2
- Supports AMQP2
- Clusterable1
- Clear documentation with different scripting language1
- Great ui1
- Inubit Integration1
- Better routing system1
- High performance1
- Runs on Open Telecom Platform1
- Delayed messages1
- Reliability1
- Open-source1
- Too complicated cluster/HA config and management9
- Needs Erlang runtime. Need ops good with Erlang runtime6
- Configuration must be done first, not by your code5
- Slow4
related RabbitMQ posts
As Sentry runs throughout the day, there are about 50 different offline tasks that we execute—anything from “process this event, pretty please” to “send all of these cool people some emails.” There are some that we execute once a day and some that execute thousands per second.
Managing this variety requires a reliably high-throughput message-passing technology. We use Celery's RabbitMQ implementation, and we stumbled upon a great feature called Federation that allows us to partition our task queue across any number of RabbitMQ servers and gives us the confidence that, if any single server gets backlogged, others will pitch in and distribute some of the backlogged tasks to their consumers.
#MessageQueue
Around the time of their Series A, Pinterest’s stack included Python and Django, with Tornado and Node.js as web servers. Memcached / Membase and Redis handled caching, with RabbitMQ handling queueing. Nginx, HAproxy and Varnish managed static-delivery and load-balancing, with persistent data storage handled by MySQL.
- Reliable for banking transactions3
- Useful for big enteprises3
- Secure2
- Broader connectivity - more protocols, APIs, Files etc1
- Many deployment options (containers, cloud, VM etc)1
- High Availability1
- Cost2
related IBM MQ posts
Want to get the differences in features and enhancement, pros and cons, and also how to Migrate from IBM MQ to Azure Service Bus.
- Easy Integration with .Net4
- Cloud Native2
- Use while high messaging need1
- Limited features in Basic tier1
- Skills can only be used in Azure - vendor lock-in1
- Lacking in JMS support1
- Observability of messages in the queue is lacking1
related Azure Service Bus posts
Want to get the differences in features and enhancement, pros and cons, and also how to Migrate from IBM MQ to Azure Service Bus.
- Easy to use18
- Open source14
- Efficient13
- JMS compliant10
- High Availability6
- Scalable5
- Distributed Network of brokers3
- Persistence3
- Support XA (distributed transactions)3
- Docker delievery1
- Highly configurable1
- RabbitMQ0
- ONLY Vertically Scalable1
- Support1
- Low resilience to exceptions and interruptions1
- Difficult to scale1
related ActiveMQ posts
I want to choose Message Queue with the following features - Highly Available, Distributed, Scalable, Monitoring. I have RabbitMQ, ActiveMQ, Kafka and Apache RocketMQ in mind. But I am confused which one to choose.
I use ActiveMQ because RabbitMQ have stopped giving the support for AMQP 1.0 or above version and the earlier version of AMQP doesn't give the functionality to support OAuth.
If OAuth is not required and we can go with AMQP 0.9 then i still recommend rabbitMq.
- Performance886
- Super fast542
- Ease of use513
- In-memory cache444
- Advanced key-value cache324
- Open source194
- Easy to deploy182
- Stable164
- Free155
- Fast121
- High-Performance42
- High Availability40
- Data Structures35
- Very Scalable32
- Replication24
- Great community22
- Pub/Sub22
- "NoSQL" key-value data store19
- Hashes16
- Sets13
- Sorted Sets11
- NoSQL10
- Lists10
- Async replication9
- BSD licensed9
- Bitmaps8
- Integrates super easy with Sidekiq for Rails background8
- Keys with a limited time-to-live7
- Open Source7
- Lua scripting6
- Strings6
- Awesomeness for Free5
- Hyperloglogs5
- Transactions4
- Outstanding performance4
- Runs server side LUA4
- LRU eviction of keys4
- Feature Rich4
- Written in ANSI C4
- Networked4
- Data structure server3
- Performance & ease of use3
- Dont save data if no subscribers are found2
- Automatic failover2
- Easy to use2
- Temporarily kept on disk2
- Scalable2
- Existing Laravel Integration2
- Channels concept2
- Object [key/value] size each 500 MB2
- Simple2
- Cannot query objects directly15
- No secondary indexes for non-numeric data types3
- No WAL1
related Redis posts
We use MongoDB as our primary #datastore. Mongo's approach to replica sets enables some fantastic patterns for operations like maintenance, backups, and #ETL.
As we pull #microservices from our #monolith, we are taking the opportunity to build them with their own datastores using PostgreSQL. We also use Redis to cache data we’d never store permanently, and to rate-limit our requests to partners’ APIs (like GitHub).
When we’re dealing with large blobs of immutable data (logs, artifacts, and test results), we store them in Amazon S3. We handle any side-effects of S3’s eventual consistency model within our own code. This ensures that we deal with user requests correctly while writes are in process.
I'm working as one of the engineering leads in RunaHR. As our platform is a Saas, we thought It'd be good to have an API (We chose Ruby and Rails for this) and a SPA (built with React and Redux ) connected. We started the SPA with Create React App since It's pretty easy to start.
We use Jest as the testing framework and react-testing-library to test React components. In Rails we make tests using RSpec.
Our main database is PostgreSQL, but we also use MongoDB to store some type of data. We started to use Redis for cache and other time sensitive operations.
We have a couple of extra projects: One is an Employee app built with React Native and the other is an internal back office dashboard built with Next.js for the client and Python in the backend side.
Since we have different frontend apps we have found useful to have Bit to document visual components and utils in JavaScript.
- Varying levels of Quality of Service to fit a range of3
- Lightweight with a relatively small data footprint2
- Very easy to configure and use with open source tools2
- Easy to configure in an unsecure manner1
related MQTT posts
Kindly suggest the best tool for generating 10Mn+ concurrent user load. The tool must support MQTT traffic, REST API, support to interfaces such as Kafka, websockets, persistence HTTP connection, auth type support to assess the support /coverage.
The tool can be integrated into CI pipelines like Azure Pipelines, GitHub, and Jenkins.
Hi Marc,
For the com part, depending of more details not provided, i'd use SSE, OR i'd run either Mosquitto or RabbitMQ running on Amazon EC2 instances and leverage MQTT or amqp 's subscribe/publish features with my users running mqtt or amqp clients (tcp or websockets) somehow. (publisher too.. you don't say how and who gets to update the document(s).
I find "a ton of end users", depending on how you define a ton (1k users ;) ?) and how frequent document updates are, that can mean a ton of ressources, can't cut it at some point, even using SSE
how many, how big, how persistant do the document(s) have to be ? Db-wise,can't say for lack of details and context, yeah could also be Redis, any RDBMS or nosql or even static json files stored on an Amazon S3 bucket .. anything really
Good luck!