What is MSMQ and what are its top alternatives?
MSMQ, also known as Microsoft Message Queuing, is a messaging protocol developed by Microsoft that allows applications running on separate servers to communicate asynchronously. Its key features include reliable message delivery, support for transactional messaging, and easy integration with Windows-based applications. However, MSMQ is limited to Windows operating systems, lacks support for modern communication protocols like RESTful APIs, and can be complex to set up and manage.
- RabbitMQ: RabbitMQ is an open-source message broker software that supports multiple messaging protocols such as AMQP, MQTT, and STOMP. Key features include message queuing, routing, clustering, and high availability. Pros: Easy to set up and use, strong community support. Cons: Requires knowledge of Erlang programming language.
- Apache Kafka: Apache Kafka is a distributed streaming platform that is used for building real-time data pipelines and streaming applications. It provides high-throughput, fault-tolerant messaging capabilities for large-scale data processing. Pros: Scalable and fault-tolerant, performs well in high-throughput environments. Cons: Complex setup and configuration.
- ActiveMQ: ActiveMQ is an open-source message broker that supports multiple messaging protocols and provides features like JMS, MQTT, and STOMP support. It offers message queuing, publish-subscribe messaging, and integration with Spring Framework. Pros: Easy integration with Java applications, robust message delivery. Cons: Limited community support.
- ZeroMQ: ZeroMQ is a high-performance messaging library that provides sockets for building distributed applications. It supports various messaging patterns like request-reply, publish-subscribe, and pipeline. Pros: Lightweight and fast, supports multiple languages. Cons: Lack of built-in message persistence.
- NATS: NATS is a high-performance messaging system designed for simplicity and performance. It offers support for publish-subscribe messaging, request-reply communication, and distributed queuing. Pros: Lightweight and easy to deploy, good performance for simple messaging scenarios. Cons: Limited features compared to other platforms.
- Redis: Redis is an in-memory data structure store that can be used as a message broker through its pub/sub capabilities. It provides high performance, reliability, and support for multiple data types. Pros: Fast message delivery, easy to set up and use. Cons: Limited to in-memory storage.
- KubeMQ: KubeMQ is a Kubernetes-native message broker that provides messaging, event streaming, and data persistence in a single platform. It offers support for various messaging patterns and can be easily deployed on Kubernetes clusters. Pros: Built for Kubernetes environments, scalable and reliable messaging. Cons: Relatively new in the market, limited community adoption.
- NSQ: NSQ is a real-time distributed messaging platform designed to operate at scale. It features a decentralized architecture, message queuing, and native support for load-balancing and message routing. Pros: Scalable and fault-tolerant, simple configuration and monitoring. Cons: Limited features compared to other platforms.
- RocketMQ: RocketMQ is a distributed messaging and streaming platform developed by Alibaba Group. It supports millions of messages per second with low latency and high availability. Pros: High performance and reliability, built for cloud-native environments. Cons: Complex to set up and manage.
- Amazon SQS: Amazon Simple Queue Service (SQS) is a fully managed message queuing service provided by AWS. It offers reliable message delivery, scalability, and integration with other AWS services. Pros: Fully managed service, scalable and reliable messaging. Cons: Limited flexibility compared to self-hosted solutions.
Top Alternatives to MSMQ
- Kafka
Kafka is a distributed, partitioned, replicated commit log service. It provides the functionality of a messaging system, but with a unique design. ...
- RabbitMQ
RabbitMQ gives your applications a common platform to send and receive messages, and your messages a safe place to live until received. ...
- IBM MQ
It is a messaging middleware that simplifies and accelerates the integration of diverse applications and business data across multiple platforms. It offers proven, enterprise-grade messaging capabilities that skillfully and safely move information. ...
- Azure Service Bus
It is a cloud messaging system for connecting apps and devices across public and private clouds. You can depend on it when you need highly-reliable cloud messaging service between applications and services, even when one or more is offline. ...
- ActiveMQ
Apache ActiveMQ is fast, supports many Cross Language Clients and Protocols, comes with easy to use Enterprise Integration Patterns and many advanced features while fully supporting JMS 1.1 and J2EE 1.4. Apache ActiveMQ is released under the Apache 2.0 License. ...
- Redis
Redis is an open source (BSD licensed), in-memory data structure store, used as a database, cache, and message broker. Redis provides data structures such as strings, hashes, lists, sets, sorted sets with range queries, bitmaps, hyperloglogs, geospatial indexes, and streams. ...
- MQTT
It was designed as an extremely lightweight publish/subscribe messaging transport. It is useful for connections with remote locations where a small code footprint is required and/or network bandwidth is at a premium. ...
- WCF
It is a framework for building service-oriented applications. Using this, you can send data as asynchronous messages from one service endpoint to another. A service endpoint can be part of a continuously available service hosted by IIS, or it can be a service hosted in an application. ...
MSMQ alternatives & related posts
- High-throughput126
- Distributed119
- Scalable92
- High-Performance86
- Durable66
- Publish-Subscribe38
- Simple-to-use19
- Open source18
- Written in Scala and java. Runs on JVM12
- Message broker + Streaming system9
- KSQL4
- Avro schema integration4
- Robust4
- Suport Multiple clients3
- Extremely good parallelism constructs2
- Partioned, replayable log2
- Simple publisher / multi-subscriber model1
- Fun1
- Flexible1
- Non-Java clients are second-class citizens32
- Needs Zookeeper29
- Operational difficulties9
- Terrible Packaging5
related Kafka posts
When I joined NYT there was already broad dissatisfaction with the LAMP (Linux Apache HTTP Server MySQL PHP) Stack and the front end framework, in particular. So, I wasn't passing judgment on it. I mean, LAMP's fine, you can do good work in LAMP. It's a little dated at this point, but it's not ... I didn't want to rip it out for its own sake, but everyone else was like, "We don't like this, it's really inflexible." And I remember from being outside the company when that was called MIT FIVE when it had launched. And been observing it from the outside, and I was like, you guys took so long to do that and you did it so carefully, and yet you're not happy with your decisions. Why is that? That was more the impetus. If we're going to do this again, how are we going to do it in a way that we're gonna get a better result?
So we're moving quickly away from LAMP, I would say. So, right now, the new front end is React based and using Apollo. And we've been in a long, protracted, gradual rollout of the core experiences.
React is now talking to GraphQL as a primary API. There's a Node.js back end, to the front end, which is mainly for server-side rendering, as well.
Behind there, the main repository for the GraphQL server is a big table repository, that we call Bodega because it's a convenience store. And that reads off of a Kafka pipeline.
To provide employees with the critical need of interactive querying, we’ve worked with Presto, an open-source distributed SQL query engine, over the years. Operating Presto at Pinterest’s scale has involved resolving quite a few challenges like, supporting deeply nested and huge thrift schemas, slow/ bad worker detection and remediation, auto-scaling cluster, graceful cluster shutdown and impersonation support for ldap authenticator.
Our infrastructure is built on top of Amazon EC2 and we leverage Amazon S3 for storing our data. This separates compute and storage layers, and allows multiple compute clusters to share the S3 data.
We have hundreds of petabytes of data and tens of thousands of Apache Hive tables. Our Presto clusters are comprised of a fleet of 450 r4.8xl EC2 instances. Presto clusters together have over 100 TBs of memory and 14K vcpu cores. Within Pinterest, we have close to more than 1,000 monthly active users (out of total 1,600+ Pinterest employees) using Presto, who run about 400K queries on these clusters per month.
Each query submitted to Presto cluster is logged to a Kafka topic via Singer. Singer is a logging agent built at Pinterest and we talked about it in a previous post. Each query is logged when it is submitted and when it finishes. When a Presto cluster crashes, we will have query submitted events without corresponding query finished events. These events enable us to capture the effect of cluster crashes over time.
Each Presto cluster at Pinterest has workers on a mix of dedicated AWS EC2 instances and Kubernetes pods. Kubernetes platform provides us with the capability to add and remove workers from a Presto cluster very quickly. The best-case latency on bringing up a new worker on Kubernetes is less than a minute. However, when the Kubernetes cluster itself is out of resources and needs to scale up, it can take up to ten minutes. Some other advantages of deploying on Kubernetes platform is that our Presto deployment becomes agnostic of cloud vendor, instance types, OS, etc.
#BigData #AWS #DataScience #DataEngineering
- It's fast and it works with good metrics/monitoring235
- Ease of configuration80
- I like the admin interface60
- Easy to set-up and start with52
- Durable22
- Standard protocols19
- Intuitive work through python19
- Written primarily in Erlang11
- Simply superb9
- Completeness of messaging patterns7
- Reliable4
- Scales to 1 million messages per second4
- Better than most traditional queue based message broker3
- Distributed3
- Supports MQTT3
- Supports AMQP3
- Clear documentation with different scripting language2
- Better routing system2
- Inubit Integration2
- Great ui2
- High performance2
- Reliability2
- Open-source2
- Runs on Open Telecom Platform2
- Clusterable2
- Delayed messages2
- Supports Streams1
- Supports STOMP1
- Supports JMS1
- Too complicated cluster/HA config and management9
- Needs Erlang runtime. Need ops good with Erlang runtime6
- Configuration must be done first, not by your code5
- Slow4
related RabbitMQ posts
As Sentry runs throughout the day, there are about 50 different offline tasks that we execute—anything from “process this event, pretty please” to “send all of these cool people some emails.” There are some that we execute once a day and some that execute thousands per second.
Managing this variety requires a reliably high-throughput message-passing technology. We use Celery's RabbitMQ implementation, and we stumbled upon a great feature called Federation that allows us to partition our task queue across any number of RabbitMQ servers and gives us the confidence that, if any single server gets backlogged, others will pitch in and distribute some of the backlogged tasks to their consumers.
#MessageQueue
Hi, I am building an enhanced web-conferencing app that will have a voice/video call, live chats, live notifications, live discussions, screen sharing, etc features. Ref: Zoom.
I need advise finalizing the tech stack for this app. I am considering below tech stack:
- Frontend: React
- Backend: Node.js
- Database: MongoDB
- IAAS: #AWS
- Containers & Orchestration: Docker / Kubernetes
- DevOps: GitLab, Terraform
- Brokers: Redis / RabbitMQ
I need advice at the platform level as to what could be considered to support concurrent video streaming seamlessly.
Also, please suggest what could be a better tech stack for my app?
#SAAS #VideoConferencing #WebAndVideoConferencing #zoom #stack
- Reliable for banking transactions3
- Useful for big enteprises3
- Secure2
- Broader connectivity - more protocols, APIs, Files etc1
- Many deployment options (containers, cloud, VM etc)1
- High Availability1
- Cost2
related IBM MQ posts
Want to get the differences in features and enhancement, pros and cons, and also how to Migrate from IBM MQ to Azure Service Bus.
- Easy Integration with .Net4
- Cloud Native2
- Use while high messaging need1
- Limited features in Basic tier1
- Skills can only be used in Azure - vendor lock-in1
- Lacking in JMS support1
- Observability of messages in the queue is lacking1
related Azure Service Bus posts
Want to get the differences in features and enhancement, pros and cons, and also how to Migrate from IBM MQ to Azure Service Bus.
- Easy to use18
- Open source14
- Efficient13
- JMS compliant10
- High Availability6
- Scalable5
- Distributed Network of brokers3
- Persistence3
- Support XA (distributed transactions)3
- Docker delievery1
- Highly configurable1
- RabbitMQ0
- ONLY Vertically Scalable1
- Support1
- Low resilience to exceptions and interruptions1
- Difficult to scale1
related ActiveMQ posts
I want to choose Message Queue with the following features - Highly Available, Distributed, Scalable, Monitoring. I have RabbitMQ, ActiveMQ, Kafka and Apache RocketMQ in mind. But I am confused which one to choose.
I use ActiveMQ because RabbitMQ have stopped giving the support for AMQP 1.0 or above version and the earlier version of AMQP doesn't give the functionality to support OAuth.
If OAuth is not required and we can go with AMQP 0.9 then i still recommend rabbitMq.
- Performance887
- Super fast542
- Ease of use514
- In-memory cache444
- Advanced key-value cache324
- Open source194
- Easy to deploy182
- Stable165
- Free156
- Fast121
- High-Performance42
- High Availability40
- Data Structures35
- Very Scalable32
- Replication24
- Pub/Sub23
- Great community22
- "NoSQL" key-value data store19
- Hashes16
- Sets13
- Sorted Sets11
- Lists10
- NoSQL10
- Async replication9
- BSD licensed9
- Integrates super easy with Sidekiq for Rails background8
- Bitmaps8
- Open Source7
- Keys with a limited time-to-live7
- Lua scripting6
- Strings6
- Awesomeness for Free5
- Hyperloglogs5
- Runs server side LUA4
- Transactions4
- Networked4
- Outstanding performance4
- Feature Rich4
- Written in ANSI C4
- LRU eviction of keys4
- Data structure server3
- Performance & ease of use3
- Temporarily kept on disk2
- Dont save data if no subscribers are found2
- Automatic failover2
- Easy to use2
- Scalable2
- Channels concept2
- Object [key/value] size each 500 MB2
- Existing Laravel Integration2
- Simple2
- Cannot query objects directly15
- No secondary indexes for non-numeric data types3
- No WAL1
related Redis posts
StackShare Feed is built entirely with React, Glamorous, and Apollo. One of our objectives with the public launch of the Feed was to enable a Server-side rendered (SSR) experience for our organic search traffic. When you visit the StackShare Feed, and you aren't logged in, you are delivered the Trending feed experience. We use an in-house Node.js rendering microservice to generate this HTML. This microservice needs to run and serve requests independent of our Rails web app. Up until recently, we had a mono-repo with our Rails and React code living happily together and all served from the same web process. In order to deploy our SSR app into a Heroku environment, we needed to split out our front-end application into a separate repo in GitHub. The driving factor in this decision was mostly due to limitations imposed by Heroku specifically with how processes can't communicate with each other. A new SSR app was created in Heroku and linked directly to the frontend repo so it stays in-sync with changes.
Related to this, we need a way to "deploy" our frontend changes to various server environments without building & releasing the entire Ruby application. We built a hybrid Amazon S3 Amazon CloudFront solution to host our Webpack bundles. A new CircleCI script builds the bundles and uploads them to S3. The final step in our rollout is to update some keys in Redis so our Rails app knows which bundles to serve. The result of these efforts were significant. Our frontend team now moves independently of our backend team, our build & release process takes only a few minutes, we are now using an edge CDN to serve JS assets, and we have pre-rendered React pages!
#StackDecisionsLaunch #SSR #Microservices #FrontEndRepoSplit
Our whole DevOps stack consists of the following tools:
- GitHub (incl. GitHub Pages/Markdown for Documentation, GettingStarted and HowTo's) for collaborative review and code management tool
- Respectively Git as revision control system
- SourceTree as Git GUI
- Visual Studio Code as IDE
- CircleCI for continuous integration (automatize development process)
- Prettier / TSLint / ESLint as code linter
- SonarQube as quality gate
- Docker as container management (incl. Docker Compose for multi-container application management)
- VirtualBox for operating system simulation tests
- Kubernetes as cluster management for docker containers
- Heroku for deploying in test environments
- nginx as web server (preferably used as facade server in production environment)
- SSLMate (using OpenSSL) for certificate management
- Amazon EC2 (incl. Amazon S3) for deploying in stage (production-like) and production environments
- PostgreSQL as preferred database system
- Redis as preferred in-memory database/store (great for caching)
The main reason we have chosen Kubernetes over Docker Swarm is related to the following artifacts:
- Key features: Easy and flexible installation, Clear dashboard, Great scaling operations, Monitoring is an integral part, Great load balancing concepts, Monitors the condition and ensures compensation in the event of failure.
- Applications: An application can be deployed using a combination of pods, deployments, and services (or micro-services).
- Functionality: Kubernetes as a complex installation and setup process, but it not as limited as Docker Swarm.
- Monitoring: It supports multiple versions of logging and monitoring when the services are deployed within the cluster (Elasticsearch/Kibana (ELK), Heapster/Grafana, Sysdig cloud integration).
- Scalability: All-in-one framework for distributed systems.
- Other Benefits: Kubernetes is backed by the Cloud Native Computing Foundation (CNCF), huge community among container orchestration tools, it is an open source and modular tool that works with any OS.
- Varying levels of Quality of Service to fit a range of3
- Lightweight with a relatively small data footprint2
- Very easy to configure and use with open source tools2
- Easy to configure in an unsecure manner1
related MQTT posts
Kindly suggest the best tool for generating 10Mn+ concurrent user load. The tool must support MQTT traffic, REST API, support to interfaces such as Kafka, websockets, persistence HTTP connection, auth type support to assess the support /coverage.
The tool can be integrated into CI pipelines like Azure Pipelines, GitHub, and Jenkins.
You can use ReductStore to keep a history of MQTT messages by using its Client SDKs. This can be useful if you use a binary format for your data and it can be recorded in a classical TSDB. You can set a FIFO quota for a bucket in your ReductStore instance so that the database removes old MQTT messages when the limit is reached.