What is Google Cloud Pub/Sub and what are its top alternatives?
Top Alternatives to Google Cloud Pub/Sub
Kafka is a distributed, partitioned, replicated commit log service. It provides the functionality of a messaging system, but with a unique design. ...
RabbitMQ gives your applications a common platform to send and receive messages, and your messages a safe place to live until received. ...
Firebase is a cloud service designed to power real-time, collaborative applications. Simply add the Firebase library to your application to gain access to a shared data structure; any changes you make to that data are automatically synchronized with the Firebase cloud and with other clients within milliseconds. ...
It enables real-time bidirectional event-based communication. It works on every platform, browser or device, focusing equally on reliability and speed. ...
Pusher is the category leader in delightful APIs for app developers building communication and collaboration features. ...
SignalR allows bi-directional communication between server and client. Servers can now push content to connected clients instantly as it becomes available. SignalR supports Web Sockets, and falls back to other compatible techniques for older browsers. SignalR includes APIs for connection management (for instance, connect and disconnect events), grouping connections, and authorization. ...
Unlike traditional enterprise messaging systems, NATS has an always-on dial tone that does whatever it takes to remain available. This forms a great base for building modern, reliable, and scalable cloud and distributed systems. ...
PubNub makes it easy for you to add real-time capabilities to your apps, without worrying about the infrastructure. Build apps that allow your users to engage in real-time across mobile, browser, desktop and server. ...
Google Cloud Pub/Sub alternatives & related posts
- Open source14
- Written in Scala and java. Runs on JVM10
- Message broker + Streaming system6
- Avro schema integration4
- Suport Multiple clients2
- Partioned, replayable log2
- Extremely good parallelism constructs1
- Simple publisher / multi-subscriber model1
- Non-Java clients are second-class citizens27
- Needs Zookeeper26
- Operational difficulties7
- Terrible Packaging2
related Kafka posts
The algorithms and data infrastructure at Stitch Fix is housed in #AWS. Data acquisition is split between events flowing through Kafka, and periodic snapshots of PostgreSQL DBs. We store data in an Amazon S3 based data warehouse. Apache Spark on Yarn is our tool of choice for data movement and #ETL. Because our storage layer (s3) is decoupled from our processing layer, we are able to scale our compute environment very elastically. We have several semi-permanent, autoscaling Yarn clusters running to serve our data processing needs. While the bulk of our compute infrastructure is dedicated to algorithmic processing, we also implemented Presto for adhoc queries and dashboards.
Beyond data movement and ETL, most #ML centric jobs (e.g. model training and execution) run in a similarly elastic environment as containers running Python and R code on Amazon EC2 Container Service clusters. The execution of batch jobs on top of ECS is managed by Flotilla, a service we built in house and open sourced (see https://github.com/stitchfix/flotilla-os).
At Stitch Fix, algorithmic integrations are pervasive across the business. We have dozens of data products actively integrated systems. That requires serving layer that is robust, agile, flexible, and allows for self-service. Models produced on Flotilla are packaged for deployment in production using Khan, another framework we've developed internally. Khan provides our data scientists the ability to quickly productionize those models they've developed with open source frameworks in Python 3 (e.g. PyTorch, sklearn), by automatically packaging them as Docker containers and deploying to Amazon ECS. This provides our data scientist a one-click method of getting from their algorithms to production. We then integrate those deployments into a service mesh, which allows us to A/B test various implementations in our product.
For more info:
- Our Algorithms Tour: https://algorithms-tour.stitchfix.com/
- Our blog: https://multithreaded.stitchfix.com/blog/
- Careers: https://multithreaded.stitchfix.com/careers/
#DataScience #DataStack #Data
As we've evolved or added additional infrastructure to our stack, we've biased towards managed services. Most new backing stores are Amazon RDS instances now. We do use self-managed PostgreSQL with TimescaleDB for time-series data—this is made HA with the use of Patroni and Consul.
We also use managed Amazon ElastiCache instances instead of spinning up Amazon EC2 instances to run Redis workloads, as well as shifting to Amazon Kinesis instead of Kafka.
- It's fast and it works with good metrics/monitoring225
- Ease of configuration79
- I like the admin interface57
- Easy to set-up and start with49
- Intuitive work through python18
- Standard protocols18
- Written primarily in Erlang10
- Simply superb7
- Completeness of messaging patterns6
- Scales to 1 million messages per second3
- Supports AMQP2
- Better than most traditional queue based message broker2
- High performance1
- Inubit Integration1
- Clear documentation with different scripting language1
- Great ui1
- Runs on Open Telecom Platform1
- Better routing system1
- Supports MQTT1
- Too complicated cluster/HA config and management9
- Needs Erlang runtime. Need ops good with Erlang runtime6
- Configuration must be done first, not by your code5
related RabbitMQ posts
As Sentry runs throughout the day, there are about 50 different offline tasks that we execute—anything from “process this event, pretty please” to “send all of these cool people some emails.” There are some that we execute once a day and some that execute thousands per second.
Managing this variety requires a reliably high-throughput message-passing technology. We use Celery's RabbitMQ implementation, and we stumbled upon a great feature called Federation that allows us to partition our task queue across any number of RabbitMQ servers and gives us the confidence that, if any single server gets backlogged, others will pitch in and distribute some of the backlogged tasks to their consumers.
We've been using RabbitMQ as Zulip's queuing system since we needed a queuing system. What I like about it is that it scales really well and has good libraries for a wide range of platforms, including our own Python. So aside from getting it running, we've had to put basically 0 effort into making it scale for our needs.
However, there's several things that could be better about it:
* It's error messages are absolutely terrible; if ever one of our users ends up getting an error with RabbitMQ (even for simple things like a misconfigured hostname), they always end up needing to get help from the Zulip team, because the errors logs are just inscrutable. As an open source project, we've handled this issue by really carefully scripting the installation to be a failure-proof configuration (in this case, setting the RabbitMQ hostname to
127.0.0.1, so that no user-controlled configuration can break it). But it was a real pain to get there and the process of determining we needed to do that caused a significant amount of pain to folks installing Zulip.
pika library for Python takes a lot of time to startup a RabbitMQ connection; this means that Zulip server restarts are more disruptive than would be ideal.
* It's annoying that you need to run the
rabbitmqctl management commands as root.
But overall, I like that it has clean, clear semanstics and high scalability, and haven't been tempted to do the work to migrate to something like Redis (which has its own downsides).
- Realtime backend made easy361
- Fast and responsive263
- Easy setup234
- Backed by google121
- Angular adaptor81
- Great customer support36
- Great documentation26
- Real-time synchronization23
- Mobile friendly20
- Rapid prototyping17
- Great security12
- Automatic scaling11
- Freakingly awesome10
- Angularfire is an amazing addition!8
- Super fast development8
- Awesome next-gen backend6
- Ios adaptor6
- Built in user auth/oauth5
- Firebase hosting5
- Speed of light4
- Very easy to use4
- It's made development super fast3
- Brilliant for startups3
- Great all-round functionality2
- Low battery consumption2
- I can quickly create static web apps with no backend2
- The concurrent updates create a great experience2
- JS Offline and Sync suport2
- Faster workflow1
- Free SSL1
- Good Free Limits1
- Push notification1
- Easy to use1
- Easy Reactjs integration1
- Can become expensive28
- Scalability is not infinite15
- No open source, you depend on external company14
- Not Flexible Enough9
- Cant filter queries5
- Very unstable server3
- Too many errors2
- No Relational Data2
related Firebase posts
This is my stack in Application & Data
My Utilities Tools
Google Analytics Postman Elasticsearch
My Devops Tools
Git GitHub GitLab npm Visual Studio Code Kibana Sentry BrowserStack
My Business Tools
- Event-based communication141
- Open source102
- Binary streaming26
- No internet dependency22
- Fallback to polling if WebSockets not supported9
- Large community7
- Ease of access and setup5
- Push notification4
- Bad documentation9
- Githubs that complement it are mostly deprecated4
- Doesn't work on React Native2
- Small community2
related Socket.IO posts
I use Socket.IO because the application has 2 frontend clients, which need to communicate in real-time. The backend-server handles the communication between these two clients via websockets. Socket.io is very easy to set up in Node.js and ExpressJS.
In the research project, the 1st client shows panoramic videos in a so called cave system (it is the VR setup of our research lab, which consists of three big screens, which are specially arranged, so the user experience the videos more immersive), the 2nd client controls the videos/locations of the 1st client.
We are starting to work on a web-based platform aiming to connect artists (clients) and professional freelancers (service providers). In-app, timeline-based, real-time communication between users (& storing it), file transfers, and push notifications are essential core features. We are considering using Node.js, ExpressJS, React, MongoDB stack with Socket.IO & Apollo, or maybe using Real-Time Database and functionalities of Firebase.
- An easy way to give customers realtime features51
- Easy to get started with27
- Free plan24
- Heroku Add-on12
- Easy and fast to configure and to understand11
- Azure Add-on6
- Push notification4
related Pusher posts
Which messaging service (Pusher vs. PubNub vs. Google Cloud Pub/Sub) to use for IoT?
Recently we finished long research on chat tool for our students and mentors. In the end we picked Mattermost Team Edition as the cheapest and most feature complete option. We did consider building everything from scratch and use something like Pusher or Twilio on a backend, but then we would have to implement all the desktop and mobile clients and all the features oursevles. Mattermost gave us flexible API, lots of built in or easy to install integrations and future-proof feature set. We are still integrating it with our main platform but so far the team, existing mentors and students are very happy.
- Supports .NET server20
- Fallback to SSE, forever frame, long polling11
- Open source5
- Ease of use3
- Requires jQuery1
- Expertise hard to get1
- Weak iOS and Android support1
related SignalR posts
- Fastest pub-sub system out there20
- Rock solid14
- Easy to grasp10
- Easy, Fast, Secure3
- Robust Security Model1
- Persistence with Jetstream supported1
- No Order1
- No Persistence1
related NATS posts
- Massively scalable & easy to use33
- Easy setup24
- Great support18
- Flexible to integrate to custom applications14
- 99.999% availability guarantees12
- Sockets at Scale12
- 70+ SDKs5
- Azure Add-on4
- Heroku Add-on3
- Easy to setup3
- Data Sync2
- AngularJS Adapter2
- PhoneGap Plugin2
- Server-Side Cache2
- Free Plan2
- Data Streams2
- Easy setup and very reliable1
- Real time and easy to use.1
- High cost, going up more in Summer '151
- CTO stephen also is A++++++1
- Angular 2+ integration1
- Documentation, easy to use, great people/service1
- Easy integration with iOS apps1
related PubNub posts
Which messaging service (Pusher vs. PubNub vs. Google Cloud Pub/Sub) to use for IoT?
I am trying to replace Socket.IO with PubNub. Provide the way to do it.