RabbitMQ

RabbitMQ

Application and Data / Data Stores / Message Queue
Avatar of jtcunning
Operations Engineer at Sentry·
Shared insights
on
CeleryCeleryRabbitMQRabbitMQ
at

As Sentry runs throughout the day, there are about 50 different offline tasks that we execute‚ÄĒanything from ‚Äúprocess this event, pretty please‚ÄĚ to ‚Äúsend all of these cool people some emails.‚ÄĚ There are some that we execute once a day and some that execute thousands per second.

Managing this variety requires a reliably high-throughput message-passing technology. We use Celery's RabbitMQ implementation, and we stumbled upon a great feature called Federation that allows us to partition our task queue across any number of RabbitMQ servers and gives us the confidence that, if any single server gets backlogged, others will pitch in and distribute some of the backlogged tasks to their consumers.

#MessageQueue

READ MORE
How Sentry Receives 20 Billion Events Per Month While Preparing to Handle Twice That - Sentry Tech Stack | StackShare (stackshare.io)
18 upvotes·856.5K views
Avatar of tabbott
Founder at Zulip·
Shared insights
on
RabbitMQRabbitMQPythonPythonRedisRedis
at

We've been using RabbitMQ as Zulip's queuing system since we needed a queuing system. What I like about it is that it scales really well and has good libraries for a wide range of platforms, including our own Python. So aside from getting it running, we've had to put basically 0 effort into making it scale for our needs.

However, there's several things that could be better about it: * It's error messages are absolutely terrible; if ever one of our users ends up getting an error with RabbitMQ (even for simple things like a misconfigured hostname), they always end up needing to get help from the Zulip team, because the errors logs are just inscrutable. As an open source project, we've handled this issue by really carefully scripting the installation to be a failure-proof configuration (in this case, setting the RabbitMQ hostname to 127.0.0.1, so that no user-controlled configuration can break it). But it was a real pain to get there and the process of determining we needed to do that caused a significant amount of pain to folks installing Zulip. * The pika library for Python takes a lot of time to startup a RabbitMQ connection; this means that Zulip server restarts are more disruptive than would be ideal. * It's annoying that you need to run the rabbitmqctl management commands as root.

But overall, I like that it has clean, clear semanstics and high scalability, and haven't been tempted to do the work to migrate to something like Redis (which has its own downsides).

READ MORE
Queue processors ‚ÄĒ Zulip 1.9.1+git documentation (zulip.readthedocs.io)
14 upvotes·647.7K views
Avatar of ydbhondekar2407
Co-Founder at weconnect.chat·
Needs advice
on
RabbitMQRabbitMQMongoDBMongoDB
and
DockerDocker

Hi, I am building an enhanced web-conferencing app that will have a voice/video call, live chats, live notifications, live discussions, screen sharing, etc features. Ref: Zoom.

I need advise finalizing the tech stack for this app. I am considering below tech stack:

  • Frontend: React
  • Backend: Node.js
  • Database: MongoDB
  • IAAS: #AWS
  • Containers & Orchestration: Docker / Kubernetes
  • DevOps: GitLab, Terraform
  • Brokers: Redis / RabbitMQ

I need advice at the platform level as to what could be considered to support concurrent video streaming seamlessly.

Also, please suggest what could be a better tech stack for my app?

#SAAS #VideoConferencing #WebAndVideoConferencing #zoom #stack

READ MORE
11 upvotes·24K views
Replies (6)
Recommends
WebRTCWebRTC

You're going to want to look hard at WebRTC. It's what almost every realtime video service uses. The appeal is that it establishes a direct connection between peers so that the massive video bandwidth doesn't need to go through your backend. That aside, actor clusters will be the other technology that handle that sort of traffic well. It was popularized by erlang for telecom backbone, akka is another choice for actor systems.

Infrastructure wise, kubernetes would be a fine choice. Just make sure to look up some benchmarks for Container Network Interface (CNI) implementations that support high bandwidth traffic.

READ MORE
8 upvotes·710 views
Avatar of BigBugLord
SDET at Ledningkart·
Recommends
DockerDocker

Kubernetes provides Auto-scaling whereas Docker Swarm doesn't support autoscaling. Kubernetes supports up to 5000 nodes whereas Docker Swarm supports more than 2000 nodes. Kubernetes is less extensive and customizable whereas Docker Swarm is more comprehensive and highly customizable. So if your main usecase is autoscaling go for kubernetes else Docker is always a good choice.

READ MORE
8 upvotes·707 views
View all (6)
Needs advice
on
RedisRedisRabbitMQRabbitMQ
and
KafkaKafka

We are going to develop a microservices-based application. It consists of AngularJS, ASP.NET Core, and MSSQL.

We have 3 types of microservices. Emailservice, Filemanagementservice, Filevalidationservice

I am a beginner in microservices. But I have read about RabbitMQ, but come to know that there are Redis and Kafka also in the market. So, I want to know which is best.

READ MORE
9 upvotes·47K views
Replies (4)
Avatar of alurimahee
Lead Architect ·
Recommends
KafkaKafka

Kafka is an Enterprise Messaging Framework whereas Redis is an Enterprise Cache Broker, in-memory database and high performance database.Both are having their own advantages, but they are different in usage and implementation. Now if you are creating microservices check the user consumption volumes, its generating logs, scalability, systems to be integrated and so on. I feel for your scenario initially you can go with KAFKA bu as the throughput, consumption and other factors are scaling then gradually you can add Redis accordingly.

READ MORE
6 upvotes·21.3K views
Recommends
NATSNATS

We found that the CNCF landscape is a good advisor when working going into the cloud / microservices space: https://landscape.cncf.io/fullscreen=yes. When choosing a technology one important criteria to me is if it is cloud native or not. Neither Redis, RabbitMQ nor Kafka is cloud native. The try to adapt but will be replaced eventually with technologies that are cloud native.

We have gone with NATS and have never looked back. We haven't spend a single minute on server maintainance in the last year and the setup of a cluster is way too easy. With the new features NATS incorporates now (and the ones still on the roadmap) it is already and will be sooo much mure than Redis, RabbitMQ and Kafka are. It can replace service discovery, load balancing, global multiclusters and failover, etc, etc.

Your thought might be: But I don't need all of that! Well, at the same time it is much more leightweight than Redis, RabbitMQ and especially Kafka.

READ MORE
3 upvotes·18.7K views
View all (4)

Around the time of their Series A, Pinterest’s stack included Python and Django, with Tornado and Node.js as web servers. Memcached / Membase and Redis handled caching, with RabbitMQ handling queueing. Nginx, HAproxy and Varnish managed static-delivery and load-balancing, with persistent data storage handled by MySQL.

READ MORE
What is the technology stack behind Pinterest? - Quora (quora.com)
8 upvotes·460.4K views
Needs advice
on
RedisRedis
and
RabbitMQRabbitMQ

Hello there, We're developing a team chat application which would consist of direct (one-to-one) conversations and channel (group) conversations. I'm not the developer (of course), but my team suggested to go with Redis.

I've seen tech stacks of BIG team chat applications like Slack and Flock...but they haven't used RabbitMQ and used Redis instead.

A quick question, what's a good choice to go with for RabbitMQ or Redis for a message queue system in our case?

READ MORE
8 upvotes·18.6K views
Replies (2)
Recommends
RabbitMQRabbitMQ

This is of course determined by the needs of your application. It is important how many of your estimated instant users in your application will be. Also, the features of the application will affect the architecture of the application. For example, if the message data would be processed on the server, I would prefer a distributed server solution such as akka actor with the rabbitmq cluster. I would definitely use Redis. Both technologies are incomparable lanes. Redis is a database and its purpose is to process data from a different memory with the memory used by the code running on the server. Rabbit is a messaging queue system. It contributes to the architecture in a different dimension. Performance and stability are keywords.

READ MORE
7 upvotes·1 comment·1.9K views
Avatar of aparedes
Lead Senior Software Engineer at InTouch Technology·
Recommends
RabbitMQRabbitMQ

Each tool supports different use cases. RabbitMQ is a middleware peace supporting message driven reqs like you are trying to accomplish and Redis, on the other hand, allows to store data if performance, cache is important. If we are taking about a message queue system approach you could use RabbitMQ, Amazon SNS/SQS or Apache Kafka

READ MORE
3 upvotes·2K views
Avatar of samcolart
Principal Developer at Colart·

We needed our e-commerce platform (built using WooCommerce) to be able to keep products in sync with our #pim (provided by #akeneo) which is built in Symfony . We hooked into the kernel.event_listener to send RabbitMQ messages to a WordPress API endpoint that triggers the updated product to rebuild with fresh data.

READ MORE
7 upvotes·179.1K views
Avatar of michaelmota
Founder at AlterEstate·

Automations are what makes a CRM powerful. With Celery and RabbitMQ we've been able to make powerful automations that truly works for our clients. Such as for example, automatic daily reports, reminders for their activities, important notifications regarding their client activities and actions on the website and more.

We use Celery basically for everything that needs to be scheduled for the future, and using RabbitMQ as our Queue-broker is amazing since it fully integrates with Django and Celery storing on our database results of the tasks done so we can see if anything fails immediately.

READ MORE
6 upvotes·195.4K views
Avatar of letmerecall
Software Engineer at FireVisor Systems·
Needs advice
on
NamekoNamekoFlaskFlask
and
BottleBottle

Which is the best Python framework for microservices?

We are using Nameko for building microservices in Python. The things we really like are dependency injection and the ease with which one can expose endpoints via RPC over RabbitMQ. We are planning to try a tool that helps us write polyglot microservices and nameko is not super compatible with it. Also, we are a bit worried about the not so good community support from nameko and looking for a python alternate to write microservices.

READ MORE
6 upvotes·12.4K views
Replies (1)
Recommends
BottleBottle

Bottle is much less bloated and fast. Its built-in templating system is one of the fastest as it compiles the templates in bytecode. Also Bottle has no depenencies, preventing dependency bloat.

READ MORE
1 upvote·5.7K views
Avatar of fdjiwani
Software Engineer at IVP·
Needs advice
on
DynatraceDynatraceDatadogDatadog
and
AppDynamicsAppDynamics

Hey there! We are looking at Datadog, Dynatrace, AppDynamics, and New Relic as options for our web application monitoring.

Current Environment: .NET Core Web app hosted on Microsoft IIS

Future Environment: Web app will be hosted on Microsoft Azure

Tech Stacks: IIS, RabbitMQ, Redis, Microsoft SQL Server

Requirement: Infra Monitoring, APM, Real - User Monitoring (User activity monitoring i.e., time spent on a page, most active page, etc.), Service Tracing, Root Cause Analysis, and Centralized Log Management.

Please advise on the above. Thanks!

READ MORE
5 upvotes·121.1K views
Replies (1)