Tarun Batra

I had been using GitHub Pages to host my website at https://tarunbatra.github.io for a long time but like any centralized service, it is prone to outages. I decided to keep a copy on IPFS to make it always available (not like it is critical, but I take it personally, since it is my digital representation). Since then I host an identical copy at https://ipfs.tarunbatra.com. I hope more and more websites are built on an immutable and distributed stack like IPFS .

We have faced the same question some time ago. Before I begin, DO NOT use Redis as a message broker. It is fast and easy to set up in the beginning but it does not scale. It is not made to be reliable in scale and that is mentioned in the official docs. This analysis of our problems with Redis may help you.

We have used Kafka and RabbitMQ both in scale. We concluded that RabbitMQ is a really good general purpose message broker (for our case) and Kafka is really fast but limited in features. That’s the trade off that we understood from using it. In-fact I blogged about the trade offs between Kafka and RabbitMQ to document it. I hope it helps you in choosing the best pub-sub layer for your use case.

RabbitMQ has been traditionally used for IoT devices since it supports MQTT protocol which is a common protocol in that space.

However I would not recommend Redis for real-time data processing. At my work Redis had problem in scaling, especially as a message queuing system. I have explained that particular problem and how we solved it with RabbitMQ here at Polling reliably at scale using DLQs .

If your are confused between Kafka and RabbitMQ, this post which explains the differences between Kafka and RabbitMQ might help.

Looks like the use case is to store JSON data. mongoDB and Postgres differ in so many aspects like scaling and consistency. Postgres has excellent JSON support now with the power of SQL. MongoDB is good in handling schema less data. However in this case it seems these differences don’t matter that much. I’d recommend you go with what you are most comfortable with.

pip is difficult to understand especially for beginners. I wrote a blog post for npm developer's point of view, to make pip more approachable.

Hi Sara,

You need to decide if this is a demo app or an app which you'll want to scale and maintain. For a quick demo, I would spin up a Node.js server with server rendered frontend using Handlebars template. This way, the whole project can be maintained in one repository without much moving parts.

If you are extracting data from image itself and not from a database, you wouldn't even need Node.js. Just a React app will do. Data extraction from image can be done on client side.

Python, PHP and Node.js all are capable of being used to create good complex software. There are many examples of similar applications built on all of them. If I have to pick one, I would say consider Python and Django. It is fairly easy to develop web applications on top of this stack. Scaling and maintaining the application should also not be a problem given a lot of resources are available online.

MongoDB and Mongoose are commonly used with Node.js and the use case doesn't seem to be requiring any special considerations as of now. However using MongoDB now will allow you to easily expand and modify your use case in future.

If not MongoDB, then my second choice will be PostgreSQL. It's a generic purpose database with jsonb support (if you need it) and lots of resources online. Nobody was fired for choosing PostgreSQL.

We have seen really good performance with frequent writes in Redis (~10k writes per second). The retrievals are also fast. I see no reason why it should not be your first choice. I know Redis doesn't have the image of a persistence database but it is capable of persistence and high availability. Moreover, MongoDB or some other database can be used as a cold storage. Even switching to MongoDB or any other NoSQL database entirely wouldn't be a big problem since Redis is a simple key-value store.

RabbitMQ is great for queuing and retrying. You can send the requests to your backend which will further queue these requests in RabbitMQ (or Kafka, too). The consumer on the other end can take care of processing . For a detailed analysis, check this blog about choosing between Kafka and RabbitMQ.