Hi everybody,
We are building a prototype of our first Hardware system. It's a sensor that we expect to send data approx. every 5 secs. If we scale it to about 1k sensors, it would mean 173M records per month.
We will offer data-retention for 7 days or 1 month based on the plan, but if by any chance the project goes well I see that we can have a lot of pain if not choosing the persistence layer correctly from day 1. Which data storage would you use for that?
Cheers, Alberto.
You might want to look into Vitess/MySQL as well. If you are interested in the inherent technology stack then https://planetscale.com might be worth exploring as an option.
We have seen really good performance with frequent writes in Redis (~10k writes per second). The retrievals are also fast. I see no reason why it should not be your first choice. I know Redis doesn't have the image of a persistence database but it is capable of persistence and high availability. Moreover, MongoDB or some other database can be used as a cold storage. Even switching to MongoDB or any other NoSQL database entirely wouldn't be a big problem since Redis is a simple key-value store.