Hi everybody,
We are building a prototype of our first Hardware system. It's a sensor that we expect to send data approx. every 5 secs. If we scale it to about 1k sensors, it would mean 173M records per month.
We will offer data-retention for 7 days or 1 month based on the plan, but if by any chance the project goes well I see that we can have a lot of pain if not choosing the persistence layer correctly from day 1. Which data storage would you use for that?
Cheers, Alberto.
Why do not you try a time series or IOT database? They usually offer very hight toughtput for ingesting and faster quering performance for this kind of data.
We have seen really good performance with frequent writes in Redis (~10k writes per second). The retrievals are also fast. I see no reason why it should not be your first choice. I know Redis doesn't have the image of a persistence database but it is capable of persistence and high availability. Moreover, MongoDB or some other database can be used as a cold storage. Even switching to MongoDB or any other NoSQL database entirely wouldn't be a big problem since Redis is a simple key-value store.
