Good day, everyone!
We recently got the request to add a feature to our systems: Forms should be saved to be continued later given the following cases:
If the connection to the server is lost, if the user closes their browser, if the user changes networks, if the user didn't finish the form and decides to take a week off and a mix of them all.
One of the best solutions we've thought so far is to save the data in the forms in a remote database every so minutes directly from the client.
A document-oriented database seems the best bet, but which database to use is the question. Since we already use Google Cloud Platform, Cloud Firestore is a "safe" option right now, but we are looking at every option we can find. The "caveat" of our use case is that we need more writes than reads, but writes are usually more expensive, and our biggest constraint is budget.
That's why I am seeking advice. What other options do we have?
In case you're wondering, we seek to save the forms bypassing the backend and our own system resources entirely, so a dedicated, remote database or Local Storage are our best options, but Local Storage is a bit controversial choice within the team, so we will explore that options later on.
Any advice or experience in this matter is highly appreciated!
You can give ClientDB a try. It simplifies storing data local, and it takes care about syncing it regularly to your remote data storage.
Advantages:
- It's cheap, because you don't have to send a write request to your database every moment. (And it's open source)
- If the user looses internet, it will sync up simply again at the moment he has internet again
- You can implement an instant experience, because you don't have to wait for database feedback e.g. saving data (Not so relevant for your Use case, but maybe overall interesting to know.)
Downside:
- You have an additional layer, that you have to maintain.
Thanks for the feedback. Considered this system for saving data
I don’t have time to go into a deeper analysis, but consider that with Cloud Firestore projects that I’ve had in the past with heavy write load, my cost was not on the writes: it was in the storage. So maybe consider that in our cost analysis - write over time + storage cost over time. Storage costs tend to be exponential and never go down unless you manually clean up your database. With Cloud Firestore this is harder than with MongoDB atlas, that has TTL for documents. By the way, I’m basing myself over an experience with Firestore from about two years ago, you might wanna check what’s the situation nowadays when it comes to archiving / deleting old data and storage costs. And last but no least, consider both Firestore and DataStore mode in your analysis. I thought there was no advantage for my use case but DataStore mode was actually more suitable at the time for a huge number of writes and deletes.
Also be mindful of performance during writes. I’ve had a bunch of performance issues with FireStore due to heavy write load and had to to batch writes. I would personally chose MongoDB Atlas as I trust MongoDB a tad more as a product, company, and always have the chance to spin up my own Mongo if costs get too high in a cloud platform.
Thanks! I truly didn't think about the storage cost. We do plan to deleting records with no activity in a week or so and whenever the form if fulfilled, likely on non-working hours. That being said, I will add the storage cost measure into consideration nevertheless. Is something that will add up, whether we clean the DB or not.
Storage cost was a huge deal for me there - but I was in scale of 8MM events per day, so it truly depends on the size of your data - but for this size of data that can be deleted - TTL for documents is a must unless you wanna write your own batch jobs to delete data 💀