Originally, we had a single MongoDB replica set that we stored everything on. As we scaled, we realized two things:
- A single Mongo replica set wasn’t going to cut it for our many quickly growing collections
- Analytics and rich searching don’t scale well in Mongo.
To solve for the first item, we now run multiple large scale Mongo deployments with a mix of replica sets and sharded replica sets (depends on the application activity for the given database). In solving for the second item, we now run multiple large Elasticsearch deployments to provide the majority of our rich searching functionality.
We also heavily use Redis across the entire platform for things like distributed locking, caching, and backing part of our job queuing layer. This has led to our most recent (and ongoing!) scaling challenge.