Shelf

Compared to MongoDB it offers predictable performance, infinite scalability, outstanding stability, and tiny cost.

When using MongoDB at scale, you encounter all sorts of issues that are not advertised in MongoDB promotional materials. You get limited by the number of connections, not compatible drivers, and unexpected bugs in new releases.

With MongoDB, it's easy to get started quickly because it doesn't require you to think about data structure in the beginning. When you hit a certain scale you either spend months reengineering your DB or paying loads of cash to scale your cluster.

On the other hand, DynamoDB is much limited in its querying API, but that's a good thing. It's very hard to build a not scalable or low-performance database with DynamoDB. Also, you get all the perks of native integration into other AWS services.

Back in 2018, we were running microservices in the ECS cluster by managing EC2 instances ourselves. It involved lots of toil work.

Without Fargate we had to maintain & monitor a pool of spot instances, choose the right size of instances, keep OS up-to-date, etc.

We want to ship features, not to manage servers. That's why as soon as AWS Fargate was announced, we migrated our microservices to this managed service. The only configuration you need is Docker image, RAM & CPU resources. The rest of the heavy lifting is taken care of by Fargate, which is wonderful.

In the R&D department, we've chosen Lambda to serve some of our ML models as part of an API for product features. This allows to develop quickly, easy scale, and avoid the unnecessary overhead for maintenance and support.

As the Shelf Dev Team grows in size, more and more newly joined engineers face questions specific to our codebase and development setup.

Imagine you try to launch a project locally for development, and then some obscure error comes up with a stack trace pointing to some internal tooling. Obviously, you won't find any help on Google, as it comes from code developed internally.

Usually, more experienced developers help newcomers in Slack. This is the fastest way to get help, but it doesn't scale, it's not repeatable, and usually is lost, since communication happens in direct messages.

That's where Stackoverflow for Teams comes in.

We migrated all our CI/CD pipelines to CircleCI back in 2017 and are particularly happy about it since!

Our top loved feature is unlimited parallelism. We can run as many builds concurrently as we want.

We also use orbs, pipeline parameters, reusable commands, build cache, test insights.

For some of the heavier repos, we use a larger resource class and mount RAMdisk to a file system to speed up builds.

As our codebase grew in size, we were looking for ways to improve code quality. We chose TypeScript over Flow due to its rapid industry adoption and overall tools support.

We noticed how different open-source projects were migrating from Flow to TypeScript. Most notably, it was Jest, even though Jest and Flow were both developed by Facebook. See this HN thread if you want to dive into an interesting discussion around this move.

Additionally, at the beginning of 2019, both Babel and ESLint enabled seamless TypeScript support, which allowed easy migration path in a backward-compatible way.

AWS Rekognition has an OCR feature but can recognize only up to 50 words per image, which is a deal-breaker for us. (see my tweet).

Also, we discovered fantastic speed and quality improvements in the 4.x versions of Tesseract. Meanwhile, the quality of AWS Rekognition's OCR remains to be mediocre in comparison.

We run Tesseract serverlessly in AWS Lambda via aws-lambda-tesseract library that we made open-source.

During preparation for the SOC 2 certification, we were looking for ways to make our infrastructure more secure. One of the ideas was to shift some low-level server maintenance responsibility from our shoulders to AWS.

Fargate helped a lot in achieving this goal. We have freed lots of time by not managing/patching/monitoring EC2 instances.