Shelf

Please visit https://shelfio.github.io/tech-radar/ for the most up-to-date version of this page.

We had to choose where to host a PostgreSQL database on AWS and the choice was between plain AWS RDS, AWS Aurora for PostgreSQL, and AWS Aurora for PostgreSQL Serverless v2.

At first, we were skeptical about Aurora Serverless v2 as past experience with Serverless v1 showed major drawbacks in terms of autoscaling performance.

But after some testing on a couple of hundreds of gigabytes of data, we were convinced Aurora Serverless v2 is the way to go.

It delivers all the promised instant scaling capabilities and provides unparalleled disk IOPS performance, up to millions of IOPS if needed.

Aurora Serverless V2 can truly handle whatever data volume you have and run even the most complex queries without doing much optimization in advance. Of course, later down the road, it gives you detailed performance metrics to help with optimization. It is similar to MongoDB in this regard - easy to start with, don't worry about structure/volume in the beginning, but you got all the tools to optimize it later.

Compared to MongoDB it offers predictable performance, infinite scalability, outstanding stability, and tiny cost.

When using MongoDB at scale, you encounter all sorts of issues that are not advertised in MongoDB promotional materials. You get limited by the number of connections, not compatible drivers, and unexpected bugs in new releases.

With MongoDB, it's easy to get started quickly because it doesn't require you to think about data structure in the beginning. When you hit a certain scale you either spend months reengineering your DB or paying loads of cash to scale your cluster.

On the other hand, DynamoDB is much limited in its querying API, but that's a good thing. It's very hard to build a not scalable or low-performance database with DynamoDB. Also, you get all the perks of native integration into other AWS services.

Back in 2018, we were running microservices in the ECS cluster by managing EC2 instances ourselves. It involved lots of toil work.

Without Fargate we had to maintain & monitor a pool of spot instances, choose the right size of instances, keep OS up-to-date, etc.

We want to ship features, not to manage servers. That's why as soon as AWS Fargate was announced, we migrated our microservices to this managed service. The only configuration you need is Docker image, RAM & CPU resources. The rest of the heavy lifting is taken care of by Fargate, which is wonderful.

In the R&D department, we've chosen Lambda to serve some of our ML models as part of an API for product features. This allows to develop quickly, easy scale, and avoid the unnecessary overhead for maintenance and support.

As the Shelf Dev Team grows in size, more and more newly joined engineers face questions specific to our codebase and development setup.

Imagine you try to launch a project locally for development, and then some obscure error comes up with a stack trace pointing to some internal tooling. Obviously, you won't find any help on Google, as it comes from code developed internally.

Usually, more experienced developers help newcomers in Slack. This is the fastest way to get help, but it doesn't scale, it's not repeatable, and usually is lost, since communication happens in direct messages.

That's where Stackoverflow for Teams comes in.

We migrated all our CI/CD pipelines to CircleCI back in 2017 and are particularly happy about it since!

Our top loved feature is unlimited parallelism. We can run as many builds concurrently as we want.

We also use orbs, pipeline parameters, reusable commands, build cache, test insights.

For some of the heavier repos, we use a larger resource class and mount RAMdisk to a file system to speed up builds.

As our codebase grew in size, we were looking for ways to improve code quality. We chose TypeScript over Flow due to its rapid industry adoption and overall tools support.

We noticed how different open-source projects were migrating from Flow to TypeScript. Most notably, it was Jest, even though Jest and Flow were both developed by Facebook. See this HN thread if you want to dive into an interesting discussion around this move.

Additionally, at the beginning of 2019, both Babel and ESLint enabled seamless TypeScript support, which allowed easy migration path in a backward-compatible way.

AWS Rekognition has an OCR feature but can recognize only up to 50 words per image, which is a deal-breaker for us. (see my tweet).

Also, we discovered fantastic speed and quality improvements in the 4.x versions of Tesseract. Meanwhile, the quality of AWS Rekognition's OCR remains to be mediocre in comparison.

We run Tesseract serverlessly in AWS Lambda via aws-lambda-tesseract library that we made open-source.

During preparation for the SOC 2 certification, we were looking for ways to make our infrastructure more secure. One of the ideas was to shift some low-level server maintenance responsibility from our shoulders to AWS.

Fargate helped a lot in achieving this goal. We have freed lots of time by not managing/patching/monitoring EC2 instances.