Load and Performance Testing on Heroku

StackShare Feed is built entirely with React, Glamorous, and Apollo. One of our objectives with the public launch of the Feed was to enable a Server-side rendered (SSR) experience for our organic search traffic. When you visit the StackShare Feed, and you aren't logged in, you are delivered the Trending feed experience. We use an in-house Node.js rendering microservice to generate this HTML. This microservice needs to run and serve requests independent of our Rails web app. Up until recently, we had a mono-repo with our Rails and React code living happily together and all served from the same web process. In order to deploy our SSR app into a Heroku environment, we needed to split out our front-end application into a separate repo in GitHub. The driving factor in this decision was mostly due to limitations imposed by Heroku specifically with how processes can't communicate with each other. A new SSR app was created in Heroku and linked directly to the frontend repo so it stays in-sync with changes.

Related to this, we need a way to "deploy" our frontend changes to various server environments without building & releasing the entire Ruby application. We built a hybrid Amazon S3 Amazon CloudFront solution to host our Webpack bundles. A new CircleCI script builds the bundles and uploads them to S3. The final step in our rollout is to update some keys in Redis so our Rails app knows which bundles to serve. The result of these efforts were significant. Our frontend team now moves independently of our backend team, our build & release process takes only a few minutes, we are now using an edge CDN to serve JS assets, and we have pre-rendered React pages!

#StackDecisionsLaunch #SSR #Microservices #FrontEndRepoSplit

We decided to use MemCachier as our Memcached provider because we were seeing some serious PostgreSQL performance issues with query-heavy pages on the site. We use MemCachier for all Rails caching and pretty aggressively too for the logged out experience (fully cached pages for the most part). We really need to move to Amazon ElastiCache as soon as possible so we can stop paying so much. The only reason we're not moving is because there are some restrictions on the network side due to our main app being hosted on Heroku.

#Caching #RailsCaching

We began our hosting journey, as many do, on Heroku because they make it easy to deploy your application and automate some of the routine tasks associated with deployments, etc. However, as our team grew and our product matured, our needs have outgrown Heroku. I will dive into the history and reasons for this in a future blog post.

We decided to migrate our infrastructure to Kubernetes running on Amazon EKS. Although Google Kubernetes Engine has a slightly more mature Kubernetes offering and is more user-friendly; we decided to go with EKS because we already using other AWS services (including a previous migration from Heroku Postgres to AWS RDS). We are still in the process of moving our main website workloads to EKS, however we have successfully migrate all our staging and testing PR apps to run in a staging cluster. We developed a Slack chatops application (also running in the cluster) which automates all the common tasks of spinning up and managing a production-like cluster for a pull request. This allows our engineering team to iterate quickly and safely test code in a full production environment. Helm plays a central role when deploying our staging apps into the cluster. We use CircleCI to build docker containers for each PR push, which are then published to Amazon EC2 Container Service (ECR). An upgrade-operator process watches the ECR repository for new containers and then uses Helm to rollout updates to the staging environments. All this happens automatically and makes it really easy for developers to get code onto servers quickly. The immutable and isolated nature of our staging environments means that we can do anything we want in that environment and quickly re-create or restore the environment to start over.

The next step in our journey is to migrate our production workloads to an EKS cluster and build out the CD workflows to get our containers promoted to that cluster after our QA testing is complete in our staging environments.

The continuous integration process for our Rails backend app starts by opening a GitHub pull request. This triggers a CircleCI build and some Code Climate checks.

The CircleCI build is a workflow that runs the following jobs:

• check for security vulnerabilities with Brakeman
• check code quality with RuboCop
• run RSpec tests in parallel with the knapsack gem, and output test coverage reports with the simplecov gem
• upload test coverage to Code Climate

Code Climate checks the following:

• code quality metrics like code complexity
• test coverage minimum thresholds

The CircleCI jobs and Code Climate checks above have corresponding GitHub status checks.

Once all the mandatory GitHub checks pass and the code+functionality have been reviewed, developers can merge their pull request into our Git master branch. Code is then ready to deploy!

#ContinuousIntegration

We currently monitor performance with the following tools:

1. Heroku Metrics: our main app is Hosted on Heroku, so it is the best place to get quick server metrics like memory usage, load averages, or response times.
2. Good old New Relic for detailed general metrics, including transaction times.
3. Skylight for more specific Rails Controller#action transaction times. Navigating those timings is much better than with New Relic, as you get a clear full breakdown of everything that happens for a given request.

Skylight offers better Rails performance insights, so why use New Relic? Because it does frontend monitoring, while Skylight doesn't. Now that we have a separate frontend app though, our frontend engineers are looking into more specialized frontend monitoring solutions.

Finally, if one of our apps go down, Pingdom alerts us on Slack and texts some of us.