Alternatives to AWS Database Migration Service

The algorithms and data infrastructure at Stitch Fix is housed in #AWS. Data acquisition is split between events flowing through Kafka, and periodic snapshots of PostgreSQL DBs. We store data in an Amazon S3 based data warehouse. Apache Spark on Yarn is our tool of choice for data movement and #ETL. Because our storage layer (s3) is decoupled from our processing layer, we are able to scale our compute environment very elastically. We have several semi-permanent, autoscaling Yarn clusters running to serve our data processing needs. While the bulk of our compute infrastructure is dedicated to algorithmic processing, we also implemented Presto for adhoc queries and dashboards.

Beyond data movement and ETL, most #ML centric jobs (e.g. model training and execution) run in a similarly elastic environment as containers running Python and R code on Amazon EC2 Container Service clusters. The execution of batch jobs on top of ECS is managed by Flotilla, a service we built in house and open sourced (see https://github.com/stitchfix/flotilla-os).

At Stitch Fix, algorithmic integrations are pervasive across the business. We have dozens of data products actively integrated systems. That requires serving layer that is robust, agile, flexible, and allows for self-service. Models produced on Flotilla are packaged for deployment in production using Khan, another framework we've developed internally. Khan provides our data scientists the ability to quickly productionize those models they've developed with open source frameworks in Python 3 (e.g. PyTorch, sklearn), by automatically packaging them as Docker containers and deploying to Amazon ECS. This provides our data scientist a one-click method of getting from their algorithms to production. We then integrate those deployments into a service mesh, which allows us to A/B test various implementations in our product.

For more info:

• Our Algorithms Tour: https://algorithms-tour.stitchfix.com/
• Our blog: https://multithreaded.stitchfix.com/blog/
• Careers: https://multithreaded.stitchfix.com/careers/

#DataScience #DataStack #Data

As we've evolved or added additional infrastructure to our stack, we've biased towards managed services. Most new backing stores are Amazon RDS instances now. We do use self-managed PostgreSQL with TimescaleDB for time-series data—this is made HA with the use of Patroni and Consul.

We also use managed Amazon ElastiCache instances instead of spinning up Amazon EC2 instances to run Redis workloads, as well as shifting to Amazon Kinesis instead of Kafka.

Flyway vs Liquibase #Migration #Backwards-compatible

We were looking for a tool to help us integrating the migration scripts as part of our Deployment. At first sight both tools look very alike, are well integrated with Spring, have a fairly frequent development activity and short release cycles.

Liquibase puts a lot of emphasis on independence with the DB, allowing you to create the scripts on formats like JSON and YML, abstracting away from SQL, which it's also supported. Since we only work with one DB type across services we wouldn't take much advantage of this feature.

Flyway on the other hand has the advantage on being actively working on the integration with PostgreSQL 11, for it's upcoming version 6. Provides a more extensive set of properties that allow us to define what's allowed on what's not on each different environment.

Instead of looking for a tool that will allow us to rollback our DB changes automatically, we decided to implement backwards-compatible DB changes, for example adding a new column instead of renaming an existing one, postponing the deletion of the deprecated column until the release has been successfully installed.

Flyway vs Liquibase #Migration #Backwards-compatible

We were looking for a tool to help us integrating the migration scripts as part of our Deployment. At first sight both tools look very alike, are well integrated with Spring, have a fairly frequent development activity and short release cycles.

Liquibase puts a lot of emphasis on independence with the DB, allowing you to create the scripts on formats like JSON and YML, abstracting away from SQL, which it's also supported. Since we only work with one DB type across services we wouldn't take much advantage of this feature.

Flyway on the other hand has the advantage on being actively working on the integration with PostgreSQL 11, for it's upcoming version 6. Provides a more extensive set of properties that allow us to define what's allowed on what's not on each different environment.

Instead of looking for a tool that will allow us to rollback our DB changes automatically, we decided to implement backwards-compatible DB changes, for example adding a new column instead of renaming an existing one, postponing the deletion of the deprecated column until the release has been successfully installed.

Heroku Docker GitHub Node.js hapi Vue.js AWS Lambda Amazon S3 PostgreSQL Knex.js Checkly is a fairly young company and we're still working hard to find the correct mix of product features, price and audience.

We are focussed on tech B2B, but I always wanted to serve solo developers too. So I decided to make a $7 plan.

Why $7? Simply put, it seems to be a sweet spot for tech companies: Heroku, Docker, Github, Appoptics (Librato) all offer $7 plans. They must have done a ton of research into this, so why not piggy back that and try it out.

Enough biz talk, onto tech. The challenges were:

• Slice of a portion of the functionality so a $7 plan is still profitable. We call this the "plan limits"
• Update API and back end services to handle and enforce plan limits.
• Update the UI to kindly state plan limits are in effect on some part of the UI.
• Update the pricing page to reflect all changes.
• Keep the actual processing backend, storage and API's as untouched as possible.

In essence, we went from strictly volume based pricing to value based pricing. Here come the technical steps & decisions we made to get there.

1. We updated our PostgreSQL schema so plans now have an array of "features". These are string constants that represent feature toggles.
2. The Vue.js frontend reads these from the vuex store on login.
3. Based on these values, the UI has simple v-if statements to either just show the feature or show a friendly "please upgrade" button.
4. The hapi API has a hook on each relevant API endpoint that checks whether a user's plan has the feature enabled, or not.

Side note: We offer 10 SMS messages per month on the developer plan. However, we were not actually counting how many people were sending. We had to update our alerting daemon (that runs on Heroku and triggers SMS messages via AWS SNS) to actually bump a counter.

What we build is basically feature-toggling based on plan features. It is very extensible for future additions. Our scheduling and storage backend that actually runs users' monitoring requests (AWS Lambda) and stores the results (S3 and Postgres) has no knowledge of all of this and remained unchanged.

Hope this helps anyone building out their SaaS and is in a similar situation.

PostgreSQL Heroku Heroku Postgres Node.js Knex.js

Last week we rolled out a simple patch that decimated the response time of a Postgres query crucial to Checkly. It quite literally went from an average of ~100ms with peaks to 1 second to a steady 1ms to 10ms.

However, that patch was just the last step of a longer journey:

1. I looked at what API endpoints were using which queries and how their response time grew over time. Specifically the customer facing API endpoints that are directly responsible for rendering the first dashboard page of the product are crucial.

2. I looked at the Heroku metrics such as those reported by heroku pg:outlier and cross references that with "slowest response time" statistics.

3. I reproduced the production situation as best as possible on a local development machine and test my hypothesis that an composite index on a uuid field and a timestampz field would reduce response times.

This method secured the victory and we rolled out a new index last week. Response times plummeted. Read the full story in the blog post.

Postman is a nice desktop #REST #API client that allows you to save requests for later use. But it does not really support GraphQL, which I use everyday at work. So it was time to look for something else.

GraphiQL is a nice toy that has a desktop client, but you cannot save requests in any organized way. Most other clients I tried were either sluggish, didn't save requests, or didn't support cookies. Lack of cookie support is a no-no for work because we use session-based authentication in our internal API.

Then I stumbled upon Insomnia REST Client, and it clicked! Cookies work, GraphQL support is pretty good, UI looks nice and goes straight to the point. The only thing it lacks is a schema explorer, but I can always use GraphiQL if I ever need one, which is almost never.

Overall, I am very happy with it, and would recommend it to anyone seriously working with GraphQL. Insomnia is a godsend!