Why I am using Haskell in my free time?
I have 3 reasons for it. I am looking for:
Improve functional programming skill.
Improve problem-solving skill.
Laziness and mathematical abstractions behind Haskell makes it a wonderful language.
It is Pure functional, it helps me to write better Scala code.
Highly expressive language gives elegant ways to solve coding puzzle.
The algorithms and data infrastructure at Stitch Fix is housed in #AWS. Data acquisition is split between events flowing through Kafka, and periodic snapshots of PostgreSQL DBs. We store data in an Amazon S3 based data warehouse. Apache Spark on Yarn is our tool of choice for data movement and #ETL. Because our storage layer (s3) is decoupled from our processing layer, we are able to scale our compute environment very elastically. We have several semi-permanent, autoscaling Yarn clusters running to serve our data processing needs. While the bulk of our compute infrastructure is dedicated to algorithmic processing, we also implemented Presto for adhoc queries and dashboards.
Beyond data movement and ETL, most #ML centric jobs (e.g. model training and execution) run in a similarly elastic environment as containers running Python and R code on Amazon EC2 Container Service clusters. The execution of batch jobs on top of ECS is managed by Flotilla, a service we built in house and open sourced (see https://github.com/stitchfix/flotilla-os).
At Stitch Fix, algorithmic integrations are pervasive across the business. We have dozens of data products actively integrated systems. That requires serving layer that is robust, agile, flexible, and allows for self-service. Models produced on Flotilla are packaged for deployment in production using Khan, another framework we've developed internally. Khan provides our data scientists the ability to quickly productionize those models they've developed with open source frameworks in Python 3 (e.g. PyTorch, sklearn), by automatically packaging them as Docker containers and deploying to Amazon ECS. This provides our data scientist a one-click method of getting from their algorithms to production. We then integrate those deployments into a service mesh, which allows us to A/B test various implementations in our product.
For more info:
#DataScience #DataStack #Data
After splitting our monolith into a Rails API + a React Redux.js frontend app, it became a necessity to monitor frontend errors. Our frontend application is not your typical website, and features a lot of interesting SPA mechanics that need to be followed closely (many async flows, redux-saga , etc.) in addition to regular browser incompatibility issues. Rollbar kicks in so that we can monitor every bug that happens on our frontend, and aggregate this with almost 0 work. The number of occurrences and affected browsers on each occurence helps us understand the priority and severity of bugs even when our users don't tell us about them, so we can decide whether we need to fix this bug that was encountered by 1k users in less than a few days days VERSUS telling this SINGLE user to switch browsers because he's using a very outdated version that no one else uses. Now we also use Rollbar with Rails, Sidekiq and even AWS Lambda errors since the interface is quite convenient.
I use LastPass because it had Android support before 1Password. Also, it's just a great product. It gives me peace of mind with 2-step auth and a YubiKey.
The only thing that drives me nuts is the password generator, sometimes it just doesn't work on certain sites. That is why I wrote/use g20 😎
Heroku vs OpenShift. I've never decided which one is better. Heroku is easier to configure. Openshift provide a better machine for free. Heroku has many addons for free. I've chosen Heroku because of easy initial set-up. I had deployment based on git push. I also tried direct deployment of jar file. Currently Heroku runs my Docker image. Heroku has very good documentation like for beginners. So if you want to start with something, let's follow Heroku. On the other hand OpenShift seems like a PRO tool supported by @RedHat.