Hudl

Measuring Availability: Instead of Nines, Let’s Count Minutes

It’s hard to find detailed explanations about how companies go about computing and tracking their availability, particularly for complex SaaS websites. Here’s how we do it for our primary web application, hudl.com.

Populating Fulla with SQL Data and Application Logs

This is the second in a series of posts on Fulla, Hudl’s data warehouse. This post discusses our methods to update Fulla daily with data from our production SQL databases and our application logs.

Migrating Millions of Users in Broad Daylight

In August we migrated our core user data (around 5.5MM user records) from SQL Server to MongoDB. We moved the data during the daytime while still taking full production traffic, maintaining nearly 100% availability for reads and writes during the course of the migration. Our CPO fittingly described it as akin to “swapping out a couple of the plane’s engines while it’s flying at 10,000 feet.” I’d like to share our approach to the migration and some of the code we used to do it.

Hello Fulla

Over the last year, the Data Engineering squad has been building a data warehouse called Fulla. Recently, the squad rethought our entire data warehouse stack. We’ve now released Fulla v2 and Hudlies are querying data like never before giving us a better understanding of our customers and our product.

Data Science on Firesquads: Classifying Emails with Naive Bayes

At Hudl, each squad on the product team takes two weeks each year to help out the coach relations team in an ongoing rotation known as Firesquads. This year, for Firesquads, the data science squad built a Naive Bayes classifier to automate the task of categorizing emails.

Faster and Cheaper: How Hudl Saved 50% and Doubled Performance

We took time to optimize our EC2 instance types. By finding the maximum load a server could handle we were able to run a quarter as many app servers. Our hourly spend dropped by 50%. Despite the huge cost savings, we also saw a 2x improvement in response times! This came about by moving to a newer instance family.

The Ultimate Test

It was a hot, sunny Thursday morning — not a cloud in sight. The smell of sunscreen hung in the air. The music pounded, shaking the ground. Hundreds of parents, friends, and fans filtered onto the sideline, anxiously awaiting the players to take the field.

Understanding User Interactions on Mobile

Understanding how users interact with your product is a key to lean software development. Seeing people trip over your navigation or completely turn around your expected use case may be cringe inducing, but it’s critical to understanding if what you built delivers. We’ll take a look at software we’ve used at Hudl to see exactly how a person uses an app in a non-intrusive manner. It’s not only saved us loads of time but also greatly increased our sample size.

Using Deep Learning to Find Basketball Highlights

Hudl stores petabytes of video. In that video there are a lot of awesome plays. Figuring out which plays are the most interesting and sifting through the uninteresting footage is a huge challenge. To solve this problem, we leveraged deep learning, Amazon Mechanical Turk, and crowd noise. The result: basketball highlights!

How Our Product Team Works

One part of Hudl I frequently have to explain to people outside the company is the structure of our product team. Fellow developers at other companies, friends I graduated with, and plenty of people in between want to know how Hudl works—and as it turns out, there’s a lot to talk about. We’re constantly evolving and learning more about how to keep our heads on straight, and as we do, we want to get the lessons learned on the table.

Exploring the Skunkworks Genius

Speed, innovation, and creativity… the key components of creative genius. Skunkworks is specifically designed to unleash the creative wrath of our product team. At Hudl, we use Skunkworks to explore new technologies and tools that make us better at what we do.

Deploying in the Multiverse

At Hudl, we like to move quickly. We are constantly fixing issues, building new features, and improving the experience for our coaches and athletes. We put a lot of thought into how we work and dedicate a lot of time to making sure we are working as efficiently as we can. So, when we began to run into major bottlenecks in our deployment process, we realized we needed a major change. We came up with a plan to break our monolithic application into smaller components, and thus The Multiverse was born.

Deploying Our Monolith Application

As a company, we understand that one of our key competitive edges is moving quickly. We develop and ship new features continuously. Before we started moving toward our microapplication architecture, we were deploying our monolithic application ten times a day. Even though the number of Monolith deployments is trending down as we break it out into multiple, smaller applications, we still rely on our deployment infrastructure to deliver multiple payloads daily.

Queuing Up Heavy Tasks to an Autoscaling Worker Farm

An autoscaling farm of AWS EC2 instances sits behind our front-facing web application, working on heavy, long-running tasks like video transcoding, thumbnail generation, and computer vision processing. It’s a battle-tested combination of queues, worker instances, and an orchestration service called Lifeguard that easily hammers through thousands of these CPU-bound jobs per minute.

Find Hidden Issues and Improvements for Your Product

Our Product team is split into several squads, each owning a different area of Hudl. At the start of a project, we focus in on that area. This allows us to move quickly and keep our goals top of mind. However, we also need to keep all areas of Hudl running smoothly. What’s the best way to prioritize fixing existing features against the new project you’re trying to deliver? At Hudl, we have a couple of processes in place to help.

Announcing Hudl.FFmpeg - a C# framework to make FFmpeg interaction simple

We have used FFmpeg for years at Hudl as our go to source for video editing. Our FFmpeg interactions were all relatively simple and were no more complex than a standard set of arguments defined by a string. This all changed when Campaign video automation came into the picture.

Better Retrospectives for Distributed Teams

In the eighteen months that I’ve been at Hudl, I’ve been on partially distributed teams. Currently, I’m the PM of a team of twelve, and six of those team members are remote. Because of that, I’ve had lots of opportunities to find out what works and what doesn’t work when you run retrospectives for distributed teams. In this post, I want to highlight the mistakes I’ve made, and then describe some of the best practices that make our retrospectives go smoothly.

Improving our automated tests 5x

Automated testing is a critical part of our deploy process at Hudl. We rely on it for feedback during deploys to production as well as our test environments. We started down the path of using automated regression testing about 2 years ago, early on trying various products until we found the right fit. We feel we have learned a lot during this time and made our process better and want to share our experiences.

Useful and Inspiring Product Management Blogs

One of my favorite things about working at Hudl is the culture of continual learning. I’ve never been with a company that puts such focus on constant improvement, genuinely striving to help everyone “level up”. Project Managers at Hudl are a very knowledge-hungry bunch. Between book discussions, weekly presentations on cutting-edge techniques and strategies, conference attendance, and more, there is never a lack of new information and always a chance to better our skills.

14 Tips for Self Learners

The self-learning path is the different from what most of us are used to, and perhaps a few tips from my journey (learned how to program 18 months ago without formal curriculum) may help you. These bits of wisdom were gained from hard-fought battles, but they were personal battles. A lot of these things will seem pretty obvious, some may seem over the top; if something doesn’t resonate with you, that’s fine. Take away what fits you.