Postmates

Following its migration from vanilla instances with autoscaling groups to Kubernetes, Postmates began facing challenges while “migrating workloads that needed to scale up very quickly.”

The built-in Horizontal Pod Autoscaler (HPA) automatically scales the number of pods in a replication controller, deployment or replica set based on observed CPU utilization. But the challenges for Postmates is that there’s no way to configure the scale velocity of one particular cluster with an HPA.

For Postmates, which runs at least three different types of applications with distinct performance and scaling characteristics, this proved problematic.

To overcome these challenges, the team created and open sourced the Configurable Horizontal Pod Autoscaler, which allows for fine-grained tuning on a per-HPA object basis. The result is that “you can configure critical services to scale down very slowly, while every other service could be configured to scale down instantly to reduce costs.”

Postmates built a tool called Bazaar that helps onboard new partners and handles several routine tasks, like nightly emails to merchants alerting them about items that are out of stock.

Since they ran Bazaar across multiple instances, the team needed to avoid sending multiple emails to their partners by obtaining lock across multiple hosts. To solve their challenge, they created and open sourced ConsulMutEx, and an Elixir module for acquiring and releasing locks with Consul and other backends.

It works with Consul’s KV store, as well as other backends, including ets, Erlang’s in-memory database.

To handle its growing observability needs, Postmates created and open sourced Cernan, a telemetry and logging aggregation server. Ceran is built on Rust and Lua, and can ingest data from many sources and then push or exposes what it’s collected to many destinations, or “sinks.” It can also create or manipulate in-flight data with programmable Lua filters.

By the end of 2014, Postmates had expanded to 18 markets with around 6,000 drivers, and had completed more than a million deliveries. To grow its reach, in December 2014 Postmates announced the opening of its API, shifting their strategy from a singular product to a logistics platforms and allowing developers to build local logistics directly into their apps.