Stack History: A Timeline of The Top 2019 Unicorn IPO's Tech Stack Evolution

In the early days of Airbnb, the question of how to pay hosts was nontrivial, since, at the time, there were few, if any, out-of-the-box providers of such services. The solution? The team built a system would generate a daily CSV detailing how much to deposit into each host’s bank account. Then, each weekday, someone would manually upload that spreadsheet via FTP to their bank.

Around the time of their Series A, Pinterest’s stack included Python and Django, with Tornado and Node.js as web servers. Memcached / Membase and Redis handled caching, with RabbitMQ handling queueing. Nginx, HAproxy and Varnish managed static-delivery and load-balancing, with persistent data storage handled by MySQL.

For many years, the marketing website was a customized WordPress instance, leaving the three-person engineering team—WeWork didn’t make its first full-time engineering hires until after its series A—to focus on building internal tooling. WordPress powered the main site until late 2014.

Uber’s early stack includes Python, MySQL and Mongo on the backend, with dispatch systems built on Node.js and Redis. Objective-C and Java powered the iPhone and Android apps, and websites were built with Backbone.js.

Early on, Lyft was powered by PHP running on Amazon EC2, with data stored in MongoDB. Client apps were native iOS and Android.

The situation in 2012 at PagerDuty was challenging: MySQL had been set up with default settings including Latin character encoding, not unicode. This started causing issues with non-romanized name users, various transliteration functions, and BLOB storage errors. PagerDuty decided to move to a universal character set once and for all.

Less than 1 minute of downtime and only a 2x increase in storage size for affected fields were goals for the migration.

It took some trial and error to get MySQL to store data the right way, and a daring few moments to flip from the old architecture to the new, but in the end the unicode migration went off without a hitch.

Early 2013

In early 2013, Airbnb tackled the problem of service discovery and load balancing in the context of a service oriented architecture (SOA) by building and releasing an open source tool called SmartStack. SmartStack is built on two other open source tools created by Airbnb called Nerve and Synapse.

Nerve is a service registration daemon that performs health checks that “creates ephemeral nodes in Zookeeper which contain information about the address/port combos for a backend available to serve requests for a particular service.”

Synapse is a transparent service discovery framework for connecting an SOA that reads the information in Zookeeper for available backends, and then uses that information to configure a local HAProxy process, which then routes requests between clients and services.

One of the earliest public references to Slack’s stack comes from a Twitter conversation. The Slack account states that “the messaging server is java, the app is php, db is mysql and solr for search,” and that uploaded files are “Stored on S3, but private files require authentication so requests go through the app.”

In late 2013, the Operations Engineering team at PagerDuty was made up of 4 engineers, and was comprised of generalists, each of whom had one or two areas of depth. Although the Operations Team ran its own on-call, each engineering team at PagerDuty also participated on the pager.

The Operations Engineering Team owned 150+ servers spanning multiple cloud providers, and used Chef to automate their infrastructure across the various cloud providers with a mix of completely custom cookbooks and customized community cookbooks.

Custom cookbooks were managed by Berkshelf, andach custom cookbook contained its own tests based on ChefSpec 3, coupled with Rspec.

Jenkins was used to GitHub for new changes and to handle unit testing of those features.

In 2014, Uber began working on Big Data Generation 1. Prior to 2014, all of the company’s data could fit into “a few traditional online transaction processing (OLTP) databases (in [their] case, MySQL and PostgreSQL ).” At that time, engineers were required to write individual code to access and join data across databases, a take made more difficult by the fact that there was no global view of all stored data, which was scattered around several OLTP databases with a “total data size on the order of a few terabytes.”

They chose Vertica as the analytical data warehouse, because of “because of its fast, scalable, and column-oriented design. We also developed multiple ad hoc ETL (Extract, Transform, and Load) jobs that copied data from different sources (i.e. Amazon S3, OLTP databases, service logs, etc.) into Vertica.”

In 2014, PagerDuty struggled with safely releasing reliable mobile applications to users due to some issues with how the code was being packaged and handled.

PagerDuty’s mobile apps are hybrid and used Cordova to share code between platforms. Coding was straightforward but packaging was not, as a separated Gulp-based build process was also being used. The PagerDuty team took a page from Java and started creating software artifacts.

Rather than checking in transformed code or publishing modules to NPM, the team started creating zipped-up build artifacts, which coincided perfectly with GitHub's Releases feature which arrived in 2013. So despite JavaScript lacking a standard packaged app format like a JAR, PagerDuty was still able to improve the build times and sizes of their mobile apps.

The early data ingestion pipeline at Pinterest used Kafka as the central message transporter, with the app servers writing messages directly to Kafka, which then uploaded log files to S3.

For databases, a custom Hadoop streamer pulled database data and wrote it to S3.

Challenges cited for this infrastructure included high operational overhead, as well as potential data loss occurring when Kafka broker outages led to an overflow of in-memory message buffering.

By mid-2014, around the time of the Series F, Pinterest users had already created more than 30 billion Pins, and the company was logging around 20 terabytes of new data daily, with around 10 petabytes of data in S3. To drive personalization for its users, and to empower engineers to build big data applications quickly, the data team built a self-serve Hadoop platform.

To start, they decoupled compute from storage, which meant teams would have to worry less about loading or synchronizing data, allowing existing or future clusters to make use of the data across a single shared file system.

A centralized Hive metastore act as the source of truth. They chose Hive for most of their Hadoop jobs “primarily because the SQL interface is simple and familiar to people across the industry.”

Dependency management takes place across three layers: Baked AMIs, which are large slow-loading dependencies pre-loaded on images; Automated Configurations (Masterless Puppets), which allows Puppet clients to “pull their configuration from S3 and set up a service that’s responsible for keeping S3 configurations in sync with the Puppet master;” and Runtime Staging on S3, which creates a working directory at runtime for each developer that pulls down its dependencies directly from S3.

Finally, they migrated their Hadoop jobs to Qubole, which “supported AWS/S3 and was relatively easy to get started on.”

By 2014, the DevOps team at Lyft decided to port their infrastructure code from Puppet to Salt. At that point, the Puppet code based included around "10,000 lines of spaghetti-code,” which was unfamiliar and challenging to the relatively new members of the DevOps team.

“The DevOps team felt that the Puppet infrastructure was too difficult to pick up quickly and would be impossible to introduce to [their] developers as the tool they’d use to manage their own services.”

To determine a path forward, the team assessed both Ansible and Salt, exploring four key areas: simplicity/ease of use, maturity, performance, and community.

They found that “Salt’s execution and state module support is more mature than Ansible’s, overall,” and that “Salt was faster than Ansible for state/playbook runs.” And while both have high levels of community support, Salt exceeded expectations in terms of friendless and responsiveness to opened issues.

On June 3, 2014 PagerDuty experienced a major issue: their Cassandra pipeline had stopped processing events and refused new ones. All in all, an outage was created that lasted 3 hours, along with additional degraded performance.

"Cassandra seems to have two modes: fine and catastrophe" said one of the PagerDuty engineers, as a seemingly routine repair had cascaded into a very bad situation. Constant memory pressure and underprovisioned amounts of RAM were isolated as a few of the factors that pointed to weaknesses in the way the cluster was set up.

After the outage, each node in the Cassandra cluster was replaced with m2.2xlarge EC2 nodes with 4 cores and 32GB of RAM. PagerDuty also moved away from using a multi-tenant Cassandra setup at that point, to help isolate failures in the future.

By the end of 2014, Postmates had expanded to 18 markets with around 6,000 drivers, and had completed more than a million deliveries. To grow its reach, in December 2014 Postmates announced the opening of its API, shifting their strategy from a singular product to a logistics platforms and allowing developers to build local logistics directly into their apps.

Late in 2014, around the time of the Series D, the WeWork engineering team had grown to 14, and while the backend was modernized with Rails and Active Admin CMS, the main website was lacking. The new headcount provided enough capacity to address the aging WordPress website.

As the team experimented with front-end technologies, they implemented a new signup flow with Angular, and other flows, including the Market Page, in React and Redux. The team says of that time: “If you’re following closely, yes, this means that in one rails app we had pages that included one or many of the following: jQuery, Angular, and React.”

The data-informed culture of Airbnb has been central to its success, and they have developed a sophisticated infrastructure to provide a reliable and scalable system for its users.

The basic flow sends source data into the system from two sources: a Kafka event stream and MySQL dumps from delivered through Sqoop. ETL and quality checks then happen in two separate Hive clusters, which are separated to isolate compute and storage resources and to provide “disaster recovery assurances if there ever was to be an outage.”

Partitioned Hive tables help ensure the data is immutable and reproducible, and Presto is used for almost all ad hoc queries on Hive managed tables.

By mid-2015, around the time of the Series E, the Digital department at WeWork had grown to more than 40 people to support the company’s growing product needs.

By then, they’d migrated the main website off of WordPress to Ruby on Rails, and a combination React, Angular, and jQuery, though there were efforts to move entirely to React for the front-end.

The backend was structured around a microservices architecture built partially in Node.js, along with a combination of Ruby, Python, Bash, and Go. Swift/Objective-C and Java powered the mobile apps.

These technologies power the listings on the website, as well as various internal tools, like community manager dashboards as well as RFID hardware for access management.

By mid-2015, Uber’s rider growth coupled with its cadence of releasing new services, like Eats and Freight, was pressuring the infrastructure. To allow the decoupling of consumption from production, and to add an abstraction layer between users, developers, and infrastructure, Uber built Catalyst, a serverless internal service mesh.

Uber decided to build their own severless solution, rather that using something like AWS Lambda, speed for its global production environments as well as introspectability.

“Make it easier for service owners to manage their local service environments” — Uber internal email from February 2015

“We want to have really easy processes and really easy infrastructure so that our feature developers can add things really quickly. One of the most important parts of that is the process of creating new services,” Casper Jensen, Uber Engineer, said. “We realized this meant Docker.”

He said it was easy to decide Docker was the route for them because it “was really easy to explain, people had read about it, understood the simple concepts about it,” and that it was an easy sell to the dev community with all the momentum surrounding everyone’s favorite container.

In late 2015, following the Series G, Pinterest began migrating their web experience to React, primarily because they “found React rendered faster than our previous template engine, had fewer obstacles to iterating on features and had a large developer community.”

The legacy setup consistent of Django, Python and Jinja on the backend, with Nunjucks handling template rendering on the client side. They wanted to move to React for handling template rendering across the board, but if they “switched the client-side rendering engine from Nunjucks to React, [they’d] also have to switch [their] server-side rendering, so they could share the same template syntax.”

They decided on an iterative approach that consolidated a single template rendering engine between client and server, since “If the server could interpret JavaScript, and use Nunjucks to render templates and share our client-side code, we could then move forward with an iterative migration to React.” The team decided to stand up a Node process, behind Nginx, and interpret JavaScript server-side.

Now, when a user agent makes a request, a latent module render requests that it needs data via an API call. Concurrently, a separate network call is made “to a co-located Node process to render the template as far as it can go with the data that it has.”

Node then responds with rendered templates, and along with a “holes” array to indicate what data was still needed to complete the render. Finally, the Python webapp makes an API call to fetch the remaining data, and each module is sent back to Node as completely independent module requests/in parallel/.

With this framework in place, Pinterest developers are in the process of replacing Nunjucks code with React components throughout the codebase.

To handle its growing observability needs, Postmates created and open sourced Cernan, a telemetry and logging aggregation server. Ceran is built on Rust and Lua, and can ingest data from many sources and then push or exposes what it’s collected to many destinations, or “sinks.” It can also create or manipulate in-flight data with programmable Lua filters.

Uber's early architecture was a monolithic Python backend on top of Postgres. Between 2014 and 2016, Uber moved many applications off of Postgres and onto MySQL, accessed through a sharding layer built in-house called Schemaless.

As the company scaled, Uber encountered issues with scaling writes due to write amplification issues (1 logical write causing 4 physical writes) and inefficient data replication with Postgres. Postgres replication works at the physical level, which means that it's not possible to replicate data between data bases running different versions. This added complexity and risk to upgrades.

MySQL, and the InnoDB storage engine in particular, can do replication at the statement level, which poses less risks and works between versions. Uber also found MySQL's caching and connection pooling capabilities to provide superior performance for their use cases.

To solve the problem of scheduling and executing arbitrary tasks in its distributed infrastructure, PagerDuty created an open-source tool called Scheduler. Scheduler is written in Scala and uses Cassandra for task persistence. It also adds Apache Kafka to handle task queuing and partitioning, with Akka to structure the library’s concurrency.

The service’s logic schedules a task by passing it to the Scheduler’s Scala API, which serializes the task metadata and enqueues it into Kafka. Scheduler then consumes the tasks, and posts them to Cassandra to prevent data loss.

In late 2016, Airbnb retired its legacy financial data pipeline. By then, they were transacting in 191 countries, with 70+ currencies and 20+ processors.

The new system decouples the financial logic from the product logic and allows for easier and more reliable horizontal scaling as transaction volume grows. It’s built on Apache Spark and store in an HDFS cluster. It’s written in Scala. They “chose Scala as the language because [they] wanted the latest features of Spark, as well as the other benefits of the language, like types, closures, immutability, lazy evaluation, etc.”

Lyft announced a major open source project in 2016, called Envoy. Envoy is a high-performance service bus for SOA, that abstracts details of the network layer and makes it easy to optimize traffic and see hotspots.

Before Envoy, Lyft used Amazon ELB for service discover and load balancing, a variety of PHP and Python apps, and some HAProxy where performance was critical. At just 30 services, enough sporadic network issues happened that the team decided to build something new, a way to get real network transparency even in a variable environment like Amazon EC2.

In an Envoy setup, each app connects through localhost, where a self-contained Envoy process is listening and then routing the traffic. The network is invisible to the app, in other words.

The Slack desktop app was originally written us the MacGap framework, which used Apple’s WebView to host web content inside of a native app frame. As this approach continued to present product limitations, Slack decided to migrate the desktop app to Electron. Electron is a platform that combines the rendering engine from Chromium and the Node.js runtime and module system. The desktop app is written as a modern ES6 + async/await React application.

For the desktop app, Slack takes a hybrid approach, wherein some of the assets ship as part of the app, but most of their assets and code are loaded remotely.

Throughout 2016, Slack began migrating from PHP5 to Hack. They cite several well-known challenges inherent to PHP, including surprise type conversions, inconsistency around reference semantics, inconsistencies in the standard library, and the fact that “PHP tries very, very hard to keep the request running, even if it has done something deeply strange.”

To overcome these challenges while maintaining the unique values of PHP, Slack turned to Hack, a gradual typing system for PHP. Hack runs on the HipHop Virtual Machine, or HHVM, an open source just-in-time (JIT) environment for PHP.

Airbnb’s web experience is powered by a Rails monolith, called Monorail, that talks to several different Java services. MySQL databases store business data and are partitioned by functionality, with messages and calendar management, for example, stored separately from the main booking flow in their own databases.

As traffic to the site continued growing, though, “one notable resource issue with MySQL databases [was] the increasing number of database connections from application servers.”

Airbnb uses AWS’s Relational Database Service (RDS) to power their MySQL instances, and “RDS uses the community edition of MySQL server, which employs a one-thread-per-connection model of connection management.” With Airbnb’s scale, this meant that their databases would hit the C10K problem, which states that “there is an upper bound in the number of connections that MySQL server can accept and serve without dramatically increasing the number of threads running, which severely degrades MySQL server performance.”

When an RDS MySQL server hits resource limits, users will have trouble connecting to the site.

MySQL does have dynamic thread pooling, but it’s only available in the enterprise edition; AWS MySQL RDS, though, doesn’t offer this feature, meaning Airbnb didn’t have access to dynamic thread pooling out-of-the-box.

After surveying several options, the team chose MariaDB MaxScale, which is “a MySQL database proxy that supports intelligent query routing in between client applications and a set of backend MySQL servers.”

Instead of using the MariaDB MaxScale off-the-shelf, however, they decided to fork it and implement their own version that would include connection pooling. Other MaxScale features, like request throttling and query blocklisting were implemented as well.

To enable horizontal scaling of the web application, the team deployed a MaxScale database proxy service in between app servers and MySQL servers. Through the service discovery system SmartStack, applications now “discover and connect to the database proxy service instead of the MySQL database,” allowing horizontal scaling to meet capacity demands.

Additionally, new Airbnb MaxScale proxy server instances can be launched to further enable horizontal scaling.

Slack’s data team works to “provide an ecosystem to help people in the company quickly and easily answer questions about usage, so they can make better and data informed decisions.” To achieve that goal, that rely on a complex data pipeline.

An in-house tool call Sqooper scrapes MySQL backups and pipe them to S3. Job queue and log data is sent to Kafka then persisted to S3 using an open source tool called Secor, which was created by Pinterest.

For compute, Amazon’s Elastic MapReduce (EMR) creates clusters preconfigured for Presto, Hive, and Spark.

Presto is then used for ad-hoc questions, validating data assumptions, exploring smaller datasets, and creating visualizations for some internal tools. Hive is used for larger data sets or longer time series data, and Spark allows teams to write efficient and robust batch and aggregation jobs. Most of the Spark pipeline is written in Scala.

Thrift binds all of these engines together with a typed schema and structured data.

Finally, the Hive Metastore serves as the ground truth for all data and its schema.

In 2015 Airbnb grew to a point that a scalable and distributed storage system was required to store data for some applications, especially search. Supporting low-latency personalized search was a major driver of the new architecture, as well as not having to index directly with the Rails and MySQL main application.

For this purpose, Airbnb created Nebula, which supports both real-time and batch access. The real-time part is powered by DynamoDB and the batch is a file format called HFileService, developed in-house at Airbnb.

Spark is used to merge all historical data together with the batch updates, with snapshots stored on S3. Nebula also can stream updates using Kinesis and Kafka, to keep other applications aware of the latest changes.

Postmates built a tool called Bazaar that helps onboard new partners and handles several routine tasks, like nightly emails to merchants alerting them about items that are out of stock.

Since they ran Bazaar across multiple instances, the team needed to avoid sending multiple emails to their partners by obtaining lock across multiple hosts. To solve their challenge, they created and open sourced ConsulMutEx, and an Elixir module for acquiring and releasing locks with Consul and other backends.

It works with Consul’s KV store, as well as other backends, including ets, Erlang’s in-memory database.

By early 2017, Airbnb was seeing around 75 million searches each day. In the midst of that growth, the team decided to rearchitect the web experience.

In the prior architecture, each page of the user’s journey, from searching, to viewing listings, to booking, was delivered standalone via Rails. The goal of the redesign was to maximize speed and fluidity.

In the new configuration, Hypernova is used to server-render React, and Redux is used for all API data and “globals,” like authentication state and experiment configurations.

In early 2017, Pinterest began moving its workload from EC2 instances to Docker containers with the goals of improving developer velocity, increasing reliability through immutable infrastructure, simplifying upgrades, and improving overall efficiency.

A single AMI (Amazon Machine Image) houses all container services, and all service-specific dependencies are put into this container.

Developers use Pinterest’s open-source tool Teletraan to launch containers, and the Docker container engine acts as the process manager to monitor and restart containers.

In mid-2017, Lyft decided to introduce a type system to its large JavaScript code base. After assessing the options, they decided to move forward with TypeScript, emphasizing its popularity both among Lyft engineers as well as among the developer community as a whole.

Since adopting TypeScript, they team has implemented TSLint for linting all TypeScript projects, and have created many TypeScript-only project, like a custom React JavaScript-to-TypeScript converter and an integration with their internal CSS library.

Following an EC2 outage in June of 2017, the WeWork team began exploring ways to scale their infrastructure while keeping things engineer-friendly.

The began dockerizing their services, using docker-compose to connect dependencies to the docker network and backend services. Jenkinsfile were added to each git repo, and Jenkins created docker [binary] images that were stored in JFrog Artifactory.

CloudFormation configuration files were created to deploy the docker images, with shell scripts handling basic tasks. In this configuration, it became possible to spin-up ECS services and tasks by definition.

The team created Environment Templates for ECS clusters, allowing services to live within environments (ECS Clusters) such as development, qa and production, and for docker images to be promoted between them.

As the WeWork footprint continued to expand, in mid-2018 the team began to explore the next generation of identity management to handle the global scale of the business.

The team decided to vet three languages for building microservices: Go, Kotlin, and Ruby. They compared the three by building a component of an identity system in each, and assessing the performance apples-to-apples.

After building out the systems and load testing each one, the team decided to implement the new system in Go for a few reasons. In addition to better performance under heavy loads, Go, according to the team, is a simpler language that will constrain developers to simpler code. Additionally, the development lifecycle is simpler with Go, since “there is little difference between running a service directly on a dev machine, to running it in a container, to running clustered instances of the service.”

In the implementation, they the Go grpc framework to handle various common infrastructure patterns, resulting in “in a clean common server pattern that we can reuse across our microservices.”

By late 2018, Pinterest was running one of the largest cloud Kafka deployments in the world, handling internal use cases like metrics, reporting, and monitoring, while also powering realtime features in the app, like content recommendations and spam detection.

To deliver that breath of coverage, Pinterest runs more than 2,000 brokers on AWS, “transporting more than 800 billion messages and more than 1.2 petabytes per day, and handling more than 15 million messages per second during the peak hours.”

They run Kafka in three AWS regions, with most of the workers in us-east-1, and with MirrorMaker transporting data among the three regions. Within each region, brokers are spread among multiple clusters for topic-level isolation, limited exposure across topics if one cluster fails.

Lyft had tried to make the switch to containers a few years before, with a home-grown provisioning system based on Salt, but ultimately waited until Kubernetes came on the scene and then matured.

Using a lot of Go for infrastructure and development (in addition to some Python), the Lyft engineering team aims to provide a unified developer experience going between dev and ops.

Lyft cites Kubernete's ecosystem and ability to be understood easily as reasons for adoption, and now uses it to support developers who have the option of deploying many different microservices with full control of the entire stack from the VM up.

“Now, [developers] just need to know that they can build an image that needs to run, and once they have that, they hand it over to the infrastructure team and then that’s it.”

Data science and engineering teams at Lyft maintain several big data pipelines that serve as the foundation for various types of analysis throughout the business.

Apache Airflow sits at the center of this big data infrastructure, allowing users to “programmatically author, schedule, and monitor data pipelines.” Airflow is an open source tool, and “Lyft is the very first Airflow adopter in production since the project was open sourced around three years ago.”

There are several key components of the architecture. A web UI allows users to view the status of their queries, along with an audit trail of any modifications the query. A metadata database stores things like job status and task instance status. A multi-process scheduler handles job requests, and triggers the executor to execute those tasks.

Airflow supports several executors, though Lyft uses CeleryExecutor to scale task execution in production. Airflow is deployed to three Amazon Auto Scaling Groups, with each associated with a celery queue.

Audit logs supplied to the web UI are powered by the existing Airflow audit logs as well as Flask signal.

Datadog, Statsd, Grafana, and PagerDuty are all used to monitor the Airflow system.

Following its migration from vanilla instances with autoscaling groups to Kubernetes, Postmates began facing challenges while “migrating workloads that needed to scale up very quickly.”

The built-in Horizontal Pod Autoscaler (HPA) automatically scales the number of pods in a replication controller, deployment or replica set based on observed CPU utilization. But the challenges for Postmates is that there’s no way to configure the scale velocity of one particular cluster with an HPA.

For Postmates, which runs at least three different types of applications with distinct performance and scaling characteristics, this proved problematic.

To overcome these challenges, the team created and open sourced the Configurable Horizontal Pod Autoscaler, which allows for fine-grained tuning on a per-HPA object basis. The result is that “you can configure critical services to scale down very slowly, while every other service could be configured to scale down instantly to reduce costs.”

By early 2019, a mix of JavaScript, ES6, and React powered the web app, with the desktop app shipping in Electron. Java and Kotlin powered the Android app with Objective-C and Swift powering iOS.

CTO Cal Henderson goes on to describe the backend: “we have our core application which powers and our API, which is written in PHP/Hacklang running on HHVM. We store data in MySQL using Vitess. For caching, we use Memcached and MCRouter. Our search service is based on SolrCloud, with various Java services for ranking. Our real-time messaging system uses WebSockets and is comprised of many services written in Java and Go.

“We use HAproxy for load balancing and Consul for configuration and some service discovery. Most of our services talk to each other over gRPC, though we have some Thrift and JSON-over-HTTP too. Our voice and video calling service is built in Elixir. A few different services are also written in Node. Our async task queue system is built on Kafka and Redis.

“Our data warehouse is built on open source tools, including Presto, Spark, Airflow, Hadoop and Kafka. For server configuration and management we use Terraform, Chef and Kubernetes. We use Prometheus for time series metrics and ELK for logging. Slack is largely hosted in AWS, in many regions globally.”