Stack History: A Timeline of Uber's Tech Stack Evolution

Uber’s early stack includes Python, MySQL and Mongo on the backend, with dispatch systems built on Node.js and Redis. Objective-C and Java powered the iPhone and Android apps, and websites were built with Backbone.js.

In 2014, Uber began working on Big Data Generation 1. Prior to 2014, all of the company’s data could fit into “a few traditional online transaction processing (OLTP) databases (in [their] case, MySQL and PostgreSQL ).” At that time, engineers were required to write individual code to access and join data across databases, a take made more difficult by the fact that there was no global view of all stored data, which was scattered around several OLTP databases with a “total data size on the order of a few terabytes.”

They chose Vertica as the analytical data warehouse, because of “because of its fast, scalable, and column-oriented design. We also developed multiple ad hoc ETL (Extract, Transform, and Load) jobs that copied data from different sources (i.e. Amazon S3, OLTP databases, service logs, etc.) into Vertica.”

In the early days, the Uber backend system was a monolithic software architecture with several app servers and a single database. The system was mainly written in Python and used SQLALchemy as the ORM-layer to the database.

In early 2014, Uber faced the fact that given their trip growth —roughly 20% per month—the solution for storing trips was going to run out of steam, both in terms of storage volume and IOPS (Input/Output Operations Per Second) by the end of the year.

To fix this problem, project Mezzanine was launched. The whole system, which Uber simply calls Schemaless in homage to its design, is written in Python. The initial version took about 5 months from idea to production deployment.

With interactions across each other and mobile devices, logging is important as it is information for internal cases like debugging and business cases like dynamic pricing.

With multiple Kafka clusters, data is archived into Hadoop before expiration. Data is ingested in realtime and indexed into an ELK stack. The ELK stack comprises of Elasticsearch, Logstash, and Kibana for searching and visualization.

Docker containers on Mesos run their microservices with consistent configurations at scale, along with Aurora for long-running services and cron jobs.

An open source tool called Phabricator powers many internal systems, from code review to documentation to process automation. Uber uses another open source tool, OpenGrok, to search through source code.

Two internal tools handle the load testing of services in a production environment. Similar to Chaos Monkey, uDestroy intentionally breaks things to provide Uber with a picture of unexpected failures. Another internal tool, Hailstorm, utilizes test accounts to simulate extra traffic on the system.

The highest-volume service in Uber’s Map Services is a service called Gurafu, which provides sophisticated routing. Gurafu runs a Java-based stack and is built on top of the DropWizard framework.

In August of 2014, Uber opened its API to all developers to access many of the primitives of powering the experience that is Uber. The API offered features such as passing a destination address to the Uber app, display pickup times, provide fare estimates, access trip history and much more.

It was launched with an Affiliate Program, which enabled developers to earn credits towards free Uber rides.

In early 2015, Uber Engineering migrated its business entities from integer identifiers to UUID identifiers as part of an initiative focused on using multiple active data centers. To do that, Uber engineers had to identify foreign key relationships between every table in the data warehouse—a nontrivial task by any accounting.

Uber’s solution was to observe and analyze incoming SQL queries to extract foreign key relationships, for which it built tool called Queryparser, which it open sourced.

Queryparser is written in Haskell, a tool that the team wasn’t previously familiar with but has strong support for language parsing. To help each other get up to speed, engineers started a weekly reading group to discuss Haskell books and documentation.

Around early 2015, Uber’s now deprecated tool for application-layer sharding, called Ringpop, incorporated Riak for high availability.

Ringpop was “a library that brings cooperation and coordination to distributed applications,” which they used, “to build scalable and fault tolerant systems for real-time dispatching by leveraging its sharding capabilities.”

Riak was used as a persistent data store for a portion of the new dispatching services. For example, as objects requiring persistence were generated, such as a new driver coming online with their associated mailbox of potential trips, Ringop would store these IDs as keys within Riak.

These layers could be utilized for additional services, such as stateful HTTP long-poll services, client/server sync services, rate limiters and geospatial services.

By mid-2015, Uber’s rider growth coupled with its cadence of releasing new services, like Eats and Freight, was pressuring the infrastructure. To allow the decoupling of consumption from production, and to add an abstraction layer between users, developers, and infrastructure, Uber built Catalyst, a serverless internal service mesh.

Uber decided to build their own severless solution, rather that using something like AWS Lambda, speed for its global production environments as well as introspectability.

In mid-2015, Uber created a solution for both iOS and Android in a platform-agnostic test runner called Octopus.

Octopus starts up two simulator/emulators: it launches the driver app and runs the driver test on one, while launching the rider app and running the rider test on the other. In addition, it handles the interprocess communication between the two tests.

In 2015, Uber released uber.github.io, highlighting their growing list of open source projects, which included TChannel and Ringpop at the time, and dozens more today.

As Uber planned to add additional services beyond rides, such as Uber Eats, the ideas of supply and demand needed to be generalized into distinct Supply and Demand services.

The logic to match all of supply and demand is a service called DISCO (dispatch optimization). It allows them to implement geospatial indexing, scale their systems, implement high availability, and to handle failures. It supports planning into the future and utilizing information as it becomes available, like revising routes on in-progress trips.

DISCO requires a geospatial index to make its decisions based on where all the supply is and where it is being routed. A geo index is also required for demand combined with a better routing engine to use all the information.

Vehicle movement is sent as location updates to geo by supply. To match riders to drivers or to display cars on a map, DISCO sends a request to geo by supply. To find the supply near a location, a circle’s worth of coverage is calculated centered on where the rider is located.

Using the cell IDs from the circle area all the relevant shards are contacted to return supply data.

Around 2015, the growing use of Uber’s data exposed limitations in the ETL and Vertica-centric setup, not to mention the increasing costs. “As our company grew, scaling our data warehouse became increasingly expensive. To cut down on costs, we started deleting older, obsolete data to free up space for new data.”

To overcome these challenges, Uber rebuilt their big data platform around Hadoop. “More specifically, we introduced a Hadoop data lake where all raw data was ingested from different online data stores only once and with no transformation during ingestion.”

“In order for users to access data in Hadoop, we introduced Presto to enable interactive ad hoc user queries, Apache Spark to facilitate programmatic access to raw data (in both SQL and non-SQL formats), and Apache Hive to serve as the workhorse for extremely large queries.”

To improve platform scalability and efficiency, Uber transitioned from JSON to Parquet, and built a central schema service to manage schemas and integrate different client libraries.

While the first generation big data platform was vulnerable to upstream data format changes, “ad hoc data ingestions jobs were replaced with a standard platform to transfer all source data in its original, nested format into the Hadoop data lake.”

These platform changes enabled the scaling challenges Uber was facing around that time: “On a daily basis, there were tens of terabytes of new data added to our data lake, and our Big Data platform grew to over 10,000 vcores with over 100,000 running batch jobs on any given day.”

"As Uber scaled its SOA, one goal was to improve the discoverability of the 500+ different services. In addition to this discoverability, other goals of the SOA included introducing a standard way of communication that provides type safety, validation, and fault tolerance.”

They decided that Apache Thrift most effectively met their needs in these areas. Thrift is a set of libraries and tools for building scalable cross-language services. To accomplish this, datatypes and service interfaces are defined in a language agnostic file. Then, code is generated to abstract the transport and encoding of RPC messages between services written in all of the languages we support (Python, Node, Go, etc.).

"Around 2015, the growing use of Uber’s data exposed limitations in the ETL and Vertica-centric setup, not to mention the increasing costs. “As our company grew, scaling our data warehouse became increasingly expensive. To cut down on costs, we started deleting older, obsolete data to free up space for new data.”

To overcome these challenges, Uber rebuilt their big data platform around Hadoop. “More specifically, we introduced a Hadoop data lake where all raw data was ingested from different online data stores only once and with no transformation during ingestion.”

“In order for users to access data in Hadoop, we introduced Presto to enable interactive ad hoc user queries, Apache Spark to facilitate programmatic access to raw data (in both SQL and non-SQL formats), and Apache Hive to serve as the workhorse for extremely large queries.”

As part of their big data efforts, Uber standardized their Hive data model in addition to providing different views of the same table. This helped provide two types of tables for all raw Hadoop data.

The first is a Changelog history table, which contains the history of all changelogs received for specific upstream table. This table is instrumental in enabling the users to scan through the history of changes for a specific table and can be merged per key to provide latest row values.

Secondly, the latest merged view of the upstream tables is saved in the merged snapshot table. The table contains the compacted merged view of all the historical changelogs received per key.

Users can join two tables more efficiently across a specific partition by avoiding a full table scan of the merged snapshot table when joining them. This helps users fetch the changed values from the changelog history table and join it against the merged snapshot table to create the full row of data. Standardizing the Hive data model improved data quality for the entire Big Data ecosystem.

“Make it easier for service owners to manage their local service environments” — Uber internal email from February 2015

“We want to have really easy processes and really easy infrastructure so that our feature developers can add things really quickly. One of the most important parts of that is the process of creating new services,” Casper Jensen, Uber Engineer, said. “We realized this meant Docker.”

He said it was easy to decide Docker was the route for them because it “was really easy to explain, people had read about it, understood the simple concepts about it,” and that it was an easy sell to the dev community with all the momentum surrounding everyone’s favorite container.

Uber's early architecture was a monolithic Python backend on top of Postgres. Between 2014 and 2016, Uber moved many applications off of Postgres and onto MySQL, accessed through a sharding layer built in-house called Schemaless.

As the company scaled, Uber encountered issues with scaling writes due to write amplification issues (1 logical write causing 4 physical writes) and inefficient data replication with Postgres. Postgres replication works at the physical level, which means that it's not possible to replicate data between data bases running different versions. This added complexity and risk to upgrades.

MySQL, and the InnoDB storage engine in particular, can do replication at the statement level, which poses less risks and works between versions. Uber also found MySQL's caching and connection pooling capabilities to provide superior performance for their use cases.

The frontline API is proxied through a HAProxy load balancer with NGINX as the fronted, which also handles SSL termination. This frontline API consist of 600 stateless endpoints that join together multiple services.

As part of the Marketplace stack, engineers in this area integrate with various other internal services, including logtron to log to disk and Kafka and uber-statsd-client, the Node.js client for statsd.

At the heart of Uber’s mobile app development are four primary apps: Android rider, Android driver, iOS rider, and iOS driver. Android developers build in Java, iOS in Objective C and Swift. Engineers across both platforms land code into a monolithic code base that ships each week.

They use some third-party libraries, but often build their own, since “Many open source libraries available are general-purpose, which can create binary bloat. For mobile engineering, every kilobyte matters.”

On Android, the build system is Gradle. For the UI, Butter Knife binds views and callbacks to fields and methods via annotation processing, and Picasso provides image loading.

As for iOS, all of the code lives in a monorepo built with Buck. For crash detection, KSCrash reports crashes to the internal reporting framework.

By mid-2016, Uber’s team was running more than one hundred thousand analytic queries daily. To keep up, they decided to redesign their analytics system, leveraging Presto, an open source SQL engine for large datasets, and Parquet, a columnar storage format for Hadoop.

Presto was chosen for a few reasons, including its scalability (according to Uber, it can access over five petabytes of data, and completes more than 90% of queries within 60 seconds).

To store its data, Uber also uses Parquet, a Hadoop storage solution that is compressible, has a columnar storage format, is encoded, and has ground-up support for nested data sets. Uber stores its data in columns instead of rows, because it removes the need to scan and discard unwanted data in rows. Columnar storage means more disk space saved, and improved query performance for large datasets.

Deep learning jobs require a unique challenge versus other jobs that run across multiple GPUs: they need every node to stay up and running till the job is complete, which is why Uber uses gang scheduling.

Gang scheduling (an optimization algorithm) means that for a cluster computing job to run, all the nodes have to be ready to run at the same time. This is especially useful in deep learning training, which involves constant feedback exchanged between nodes. Uber implemented gang scheduling in an Open Source framework called Horovod, to run Google’s TensorFlow machine learning software across multiple nodes.

Because they needed GPUs in upstream releases as well, Uber’s engineers chose to use Mesos containers over Docker.

The engineers at Uber used Horovod (and the TensorFlow package compatible with it) because it was easier to learn the rules of the MPI library in Horovod, than learning an entirely new system.

After adopting Apache Hadoop framework as its data platform, Uber created Marmaray, it’s open source framework for reliability ingesting and diapering data throughout its platform. “Marmaray ensures that all ingested raw data conforms to an appropriate source schema, maintaining a high level of quality so that analytical results are reliable.”

In mid-2015, Uber began exploring ways to scale ML across the organization, avoiding ML anti-patterns while standardizing workflows and tools. This effort led to Michelangelo.

Michelangelo consists of a mix of open source systems and components built in-house. The primary open sourced components used are HDFS, Spark, Samza, Cassandra, MLLib, XGBoost, and TensorFlow.