Lyft

Data science and engineering teams at Lyft maintain several big data pipelines that serve as the foundation for various types of analysis throughout the business.

Apache Airflow sits at the center of this big data infrastructure, allowing users to “programmatically author, schedule, and monitor data pipelines.” Airflow is an open source tool, and “Lyft is the very first Airflow adopter in production since the project was open sourced around three years ago.”

There are several key components of the architecture. A web UI allows users to view the status of their queries, along with an audit trail of any modifications the query. A metadata database stores things like job status and task instance status. A multi-process scheduler handles job requests, and triggers the executor to execute those tasks.

Airflow supports several executors, though Lyft uses CeleryExecutor to scale task execution in production. Airflow is deployed to three Amazon Auto Scaling Groups, with each associated with a celery queue.

Audit logs supplied to the web UI are powered by the existing Airflow audit logs as well as Flask signal.

Datadog, Statsd, Grafana, and PagerDuty are all used to monitor the Airflow system.

Lyft had tried to make the switch to containers a few years before, with a home-grown provisioning system based on Salt, but ultimately waited until Kubernetes came on the scene and then matured.

Using a lot of Go for infrastructure and development (in addition to some Python), the Lyft engineering team aims to provide a unified developer experience going between dev and ops.

Lyft cites Kubernete's ecosystem and ability to be understood easily as reasons for adoption, and now uses it to support developers who have the option of deploying many different microservices with full control of the entire stack from the VM up.

“Now, [developers] just need to know that they can build an image that needs to run, and once they have that, they hand it over to the infrastructure team and then that’s it.”

In mid-2017, Lyft decided to introduce a type system to its large JavaScript code base. After assessing the options, they decided to move forward with TypeScript, emphasizing its popularity both among Lyft engineers as well as among the developer community as a whole.

Since adopting TypeScript, they team has implemented TSLint for linting all TypeScript projects, and have created many TypeScript-only project, like a custom React JavaScript-to-TypeScript converter and an integration with their internal CSS library.

Lyft announced a major open source project in 2016, called Envoy. Envoy is a high-performance service bus for SOA, that abstracts details of the network layer and makes it easy to optimize traffic and see hotspots.

Before Envoy, Lyft used Amazon ELB for service discover and load balancing, a variety of PHP and Python apps, and some HAProxy where performance was critical. At just 30 services, enough sporadic network issues happened that the team decided to build something new, a way to get real network transparency even in a variable environment like Amazon EC2.

In an Envoy setup, each app connects through localhost, where a self-contained Envoy process is listening and then routing the traffic. The network is invisible to the app, in other words.

By 2014, the DevOps team at Lyft decided to port their infrastructure code from Puppet to Salt. At that point, the Puppet code based included around "10,000 lines of spaghetti-code,” which was unfamiliar and challenging to the relatively new members of the DevOps team.

“The DevOps team felt that the Puppet infrastructure was too difficult to pick up quickly and would be impossible to introduce to [their] developers as the tool they’d use to manage their own services.”

To determine a path forward, the team assessed both Ansible and Salt, exploring four key areas: simplicity/ease of use, maturity, performance, and community.

They found that “Salt’s execution and state module support is more mature than Ansible’s, overall,” and that “Salt was faster than Ansible for state/playbook runs.” And while both have high levels of community support, Salt exceeded expectations in terms of friendless and responsiveness to opened issues.

Early on, Lyft was powered by PHP running on Amazon EC2, with data stored in MongoDB. Client apps were native iOS and Android.