StatsD

I see StatsD is commonly used in conjunction with Datadog. In fact, Datadog even has their own StatsD daemon (called DogStatsD) embedded in the DataDog agent. Can someone explain to me what it is that StatsD gives you which you don't already have with Datadog's APM and distributed tracing functionality?

A huge part of our continuous deployment practices is to have granular alerting and monitoring across the platform. To do this, we run Sentry on-premise, inside our VPCs, for our event alerting, and we run an awesome observability and monitoring system consisting of StatsD, Graphite and Grafana. We have dashboards using this system to monitor our core subsystems so that we can know the health of any given subsystem at any moment. This system ties into our PagerDuty rotation, as well as alerts from some of our Amazon CloudWatch alarms (we’re looking to migrate all of these to our internal monitoring system soon).

We use collectd because of it's low footprint and great capabilities. We use it to monitor our Google Compute Engine machines. More interestingly we setup collectd as StatsD replacement - all our Clojure services push application-level metrics using our own metrics library and collectd pushes them to Stackdriver

Data science and engineering teams at Lyft maintain several big data pipelines that serve as the foundation for various types of analysis throughout the business.

Apache Airflow sits at the center of this big data infrastructure, allowing users to “programmatically author, schedule, and monitor data pipelines.” Airflow is an open source tool, and “Lyft is the very first Airflow adopter in production since the project was open sourced around three years ago.”

There are several key components of the architecture. A web UI allows users to view the status of their queries, along with an audit trail of any modifications the query. A metadata database stores things like job status and task instance status. A multi-process scheduler handles job requests, and triggers the executor to execute those tasks.

Airflow supports several executors, though Lyft uses CeleryExecutor to scale task execution in production. Airflow is deployed to three Amazon Auto Scaling Groups, with each associated with a celery queue.

Audit logs supplied to the web UI are powered by the existing Airflow audit logs as well as Flask signal.

Datadog, Statsd, Grafana, and PagerDuty are all used to monitor the Airflow system.

The frontline API is proxied through a HAProxy load balancer with NGINX as the fronted, which also handles SSL termination. This frontline API consist of 600 stateless endpoints that join together multiple services.

As part of the Marketplace stack, engineers in this area integrate with various other internal services, including logtron to log to disk and Kafka and uber-statsd-client, the Node.js client for statsd.

Business and system counters go through StatsD and are pushed to InfluxDB StatsD