Prometheus vs StatsD: What are the differences?
Developers describe Prometheus as "An open-source service monitoring system and time series database, developed by SoundCloud". Prometheus is a systems and service monitoring system. It collects metrics from configured targets at given intervals, evaluates rule expressions, displays the results, and can trigger alerts if some condition is observed to be true. On the other hand, StatsD is detailed as "Simple daemon for easy stats aggregation". StatsD is a front-end proxy for the Graphite/Carbon metrics server, originally written by Etsy's Erik Kastner. StatsD is a network daemon that runs on the Node.js platform and listens for statistics, like counters and timers, sent over UDP and sends aggregates to one or more pluggable backend services (e.g., Graphite).
Prometheus and StatsD can be categorized as "Monitoring" tools.
Some of the features offered by Prometheus are:
- a multi-dimensional data model (timeseries defined by metric name and set of key/value dimensions)
- a flexible query language to leverage this dimensionality
- no dependency on distributed storage
On the other hand, StatsD provides the following key features:
- buckets: Each stat is in its own "bucket". They are not predefined anywhere. Buckets can be named anything that will translate to Graphite (periods make folders, etc)
- values: Each stat will have a value. How it is interpreted depends on modifiers. In general values should be integer.
- flush: After the flush interval timeout (defined by config.flushInterval, default 10 seconds), stats are aggregated and sent to an upstream backend service.
"Powerful easy to use monitoring" is the top reason why over 32 developers like Prometheus, while over 6 developers mention "Single responsibility" as the leading cause for choosing StatsD.
Prometheus and StatsD are both open source tools. It seems that Prometheus with 24.6K GitHub stars and 3.49K forks on GitHub has more adoption than StatsD with 14.1K GitHub stars and 1.83K GitHub forks.
Slack, Docplanner, and Uber Technologies are some of the popular companies that use Prometheus, whereas StatsD is used by Lyft, Shopify, and SendGrid. Prometheus has a broader approval, being mentioned in 235 company stacks & 84 developers stacks; compared to StatsD, which is listed in 72 company stacks and 16 developer stacks.
What is Prometheus?
What is StatsD?
Need advice about which tool to choose?Ask the StackShare community!
Sign up to add, upvote and see more prosMake informed product decisions
Sign up to get full access to all the companiesMake informed product decisions
Sign up to get full access to all the tool integrationsMake informed product decisions
We recently implemented Thanos alongside Prometheus into our Kubernetes clusters, we had previously used a variety of different metrics systems and we wanted to make life simpler for everyone by just picking one.
Prometheus seemed like an obvious choice due to its powerful querying language, native Kubernetes support and great community. However we found it somewhat lacking when it came to being highly available, something that would be very important if we wanted this to be the single source of all our metrics.
Thanos came along and solved a lot of these problems. It allowed us to run multiple Prometheis without duplicating metrics, query multiple Prometheus clusters at once, and easily back up data and then query it. Now we have a single place to go if you want to view metrics across all our clusters, with many layers of redundancy to make sure this monitoring solution is as reliable and resilient as we could reasonably make it.
If you're interested in a bit more detail feel free to check out the blog I wrote on the subject that's linked.
Why we spent several years building an open source, large-scale metrics alerting system, M3, built for Prometheus:
By late 2014, all services, infrastructure, and servers at Uber emitted metrics to a Graphite stack that stored them using the Whisper file format in a sharded Carbon cluster. We used Grafana for dashboarding and Nagios for alerting, issuing Graphite threshold checks via source-controlled scripts. While this worked for a while, expanding the Carbon cluster required a manual resharding process and, due to lack of replication, any single node’s disk failure caused permanent loss of its associated metrics. In short, this solution was not able to meet our needs as the company continued to grow.
To ensure the scalability of Uber’s metrics backend, we decided to build out a system that provided fault tolerant metrics ingestion, storage, and querying as a managed platform...
(GitHub : https://github.com/m3db/m3)
Data science and engineering teams at Lyft maintain several big data pipelines that serve as the foundation for various types of analysis throughout the business.
Apache Airflow sits at the center of this big data infrastructure, allowing users to “programmatically author, schedule, and monitor data pipelines.” Airflow is an open source tool, and “Lyft is the very first Airflow adopter in production since the project was open sourced around three years ago.”
There are several key components of the architecture. A web UI allows users to view the status of their queries, along with an audit trail of any modifications the query. A metadata database stores things like job status and task instance status. A multi-process scheduler handles job requests, and triggers the executor to execute those tasks.
Airflow supports several executors, though Lyft uses CeleryExecutor to scale task execution in production. Airflow is deployed to three Amazon Auto Scaling Groups, with each associated with a celery queue.
Audit logs supplied to the web UI are powered by the existing Airflow audit logs as well as Flask signal.
Datadog, Statsd, Grafana, and PagerDuty are all used to monitor the Airflow system.
We use collectd because of it's low footprint and great capabilities. We use it to monitor our Google Compute Engine machines. More interestingly we setup collectd as StatsD replacement - all our Clojure services push application-level metrics using our own metrics library and collectd pushes them to Stackdriver
A huge part of our continuous deployment practices is to have granular alerting and monitoring across the platform. To do this, we run Sentry on-premise, inside our VPCs, for our event alerting, and we run an awesome observability and monitoring system consisting of StatsD, Graphite and Grafana. We have dashboards using this system to monitor our core subsystems so that we can know the health of any given subsystem at any moment. This system ties into our PagerDuty rotation, as well as alerts from some of our Amazon CloudWatch alarms (we’re looking to migrate all of these to our internal monitoring system soon).
We have Prometheus as a monitoring engine as a part of our stack which contains Kubernetes cluster, container images and other open source tools. Also, I am aware that Sysdig can be integrated with Prometheus but I really wanted to know whether Sysdig or sysdig+prometheus will make better monitoring solution.
StatsD is used to track the number of messages we're publishing and the type of realtime subscribers. So it shows the number of longpoll connections, the number of websocket connections etc. It also tracks how Redis is performing.
We primarily use Prometheus to gather metrics and statistics to display them in Grafana. Aside from that we poll Prometheus for our orchestration-solution "JCOverseer" to determine, which host is least occupied at the moment.
Gather metrics from systems and applications. Evaluate alerting rules. Alerts are pushed to OpsGenie and Slack.
We primarily use Prometheus to gather metrics and statistics to display them in Grafana.