Logstash vs Prometheus: What are the differences?
Developers describe Logstash as "Collect, Parse, & Enrich Data". Logstash is a tool for managing events and logs. You can use it to collect logs, parse them, and store them for later use (like, for searching). If you store them in Elasticsearch, you can view and analyze them with Kibana. On the other hand, Prometheus is detailed as "An open-source service monitoring system and time series database, developed by SoundCloud". Prometheus is a systems and service monitoring system. It collects metrics from configured targets at given intervals, evaluates rule expressions, displays the results, and can trigger alerts if some condition is observed to be true.
Logstash can be classified as a tool in the "Log Management" category, while Prometheus is grouped under "Monitoring Tools".
Some of the features offered by Logstash are:
- Centralize data processing of all types
- Normalize varying schema and formats
- Quickly extend to custom log formats
On the other hand, Prometheus provides the following key features:
- a multi-dimensional data model (timeseries defined by metric name and set of key/value dimensions)
- a flexible query language to leverage this dimensionality
- no dependency on distributed storage
"Free" is the top reason why over 60 developers like Logstash, while over 32 developers mention "Powerful easy to use monitoring" as the leading cause for choosing Prometheus.
Logstash and Prometheus are both open source tools. It seems that Prometheus with 25K GitHub stars and 3.55K forks on GitHub has more adoption than Logstash with 10.3K GitHub stars and 2.78K GitHub forks.
Airbnb, reddit, and Typeform are some of the popular companies that use Logstash, whereas Prometheus is used by Uber Technologies, Slack, and DigitalOcean. Logstash has a broader approval, being mentioned in 561 company stacks & 278 developers stacks; compared to Prometheus, which is listed in 243 company stacks and 85 developer stacks.
What is Logstash?
What is Prometheus?
Need advice about which tool to choose?Ask the StackShare community!
Sign up to add, upvote and see more prosMake informed product decisions
Sign up to get full access to all the companiesMake informed product decisions
Sign up to get full access to all the tool integrationsMake informed product decisions
We recently implemented Thanos alongside Prometheus into our Kubernetes clusters, we had previously used a variety of different metrics systems and we wanted to make life simpler for everyone by just picking one.
Prometheus seemed like an obvious choice due to its powerful querying language, native Kubernetes support and great community. However we found it somewhat lacking when it came to being highly available, something that would be very important if we wanted this to be the single source of all our metrics.
Thanos came along and solved a lot of these problems. It allowed us to run multiple Prometheis without duplicating metrics, query multiple Prometheus clusters at once, and easily back up data and then query it. Now we have a single place to go if you want to view metrics across all our clusters, with many layers of redundancy to make sure this monitoring solution is as reliable and resilient as we could reasonably make it.
If you're interested in a bit more detail feel free to check out the blog I wrote on the subject that's linked.
Why we spent several years building an open source, large-scale metrics alerting system, M3, built for Prometheus:
By late 2014, all services, infrastructure, and servers at Uber emitted metrics to a Graphite stack that stored them using the Whisper file format in a sharded Carbon cluster. We used Grafana for dashboarding and Nagios for alerting, issuing Graphite threshold checks via source-controlled scripts. While this worked for a while, expanding the Carbon cluster required a manual resharding process and, due to lack of replication, any single node’s disk failure caused permanent loss of its associated metrics. In short, this solution was not able to meet our needs as the company continued to grow.
To ensure the scalability of Uber’s metrics backend, we decided to build out a system that provided fault tolerant metrics ingestion, storage, and querying as a managed platform...
(GitHub : https://github.com/m3db/m3)
At Kong while building an internal tool, we struggled to route metrics to Prometheus and logs to Logstash without incurring too much latency in our metrics collection.
We replaced nginx with OpenResty on the edge of our tool which allowed us to use the lua-nginx-module to run Lua code that captures metrics and records telemetry data during every request’s log phase. Our code then pushes the metrics to a local aggregator process (written in Go) which in turn exposes them in Prometheus Exposition Format for consumption by Prometheus. This solution reduced the number of components we needed to maintain and is fast thanks to NGINX and LuaJIT.
We have Prometheus as a monitoring engine as a part of our stack which contains Kubernetes cluster, container images and other open source tools. Also, I am aware that Sysdig can be integrated with Prometheus but I really wanted to know whether Sysdig or sysdig+prometheus will make better monitoring solution.
We primarily use Prometheus to gather metrics and statistics to display them in Grafana. Aside from that we poll Prometheus for our orchestration-solution "JCOverseer" to determine, which host is least occupied at the moment.
Gather metrics from systems and applications. Evaluate alerting rules. Alerts are pushed to OpsGenie and Slack.
We primarily use Prometheus to gather metrics and statistics to display them in Grafana.