Grafana

Grafana

DevOps / Monitoring / Monitoring Tools
Avatar of conor
Tech Brand Mgr, Office of CTO at Uber·

Why we spent several years building an open source, large-scale metrics alerting system, M3, built for Prometheus:

By late 2014, all services, infrastructure, and servers at Uber emitted metrics to a Graphite stack that stored them using the Whisper file format in a sharded Carbon cluster. We used Grafana for dashboarding and Nagios for alerting, issuing Graphite threshold checks via source-controlled scripts. While this worked for a while, expanding the Carbon cluster required a manual resharding process and, due to lack of replication, any single node’s disk failure caused permanent loss of its associated metrics. In short, this solution was not able to meet our needs as the company continued to grow.

To ensure the scalability of Uber’s metrics backend, we decided to build out a system that provided fault tolerant metrics ingestion, storage, and querying as a managed platform...

https://eng.uber.com/m3/

(GitHub : https://github.com/m3db/m3)

READ MORE
M3: Uber’s Open Source, Large-scale Metrics Platform for Prometheus (eng.uber.com)
13 upvotes·1.7M views
Recommends
GrafanaGrafana
at

For our Predictive Analytics platform, we have used both Grafana and Kibana

Kibana has predictions and ML algorithms support, so if you need them, you may be better off with Kibana . The multi-variate analysis features it provide are very unique (not available in Grafana).

For everything else, definitely Grafana . Especially the number of supported data sources, and plugins clearly makes Grafana a winner (in just visualization and reporting sense). Creating your own plugin is also very easy. The top pros of Grafana (which it does better than Kibana ) are:

  • Creating and organizing visualization panels
  • Templating the panels on dashboards for repetetive tasks
  • Realtime monitoring, filtering of charts based on conditions and variables
  • Export / Import in JSON format (that allows you to version and save your dashboard as part of git)
READ MORE
Cenacle Technology & Consultation Services (cenacle.co.in)
9 upvotes·366.9K views
Recommends
KibanaKibana

I use both Kibana and Grafana on my workplace: Kibana for logging and Grafana for monitoring. Since you already work with Elasticsearch, I think Kibana is the safest choice in terms of ease of use and variety of messages it can manage, while Grafana has still (in my opinion) a strong link to metrics

READ MORE
8 upvotes·83.8K views

A huge part of our continuous deployment practices is to have granular alerting and monitoring across the platform. To do this, we run Sentry on-premise, inside our VPCs, for our event alerting, and we run an awesome observability and monitoring system consisting of StatsD, Graphite and Grafana. We have dashboards using this system to monitor our core subsystems so that we can know the health of any given subsystem at any moment. This system ties into our PagerDuty rotation, as well as alerts from some of our Amazon CloudWatch alarms (we’re looking to migrate all of these to our internal monitoring system soon).

READ MORE
How Mixmax Uses Node and Go to Process 250M Events a day - Mixmax Tech Stack (stackshare.io)
5 upvotes·478.1K views

One size definitely doesn’t fit all when it comes to open source monitoring solutions, and executing generally understood best practices in the context of unique distributed systems presents all sorts of problems. Megan Anctil, a senior engineer on the Technical Operations team at Slack gave a talk at an O’Reilly Velocity Conference sharing pain points and lessons learned at wrangling known technologies such as Icinga, Graphite, Grafana, and the Elastic Stack to best fit the company’s use cases.

At the time, Slack used a few well-known monitoring tools since it’s Technical Operations team wasn’t large enough to build an in-house solution for all of these. Nor did the team think it’s sustainable to throw money at the problem, given the volume of information processed and the not-insignificant price and rigidity of many vendor solutions. With thousands of servers across multiple regions and millions of metrics and documents being processed and indexed per second, the team had to figure out how to scale these technologies to fit Slack’s needs.

On the backend, they experimented with multiple clusters in both Graphite and ELK, distributed Icinga nodes, and more. At the same time, they’ve tried to build usability into Grafana that reflects the team’s mental models of the system and have found ways to make alerts from Icinga more insightful and actionable.

READ MORE
Our many monitoring monsters: Systems engineering conference: O'Reilly Velocity, San Jose, CA, June 19 - 22, 2017 (conferences.oreilly.com)
5 upvotes·298.3K views
Avatar of bramzor
Founder at CloudvCard·

After looking for a way to monitor or at least get a better overview of our infrastructure, we found out that Grafana (which I previously only used in ELK stacks) has a plugin available to fully integrate with Amazon CloudWatch . Which makes it way better for our use-case than the offer of the different competitors (most of them are even paid). There is also a CloudFlare plugin available, the platform we use to serve our DNS requests. Although we are a big fan of https://smashing.github.io/ (previously dashing), for now we are starting with Grafana .

READ MORE
5 upvotes·42.4K views
Needs advice
on
SysdigSysdig
vs
New RelicNew Relic
vs
DatadogDatadog

We are looking for a centralised monitoring solution for our application deployed on Amazon EKS. We would like to monitor using metrics from Kubernetes, AWS services (NeptuneDB, AWS Elastic Load Balancing (ELB), Amazon EBS, Amazon S3, etc) and application microservice's custom metrics.

We are expected to use around 80 microservices (not replicas). I think a total of 200-250 microservices will be there in the system with 10-12 slave nodes.

We tried Prometheus but it looks like maintenance is a big issue. We need to manage scaling, maintaining the storage, and dealing with multiple exporters and Grafana. I felt this itself needs few dedicated resources (at least 2-3 people) to manage. Not sure if I am thinking in the correct direction. Please confirm.

You mentioned Datadog and Sysdig charges per host. Does it charge per slave node?

READ MORE
5 upvotes·4.7K views
Replies (3)
Recommends
DatadogDatadog

Can't say anything to Sysdig. I clearly prefer Datadog as

  • they provide plenty of easy to "switch-on" plugins for various technologies (incl. most of AWS)
  • easy to code (python) agent plugins / api for own metrics
  • brillant dashboarding / alarms with many customization options
  • pricing is OK, there are cheaper options for specific use cases but if you want superior dashboarding / alarms I haven't seen a good competitor (despite your own Prometheus / Grafana / Kibana dog food)

IMHO NewRelic is "promising since years" ;) good ideas but bad integration between their products. Their Dashboard query language is really nice but lacks critical functions like multiple data sets or advanced calculations. Needless to say you get all of that with Datadog.

Need help setting up a monitoring / logging / alarm infrastructure? Send me a message!

READ MORE
5 upvotes·2 comments·1 view
Avatar of maikschroeder5741
CIO at Instana·
Recommends
InstanaInstana

Hi Medeti,

you are right. Building based on your stack something with open source is heavy lifting. A lot of people I know start with such a set-up, but quickly run into frustration as they need to dedicated their best people to build a monitoring which is doing the job in a professional way.

As you are microservice focussed and are looking for 'low implementation and maintenance effort', you might want to have a look at INSTANA, which was built with modern tool stacks in mind. https://www.instana.com/apm-for-microservices/

We have a public sand-box available if you just want to have a look at the product once and of course also a free-trial: https://www.instana.com/getting-started-with-apm/

Let me know if you need anything on top.

READ MORE
4 upvotes
View all (3)
Recommends
KibanaKibana

I use Kibana because it ships with the ELK stack. I don't find it as powerful as Splunk however it is light years above grepping through log files. We previously used Grafana but found it to be annoying to maintain a separate tool outside of the ELK stack. We were able to get everything we needed from Kibana.

READ MORE
4 upvotes·624.2K views

Data science and engineering teams at Lyft maintain several big data pipelines that serve as the foundation for various types of analysis throughout the business.

Apache Airflow sits at the center of this big data infrastructure, allowing users to “programmatically author, schedule, and monitor data pipelines.” Airflow is an open source tool, and “Lyft is the very first Airflow adopter in production since the project was open sourced around three years ago.”

There are several key components of the architecture. A web UI allows users to view the status of their queries, along with an audit trail of any modifications the query. A metadata database stores things like job status and task instance status. A multi-process scheduler handles job requests, and triggers the executor to execute those tasks.

Airflow supports several executors, though Lyft uses CeleryExecutor to scale task execution in production. Airflow is deployed to three Amazon Auto Scaling Groups, with each associated with a celery queue.

Audit logs supplied to the web UI are powered by the existing Airflow audit logs as well as Flask signal.

Datadog, Statsd, Grafana, and PagerDuty are all used to monitor the Airflow system.

READ MORE
Running Apache Airflow At Lyft – Lyft Engineering (eng.lyft.com)
4 upvotes·444.7K views
Avatar of sunilmchaudhari
Team Lead at DBS·
Needs advice
on
PrometheusPrometheus
vs
MetricbeatMetricbeat

Hi, We have a situation, where we are using Prometheus to get system metrics from PCF (Pivotal Cloud Foundry) platform. We send that as time-series data to Cortex via a Prometheus server and built a dashboard using Grafana. There is another pipeline where we need to read metrics from a Linux server using Metricbeat, CPU, memory, and Disk. That will be sent to Elasticsearch and Grafana will pull and show the data in a dashboard.

Is it OK to use Metricbeat for Linux server or can we use Prometheus?

What is the difference in system metrics sent by Metricbeat and Prometheus node exporters?

Regards, Sunil.

READ MORE
2 upvotes·8.2K views