Fluentd vs Prometheus

Need advice about which tool to choose?Ask the StackShare community!


+ 1

+ 1
Add tool

Fluentd vs Prometheus: What are the differences?

Introduction: Fluentd and Prometheus are both popular open-source data collection and monitoring tools used in the field of DevOps. Although they serve similar purposes, there are some key differences between these two tools.

  1. Data Collection Approach: Fluentd is a log collector and aggregator that operates on logs in real-time. It collects logs from various sources, transforms them, and sends them to various destinations for further processing. On the other hand, Prometheus is a time-series database and monitoring system that is primarily used for monitoring and alerting. It collects metrics data from configured targets periodically and stores it for analysis and visualization.

  2. Data Types: Fluentd is more oriented towards collecting and processing unstructured log data. It accepts logs in various formats such as text, JSON, and others. Prometheus, on the other hand, focuses on collecting and analyzing numeric time-series data. It is designed to monitor and analyze metrics related to system performance, resource utilization, and application behavior.

  3. Query Language: Fluentd uses its own query language called Fluent Query Language (FLQL). It allows users to filter and manipulate log data using a SQL-like syntax. Prometheus, on the other hand, uses Prometheus Query Language (PromQL) for querying and analyzing time-series data. PromQL provides a powerful set of operators and functions specifically tailored for time-series analysis.

  4. Monitoring Architecture: Fluentd follows a centralized architecture where logs are collected from various sources and sent to a centralized server for processing and analysis. It provides a unified view of logs across the system. In contrast, Prometheus follows a decentralized architecture where it scrapes metrics data directly from configured targets at regular intervals. Each target maintains its own metrics data, and Prometheus queries these targets individually.

  5. Alerting and Monitoring Capabilities: Fluentd focuses on log aggregation and routing, and does not have built-in support for alerting and monitoring. Prometheus, on the other hand, has powerful alerting and monitoring capabilities. It allows users to define alert rules based on metric conditions and send alerts to various notification channels. It also provides a flexible dashboard for visualizing and analyzing metrics data.

  6. Integration with Ecosystem: Fluentd is highly extensible and can be integrated with various other tools, services, and platforms. It provides plugins for different log sources and destinations, allowing seamless integration with existing infrastructure. Prometheus also has a wide range of integrations with different systems and frameworks. It provides exporters for collecting metrics from various sources and supports integrations with popular monitoring and visualization tools.

**In Summary, Fluentd is a log collector and aggregator that operates on unstructured log data, while Prometheus is a monitoring system that specializes in time-series data analysis and alerting. Fluentd uses FLQL for log querying, follows a centralized architecture, and does not have built-in monitoring capabilities. Prometheus uses PromQL for time-series analysis, follows a decentralized architecture, and provides powerful monitoring and alerting features. Both tools have extensive integrations with different systems and can be used together to create a comprehensive monitoring solution.

Advice on Fluentd and Prometheus
Susmita Meher
Senior SRE at African Bank · | 4 upvotes · 797.9K views
Needs advice

Looking for a tool which can be used for mainly dashboard purposes, but here are the main requirements:

  • Must be able to get custom data from AS400,
  • Able to display automation test results,
  • System monitoring / Nginx API,
  • Able to get data from 3rd parties DB.

Grafana is almost solving all the problems, except AS400 and no database to get automation test results.

See more
Replies (1)
Sakti Behera
Technical Specialist, Software Engineering at AT&T · | 3 upvotes · 583.3K views

You can look out for Prometheus Instrumentation (https://prometheus.io/docs/practices/instrumentation/) Client Library available in various languages https://prometheus.io/docs/instrumenting/clientlibs/ to create the custom metric you need for AS4000 and then Grafana can query the newly instrumented metric to show on the dashboard.

See more
Sunil Chaudhari
Needs advice

Hi, We have a situation, where we are using Prometheus to get system metrics from PCF (Pivotal Cloud Foundry) platform. We send that as time-series data to Cortex via a Prometheus server and built a dashboard using Grafana. There is another pipeline where we need to read metrics from a Linux server using Metricbeat, CPU, memory, and Disk. That will be sent to Elasticsearch and Grafana will pull and show the data in a dashboard.

Is it OK to use Metricbeat for Linux server or can we use Prometheus?

What is the difference in system metrics sent by Metricbeat and Prometheus node exporters?

Regards, Sunil.

See more
Replies (2)
Matthew Rothstein

If you're already using Prometheus for your system metrics, then it seems like standing up Elasticsearch just for Linux host monitoring is excessive. The node_exporter is probably sufficient if you'e looking for standard system metrics.

Another thing to consider is that Metricbeat / ELK use a push model for metrics delivery, whereas Prometheus pulls metrics from each node it is monitoring. Depending on how you manage your network security, opting for one solution over two may make things simpler.

See more

Hi Sunil! Unfortunately, I don´t have much experience with Metricbeat so I can´t advise on the diffs with Prometheus...for Linux server, I encourage you to use Prometheus node exporter and for PCF, I would recommend using the instana tile (https://www.instana.com/supported-technologies/pivotal-cloud-foundry/). Let me know if you have further questions! Regards Jose

See more
Mat Jovanovic
Head of Cloud at Mats Cloud · | 3 upvotes · 726.8K views
Needs advice

We're looking for a Monitoring and Logging tool. It has to support AWS (mostly 100% serverless, Lambdas, SNS, SQS, API GW, CloudFront, Autora, etc.), as well as Azure and GCP (for now mostly used as pure IaaS, with a lot of cognitive services, and mostly managed DB). Hopefully, something not as expensive as Datadog or New relic, as our SRE team could support the tool inhouse. At the moment, we primarily use CloudWatch for AWS and Pandora for most on-prem.

See more
Replies (2)
Lucas Rincon

this is quite affordable and provides what you seem to be looking for. you can see a whole thing about the APM space here https://www.apmexperts.com/observability/ranking-the-observability-offerings/

See more

I worked with Datadog at least one year and my position is that commercial tools like Datadog are the best option to consolidate and analyze your metrics. Obviously, if you can't pay the tool, the best free options are the mix of Prometheus with their Alert Manager and Grafana to visualize (that are complementary not substitutable). But I think that no use a good tool it's finally more expensive that use a not really good implementation of free tools and you will pay also to maintain its.

See more
Decisions about Fluentd and Prometheus
Leonardo Henrique da Paixão
Junior QA Tester at SolarMarket · | 15 upvotes · 362.2K views

The objective of this work was to develop a system to monitor the materials of a production line using IoT technology. Currently, the process of monitoring and replacing parts depends on manual services. For this, load cells, microcontroller, Broker MQTT, Telegraf, InfluxDB, and Grafana were used. It was implemented in a workflow that had the function of collecting sensor data, storing it in a database, and visualizing it in the form of weight and quantity. With these developed solutions, he hopes to contribute to the logistics area, in the replacement and control of materials.

See more
Get Advice from developers at your company using StackShare Enterprise. Sign up for StackShare Enterprise.
Learn More
Pros of Fluentd
Pros of Prometheus
  • 11
  • 9
    Great for Kubernetes node container log forwarding
  • 9
  • 8
  • 47
    Powerful easy to use monitoring
  • 38
    Flexible query language
  • 32
    Dimensional data model
  • 27
  • 23
    Active and responsive community
  • 22
    Extensive integrations
  • 19
    Easy to setup
  • 12
    Beautiful Model and Query language
  • 7
    Easy to extend
  • 6
  • 3
    Written in Go
  • 2
    Good for experimentation
  • 1
    Easy for monitoring

Sign up to add or upvote prosMake informed product decisions

Cons of Fluentd
Cons of Prometheus
    Be the first to leave a con
    • 12
      Just for metrics
    • 6
      Bad UI
    • 6
      Needs monitoring to access metrics endpoints
    • 4
      Not easy to configure and use
    • 3
      Supports only active agents
    • 2
      Written in Go
    • 2
      TLS is quite difficult to understand
    • 2
      Requires multiple applications and tools
    • 1
      Single point of failure

    Sign up to add or upvote consMake informed product decisions

    What is Fluentd?

    Fluentd collects events from various data sources and writes them to files, RDBMS, NoSQL, IaaS, SaaS, Hadoop and so on. Fluentd helps you unify your logging infrastructure.

    What is Prometheus?

    Prometheus is a systems and service monitoring system. It collects metrics from configured targets at given intervals, evaluates rule expressions, displays the results, and can trigger alerts if some condition is observed to be true.

    Need advice about which tool to choose?Ask the StackShare community!

    What companies use Fluentd?
    What companies use Prometheus?
    See which teams inside your own company are using Fluentd or Prometheus.
    Sign up for StackShare EnterpriseLearn More

    Sign up to get full access to all the companiesMake informed product decisions

    What tools integrate with Fluentd?
    What tools integrate with Prometheus?

    Sign up to get full access to all the tool integrationsMake informed product decisions

    Blog Posts

    Dec 8 2020 at 5:50PM


    May 21 2020 at 12:02AM

    Rancher Labs

    KubernetesAmazon EC2Grafana+12
    What are some alternatives to Fluentd and Prometheus?
    Logstash is a tool for managing events and logs. You can use it to collect logs, parse them, and store them for later use (like, for searching). If you store them in Elasticsearch, you can view and analyze them with Kibana.
    It provides the leading platform for Operational Intelligence. Customers use it to search, monitor, analyze and visualize machine data.
    collectd gathers statistics about the system it is running on and stores this information. Those statistics can then be used to find current performance bottlenecks (i.e. performance analysis) and predict future system load (i.e. capacity planning). Or if you just want pretty graphs of your private server and are fed up with some homegrown solution you're at the right place, too.
    It helps you keep the simple things simple by offering a lightweight way to forward and centralize logs and files.
    Elasticsearch is a distributed, RESTful search and analytics engine capable of storing data and searching it in near real time. Elasticsearch, Kibana, Beats and Logstash are the Elastic Stack (sometimes called the ELK Stack).
    See all alternatives