What is ELK and what are its top alternatives?
ELK, which stands for Elasticsearch, Logstash, and Kibana, is a popular open-source stack used for log management and analytics. Elasticsearch is a distributed, RESTful search and analytics engine capable of storing and searching large volumes of data in near real-time. Logstash is a flexible data collection, processing, and pipeline tool that can ingest data from multiple sources. Kibana is a data visualization platform that allows users to explore, analyze, and visualize data stored in Elasticsearch. Key features of ELK include scalability, flexibility, and a strong community support. However, setting up and maintaining the ELK stack can be complex for beginners, and it may require considerable resources in terms of hardware and maintenance.
- Grafana: Grafana is a multi-platform open-source analytics and visualization platform that allows users to query, visualize, alert on, and understand data. Key features include support for multiple data sources, high flexibility in creating dashboards, and a large plugin ecosystem. Pros of Grafana compared to ELK include easier setup and a more user-friendly interface, while a potential con could be less robust log management capabilities.
- Splunk: Splunk is a proprietary software platform for searching, monitoring, and analyzing machine-generated data. Key features include real-time visibility, machine learning capabilities, and a user-friendly interface. Pros of Splunk compared to ELK include comprehensive out-of-the-box features and ease of use, while a potential con could be higher cost for enterprise use.
- Sumo Logic: Sumo Logic is a cloud-based log management and analytics service that offers real-time insights into application and infrastructure logs. Key features include easy scalability, machine learning-based analytics, and centralized log management. Pros of Sumo Logic compared to ELK include ease of use and maintenance, while a potential con could be limited customization options.
- Logz.io: Logz.io is a cloud-native observability platform that offers log management, monitoring, and security analytics. Key features include AI-powered log analysis, built-in integrations with popular tools, and support for Kubernetes monitoring. Pros of Logz.io compared to ELK include ease of setup and management, while a potential con could be higher cost for enterprise features.
- Graylog: Graylog is an open-source log management platform that allows users to collect, index, and analyze log data in real-time. Key features include a robust search functionality, alerting capabilities, and support for plugins. Pros of Graylog compared to ELK include easier setup and configuration, while a potential con could be a less polished user interface.
- Fluentd: Fluentd is an open-source data collector that allows users to unify log data collection and consumption for better use and understanding of data. Key features include flexibility in data source inputs and outputs, plugins for various data sources, and easy integration with other tools. Pros of Fluentd compared to ELK include lightweight resource usage and high performance, while a potential con could be a steeper learning curve for beginners.
- Loggly: Loggly is a cloud-based log management service that offers real-time log analysis, monitoring, and alerting. Key features include centralized log data storage, searchable log data, and customizable dashboards. Pros of Loggly compared to ELK include ease of use and setup, while a potential con could be limited customization options.
- Datadog: Datadog is a cloud monitoring and analytics platform that provides visibility into the performance of applications, infrastructure, and logs. Key features include integrations with over 400 technologies, alerting capabilities, and customizable dashboards. Pros of Datadog compared to ELK include a centralized platform for monitoring and logs, while a potential con could be higher cost for enterprise features.
- Scalyr: Scalyr is a log management and observability platform that offers high-speed log search and analysis. Key features include advanced search capabilities, real-time log monitoring, and integrations with popular tools. Pros of Scalyr compared to ELK include high performance and speed in log processing, while a potential con could be limited scalability options.
- Logstash-Forwarder: Logstash-Forwarder, now known as Filebeat, is a lightweight log shipper for forwarding and centralizing log data. Key features include scalability, extensibility, and support for multiple data sources. Pros of Logstash-Forwarder compared to ELK include easy integration with Elasticsearch and Kibana, while a potential con could be limited data processing capabilities.
Top Alternatives to ELK
- Datadog
Datadog is the leading service for cloud-scale monitoring. It is used by IT, operations, and development teams who build and operate applications that run on dynamic or hybrid cloud infrastructure. Start monitoring in minutes with Datadog! ...
- Splunk
It provides the leading platform for Operational Intelligence. Customers use it to search, monitor, analyze and visualize machine data. ...
- Graylog
Centralize and aggregate all your log files for 100% visibility. Use our powerful query language to search through terabytes of log data to discover and analyze important information. ...
- New Relic
The world’s best software and DevOps teams rely on New Relic to move faster, make better decisions and create best-in-class digital experiences. If you run software, you need to run New Relic. More than 50% of the Fortune 100 do too. ...
- Kibana
Kibana is an open source (Apache Licensed), browser based analytics and search dashboard for Elasticsearch. Kibana is a snap to setup and start using. Kibana strives to be easy to get started with, while also being flexible and powerful, just like Elasticsearch. ...
- Grafana
Grafana is a general purpose dashboard and graph composer. It's focused on providing rich ways to visualize time series metrics, mainly though graphs but supports other ways to visualize data through a pluggable panel architecture. It currently has rich support for for Graphite, InfluxDB and OpenTSDB. But supports other data sources via plugins. ...
- Sentry
Sentry’s Application Monitoring platform helps developers see performance issues, fix errors faster, and optimize their code health. ...
- Amazon CloudWatch
It helps you gain system-wide visibility into resource utilization, application performance, and operational health. It retrieve your monitoring data, view graphs to help take automated action based on the state of your cloud environment. ...
ELK alternatives & related posts
- Monitoring for many apps (databases, web servers, etc)139
- Easy setup107
- Powerful ui87
- Powerful integrations84
- Great value70
- Great visualization54
- Events + metrics = clarity46
- Notifications41
- Custom metrics41
- Flexibility39
- Free & paid plans19
- Great customer support16
- Makes my life easier15
- Adapts automatically as i scale up10
- Easy setup and plugins9
- Super easy and powerful8
- AWS support7
- In-context collaboration7
- Rich in features6
- Docker support5
- Cost4
- Full visibility of applications4
- Monitor almost everything4
- Cute logo4
- Automation tools4
- Source control and bug tracking4
- Simple, powerful, great for infra4
- Easy to Analyze4
- Best than others4
- Best in the field3
- Expensive3
- Good for Startups3
- Free setup3
- APM2
- Expensive20
- No errors exception tracking4
- External Network Goes Down You Wont Be Logging2
- Complicated1
related Datadog posts
Our primary source of monitoring and alerting is Datadog. We’ve got prebuilt dashboards for every scenario and integration with PagerDuty to manage routing any alerts. We’ve definitely scaled past the point where managing dashboards is easy, but we haven’t had time to invest in using features like Anomaly Detection. We’ve started using Honeycomb for some targeted debugging of complex production issues and we are liking what we’ve seen. We capture any unhandled exceptions with Rollbar and, if we realize one will keep happening, we quickly convert the metrics to point back to Datadog, to keep Rollbar as clean as possible.
We use Segment to consolidate all of our trackers, the most important of which goes to Amplitude to analyze user patterns. However, if we need a more consolidated view, we push all of our data to our own data warehouse running PostgreSQL; this is available for analytics and dashboard creation through Looker.
Hey there! We are looking at Datadog, Dynatrace, AppDynamics, and New Relic as options for our web application monitoring.
Current Environment: .NET Core Web app hosted on Microsoft IIS
Future Environment: Web app will be hosted on Microsoft Azure
Tech Stacks: IIS, RabbitMQ, Redis, Microsoft SQL Server
Requirement: Infra Monitoring, APM, Real - User Monitoring (User activity monitoring i.e., time spent on a page, most active page, etc.), Service Tracing, Root Cause Analysis, and Centralized Log Management.
Please advise on the above. Thanks!
- API for searching logs, running reports3
- Alert system based on custom query results3
- Splunk language supports string, date manip, math, etc2
- Dashboarding on any log contents2
- Custom log parsing as well as automatic parsing2
- Query engine supports joining, aggregation, stats, etc2
- Rich GUI for searching live logs2
- Ability to style search results into reports2
- Granular scheduling and time window support1
- Query any log as key-value pairs1
- Splunk query language rich so lots to learn1
related Splunk posts
I use Kibana because it ships with the ELK stack. I don't find it as powerful as Splunk however it is light years above grepping through log files. We previously used Grafana but found it to be annoying to maintain a separate tool outside of the ELK stack. We were able to get everything we needed from Kibana.
We are currently exploring Elasticsearch and Splunk for our centralized logging solution. I need some feedback about these two tools. We expect our logs in the range of upwards > of 10TB of logging data.
- Open source19
- Powerfull13
- Well documented8
- Alerts6
- User authentification5
- Flexibel query and parsing language5
- Alerts and dashboards3
- User management3
- Easy query language and english parsing3
- Easy to install2
- Manage users and permissions1
- A large community1
- Free Version1
- Does not handle frozen indices at all1
related Graylog posts
New Relic
- Easy setup415
- Really powerful344
- Awesome visualization245
- Ease of use194
- Great ui151
- Free tier106
- Great tool for insights80
- Heroku Integration66
- Market leader55
- Peace of mind49
- Push notifications21
- Email notifications20
- Heroku Add-on17
- Error Detection and Alerting16
- Multiple language support13
- SQL Analysis11
- Server Resources Monitoring11
- Transaction Tracing9
- Apdex Scores8
- Azure Add-on8
- Analysis of CPU, Disk, Memory, and Network7
- Detailed reports7
- Performance of External Services6
- Error Analysis6
- Application Availability Monitoring and Alerting6
- Application Response Times6
- Most Time Consuming Transactions5
- JVM Performance Analyzer (Java)5
- Browser Transaction Tracing4
- Top Database Operations4
- Easy to use4
- Application Map3
- Weekly Performance Email3
- Pagoda Box integration3
- Custom Dashboards3
- Easy to setup2
- Background Jobs Transaction Analysis2
- App Speed Index2
- Super Expensive1
- Team Collaboration Tools1
- Metric Data Retention1
- Metric Data Resolution1
- Worst Transactions by User Dissatisfaction1
- Real User Monitoring Overview1
- Real User Monitoring Analysis and Breakdown1
- Time Comparisons1
- Access to Performance Data API1
- Incident Detection and Alerting1
- Best of the best, what more can you ask for1
- Best monitoring on the market1
- Rails integration1
- Free1
- Proce0
- Price0
- Exceptions0
- Cost0
- Pricing model doesn't suit microservices20
- UI isn't great10
- Expensive7
- Visualizations aren't very helpful7
- Hard to understand why things in your app are breaking5
related New Relic posts
Hey there! We are looking at Datadog, Dynatrace, AppDynamics, and New Relic as options for our web application monitoring.
Current Environment: .NET Core Web app hosted on Microsoft IIS
Future Environment: Web app will be hosted on Microsoft Azure
Tech Stacks: IIS, RabbitMQ, Redis, Microsoft SQL Server
Requirement: Infra Monitoring, APM, Real - User Monitoring (User activity monitoring i.e., time spent on a page, most active page, etc.), Service Tracing, Root Cause Analysis, and Centralized Log Management.
Please advise on the above. Thanks!
I need to choose a monitoring tool for my project, but currently, my application doesn't have much load or many users. My application is not generating GBs of data. We don't want to send the user information to New Relic because it's a 3rd party tool. And we can deploy Kibana locally on our server. What should I use, Kibana or New Relic?
- Easy to setup88
- Free65
- Can search text45
- Has pie chart21
- X-axis is not restricted to timestamp13
- Easy queries and is a good way to view logs9
- Supports Plugins6
- Dev Tools4
- More "user-friendly"3
- Can build dashboards3
- Out-of-Box Dashboards/Analytics for Metrics/Heartbeat2
- Easy to drill-down2
- Up and running1
- Unintuituve7
- Works on top of elastic only4
- Elasticsearch is huge4
- Hardweight UI3
related Kibana posts
Often enough I have to explain my way of going about setting up a CI/CD pipeline with multiple deployment platforms. Since I am a bit tired of yapping the same every single time, I've decided to write it up and share with the world this way, and send people to read it instead ;). I will explain it on "live-example" of how the Rome got built, basing that current methodology exists only of readme.md and wishes of good luck (as it usually is ;)).
It always starts with an app, whatever it may be and reading the readmes available while Vagrant and VirtualBox is installing and updating. Following that is the first hurdle to go over - convert all the instruction/scripts into Ansible playbook(s), and only stopping when doing a clear vagrant up
or vagrant reload
we will have a fully working environment. As our Vagrant environment is now functional, it's time to break it! This is the moment to look for how things can be done better (too rigid/too lose versioning? Sloppy environment setup?) and replace them with the right way to do stuff, one that won't bite us in the backside. This is the point, and the best opportunity, to upcycle the existing way of doing dev environment to produce a proper, production-grade product.
I should probably digress here for a moment and explain why. I firmly believe that the way you deploy production is the same way you should deploy develop, shy of few debugging-friendly setting. This way you avoid the discrepancy between how production work vs how development works, which almost always causes major pains in the back of the neck, and with use of proper tools should mean no more work for the developers. That's why we start with Vagrant as developer boxes should be as easy as vagrant up
, but the meat of our product lies in Ansible which will do meat of the work and can be applied to almost anything: AWS, bare metal, docker, LXC, in open net, behind vpn - you name it.
We must also give proper consideration to monitoring and logging hoovering at this point. My generic answer here is to grab Elasticsearch, Kibana, and Logstash. While for different use cases there may be better solutions, this one is well battle-tested, performs reasonably and is very easy to scale both vertically (within some limits) and horizontally. Logstash rules are easy to write and are well supported in maintenance through Ansible, which as I've mentioned earlier, are at the very core of things, and creating triggers/reports and alerts based on Elastic and Kibana is generally a breeze, including some quite complex aggregations.
If we are happy with the state of the Ansible it's time to move on and put all those roles and playbooks to work. Namely, we need something to manage our CI/CD pipelines. For me, the choice is obvious: TeamCity. It's modern, robust and unlike most of the light-weight alternatives, it's transparent. What I mean by that is that it doesn't tell you how to do things, doesn't limit your ways to deploy, or test, or package for that matter. Instead, it provides a developer-friendly and rich playground for your pipelines. You can do most the same with Jenkins, but it has a quite dated look and feel to it, while also missing some key functionality that must be brought in via plugins (like quality REST API which comes built-in with TeamCity). It also comes with all the common-handy plugins like Slack or Apache Maven integration.
The exact flow between CI and CD varies too greatly from one application to another to describe, so I will outline a few rules that guide me in it: 1. Make build steps as small as possible. This way when something breaks, we know exactly where, without needing to dig and root around. 2. All security credentials besides development environment must be sources from individual Vault instances. Keys to those containers should exist only on the CI/CD box and accessible by a few people (the less the better). This is pretty self-explanatory, as anything besides dev may contain sensitive data and, at times, be public-facing. Because of that appropriate security must be present. TeamCity shines in this department with excellent secrets-management. 3. Every part of the build chain shall consume and produce artifacts. If it creates nothing, it likely shouldn't be its own build. This way if any issue shows up with any environment or version, all developer has to do it is grab appropriate artifacts to reproduce the issue locally. 4. Deployment builds should be directly tied to specific Git branches/tags. This enables much easier tracking of what caused an issue, including automated identifying and tagging the author (nothing like automated regression testing!).
Speaking of deployments, I generally try to keep it simple but also with a close eye on the wallet. Because of that, I am more than happy with AWS or another cloud provider, but also constantly peeking at the loads and do we get the value of what we are paying for. Often enough the pattern of use is not constantly erratic, but rather has a firm baseline which could be migrated away from the cloud and into bare metal boxes. That is another part where this approach strongly triumphs over the common Docker and CircleCI setup, where you are very much tied in to use cloud providers and getting out is expensive. Here to embrace bare-metal hosting all you need is a help of some container-based self-hosting software, my personal preference is with Proxmox and LXC. Following that all you must write are ansible scripts to manage hardware of Proxmox, similar way as you do for Amazon EC2 (ansible supports both greatly) and you are good to go. One does not exclude another, quite the opposite, as they can live in great synergy and cut your costs dramatically (the heavier your base load, the bigger the savings) while providing production-grade resiliency.
This is my stack in Application & Data
JavaScript PHP HTML5 jQuery Redis Amazon EC2 Ubuntu Sass Vue.js Firebase Laravel Lumen Amazon RDS GraphQL MariaDB
My Utilities Tools
Google Analytics Postman Elasticsearch
My Devops Tools
Git GitHub GitLab npm Visual Studio Code Kibana Sentry BrowserStack
My Business Tools
Slack
- Beautiful89
- Graphs are interactive68
- Free57
- Easy56
- Nicer than the Graphite web interface34
- Many integrations26
- Can build dashboards18
- Easy to specify time window10
- Can collaborate on dashboards10
- Dashboards contain number tiles9
- Open Source5
- Integration with InfluxDB5
- Click and drag to zoom in5
- Authentification and users management4
- Threshold limits in graphs4
- Alerts3
- It is open to cloud watch and many database3
- Simple and native support to Prometheus3
- Great community support2
- You can use this for development to check memcache2
- You can visualize real time data to put alerts2
- Grapsh as code0
- Plugin visualizationa0
- No interactive query builder1
related Grafana posts
Grafana and Prometheus together, running on Kubernetes , is a powerful combination. These tools are cloud-native and offer a large community and easy integrations. At PayIt we're using exporting Java application metrics using a Dropwizard metrics exporter, and our Node.js services now use the prom-client npm library to serve metrics.
Why we spent several years building an open source, large-scale metrics alerting system, M3, built for Prometheus:
By late 2014, all services, infrastructure, and servers at Uber emitted metrics to a Graphite stack that stored them using the Whisper file format in a sharded Carbon cluster. We used Grafana for dashboarding and Nagios for alerting, issuing Graphite threshold checks via source-controlled scripts. While this worked for a while, expanding the Carbon cluster required a manual resharding process and, due to lack of replication, any single node’s disk failure caused permanent loss of its associated metrics. In short, this solution was not able to meet our needs as the company continued to grow.
To ensure the scalability of Uber’s metrics backend, we decided to build out a system that provided fault tolerant metrics ingestion, storage, and querying as a managed platform...
(GitHub : https://github.com/m3db/m3)
- Consolidates similar errors and makes resolution easy237
- Email Notifications121
- Open source108
- Slack integration84
- Github integration71
- Easy49
- User-friendly interface44
- The most important tool we use in production28
- Hipchat integration18
- Heroku Integration17
- Good documentation15
- Free tier14
- Self-hosted11
- Easy setup9
- Realiable7
- Provides context, and great stack trace6
- Feedback form on error pages4
- Love it baby4
- Gitlab integration3
- Filter by custom tags3
- Super user friendly3
- Captures local variables at each frame in backtraces3
- Easy Integration3
- Performance measurements1
- Confusing UI12
- Bundle size4
related Sentry posts
For my portfolio websites and my personal OpenSource projects I had started exclusively using React and JavaScript so I needed a way to track any errors that we're happening for my users that I didn't uncover during my personal UAT.
I had narrowed it down to two tools LogRocket and Sentry (I also tried Bugsnag but it did not make the final two). Before I get into this I want to say that both of these tools are amazing and whichever you choose will suit your needs well.
I firstly decided to go with LogRocket the fact that they had a recorded screen capture of what the user was doing when the bug happened was amazing... I could go back and rewatch what the user did to replicate that error, this was fantastic. It was also very easy to setup and get going. They had options for React and Redux.js so you can track all your Redux.js actions. I had a fairly large Redux.js store, this was ended up being a issue, it killed the processing power on my machine, Chrome ended up using 2-4gb of ram, so I quickly disabled the Redux.js option.
After using LogRocket for a month or so I decided to switch to Sentry. I noticed that Sentry was openSorce and everyone was talking about Sentry so I thought I may as well give it a test drive. Setting it up was so easy, I had everything up and running within seconds. It also gives you the option to wrap an errorBoundry in React so get more specific errors. The simplicity of Sentry was a breath of fresh air, it allowed me find the bug that was shown to the user and fix that very simply. The UI for Sentry is beautiful and just really clean to look at, and their emails are also just perfect.
I have decided to stick with Sentry for the long run, I tested pretty much all the JS error loggers and I find Sentry the best.
This is my stack in Application & Data
JavaScript PHP HTML5 jQuery Redis Amazon EC2 Ubuntu Sass Vue.js Firebase Laravel Lumen Amazon RDS GraphQL MariaDB
My Utilities Tools
Google Analytics Postman Elasticsearch
My Devops Tools
Git GitHub GitLab npm Visual Studio Code Kibana Sentry BrowserStack
My Business Tools
Slack
Amazon CloudWatch
- Monitor aws resources76
- Zero setup46
- Detailed Monitoring30
- Backed by Amazon23
- Auto Scaling groups19
- SNS and autoscaling integrations11
- Burstable instances metrics (t2 cpu credit balance)5
- HIPAA/PCI/SOC Compliance-friendly3
- Native tool for AWS so understand AWS out of the box1
- Poor Search Capabilities2
related Amazon CloudWatch posts
A huge part of our continuous deployment practices is to have granular alerting and monitoring across the platform. To do this, we run Sentry on-premise, inside our VPCs, for our event alerting, and we run an awesome observability and monitoring system consisting of StatsD, Graphite and Grafana. We have dashboards using this system to monitor our core subsystems so that we can know the health of any given subsystem at any moment. This system ties into our PagerDuty rotation, as well as alerts from some of our Amazon CloudWatch alarms (we’re looking to migrate all of these to our internal monitoring system soon).
After looking for a way to monitor or at least get a better overview of our infrastructure, we found out that Grafana (which I previously only used in ELK stacks) has a plugin available to fully integrate with Amazon CloudWatch . Which makes it way better for our use-case than the offer of the different competitors (most of them are even paid). There is also a CloudFlare plugin available, the platform we use to serve our DNS requests. Although we are a big fan of https://smashing.github.io/ (previously dashing), for now we are starting with Grafana .