Need advice about which tool to choose?Ask the StackShare community!
Kibana vs Splunk Cloud: What are the differences?
Developers describe Kibana as "Explore & Visualize Your Data". Kibana is an open source (Apache Licensed), browser based analytics and search dashboard for Elasticsearch. Kibana is a snap to setup and start using. Kibana strives to be easy to get started with, while also being flexible and powerful, just like Elasticsearch. On the other hand, Splunk Cloud is detailed as "Easy and fast way to analyze valuable machine data with the convenience of software as a service (SaaS)". If you're looking for all the benefits of Splunk® Enterprise with all the benefits of software-as-a-service, then look no further. Splunk Cloud is backed by a 100% uptime SLA, scales to over 10TB/day, and offers a highly secure environment.
Kibana belongs to "Monitoring Tools" category of the tech stack, while Splunk Cloud can be primarily classified under "Log Management".
Some of the features offered by Kibana are:
- Flexible analytics and visualization platform
- Real-time summary and charting of streaming data
- Intuitive interface for a variety of users
On the other hand, Splunk Cloud provides the following key features:
- Splunk Cloud delivers all the features of award-winning Splunk® Enterprise, as a cloud-based service. The platform provides access to various apps and enables centralized visibility across cloud, hybrid and on-premises environments
- Instant: Instant trial and instant conversion from POC to production
- Secure: Completed SOC2 Type 2 Attestation*. Dedicated cloud environments for each customer
"Easy to setup" is the top reason why over 76 developers like Kibana, while over 3 developers mention "More powerful & Integrates with on-prem & off-prem" as the leading cause for choosing Splunk Cloud.
Kibana is an open source tool with 12.4K GitHub stars and 4.81K GitHub forks. Here's a link to Kibana's open source repository on GitHub.
Airbnb, DigitalOcean, and 9GAG are some of the popular companies that use Kibana, whereas Splunk Cloud is used by Splunk, All Events in City, and SendHub. Kibana has a broader approval, being mentioned in 907 company stacks & 480 developers stacks; compared to Splunk Cloud, which is listed in 10 company stacks and 5 developer stacks.
We would like to detect unusual config changes that can potentially cause production outage.
Such as, SecurityGroup new allow/deny rule, AuthZ policy change, Secret key/certificate rotation, IP subnet add/drop. The problem is the source of all of these activities is different, i.e., AWS IAM, Amazon EC2, internal prod services, envoy sidecar, etc.
Which of the technology would be best suitable to detect only IMP events (not all activity) from various sources all workload running on AWS and also Splunk Cloud?
For continuous monitoring and detecting unusual configuration changes, I would suggest you look into AWS Config.
AWS Config enables you to assess, audit, and evaluate the configurations of your AWS resources. Config continuously monitors and records your AWS resource configurations and allows you to automate the evaluation of recorded configurations against desired configurations. Here is a list of supported AWS resources types and resource relationships with AWS Config https://docs.aws.amazon.com/config/latest/developerguide/resource-config-reference.html
Also as of Nov, 2019 - AWS Config launches support for third-party resources. You can now publish the configuration of third-party resources, such as GitHub repositories, Microsoft Active Directory resources, or any on-premises server into AWS Config using the new API. Here is more detail: https://docs.aws.amazon.com/config/latest/developerguide/customresources.html
If you have multiple AWS Account in your organization and want to detect changes there: https://docs.aws.amazon.com/config/latest/developerguide/aggregate-data.html
Lastly, if you already use Splunk Cloud in your enterprise and are looking for a consolidated view then, AWS Config is supported by Splunk Cloud as per their documentation too. https://aws.amazon.com/marketplace/pp/Splunk-Inc-Splunk-Cloud/B06XK299KV https://aws.amazon.com/marketplace/pp/Splunk-Inc-Splunk-Cloud/B06XK299KV
While it won't detect events as they happen a good stop gap would be to define your infrastructure config using terraform. You can then periodically run the terraform config against your environment and alert if there are any changes.
Consider using a combination of Netflix Security Monkey and AWS Guard Duty.
You can achieve automated detection and alerting, as well as automated recovery based on policies with these tools.
For instance, you could detect SecurityGroup rule changes that allow unrestricted egress from EC2 instances and then revert those changes automatically.
It's unclear from your post whether you want to detect events within the Splunk Cloud infrastructure or if you want to detect events indicated in data going to Splunk using the Splunk capabilities. If the latter, then Splunk has extremely rich capabilities in their query language and integrated alerting functions. With Splunk you can also run arbitrary Python scripts in response to certain events, so what you can't analyze and alert on with native functionality or plugins, you could write code to achieve.
Well there are clear advantages of using either tools, it all boils down to what exactly are you trying to achieve with this i.e do you want to proactive monitoring or do you want debug an incident/issue. Splunk definitely is superior in terms of proactively monitoring your logs for unusal events, but getting the cloudtrail logs across to splunk would require some not so straight forward setup (Splunk has a blueprint for this setup which uses AWS kinesis/Firehose). Cloudtrail on the other had is available out of the box from AWS, the setup is quite simple and straight forward. But analysing the log could require you setup Glue crawlers and you might have to use AWS Athena to run SQL Like query.
Refer: https://docs.aws.amazon.com/athena/latest/ug/cloudtrail-logs.html
In my personal experience the cost/effort involved in setting up splunk is not worth it for smaller workloads, whereas the AWS Cloudtrail/Glue/Athena would be less expensive setup(comparatively).
Alternatively you could look at something like sumologic, which has better integration with cloudtrail as opposed to splunk. Hope that helps.
I'd recommend using CloudTrail, it helped me a lot. But depending on your situation I'd recommed building a custom solution(like aws amazon-ssm-agent) which on configuration change makes an API call and logs them in grafana or kibana.
From a StackShare Community member: “We need better analytics & insights into our Elasticsearch cluster. Grafana, which ships with advanced support for Elasticsearch, looks great but isn’t officially supported/endorsed by Elastic. Kibana, on the other hand, is made and supported by Elastic. I’m wondering what people suggest in this situation."
For our Predictive Analytics platform, we have used both Grafana and Kibana
- Grafana based demo video: https://www.youtube.com/watch?v=tdTB2AcU4Sg
- Kibana based reporting screenshot: https://imgur.com/vuVvZKN
Kibana has predictions
and ML algorithms support, so if you need them, you may be better off with Kibana . The multi-variate analysis features it provide are very unique (not available in Grafana).
For everything else, definitely Grafana . Especially the number of supported data sources, and plugins clearly makes Grafana a winner (in just visualization and reporting sense). Creating your own plugin is also very easy. The top pros of Grafana (which it does better than Kibana ) are:
- Creating and organizing visualization panels
- Templating the panels on dashboards for repetetive tasks
- Realtime monitoring, filtering of charts based on conditions and variables
- Export / Import in JSON format (that allows you to version and save your dashboard as part of git)
I use both Kibana and Grafana on my workplace: Kibana for logging and Grafana for monitoring. Since you already work with Elasticsearch, I think Kibana is the safest choice in terms of ease of use and variety of messages it can manage, while Grafana has still (in my opinion) a strong link to metrics
After looking for a way to monitor or at least get a better overview of our infrastructure, we found out that Grafana (which I previously only used in ELK stacks) has a plugin available to fully integrate with Amazon CloudWatch . Which makes it way better for our use-case than the offer of the different competitors (most of them are even paid). There is also a CloudFlare plugin available, the platform we use to serve our DNS requests. Although we are a big fan of https://smashing.github.io/ (previously dashing), for now we are starting with Grafana .
I use Kibana because it ships with the ELK stack. I don't find it as powerful as Splunk however it is light years above grepping through log files. We previously used Grafana but found it to be annoying to maintain a separate tool outside of the ELK stack. We were able to get everything we needed from Kibana.
Kibana should be sufficient in this architecture for decent analytics, if stronger metrics is needed then combine with Grafana. Datadog also offers nice overview but there's no need for it in this case unless you need more monitoring and alerting (and more technicalities).
@Kibana, of course, because @Grafana looks like amateur sort of solution, crammed with query builder grouping aggregates, but in essence, as recommended by CERN - KIbana is the corporate (startup vectored) decision.
Furthermore, @Kibana comes with complexity adhering ELK stack, whereas @InfluxDB + @Grafana & co. recently have become sophisticated development conglomerate instead of advancing towards a understandable installation step by step inheritance.
Pros of Kibana
- Easy to setup88
- Free65
- Can search text45
- Has pie chart21
- X-axis is not restricted to timestamp13
- Easy queries and is a good way to view logs9
- Supports Plugins6
- Dev Tools4
- More "user-friendly"3
- Can build dashboards3
- Out-of-Box Dashboards/Analytics for Metrics/Heartbeat2
- Easy to drill-down2
- Up and running1
Pros of Splunk Cloud
- More powerful & Integrates with on-prem & off-prem7
- Free3
- Powerful log analytics3
- Pci compliance1
- Production debugger1
Sign up to add or upvote prosMake informed product decisions
Cons of Kibana
- Unintuituve7
- Works on top of elastic only4
- Elasticsearch is huge4
- Hardweight UI3