Need advice about which tool to choose?Ask the StackShare community!
AWS CloudTrail vs Splunk Cloud: What are the differences?
Key Differences between AWS CloudTrail and Splunk Cloud
AWS CloudTrail and Splunk Cloud are both widely used platforms for log management and analysis, but they differ in several key aspects. The following are the key differences between AWS CloudTrail and Splunk Cloud.
-
Data Sources:
- AWS CloudTrail primarily collects and logs data related to API activity within the AWS platform. It captures events like user logins, service events, and resource changes.
- Splunk Cloud, on the other hand, is a comprehensive log management and analysis platform that can collect data from a wide range of sources, including AWS CloudTrail, security appliances, application logs, and many others. It enables organizations to centralize their logs from multiple systems.
-
Deployment and Management:
- AWS CloudTrail is a fully managed service by Amazon Web Services (AWS), meaning that the infrastructure and management responsibilities are handled by AWS. Users only need to enable and configure CloudTrail through the AWS Management Console.
- Splunk Cloud, on the other hand, requires users to deploy and manage their own instance of Splunk Cloud. This involves setting up the infrastructure, configuring the environment, and managing the platform's ongoing operations.
-
Scalability and Elasticity:
- AWS CloudTrail scales automatically based on the demand of API activity. It can handle high volumes of events and data without requiring users to provision additional resources.
- Splunk Cloud provides scalability and elasticity by allowing users to dynamically add resources to their deployment as needed. Users can allocate additional resources to handle increased log ingestion and analysis requirements.
-
Pricing Model:
- AWS CloudTrail pricing is based on the volume of API events captured and stored, as well as additional charges for data transfer and storage. Users only pay for the resources they consume.
- Splunk Cloud pricing is based on the volume of data ingested and stored within the platform. The pricing model includes both the data ingestion and storage costs, with different tiers available based on the amount of data being processed.
-
Out-of-the-Box Functionality:
- AWS CloudTrail provides built-in integration with other AWS services and features, such as AWS Identity and Access Management (IAM) and AWS CloudWatch. It offers native integration and the ability to trigger actions based on events.
- Splunk Cloud offers extensive out-of-the-box functionality for log analysis, including pre-built dashboards, visualizations, and search capabilities. It provides a wide range of features and tools that cater to different use cases and log analysis requirements.
-
Customizability and Extensibility:
- AWS CloudTrail allows users to customize the retention period for their log data and configure notifications for specific events. However, customization options are limited compared to Splunk Cloud.
- Splunk Cloud offers extensive customization capabilities, including the ability to create custom dashboards, alerts, reports, and data visualizations. It also supports the development of custom applications and integrations through its rich ecosystem of apps and plugins.
In summary, AWS CloudTrail is primarily focused on logging AWS API activity, while Splunk Cloud provides a more comprehensive log management and analysis platform that can collect data from various sources. AWS CloudTrail is fully managed by AWS, while Splunk Cloud requires users to deploy and manage their own instance. AWS CloudTrail has a more limited customization scope compared to the extensive customization options offered by Splunk Cloud.
We would like to detect unusual config changes that can potentially cause production outage.
Such as, SecurityGroup new allow/deny rule, AuthZ policy change, Secret key/certificate rotation, IP subnet add/drop. The problem is the source of all of these activities is different, i.e., AWS IAM, Amazon EC2, internal prod services, envoy sidecar, etc.
Which of the technology would be best suitable to detect only IMP events (not all activity) from various sources all workload running on AWS and also Splunk Cloud?
For continuous monitoring and detecting unusual configuration changes, I would suggest you look into AWS Config.
AWS Config enables you to assess, audit, and evaluate the configurations of your AWS resources. Config continuously monitors and records your AWS resource configurations and allows you to automate the evaluation of recorded configurations against desired configurations. Here is a list of supported AWS resources types and resource relationships with AWS Config https://docs.aws.amazon.com/config/latest/developerguide/resource-config-reference.html
Also as of Nov, 2019 - AWS Config launches support for third-party resources. You can now publish the configuration of third-party resources, such as GitHub repositories, Microsoft Active Directory resources, or any on-premises server into AWS Config using the new API. Here is more detail: https://docs.aws.amazon.com/config/latest/developerguide/customresources.html
If you have multiple AWS Account in your organization and want to detect changes there: https://docs.aws.amazon.com/config/latest/developerguide/aggregate-data.html
Lastly, if you already use Splunk Cloud in your enterprise and are looking for a consolidated view then, AWS Config is supported by Splunk Cloud as per their documentation too. https://aws.amazon.com/marketplace/pp/Splunk-Inc-Splunk-Cloud/B06XK299KV https://aws.amazon.com/marketplace/pp/Splunk-Inc-Splunk-Cloud/B06XK299KV
While it won't detect events as they happen a good stop gap would be to define your infrastructure config using terraform. You can then periodically run the terraform config against your environment and alert if there are any changes.
Consider using a combination of Netflix Security Monkey and AWS Guard Duty.
You can achieve automated detection and alerting, as well as automated recovery based on policies with these tools.
For instance, you could detect SecurityGroup rule changes that allow unrestricted egress from EC2 instances and then revert those changes automatically.
It's unclear from your post whether you want to detect events within the Splunk Cloud infrastructure or if you want to detect events indicated in data going to Splunk using the Splunk capabilities. If the latter, then Splunk has extremely rich capabilities in their query language and integrated alerting functions. With Splunk you can also run arbitrary Python scripts in response to certain events, so what you can't analyze and alert on with native functionality or plugins, you could write code to achieve.
Well there are clear advantages of using either tools, it all boils down to what exactly are you trying to achieve with this i.e do you want to proactive monitoring or do you want debug an incident/issue. Splunk definitely is superior in terms of proactively monitoring your logs for unusal events, but getting the cloudtrail logs across to splunk would require some not so straight forward setup (Splunk has a blueprint for this setup which uses AWS kinesis/Firehose). Cloudtrail on the other had is available out of the box from AWS, the setup is quite simple and straight forward. But analysing the log could require you setup Glue crawlers and you might have to use AWS Athena to run SQL Like query.
Refer: https://docs.aws.amazon.com/athena/latest/ug/cloudtrail-logs.html
In my personal experience the cost/effort involved in setting up splunk is not worth it for smaller workloads, whereas the AWS Cloudtrail/Glue/Athena would be less expensive setup(comparatively).
Alternatively you could look at something like sumologic, which has better integration with cloudtrail as opposed to splunk. Hope that helps.
I'd recommend using CloudTrail, it helped me a lot. But depending on your situation I'd recommed building a custom solution(like aws amazon-ssm-agent) which on configuration change makes an API call and logs them in grafana or kibana.
Pros of AWS CloudTrail
- Very easy setup7
- Good integrations with 3rd party tools3
- Very powerful2
- Backup to S32
Pros of Splunk Cloud
- More powerful & Integrates with on-prem & off-prem7
- Free3
- Powerful log analytics3
- Pci compliance1
- Production debugger1