Need advice about which tool to choose?Ask the StackShare community!

Amazon Kinesis

731
604
+ 1
9
AWS CloudTrail

295
280
+ 1
14
Add tool

AWS CloudTrail vs Amazon Kinesis: What are the differences?

Key Differences between AWS CloudTrail and Amazon Kinesis

1. AWS CloudTrail: AWS CloudTrail is a service that enables governance, compliance, operational auditing, and risk auditing of AWS account activity. It provides a detailed history of all API calls made within an AWS account, including who made the call, when it was made, and which resources were affected.

2. Amazon Kinesis: Amazon Kinesis is a fully managed service that makes it easy to collect, process, and analyze real-time, streaming data from various sources such as website clickstreams, IoT devices, social media, and more. It enables the processing of large amounts of data to generate real-time insights.

3. CloudTrail Focus: The main focus of AWS CloudTrail is to provide a comprehensive record and monitoring of all API activity within an AWS account. It captures both management events and data events, allowing organizations to ensure compliance with regulatory requirements, troubleshoot operational issues, and detect unauthorized activity.

4. Kinesis Focus: On the other hand, Amazon Kinesis is primarily designed for real-time data streaming and processing. It allows organizations to ingest, store, and process large-scale data streams in real-time, enabling actions such as real-time analytics, machine learning, and alerting.

5. Event Data vs. Stream Data: CloudTrail captures API events and generates logs in JSON format, storing them in an S3 bucket or sending them to CloudWatch Logs. It provides detailed information about actions performed within the AWS environment. In contrast, Kinesis deals with stream data, which is continuous, ordered, and scalable. It can handle massive amounts of data in real-time to enable real-time application processing.

6. Use Cases: AWS CloudTrail is commonly used for security and compliance purposes, as it allows organizations to monitor and track all access and changes made within their AWS accounts. It is essential for forensic investigations, audit trail analysis, and compliance reporting. On the other hand, Amazon Kinesis is widely used for real-time analytics, streaming data processing, and building real-time applications that require continuous ingestion, processing, and analysis of data.

In Summary, CloudTrail focuses on capturing and monitoring API activity within an AWS account for security and compliance purposes, while Kinesis is designed for real-time data streaming and processing for real-time analytics and application development.

Advice on Amazon Kinesis and AWS CloudTrail
Jigar Shah
Security Software Engineer at Pinterest · | 8 upvotes · 166.5K views

We would like to detect unusual config changes that can potentially cause production outage.

Such as, SecurityGroup new allow/deny rule, AuthZ policy change, Secret key/certificate rotation, IP subnet add/drop. The problem is the source of all of these activities is different, i.e., AWS IAM, Amazon EC2, internal prod services, envoy sidecar, etc.

Which of the technology would be best suitable to detect only IMP events (not all activity) from various sources all workload running on AWS and also Splunk Cloud?

See more
Replies (5)
Nati Abebe
Recommends
on
AWS ConfigAWS Config

For continuous monitoring and detecting unusual configuration changes, I would suggest you look into AWS Config.

AWS Config enables you to assess, audit, and evaluate the configurations of your AWS resources. Config continuously monitors and records your AWS resource configurations and allows you to automate the evaluation of recorded configurations against desired configurations. Here is a list of supported AWS resources types and resource relationships with AWS Config https://docs.aws.amazon.com/config/latest/developerguide/resource-config-reference.html

Also as of Nov, 2019 - AWS Config launches support for third-party resources. You can now publish the configuration of third-party resources, such as GitHub repositories, Microsoft Active Directory resources, or any on-premises server into AWS Config using the new API. Here is more detail: https://docs.aws.amazon.com/config/latest/developerguide/customresources.html

If you have multiple AWS Account in your organization and want to detect changes there: https://docs.aws.amazon.com/config/latest/developerguide/aggregate-data.html

Lastly, if you already use Splunk Cloud in your enterprise and are looking for a consolidated view then, AWS Config is supported by Splunk Cloud as per their documentation too. https://aws.amazon.com/marketplace/pp/Splunk-Inc-Splunk-Cloud/B06XK299KV https://aws.amazon.com/marketplace/pp/Splunk-Inc-Splunk-Cloud/B06XK299KV

See more
Isaac Povey
Casual Software Engineer at Skedulo · | 6 upvotes · 73.6K views
Recommends
on
TerraformTerraform

While it won't detect events as they happen a good stop gap would be to define your infrastructure config using terraform. You can then periodically run the terraform config against your environment and alert if there are any changes.

See more
Matthew Rothstein
Recommends
on
Security MonkeySecurity Monkey

Consider using a combination of Netflix Security Monkey and AWS Guard Duty.

You can achieve automated detection and alerting, as well as automated recovery based on policies with these tools.

For instance, you could detect SecurityGroup rule changes that allow unrestricted egress from EC2 instances and then revert those changes automatically.

It's unclear from your post whether you want to detect events within the Splunk Cloud infrastructure or if you want to detect events indicated in data going to Splunk using the Splunk capabilities. If the latter, then Splunk has extremely rich capabilities in their query language and integrated alerting functions. With Splunk you can also run arbitrary Python scripts in response to certain events, so what you can't analyze and alert on with native functionality or plugins, you could write code to achieve.

See more
Vijayanand Narayanasharma
DevOps/TechOps Consultant at Qantas Loyalty · | 3 upvotes · 64.5K views
Recommends
on
AWS CloudTrailAWS CloudTrail

Well there are clear advantages of using either tools, it all boils down to what exactly are you trying to achieve with this i.e do you want to proactive monitoring or do you want debug an incident/issue. Splunk definitely is superior in terms of proactively monitoring your logs for unusal events, but getting the cloudtrail logs across to splunk would require some not so straight forward setup (Splunk has a blueprint for this setup which uses AWS kinesis/Firehose). Cloudtrail on the other had is available out of the box from AWS, the setup is quite simple and straight forward. But analysing the log could require you setup Glue crawlers and you might have to use AWS Athena to run SQL Like query.

Refer: https://docs.aws.amazon.com/athena/latest/ug/cloudtrail-logs.html

In my personal experience the cost/effort involved in setting up splunk is not worth it for smaller workloads, whereas the AWS Cloudtrail/Glue/Athena would be less expensive setup(comparatively).

Alternatively you could look at something like sumologic, which has better integration with cloudtrail as opposed to splunk. Hope that helps.

See more
Recommends
on
AWS CloudTrailAWS CloudTrail

I'd recommend using CloudTrail, it helped me a lot. But depending on your situation I'd recommed building a custom solution(like aws amazon-ssm-agent) which on configuration change makes an API call and logs them in grafana or kibana.

See more
Manage your open source components, licenses, and vulnerabilities
Learn More
Pros of Amazon Kinesis
Pros of AWS CloudTrail
  • 9
    Scalable
  • 7
    Very easy setup
  • 3
    Good integrations with 3rd party tools
  • 2
    Very powerful
  • 2
    Backup to S3

Sign up to add or upvote prosMake informed product decisions

Cons of Amazon Kinesis
Cons of AWS CloudTrail
  • 3
    Cost
    Be the first to leave a con

    Sign up to add or upvote consMake informed product decisions

    What is Amazon Kinesis?

    Amazon Kinesis can collect and process hundreds of gigabytes of data per second from hundreds of thousands of sources, allowing you to easily write applications that process information in real-time, from sources such as web site click-streams, marketing and financial information, manufacturing instrumentation and social media, and operational logs and metering data.

    What is AWS CloudTrail?

    With CloudTrail, you can get a history of AWS API calls for your account, including API calls made via the AWS Management Console, AWS SDKs, command line tools, and higher-level AWS services (such as AWS CloudFormation). The AWS API call history produced by CloudTrail enables security analysis, resource change tracking, and compliance auditing. The recorded information includes the identity of the API caller, the time of the API call, the source IP address of the API caller, the request parameters, and the response elements returned by the AWS service.

    Need advice about which tool to choose?Ask the StackShare community!

    What companies use Amazon Kinesis?
    What companies use AWS CloudTrail?
    Manage your open source components, licenses, and vulnerabilities
    Learn More

    Sign up to get full access to all the companiesMake informed product decisions

    What tools integrate with Amazon Kinesis?
    What tools integrate with AWS CloudTrail?

    Sign up to get full access to all the tool integrationsMake informed product decisions

    Blog Posts

    Jul 2 2019 at 9:34PM

    Segment

    Google AnalyticsAmazon S3New Relic+25
    10
    7008
    GitHubPythonNode.js+47
    55
    73175
    GitHubDockerAmazon EC2+23
    12
    6748
    GitHubMySQLSlack+44
    109
    50930
    What are some alternatives to Amazon Kinesis and AWS CloudTrail?
    Kafka
    Kafka is a distributed, partitioned, replicated commit log service. It provides the functionality of a messaging system, but with a unique design.
    Apache Spark
    Spark is a fast and general processing engine compatible with Hadoop data. It can run in Hadoop clusters through YARN or Spark's standalone mode, and it can process data in HDFS, HBase, Cassandra, Hive, and any Hadoop InputFormat. It is designed to perform both batch processing (similar to MapReduce) and new workloads like streaming, interactive queries, and machine learning.
    Amazon SQS
    Transmit any volume of data, at any level of throughput, without losing messages or requiring other services to be always available. With SQS, you can offload the administrative burden of operating and scaling a highly available messaging cluster, while paying a low price for only what you use.
    Amazon Kinesis Firehose
    Amazon Kinesis Firehose is the easiest way to load streaming data into AWS. It can capture and automatically load streaming data into Amazon S3 and Amazon Redshift, enabling near real-time analytics with existing business intelligence tools and dashboards you’re already using today.
    Firehose.io
    Firehose is both a Rack application and JavaScript library that makes building real-time web applications possible.
    See all alternatives