AWS CloudTrail vs Elasticsearch

Need advice about which tool to choose?Ask the StackShare community!

AWS CloudTrail

266
217
+ 1
14
Elasticsearch

24.4K
18.3K
+ 1
1.6K
Add tool

AWS CloudTrail vs Elasticsearch: What are the differences?

AWS CloudTrail: Record AWS API calls for your account and have log files delivered to you. With CloudTrail, you can get a history of AWS API calls for your account, including API calls made via the AWS Management Console, AWS SDKs, command line tools, and higher-level AWS services (such as AWS CloudFormation). The AWS API call history produced by CloudTrail enables security analysis, resource change tracking, and compliance auditing. The recorded information includes the identity of the API caller, the time of the API call, the source IP address of the API caller, the request parameters, and the response elements returned by the AWS service; Elasticsearch: Open Source, Distributed, RESTful Search Engine. Elasticsearch is a distributed, RESTful search and analytics engine capable of storing data and searching it in near real time. Elasticsearch, Kibana, Beats and Logstash are the Elastic Stack (sometimes called the ELK Stack).

AWS CloudTrail can be classified as a tool in the "Log Management" category, while Elasticsearch is grouped under "Search as a Service".

Some of the features offered by AWS CloudTrail are:

  • Increased Visibility- CloudTrail provides increased visibility into your user activity by recording AWS API calls. You can answer questions such as, what actions did a given user take over a given time period? For a given resource, which user has taken actions on it over a given time period? What is the source IP address of a given activity? Which activities failed due to inadequate permissions?
  • Durable and Inexpensive Log File Storage- CloudTrail uses Amazon S3 for log file storage and delivery, so log files are stored durably and inexpensively. You can use Amazon S3 lifecycle configuration rules to further reduce storage costs. For example, you can define rules to automatically delete old log files or archive them to Amazon Glacier for additional savings.
  • Easy Administration- CloudTrail is a fully managed service

On the other hand, Elasticsearch provides the following key features:

  • Distributed and Highly Available Search Engine.
  • Multi Tenant with Multi Types.
  • Various set of APIs including RESTful

"Very easy setup" is the primary reason why developers consider AWS CloudTrail over the competitors, whereas "Powerful api" was stated as the key factor in picking Elasticsearch.

Elasticsearch is an open source tool with 41.9K GitHub stars and 14K GitHub forks. Here's a link to Elasticsearch's open source repository on GitHub.

According to the StackShare community, Elasticsearch has a broader approval, being mentioned in 1976 company stacks & 937 developers stacks; compared to AWS CloudTrail, which is listed in 37 company stacks and 11 developer stacks.

Advice on AWS CloudTrail and Elasticsearch
Jigar Shah
Security Software Engineer at Pinterest · | 7 upvotes · 56.2K views

We would like to detect unusual config changes that can potentially cause production outage.

Such as, SecurityGroup new allow/deny rule, AuthZ policy change, Secret key/certificate rotation, IP subnet add/drop. The problem is the source of all of these activities is different, i.e., AWS IAM, Amazon EC2, internal prod services, envoy sidecar, etc.

Which of the technology would be best suitable to detect only IMP events (not all activity) from various sources all workload running on AWS and also Splunk Cloud?

See more
Replies (5)
Nati Abebe
Recommends
AWS Config

For continuous monitoring and detecting unusual configuration changes, I would suggest you look into AWS Config.

AWS Config enables you to assess, audit, and evaluate the configurations of your AWS resources. Config continuously monitors and records your AWS resource configurations and allows you to automate the evaluation of recorded configurations against desired configurations. Here is a list of supported AWS resources types and resource relationships with AWS Config https://docs.aws.amazon.com/config/latest/developerguide/resource-config-reference.html

Also as of Nov, 2019 - AWS Config launches support for third-party resources. You can now publish the configuration of third-party resources, such as GitHub repositories, Microsoft Active Directory resources, or any on-premises server into AWS Config using the new API. Here is more detail: https://docs.aws.amazon.com/config/latest/developerguide/customresources.html

If you have multiple AWS Account in your organization and want to detect changes there: https://docs.aws.amazon.com/config/latest/developerguide/aggregate-data.html

Lastly, if you already use Splunk Cloud in your enterprise and are looking for a consolidated view then, AWS Config is supported by Splunk Cloud as per their documentation too. https://aws.amazon.com/marketplace/pp/Splunk-Inc-Splunk-Cloud/B06XK299KV https://aws.amazon.com/marketplace/pp/Splunk-Inc-Splunk-Cloud/B06XK299KV

See more
Isaac Povey
Casual Software Engineer at Skedulo · | 5 upvotes · 37.9K views
Recommends
Terraform

While it won't detect events as they happen a good stop gap would be to define your infrastructure config using terraform. You can then periodically run the terraform config against your environment and alert if there are any changes.

See more
Matthew Rothstein
Recommends
Security Monkey

Consider using a combination of Netflix Security Monkey and AWS Guard Duty.

You can achieve automated detection and alerting, as well as automated recovery based on policies with these tools.

For instance, you could detect SecurityGroup rule changes that allow unrestricted egress from EC2 instances and then revert those changes automatically.

It's unclear from your post whether you want to detect events within the Splunk Cloud infrastructure or if you want to detect events indicated in data going to Splunk using the Splunk capabilities. If the latter, then Splunk has extremely rich capabilities in their query language and integrated alerting functions. With Splunk you can also run arbitrary Python scripts in response to certain events, so what you can't analyze and alert on with native functionality or plugins, you could write code to achieve.

See more
Vijayanand Narayanasharma
DevOps/TechOps Consultant at Qantas Loyalty · | 3 upvotes · 29.6K views
Recommends
AWS CloudTrail

Well there are clear advantages of using either tools, it all boils down to what exactly are you trying to achieve with this i.e do you want to proactive monitoring or do you want debug an incident/issue. Splunk definitely is superior in terms of proactively monitoring your logs for unusal events, but getting the cloudtrail logs across to splunk would require some not so straight forward setup (Splunk has a blueprint for this setup which uses AWS kinesis/Firehose). Cloudtrail on the other had is available out of the box from AWS, the setup is quite simple and straight forward. But analysing the log could require you setup Glue crawlers and you might have to use AWS Athena to run SQL Like query.

Refer: https://docs.aws.amazon.com/athena/latest/ug/cloudtrail-logs.html

In my personal experience the cost/effort involved in setting up splunk is not worth it for smaller workloads, whereas the AWS Cloudtrail/Glue/Athena would be less expensive setup(comparatively).

Alternatively you could look at something like sumologic, which has better integration with cloudtrail as opposed to splunk. Hope that helps.

See more
Ajay M
Recommends
AWS CloudTrail

I'd recommend using CloudTrail, it helped me a lot. But depending on your situation I'd recommed building a custom solution(like aws amazon-ssm-agent) which on configuration change makes an API call and logs them in grafana or kibana.

See more
Rana Usman Shahid
Chief Technology Officer at TechAvanza · | 5 upvotes · 136.9K views
Needs advice
on
Firebase
Elasticsearch
and
Algolia

Hey everybody! (1) I am developing an android application. I have data of around 3 million record (less than a TB). I want to save that data in the cloud. Which company provides the best cloud database services that would suit my scenario? It should be secured, long term useable, and provide better services. I decided to use Firebase Realtime database. Should I stick with Firebase or are there any other companies that provide a better service?

(2) I have the functionality of searching data in my app. Same data (less than a TB). Which search solution should I use in this case? I found Elasticsearch and Algolia search. It should be secure and fast. If any other company provides better services than these, please feel free to suggest them.

Thank you!

See more
Replies (2)
Josh Dzielak
Co-Founder & CTO at Orbit · | 6 upvotes · 103.1K views
Recommends
Algolia

Hi Rana, good question! From my Firebase experience, 3 million records is not too big at all, as long as the cost is within reason for you. With Firebase you will be able to access the data from anywhere, including an android app, and implement fine-grained security with JSON rules. The real-time-ness works perfectly. As a fully managed database, Firebase really takes care of everything. The only thing to watch out for is if you need complex query patterns - Firestore (also in the Firebase family) can be a better fit there.

To answer question 2: the right answer will depend on what's most important to you. Algolia is like Firebase is that it is fully-managed, very easy to set up, and has great SDKs for Android. Algolia is really a full-stack search solution in this case, and it is easy to connect with your Firebase data. Bear in mind that Algolia does cost money, so you'll want to make sure the cost is okay for you, but you will save a lot of engineering time and never have to worry about scale. The search-as-you-type performance with Algolia is flawless, as that is a primary aspect of its design. Elasticsearch can store tons of data and has all the flexibility, is hosted for cheap by many cloud services, and has many users. If you haven't done a lot with search before, the learning curve is higher than Algolia for getting the results ranked properly, and there is another learning curve if you want to do the DevOps part yourself. Both are very good platforms for search, Algolia shines when buliding your app is the most important and you don't want to spend many engineering hours, Elasticsearch shines when you have a lot of data and don't mind learning how to run and optimize it.

See more
Mike Endale
Recommends
Cloud Firestore

Rana - we use Cloud Firestore at our startup. It handles many million records without any issues. It provides you the same set of features that the Firebase Realtime Database provides on top of the indexing and security trims. The only thing to watch out for is to make sure your Cloud Functions have proper exception handling and there are no infinite loop in the code. This will be too costly if not caught quickly.

For search; Algolia is a great option, but cost is a real consideration. Indexing large number of records can be cost prohibitive for most projects. Elasticsearch is a solid alternative, but requires a little additional work to configure and maintain if you want to self-host.

Hope this helps.

See more
Get Advice from developers at your company using Private StackShare. Sign up for Private StackShare.
Learn More