Splunk

It is a data processing, storage and analysis platform. It is scalable - just add more CPUs / servers for greater throughput. It is suitable for processing high volume data such as system logs, to provide valuable insights into IT performance and usage.

Matano is the open source security lake platform for AWS. It lets you ingest petabytes of security and log data from various sources, store and query them in a data lake, and create Python detections as code for realtime alerting.

Logstash is a tool for managing events and logs. You can use it to collect logs, parse them, and store them for later use (like, for searching). If you store them in Elasticsearch, you can view and analyze them with Kibana.

It is intended as a successor to the popular log4j project. It is divided into three modules, logback-core, logback-classic and logback-access. The logback-core module lays the groundwork for the other two modules, logback-classic natively implements the SLF4J API so that you can readily switch back and forth between logback and other logging frameworks and logback-access module integrates with Servlet containers, such as Tomcat and Jetty, to provide HTTP-access log functionality.

It is a simple Logging Facade for Java (SLF4J) serves as a simple facade or abstraction for various logging frameworks allowing the end user to plug in the desired logging framework at deployment time.

Spark is a fast and general processing engine compatible with Hadoop data. It can run in Hadoop clusters through YARN or Spark's standalone mode, and it can process data in HDFS, HBase, Cassandra, Hive, and any Hadoop InputFormat. It is designed to perform both batch processing (similar to MapReduce) and new workloads like streaming, interactive queries, and machine learning.

It provides diagnostic logging to files, the console, and elsewhere. It is easy to set up, has a clean API, and is portable between recent .NET platforms.

It is the acronym for three open source projects: Elasticsearch, Logstash, and Kibana. Elasticsearch is a search and analytics engine. Logstash is a server‑side data processing pipeline that ingests data from multiple sources simultaneously, transforms it, and then sends it to a "stash" like Elasticsearch. Kibana lets users visualize data with charts and graphs in Elasticsearch.

Fluentd collects events from various data sources and writes them to files, RDBMS, NoSQL, IaaS, SaaS, Hadoop and so on. Fluentd helps you unify your logging infrastructure.

Papertrail helps detect, resolve, and avoid infrastructure problems using log messages. Papertrail's practicality comes from our own experience as sysadmins, developers, and entrepreneurs.

Centralize and aggregate all your log files for 100% visibility. Use our powerful query language to search through terabytes of log data to discover and analyze important information.

Apache Flink is an open source system for fast and versatile data analytics in clusters. Flink supports batch and streaming analytics, in one system. Analytical programs can be written in concise and elegant APIs in Java and Scala.

Amazon Athena is an interactive query service that makes it easy to analyze data in Amazon S3 using standard SQL. Athena is serverless, so there is no infrastructure to manage, and you pay only for the queries that you run.

Hive facilitates reading, writing, and managing large datasets residing in distributed storage using SQL. Structure can be projected onto data already in storage.

A fully managed extract, transform, and load (ETL) service that makes it easy for customers to prepare and load their data for analytics.

Distributed SQL Query Engine for Big Data

Druid is a distributed, column-oriented, real-time analytics data store that is commonly used to power exploratory dashboards in multi-tenant environments. Druid excels as a data warehousing solution for fast aggregate queries on petabyte sized data sets. Druid supports a variety of flexible filters, exact calculations, approximate algorithms, and other useful calculations.

With CloudTrail, you can get a history of AWS API calls for your account, including API calls made via the AWS Management Console, AWS SDKs, command line tools, and higher-level AWS services (such as AWS CloudFormation). The AWS API call history produced by CloudTrail enables security analysis, resource change tracking, and compliance auditing. The recorded information includes the identity of the API caller, the time of the API call, the source IP address of the API caller, the request parameters, and the response elements returned by the AWS service.

It is an open source software integration platform helps you in effortlessly turning data into business insights. It uses native code generation that lets you run your data pipelines seamlessly across all cloud providers and get optimized performance on all platforms.

Logentries makes machine-generated log data easily accessible to IT operations, development, and business analysis teams of all sizes. With the broadest platform support and an open API, Logentries brings the value of log-level data to any system, to any team member, and to a community of more than 25,000 worldwide users.

It is a SaaS solution to manage your log data. There is nothing to install and updates are automatically applied to your Loggly subdomain.

It is a service designed to allow developers to integrate disparate data sources. It is a platform somewhat like SSIS in the cloud to manage the data you have both on-prem and in the cloud.

Cloud-based machine data analytics platform that enables companies to proactively identify availability and performance issues in their infrastructure, improve their security posture and enhance application rollouts. Companies using Sumo Logic reduce their mean-time-to-resolution by 50% and can save hundreds of thousands of dollars, annually. Customers include Netflix, Medallia, Orange, and GoGo Inflight.

Impala is a modern, open source, MPP SQL query engine for Apache Hadoop. Impala is shipped by Cloudera, MapR, and Amazon. With Impala, you can query data, whether stored in HDFS or Apache HBase – including SELECT, JOIN, and aggregate functions – in real time.

It helps you keep the simple things simple by offering a lightweight way to forward and centralize logs and files.

Its mission is to connect the world’s applications, data and devices. It makes connecting anything easy with Anypoint Platform™, the only complete integration platform for SaaS, SOA and APIs. Thousands of organizations in 60 countries, from emerging brands to Global 500 enterprises, use it to innovate faster and gain competitive advantage.

Dremio—the data lake engine, operationalizes your data lake storage and speeds your analytics processes with a high-performance and high-efficiency query engine while also democratizing data access for data scientists and analysts.

Splunk Enterprise delivers massive scale and speed to give you the real-time insights needed to boost productivity, security, profitability and competitiveness.

An open-source storage layer that brings ACID transactions to Apache Spark™ and big data workloads.

It is an analytics service that brings together enterprise data warehousing and Big Data analytics. It gives you the freedom to query data on your terms, using either serverless on-demand or provisioned resources—at scale. It brings these two worlds together with a unified experience to ingest, prepare, manage, and serve data for immediate BI and machine learning needs.

With Redshift Spectrum, you can extend the analytic power of Amazon Redshift beyond data stored on local disks in your data warehouse to query vast amounts of unstructured data in your Amazon S3 “data lake” -- without having to load or transform any data.

The easiest log management system you will ever use! LogDNA is a cloud-based log management system that allows engineering and devops to aggregate all system and application logs into one efficient platform. Save, store, tail and search app

It is a columnar storage format available to any project in the Hadoop ecosystem, regardless of the choice of data processing framework, data model or programming language.

It provides a best-in-class, unified analytics platform that will forever be independent from underlying infrastructure.

It uses a custom-built technology to process and store your logs. Collect logs across your stack with SQL-compatible structured logging.

A new addition to the open source Apache Hadoop ecosystem, Kudu completes Hadoop's storage layer to enable fast analytics on fast data.

It is a super fast, lightweight, and highly scalable logging and metrics processor and forwarder. It is the preferred choice for cloud and containerized environments.

Timber.io is a cloud based logging platform, specifically engineered for Ruby, Elixir and Node apps. It automatically structures your log data with it's open source packages, makes the lines readable, adds valuable context, and lets you easily create graphs and alerts from your logs.

Get a clear overview of what is happening across your distributed environments, and spot the needle in the haystack in no time. Build dynamic analyses and identify improvements for your software, your user experience and your business.

It is an open-source alternative to BigQuery, Redshift, and Snowflake. It is a wrapper around Clickhouse that lets you input arbitrary JSON and perform analytical queries against it. It automatically creates tables and columns when new data is added.

Apache Kylin™ is an open source Distributed Analytics Engine designed to provide SQL interface and multi-dimensional analysis (OLAP) on Hadoop/Spark supporting extremely large datasets, originally contributed from eBay Inc.

Pig is a dataflow programming environment for processing very large files. Pig's language is called Pig Latin. A Pig Latin program consists of a directed acyclic graph where each node represents an operation that transforms data. Operations are of two flavors: (1) relational-algebra style operations such as join, filter, project; (2) functional-programming style operators such as map, reduce.

It is open source and lets regular users import their big data, query it, search it, visualize it and build dashboards on top of it, all from their browser.

It provides Elasticsearch, Logstash and Kibana on the cloud with alerts, unlimited scalability and free ELK apps. Index, search & visualize your data.

An end-to-end data integration platform to build, run, monitor and manage smart data pipelines that deliver continuous data for DataOps.

It is a distributed, reliable, and available service for efficiently collecting, aggregating, and moving large amounts of log data. It has a simple and flexible architecture based on streaming data flows. It is robust and fault tolerant with tunable reliability mechanisms and many failover and recovery mechanisms. It uses a simple extensible data model that allows for online analytic application.

Cask Data Application Platform (CDAP) is an open source application development platform for the Hadoop ecosystem that provides developers with data and application virtualization to accelerate application development, address a broader range of real-time and batch use cases, and deploy applications into production while satisfying enterprise requirements.

The Sumo Logic platform helps you make data-driven decisions and reduce the time to investigate security and operational issues so you can free up resources for more important activities.

Scalyr is log search and management so fast you actually use it. Custom dashboards, graphs, alerts and log parsers allow you to monitor what's important to you. We're proud to serve customers like Business Insider, Opendoor, and Grab.

It offers high-performance, great security features and a modular design. It is able to accept inputs from a wide variety of sources, transform them, and output to the results to diverse destinations.