AWS DataOps Development Kit

Spark is a fast and general processing engine compatible with Hadoop data. It can run in Hadoop clusters through YARN or Spark's standalone mode, and it can process data in HDFS, HBase, Cassandra, Hive, and any Hadoop InputFormat. It is designed to perform both batch processing (similar to MapReduce) and new workloads like streaming, interactive queries, and machine learning.

It provides the leading platform for Operational Intelligence. Customers use it to search, monitor, analyze and visualize machine data.

Apache Flink is an open source system for fast and versatile data analytics in clusters. Flink supports batch and streaming analytics, in one system. Analytical programs can be written in concise and elegant APIs in Java and Scala.

Amazon Athena is an interactive query service that makes it easy to analyze data in Amazon S3 using standard SQL. Athena is serverless, so there is no infrastructure to manage, and you pay only for the queries that you run.

Hive facilitates reading, writing, and managing large datasets residing in distributed storage using SQL. Structure can be projected onto data already in storage.

A fully managed extract, transform, and load (ETL) service that makes it easy for customers to prepare and load their data for analytics.

Distributed SQL Query Engine for Big Data

Druid is a distributed, column-oriented, real-time analytics data store that is commonly used to power exploratory dashboards in multi-tenant environments. Druid excels as a data warehousing solution for fast aggregate queries on petabyte sized data sets. Druid supports a variety of flexible filters, exact calculations, approximate algorithms, and other useful calculations.

It is an open source software integration platform helps you in effortlessly turning data into business insights. It uses native code generation that lets you run your data pipelines seamlessly across all cloud providers and get optimized performance on all platforms.

It is a service designed to allow developers to integrate disparate data sources. It is a platform somewhat like SSIS in the cloud to manage the data you have both on-prem and in the cloud.

Impala is a modern, open source, MPP SQL query engine for Apache Hadoop. Impala is shipped by Cloudera, MapR, and Amazon. With Impala, you can query data, whether stored in HDFS or Apache HBase – including SELECT, JOIN, and aggregate functions – in real time.

Its mission is to connect the world’s applications, data and devices. It makes connecting anything easy with Anypoint Platform™, the only complete integration platform for SaaS, SOA and APIs. Thousands of organizations in 60 countries, from emerging brands to Global 500 enterprises, use it to innovate faster and gain competitive advantage.

Dremio—the data lake engine, operationalizes your data lake storage and speeds your analytics processes with a high-performance and high-efficiency query engine while also democratizing data access for data scientists and analysts.

An open-source storage layer that brings ACID transactions to Apache Spark™ and big data workloads.

It is an analytics service that brings together enterprise data warehousing and Big Data analytics. It gives you the freedom to query data on your terms, using either serverless on-demand or provisioned resources—at scale. It brings these two worlds together with a unified experience to ingest, prepare, manage, and serve data for immediate BI and machine learning needs.

With Redshift Spectrum, you can extend the analytic power of Amazon Redshift beyond data stored on local disks in your data warehouse to query vast amounts of unstructured data in your Amazon S3 “data lake” -- without having to load or transform any data.

It is a columnar storage format available to any project in the Hadoop ecosystem, regardless of the choice of data processing framework, data model or programming language.

It provides a best-in-class, unified analytics platform that will forever be independent from underlying infrastructure.

A new addition to the open source Apache Hadoop ecosystem, Kudu completes Hadoop's storage layer to enable fast analytics on fast data.

It is an open-source alternative to BigQuery, Redshift, and Snowflake. It is a wrapper around Clickhouse that lets you input arbitrary JSON and perform analytical queries against it. It automatically creates tables and columns when new data is added.

Apache Kylin™ is an open source Distributed Analytics Engine designed to provide SQL interface and multi-dimensional analysis (OLAP) on Hadoop/Spark supporting extremely large datasets, originally contributed from eBay Inc.

Pig is a dataflow programming environment for processing very large files. Pig's language is called Pig Latin. A Pig Latin program consists of a directed acyclic graph where each node represents an operation that transforms data. Operations are of two flavors: (1) relational-algebra style operations such as join, filter, project; (2) functional-programming style operators such as map, reduce.

It is open source and lets regular users import their big data, query it, search it, visualize it and build dashboards on top of it, all from their browser.

An end-to-end data integration platform to build, run, monitor and manage smart data pipelines that deliver continuous data for DataOps.

Cask Data Application Platform (CDAP) is an open source application development platform for the Hadoop ecosystem that provides developers with data and application virtualization to accelerate application development, address a broader range of real-time and batch use cases, and deploy applications into production while satisfying enterprise requirements.

It is a fast distributed SQL query engine for big data analytics that helps you explore your data universe. It is designed to query large data sets distributed over one or more heterogeneous data sources.

It is a managed Spark and Hadoop service that lets you take advantage of open source data tools for batch processing, querying, streaming, and machine learning. It helps you create clusters quickly, manage them easily, and save money by turning clusters off when you don't need them.

It is a powerful tool for working with messy data: cleaning it; transforming it from one format into another; and extending it with web services and external data.

It is a cloud-based service from Microsoft for big data analytics that helps organizations process large amounts of streaming or historical data.

A fully managed, cloud-native data integration service that helps users efficiently build and manage ETL/ELT data pipelines. With a graphical interface and a broad open-source library of preconfigured connectors and transformations, and more.

Its Virtual Data Warehouse delivers performance, security and agility to exceed the demands of modern-day operational analytics.

Pachyderm is an open source MapReduce engine that uses Docker containers for distributed computations.

Singer powers data extraction and consolidation for all of your organization’s tools: advertising platforms, web analytics, payment processors, email service providers, marketing automation, databases, and more.

It is an Intelligent Platform that Interoperates with Your Data Investments. It sits between the data storage and processing environments and the visualization, statistical or machine learning tools used downstream

It is a Hitachi Group Company, data integration and business analytics company with an enterprise, Online Analytical Processing server (OLAP). Allows business users to analyze large and complex amounts of data in real-time.

It is the leading Reverse ETL platform. Sync customer data from your warehouse into tools your business teams rely on.

It is a metadata driven application for improving the productivity of data analysts, data scientists and engineers when interacting with data.

It is an open source enterprise-ready data lake management software platform for self-service data ingest and data preparation with integrated metadata management, governance, security and best practices inspired by Think Big's 150+ big data implementation projects.

AresDB is a GPU-powered real-time analytics storage and query engine. It features low query latency, high data freshness and highly efficient in-memory and on disk storage management.

The leader in collaborative data cataloging, it empowers analysts & information stewards to search, query & collaborate for fast and accurate insights.

Connect, integrate and automate all of your systems, APIs and apps, including cloud and legacy ones, using an open-source integration platform in Python. ESB, SOA, REST, API and Cloud Integrations in Python.

Vespa is an engine for low-latency computation over large data sets. It stores and indexes your data such that queries, selection and processing over the data can be performed at serving time.

It is a high-performance format for huge analytic tables. It brings the reliability and simplicity of SQL tables to big data while making it possible for engines like Spark, Trino, Flink, Presto, Hive, and Impala to work safely with the same tables simultaneously.

It is a fully managed integration service that enables you to securely transfer data between Software-as-a-Service (SaaS) applications like Salesforce, Marketo, Slack, and ServiceNow, and AWS services like Amazon S3 and Amazon Redshift, in just a few clicks. With AppFlow, you can run data flows at nearly any scale at the frequency you choose - on a schedule, in response to a business event, or on demand. You can configure data transformation capabilities like filtering and validation to generate rich, ready-to-use data as part of the flow itself, without additional steps. AppFlow automatically encrypts data in motion, and allows users to restrict data from flowing over the public Internet for SaaS applications that are integrated with AWS PrivateLink, reducing exposure to security threats.

A lightweight ETL framework with a focus on transparency and complexity reduction.

The open-source SQL interface to your favorite cloud APIs. Select * from AWS, Azure, GCP, Github, Slack, and more. It exposes APIs and services as a high-performance relational database, giving you the ability to write SQL-based queries and controls to explore, assess and report on dynamic data.

It is a single application that helps you get any data into Hadoop, bring it together, analyze it, and visualize it as quickly and easily as possible. No coding required. Everything in it is self-service and intuitive, from our wizard-based data integration, to a spreadsheet with point-and-click analytics, to our blank canvas to for building custom visualizations.

It enables you to quickly develop, orchestrate, and operate distributed streaming applications on Kubernetes. With Cloudflow, streaming applications are comprised of small composable components wired together with schema-based contracts. It can dramatically accelerate streaming application development—​reducing the time required to create, package, and deploy—​from weeks to hours.

Apache Pinot is a fast, scalable real-time analytics database. It is a column-oriented distributed Online Analytics Processing (OLAP) database designed for high concurrency and low latency. It can scan petabyte-scale data and produce results even as fast as single-digit milliseconds.

It is a unique product that puts your most valuable asset at the core of your business: YOUR DATA. It serves as the backbone for the Digital Transformation of companies. It brings together the latest, most disruptive technologies into a single product that responds to the needs of today’s market: