Overview

StreamSets

Stacks53

Followers133

Votes0

Dremio

Stacks116

Followers348

Votes8

Dremio vs StreamSets: What are the differences?

# Introduction

1. **Data Processing Paradigm**: Dremio is primarily designed for data analytics and BI use cases, whereas StreamSets is focused on data integration and pipeline orchestration. Dremio provides a SQL interface for querying and analyzing data, while StreamSets offers a visual interface for building data pipelines.
2. **Data Sources**: Dremio supports a wide range of data sources including traditional databases, cloud storage, and data lakes, while StreamSets focuses on ingesting data from diverse sources and transforming it before loading it into destinations.
3. **Real-time Processing**: StreamSets excels in real-time data processing scenarios, enabling users to ingest and process data continuously with low latency, while Dremio is more suited for batch processing and interactive analytics.
4. **Deployment Options**: Dremio can be deployed on-premises, in the cloud, or in hybrid environments, offering flexibility in deployment options, while StreamSets is typically deployed on-premises or in the cloud but lacks hybrid deployment capabilities.
5. **Data Governance and Security**: Dremio provides advanced data governance features such as fine-grained access control and data lineage tracking, ensuring data security and compliance, whereas StreamSets focuses more on data movement and transformation with limited governance features.
6. **Open Source vs. Commercial**: Dremio offers both open-source and commercial editions, providing users with the option to choose based on their requirements, whereas StreamSets is primarily a commercial product with a free version for limited functionality.

In Summary, Dremio and StreamSets differ in their data processing paradigm, supported data sources, real-time processing capabilities, deployment options, data governance features, and licensing models.

🔥 Trending in Background Jobs on StackShare

Kafka Background Jobs

Distributed, fault tolerant, high throughput pub-sub messaging system

Try it View Docs Alternatives

Try

84

22.3k

Advice on StreamSets, Dremio

karunakaran

Consultant

Jun 26, 2020

Needs advice

I am trying to build a data lake by pulling data from multiple data sources ( custom-built tools, excel files, CSV files, etc) and use the data lake to generate dashboards.

My question is which is the best tool to do the following:

Create pipelines to ingest the data from multiple sources into the data lake
Help me in aggregating and filtering data available in the data lake.
Create new reports by combining different data elements from the data lake.

I need to use only open-source tools for this activity.

I appreciate your valuable inputs and suggestions. Thanks in Advance.

80.4k views80.4k

Comments

datocrats-org

Jul 29, 2020

Needs adviceon

Amazon EC2

Tableau

PowerBI

We need to perform ETL from several databases into a data warehouse or data lake. We want to

keep raw and transformed data available to users to draft their own queries efficiently
give users the ability to give custom permissions and SSO
move between open-source on-premises development and cloud-based production environments

We want to use inexpensive Amazon EC2 instances only on medium-sized data set 16GB to 32GB feeding into Tableau Server or PowerBI for reporting and data analysis purposes.

319k views319k

Comments

Detailed Comparison

StreamSets	Dremio
An end-to-end data integration platform to build, run, monitor and manage smart data pipelines that deliver continuous data for DataOps.	Dremio—the data lake engine, operationalizes your data lake storage and speeds your analytics processes with a high-performance and high-efficiency query engine while also democratizing data access for data scientists and analysts.
Only StreamSets provides a single design experience for all design patterns (batch, streaming, CDC, ETL, ELT, and ML pipelines) for 10x greater developer productivity; smart data pipelines that are resilient to change for 80% less breakages; and a single pane of glass for managing and monitoring all pipelines across hybrid and cloud architectures to eliminate blind spots and control gaps.	Democratize all your data; Make your data engineers more productive; Accelerate your favorite tools; Self service, for everybody
Statistics
Stacks 53	Stacks 116
Followers 133	Followers 348
Votes 0	Votes 8
Pros & Cons
Cons 2 No user community 1 Crashes	Pros 3 Nice GUI to enable more people to work with Data 2 Easier to Deploy 2 Connect NoSQL databases with RDBMS 1 Free Cons 1 Works only on Iceberg structured data
Integrations
HBase Databricks Amazon Redshift MySQL gRPC Google BigQuery Amazon Kinesis Cassandra Hadoop Redis	Amazon S3 Python Tableau Azure Database for PostgreSQL Qlik Sense PowerBI

What are some alternatives to StreamSets, Dremio?

Kafka

Kafka is a distributed, partitioned, replicated commit log service. It provides the functionality of a messaging system, but with a unique design.

RabbitMQ

RabbitMQ gives your applications a common platform to send and receive messages, and your messages a safe place to live until received.

Celery

Celery is an asynchronous task queue/job queue based on distributed message passing. It is focused on real-time operation, but supports scheduling as well.

Amazon SQS

Transmit any volume of data, at any level of throughput, without losing messages or requiring other services to be always available. With SQS, you can offload the administrative burden of operating and scaling a highly available messaging cluster, while paying a low price for only what you use.

Google BigQuery

Run super-fast, SQL-like queries against terabytes of data in seconds, using the processing power of Google's infrastructure. Load data with ease. Bulk load your data using Google Cloud Storage or stream it in. Easy access. Access BigQuery by using a browser tool, a command-line tool, or by making calls to the BigQuery REST API with client libraries such as Java, PHP or Python.

NSQ

NSQ is a realtime distributed messaging platform designed to operate at scale, handling billions of messages per day. It promotes distributed and decentralized topologies without single points of failure, enabling fault tolerance and high availability coupled with a reliable message delivery guarantee. See features & guarantees.

Apache Spark

Spark is a fast and general processing engine compatible with Hadoop data. It can run in Hadoop clusters through YARN or Spark's standalone mode, and it can process data in HDFS, HBase, Cassandra, Hive, and any Hadoop InputFormat. It is designed to perform both batch processing (similar to MapReduce) and new workloads like streaming, interactive queries, and machine learning.

Amazon Redshift

It is optimized for data sets ranging from a few hundred gigabytes to a petabyte or more and costs less than $1,000 per terabyte per year, a tenth the cost of most traditional data warehousing solutions.

ActiveMQ

Apache ActiveMQ is fast, supports many Cross Language Clients and Protocols, comes with easy to use Enterprise Integration Patterns and many advanced features while fully supporting JMS 1.1 and J2EE 1.4. Apache ActiveMQ is released under the Apache 2.0 License.

ZeroMQ

The 0MQ lightweight messaging kernel is a library which extends the standard socket interfaces with features traditionally provided by specialised messaging middleware products. 0MQ sockets provide an abstraction of asynchronous message queues, multiple messaging patterns, message filtering (subscriptions), seamless access to multiple transport protocols and more.