Amazon S3 vs Hadoop vs Minio

Overview

Amazon S3

Stacks55.1K

Followers40.2K

Votes2.0K

Hadoop

Stacks2.7K

Followers2.3K

Votes56

GitHub Stars15.3K

Forks9.1K

Minio

Stacks637

Followers670

Votes43

GitHub Stars57.8K

Forks6.4K

Amazon S3 vs Hadoop vs Minio: What are the differences?

Key differences between Amazon S3, Hadoop, and Minio

Amazon S3, Hadoop, and Minio are all popular distributed storage and object storage systems. While they serve similar purposes, there are several key differences between them:

Scalability: Amazon S3 and Minio are both highly scalable, allowing for the storage of large amounts of data. However, Hadoop's scalability is primarily focused on processing and analyzing large datasets, rather than storing them.
Data Redundancy and Durability: Amazon S3 and Minio provide high data redundancy and durability through replication and data integrity checks. Hadoop, on the other hand, relies on the underlying file system for data redundancy and durability and does not have built-in replication mechanisms.
Pricing Model: Amazon S3 follows a pay-as-you-go pricing model, where users are charged based on the amount of stored data and data transfer. Minio, on the other hand, is an open-source software with no direct pricing model. Hadoop is also open-source, but the costs associated with running and managing a Hadoop cluster can vary.
Data Access: Amazon S3 and Minio provide RESTful APIs for accessing stored data, allowing for easy integration with various applications. Hadoop, on the other hand, uses its own file system (HDFS) and requires the use of its APIs (such as Hadoop Distributed File System commands and MapReduce) for data access and processing.
Ease of Setup and Configuration: Amazon S3 is a fully managed service, making it easy to set up and configure without the need for managing infrastructure. Minio is relatively easy to set up and configure, but it requires the user to manage the infrastructure. Hadoop, however, requires significant setup and configuration, including the deployment and management of a Hadoop cluster.
Data Processing Capabilities: While Amazon S3 and Minio focus primarily on data storage, Hadoop provides a comprehensive data processing framework, with support for distributed processing and various big data processing frameworks such as MapReduce, Spark, and Hive.

In summary, Amazon S3 is a scalable and cost-effective object storage service with a pay-as-you-go pricing model, while Minio is an open-source alternative with similar features. Hadoop, on the other hand, is a comprehensive distributed processing framework with its own file system and data processing capabilities.

Share your Stack

Help developers discover the tools you use. Get visibility for your team's tech choices and contribute to the community's knowledge.

View Docs

CLI (Node.js)

Manual

Advice on Amazon S3, Hadoop, Minio

Mohammad

Aug 30, 2020

Needs adviceon

Backblaze B2 Cloud Storage

PHP

Laravel

Hello! I have a mobile app with nearly 100k MAU, and I want to add a cloud file storage service to my app.

My app will allow users to store their image, video, and audio files and retrieve them to their device when necessary.

I have already decided to use PHP & Laravel as my backend, and I use Contabo VPS. Now, I need an object storage service for my app, and my options are:

Amazon S3 : It sounds to me like the best option but the most expensive. Closest to my users (MENA Region) for other services, I will have to go to Europe. Not sure how important this is?
DigitalOcean Spaces : Seems like my best option for price/service, but I am still not sure
Wasabi: the best price (6 USD/MONTH/TB) and free bandwidth, but I am not sure if it fits my needs as I want to allow my users to preview audio and video files. They don't recommend their service for streaming videos.
Backblaze B2 Cloud Storage: Good price but not sure about them.
There is also the self-hosted s3 compatible option, but I am not sure about that.

Any thoughts will be helpful. Also, if you think I should post in a different sub, please tell me.

180k views180k

Comments

Dalton

Oct 23, 2020

Decided

Minio is a free and open source object storage system. It can be self-hosted and is S3 compatible. During the early stage it would save cost and allow us to move to a different object storage when we scale up. It is also fast and easy to set up. This is very useful during development since it can be run on localhost.

143k views143k

Comments

Gabriel

CEO at NaoLogic Inc

Dec 24, 2019

Decided

We offer our customer HIPAA compliant storage. After analyzing the market, we decided to go with Google Storage. The Nodejs API is ok, still not ES6 and can be very confusing to use. For each new customer, we created a different bucket so they can have individual data and not have to worry about data loss. After 1000+ customers we started seeing many problems with the creation of new buckets, with saving or retrieving a new file. Many false positive: the Promise returned ok, but in reality, it failed.

That's why we switched to S3 that just works.

330k views330k

Comments

Detailed Comparison

Amazon S3	Hadoop	Minio
Amazon Simple Storage Service provides a fully redundant data storage infrastructure for storing and retrieving any amount of data, at any time, from anywhere on the web	The Apache Hadoop software library is a framework that allows for the distributed processing of large data sets across clusters of computers using simple programming models. It is designed to scale up from single servers to thousands of machines, each offering local computation and storage.	Minio is an object storage server compatible with Amazon S3 and licensed under Apache 2.0 License
Write, read, and delete objects containing from 1 byte to 5 terabytes of data each. The number of objects you can store is unlimited.;Each object is stored in a bucket and retrieved via a unique, developer-assigned key.;A bucket can be stored in one of several Regions. You can choose a Region to optimize for latency, minimize costs, or address regulatory requirements. Amazon S3 is currently available in the US Standard, US West (Oregon), US West (Northern California), EU (Ireland), Asia Pacific (Singapore), Asia Pacific (Tokyo), Asia Pacific (Sydney), South America (Sao Paulo), and GovCloud (US) Regions. The US Standard Region automatically routes requests to facilities in Northern Virginia or the Pacific Northwest using network maps.;Objects stored in a Region never leave the Region unless you transfer them out. For example, objects stored in the EU (Ireland) Region never leave the EU.;Authentication mechanisms are provided to ensure that data is kept secure from unauthorized access. Objects can be made private or public, and rights can be granted to specific users.;Options for secure data upload/download and encryption of data at rest are provided for additional data protection.;Uses standards-based REST and SOAP interfaces designed to work with any Internet-development toolkit.;Built to be flexible so that protocol or functional layers can easily be added. The default download protocol is HTTP. A BitTorrent protocol interface is provided to lower costs for high-scale distribution.;Provides functionality to simplify manageability of data through its lifetime. Includes options for segregating data by buckets, monitoring and controlling spend, and automatically archiving data to even lower cost storage options. These options can be easily administered from the Amazon S3 Management Console.;Reliability backed with the Amazon S3 Service Level Agreement.	-	-
Statistics
GitHub Stars -	GitHub Stars 15.3K	GitHub Stars 57.8K
GitHub Forks -	GitHub Forks 9.1K	GitHub Forks 6.4K
Stacks 55.1K	Stacks 2.7K	Stacks 637
Followers 40.2K	Followers 2.3K	Followers 670
Votes 2.0K	Votes 56	Votes 43
Pros & Cons
Pros 590 Reliable 492 Scalable 456 Cheap 329 Simple & easy 83 Many sdks Cons 7 Permissions take some time to get right 6 Requires a credit card 6 Takes time/work to organize buckets & folders properly 3 Complex to set up	Pros 39 Great ecosystem 11 One stack to rule them all 4 Great load balancer 1 Amazon aws 1 Java syntax	Pros 10 Store and Serve Resumes & Job Description PDF, Backups 8 S3 Compatible 4 Open Source 4 Simple 3 Encryption and Tamper-Proof Cons 3 Deletion of huge buckets is not possible

What are some alternatives to Amazon S3, Hadoop, Minio?

MongoDB

MongoDB stores data in JSON-like documents that can vary in structure, offering a dynamic, flexible schema. MongoDB was also designed for high availability and scalability, with built-in replication and auto-sharding.

MySQL

The MySQL software delivers a very fast, multi-threaded, multi-user, and robust SQL (Structured Query Language) database server. MySQL Server is intended for mission-critical, heavy-load production systems as well as for embedding into mass-deployed software.

PostgreSQL

PostgreSQL is an advanced object-relational database management system that supports an extended subset of the SQL standard, including transactions, foreign keys, subqueries, triggers, user-defined types and functions.

Microsoft SQL Server

Microsoft® SQL Server is a database management and analysis system for e-commerce, line-of-business, and data warehousing solutions.

SQLite

SQLite is an embedded SQL database engine. Unlike most other SQL databases, SQLite does not have a separate server process. SQLite reads and writes directly to ordinary disk files. A complete SQL database with multiple tables, indices, triggers, and views, is contained in a single disk file.

Cassandra

Partitioning means that Cassandra can distribute your data across multiple machines in an application-transparent matter. Cassandra will automatically repartition as machines are added and removed from the cluster. Row store means that like relational databases, Cassandra organizes data by rows and columns. The Cassandra Query Language (CQL) is a close relative of SQL.

Memcached

Memcached is an in-memory key-value store for small chunks of arbitrary data (strings, objects) from results of database calls, API calls, or page rendering.

MariaDB

Started by core members of the original MySQL team, MariaDB actively works with outside developers to deliver the most featureful, stable, and sanely licensed open SQL server in the industry. MariaDB is designed as a drop-in replacement of MySQL(R) with more features, new storage engines, fewer bugs, and better performance.

RethinkDB

RethinkDB is built to store JSON documents, and scale to multiple machines with very little effort. It has a pleasant query language that supports really useful queries like table joins and group by, and is easy to setup and learn.

ArangoDB

A distributed free and open-source database with a flexible data model for documents, graphs, and key-values. Build high performance applications using a convenient SQL-like query language or JavaScript extensions.

Related Comparisons

Amazon S3 vs Hadoop vs Minio: What are the differences?

Key differences between Amazon S3, Hadoop, and Minio

Amazon S3, Hadoop, and Minio are all popular distributed storage and object storage systems. While they serve similar purposes, there are several key differences between them:

Scalability: Amazon S3 and Minio are both highly scalable, allowing for the storage of large amounts of data. However, Hadoop's scalability is primarily focused on processing and analyzing large datasets, rather than storing them.
Data Redundancy and Durability: Amazon S3 and Minio provide high data redundancy and durability through replication and data integrity checks. Hadoop, on the other hand, relies on the underlying file system for data redundancy and durability and does not have built-in replication mechanisms.
Pricing Model: Amazon S3 follows a pay-as-you-go pricing model, where users are charged based on the amount of stored data and data transfer. Minio, on the other hand, is an open-source software with no direct pricing model. Hadoop is also open-source, but the costs associated with running and managing a Hadoop cluster can vary.
Data Access: Amazon S3 and Minio provide RESTful APIs for accessing stored data, allowing for easy integration with various applications. Hadoop, on the other hand, uses its own file system (HDFS) and requires the use of its APIs (such as Hadoop Distributed File System commands and MapReduce) for data access and processing.
Ease of Setup and Configuration: Amazon S3 is a fully managed service, making it easy to set up and configure without the need for managing infrastructure. Minio is relatively easy to set up and configure, but it requires the user to manage the infrastructure. Hadoop, however, requires significant setup and configuration, including the deployment and management of a Hadoop cluster.
Data Processing Capabilities: While Amazon S3 and Minio focus primarily on data storage, Hadoop provides a comprehensive data processing framework, with support for distributed processing and various big data processing frameworks such as MapReduce, Spark, and Hive.

Amazon S3 vs Hadoop vs Minio

Overview

Amazon S3 vs Hadoop vs Minio: What are the differences?