Need advice about which tool to choose?Ask the StackShare community!
Amazon S3 vs Hadoop vs Minio: What are the differences?
Key differences between Amazon S3, Hadoop, and Minio
Amazon S3, Hadoop, and Minio are all popular distributed storage and object storage systems. While they serve similar purposes, there are several key differences between them:
Scalability: Amazon S3 and Minio are both highly scalable, allowing for the storage of large amounts of data. However, Hadoop's scalability is primarily focused on processing and analyzing large datasets, rather than storing them.
Data Redundancy and Durability: Amazon S3 and Minio provide high data redundancy and durability through replication and data integrity checks. Hadoop, on the other hand, relies on the underlying file system for data redundancy and durability and does not have built-in replication mechanisms.
Pricing Model: Amazon S3 follows a pay-as-you-go pricing model, where users are charged based on the amount of stored data and data transfer. Minio, on the other hand, is an open-source software with no direct pricing model. Hadoop is also open-source, but the costs associated with running and managing a Hadoop cluster can vary.
Data Access: Amazon S3 and Minio provide RESTful APIs for accessing stored data, allowing for easy integration with various applications. Hadoop, on the other hand, uses its own file system (HDFS) and requires the use of its APIs (such as Hadoop Distributed File System commands and MapReduce) for data access and processing.
Ease of Setup and Configuration: Amazon S3 is a fully managed service, making it easy to set up and configure without the need for managing infrastructure. Minio is relatively easy to set up and configure, but it requires the user to manage the infrastructure. Hadoop, however, requires significant setup and configuration, including the deployment and management of a Hadoop cluster.
Data Processing Capabilities: While Amazon S3 and Minio focus primarily on data storage, Hadoop provides a comprehensive data processing framework, with support for distributed processing and various big data processing frameworks such as MapReduce, Spark, and Hive.
In summary, Amazon S3 is a scalable and cost-effective object storage service with a pay-as-you-go pricing model, while Minio is an open-source alternative with similar features. Hadoop, on the other hand, is a comprehensive distributed processing framework with its own file system and data processing capabilities.
I have a lot of data that's currently sitting in a MariaDB database, a lot of tables that weigh 200gb with indexes. Most of the large tables have a date column which is always filtered, but there are usually 4-6 additional columns that are filtered and used for statistics. I'm trying to figure out the best tool for storing and analyzing large amounts of data. Preferably self-hosted or a cheap solution. The current problem I'm running into is speed. Even with pretty good indexes, if I'm trying to load a large dataset, it's pretty slow.
Druid Could be an amazing solution for your use case, My understanding, and the assumption is you are looking to export your data from MariaDB for Analytical workload. It can be used for time series database as well as a data warehouse and can be scaled horizontally once your data increases. It's pretty easy to set up on any environment (Cloud, Kubernetes, or Self-hosted nix system). Some important features which make it a perfect solution for your use case. 1. It can do streaming ingestion (Kafka, Kinesis) as well as batch ingestion (Files from Local & Cloud Storage or Databases like MySQL, Postgres). In your case MariaDB (which has the same drivers to MySQL) 2. Columnar Database, So you can query just the fields which are required, and that runs your query faster automatically. 3. Druid intelligently partitions data based on time and time-based queries are significantly faster than traditional databases. 4. Scale up or down by just adding or removing servers, and Druid automatically rebalances. Fault-tolerant architecture routes around server failures 5. Gives ana amazing centralized UI to manage data sources, query, tasks.
Hello! I have a mobile app with nearly 100k MAU, and I want to add a cloud file storage service to my app.
My app will allow users to store their image, video, and audio files and retrieve them to their device when necessary.
I have already decided to use PHP & Laravel as my backend, and I use Contabo VPS. Now, I need an object storage service for my app, and my options are:
Amazon S3 : It sounds to me like the best option but the most expensive. Closest to my users (MENA Region) for other services, I will have to go to Europe. Not sure how important this is?
DigitalOcean Spaces : Seems like my best option for price/service, but I am still not sure
Wasabi: the best price (6 USD/MONTH/TB) and free bandwidth, but I am not sure if it fits my needs as I want to allow my users to preview audio and video files. They don't recommend their service for streaming videos.
Backblaze B2 Cloud Storage: Good price but not sure about them.
There is also the self-hosted s3 compatible option, but I am not sure about that.
Any thoughts will be helpful. Also, if you think I should post in a different sub, please tell me.
If pricing is the issue i'd suggest you use digital ocean, but if its not use amazon was digital oceans API is s3 compatible
Hello Mohammad, I am using : Cloudways >> AWS >> Bahrain for last 2 years. This is best I consider out of my 10 year research on Laravel hosting.
Minio is a free and open source object storage system. It can be self-hosted and is S3 compatible. During the early stage it would save cost and allow us to move to a different object storage when we scale up. It is also fast and easy to set up. This is very useful during development since it can be run on localhost.
We offer our customer HIPAA compliant storage. After analyzing the market, we decided to go with Google Storage. The Nodejs API is ok, still not ES6 and can be very confusing to use. For each new customer, we created a different bucket so they can have individual data and not have to worry about data loss. After 1000+ customers we started seeing many problems with the creation of new buckets, with saving or retrieving a new file. Many false positive: the Promise returned ok, but in reality, it failed.
That's why we switched to S3 that just works.
Pros of Amazon S3
- Reliable590
- Scalable492
- Cheap456
- Simple & easy329
- Many sdks83
- Logical30
- Easy Setup13
- REST API11
- 1000+ POPs11
- Secure6
- Easy4
- Plug and play4
- Web UI for uploading files3
- Faster on response2
- Flexible2
- GDPR ready2
- Easy to use1
- Plug-gable1
- Easy integration with CloudFront1
Pros of Hadoop
- Great ecosystem39
- One stack to rule them all11
- Great load balancer4
- Amazon aws1
- Java syntax1
Pros of Minio
- Store and Serve Resumes & Job Description PDF, Backups10
- S3 Compatible8
- Simple4
- Open Source4
- Encryption and Tamper-Proof3
- Lambda Compute3
- Private Cloud Storage2
- Pluggable Storage Backend2
- Scalable2
- Data Protection2
- Highly Available2
- Performance1
Sign up to add or upvote prosMake informed product decisions
Cons of Amazon S3
- Permissions take some time to get right7
- Requires a credit card6
- Takes time/work to organize buckets & folders properly6
- Complex to set up3
Cons of Hadoop
Cons of Minio
- Deletion of huge buckets is not possible3
Sign up to add or upvote consMake informed product decisions
What is Amazon S3?
What is Hadoop?
What is Minio?
Need advice about which tool to choose?Ask the StackShare community!
What companies use Minio?
Sign up to get full access to all the companiesMake informed product decisions
What tools integrate with Amazon S3?
What tools integrate with Hadoop?
Sign up to get full access to all the tool integrationsMake informed product decisions
Blog Posts
+4
+5
+5
+7
+6
+4
+4
+17