Amazon S3 vs Google Cloud Bigtable vs Hadoop

Need advice about which tool to choose?Ask the StackShare community!

Amazon S3

44.6K
32.3K
+ 1
2K
Google Cloud Bigtable

119
311
+ 1
24
Hadoop

2.1K
2.1K
+ 1
55
Advice on Amazon S3, Google Cloud Bigtable, and Hadoop
Needs advice
on
KafkaKafkaInfluxDBInfluxDB
and
HadoopHadoop

I have a lot of data that's currently sitting in a MariaDB database, a lot of tables that weigh 200gb with indexes. Most of the large tables have a date column which is always filtered, but there are usually 4-6 additional columns that are filtered and used for statistics. I'm trying to figure out the best tool for storing and analyzing large amounts of data. Preferably self-hosted or a cheap solution. The current problem I'm running into is speed. Even with pretty good indexes, if I'm trying to load a large dataset, it's pretty slow.

See more
Replies (1)
Recommends
DruidDruid

Druid Could be an amazing solution for your use case, My understanding, and the assumption is you are looking to export your data from MariaDB for Analytical workload. It can be used for time series database as well as a data warehouse and can be scaled horizontally once your data increases. It's pretty easy to set up on any environment (Cloud, Kubernetes, or Self-hosted nix system). Some important features which make it a perfect solution for your use case. 1. It can do streaming ingestion (Kafka, Kinesis) as well as batch ingestion (Files from Local & Cloud Storage or Databases like MySQL, Postgres). In your case MariaDB (which has the same drivers to MySQL) 2. Columnar Database, So you can query just the fields which are required, and that runs your query faster automatically. 3. Druid intelligently partitions data based on time and time-based queries are significantly faster than traditional databases. 4. Scale up or down by just adding or removing servers, and Druid automatically rebalances. Fault-tolerant architecture routes around server failures 5. Gives ana amazing centralized UI to manage data sources, query, tasks.

See more

Hello! I have a mobile app with nearly 100k MAU, and I want to add a cloud file storage service to my app.

My app will allow users to store their image, video, and audio files and retrieve them to their device when necessary.

I have already decided to use PHP & Laravel as my backend, and I use Contabo VPS. Now, I need an object storage service for my app, and my options are:

  • Amazon S3 : It sounds to me like the best option but the most expensive. Closest to my users (MENA Region) for other services, I will have to go to Europe. Not sure how important this is?

  • DigitalOcean Spaces : Seems like my best option for price/service, but I am still not sure

  • Wasabi: the best price (6 USD/MONTH/TB) and free bandwidth, but I am not sure if it fits my needs as I want to allow my users to preview audio and video files. They don't recommend their service for streaming videos.

  • Backblaze B2 Cloud Storage: Good price but not sure about them.

  • There is also the self-hosted s3 compatible option, but I am not sure about that.

Any thoughts will be helpful. Also, if you think I should post in a different sub, please tell me.

See more
Replies (2)
Recommends
Cloudways Cloudways

Hello Mohammad, I am using : Cloudways >> AWS >> Bahrain for last 2 years. This is best I consider out of my 10 year research on Laravel hosting.

See more
Michira Griffins
Software Developer at Codeshares Ltd · | 1 upvotes · 67.4K views

If pricing is the issue i'd suggest you use digital ocean, but if its not use amazon was digital oceans API is s3 compatible

See more
Decisions about Amazon S3, Google Cloud Bigtable, and Hadoop

Minio is a free and open source object storage system. It can be self-hosted and is S3 compatible. During the early stage it would save cost and allow us to move to a different object storage when we scale up. It is also fast and easy to set up. This is very useful during development since it can be run on localhost.

See more
Gabriel Pa

We offer our customer HIPAA compliant storage. After analyzing the market, we decided to go with Google Storage. The Nodejs API is ok, still not ES6 and can be very confusing to use. For each new customer, we created a different bucket so they can have individual data and not have to worry about data loss. After 1000+ customers we started seeing many problems with the creation of new buckets, with saving or retrieving a new file. Many false positive: the Promise returned ok, but in reality, it failed.

That's why we switched to S3 that just works.

See more
Get Advice from developers at your company using Private StackShare. Sign up for Private StackShare.
Learn More
Pros of Amazon S3
Pros of Google Cloud Bigtable
Pros of Hadoop
  • 591
    Reliable
  • 492
    Scalable
  • 456
    Cheap
  • 328
    Simple & easy
  • 83
    Many sdks
  • 29
    Logical
  • 12
    Easy Setup
  • 11
    1000+ POPs
  • 10
    REST API
  • 5
    Secure
  • 3
    Easy
  • 2
    Web UI for uploading files
  • 2
    Plug and play
  • 1
    GDPR ready
  • 1
    Flexible
  • 1
    Faster on response
  • 1
    Plug-gable
  • 1
    Easy to use
  • 1
    Easy integration with CloudFront
  • 10
    High performance
  • 9
    Fully managed
  • 5
    High scalability
  • 38
    Great ecosystem
  • 11
    One stack to rule them all
  • 4
    Great load balancer
  • 1
    Amazon aws
  • 1
    Java syntax

Sign up to add or upvote prosMake informed product decisions

Cons of Amazon S3
Cons of Google Cloud Bigtable
Cons of Hadoop
  • 7
    Permissions take some time to get right
  • 6
    Takes time/work to organize buckets & folders properly
  • 5
    Requires a credit card
  • 3
    Complex to set up
    Be the first to leave a con
      Be the first to leave a con

      Sign up to add or upvote consMake informed product decisions

      - No public GitHub repository available -
      - No public GitHub repository available -

      What is Amazon S3?

      Amazon Simple Storage Service provides a fully redundant data storage infrastructure for storing and retrieving any amount of data, at any time, from anywhere on the web

      What is Google Cloud Bigtable?

      Google Cloud Bigtable offers you a fast, fully managed, massively scalable NoSQL database service that's ideal for web, mobile, and Internet of Things applications requiring terabytes to petabytes of data. Unlike comparable market offerings, Cloud Bigtable doesn't require you to sacrifice speed, scale, or cost efficiency when your applications grow. Cloud Bigtable has been battle-tested at Google for more than 10 years—it's the database driving major applications such as Google Analytics and Gmail.

      What is Hadoop?

      The Apache Hadoop software library is a framework that allows for the distributed processing of large data sets across clusters of computers using simple programming models. It is designed to scale up from single servers to thousands of machines, each offering local computation and storage.

      Need advice about which tool to choose?Ask the StackShare community!

      What companies use Amazon S3?
      What companies use Google Cloud Bigtable?
      What companies use Hadoop?

      Sign up to get full access to all the companiesMake informed product decisions

      What tools integrate with Amazon S3?
      What tools integrate with Google Cloud Bigtable?
      What tools integrate with Hadoop?

      Sign up to get full access to all the tool integrationsMake informed product decisions

      What are some alternatives to Amazon S3, Google Cloud Bigtable, and Hadoop?
      Amazon Glacier
      In order to keep costs low, Amazon Glacier is optimized for data that is infrequently accessed and for which retrieval times of several hours are suitable. With Amazon Glacier, customers can reliably store large or small amounts of data for as little as $0.01 per gigabyte per month, a significant savings compared to on-premises solutions.
      Amazon EBS
      Amazon EBS volumes are network-attached, and persist independently from the life of an instance. Amazon EBS provides highly available, highly reliable, predictable storage volumes that can be attached to a running Amazon EC2 instance and exposed as a device within the instance. Amazon EBS is particularly suited for applications that require a database, file system, or access to raw block level storage.
      Amazon EC2
      It is a web service that provides resizable compute capacity in the cloud. It is designed to make web-scale computing easier for developers.
      Google Drive
      Keep photos, stories, designs, drawings, recordings, videos, and more. Your first 15 GB of storage are free with a Google Account. Your files in Drive can be reached from any smartphone, tablet, or computer.
      Microsoft Azure
      Azure is an open and flexible cloud platform that enables you to quickly build, deploy and manage applications across a global network of Microsoft-managed datacenters. You can build applications using any language, tool or framework. And you can integrate your public cloud applications with your existing IT environment.
      See all alternatives