Google BigQuery vs Hadoop

Need advice about which tool to choose?Ask the StackShare community!

Google BigQuery

1K
867
+ 1
144
Hadoop

1.9K
1.8K
+ 1
54
Add tool

Google BigQuery vs Hadoop: What are the differences?

Developers describe Google BigQuery as "Analyze terabytes of data in seconds". Run super-fast, SQL-like queries against terabytes of data in seconds, using the processing power of Google's infrastructure Load data with ease. Bulk load your data using Google Cloud Storage or stream it in. Easy access. Access BigQuery by using a browser tool, a command-line tool, or by making calls to the BigQuery REST API with client libraries such as Java, PHP or Python.. On the other hand, Hadoop is detailed as "Open-source software for reliable, scalable, distributed computing". The Apache Hadoop software library is a framework that allows for the distributed processing of large data sets across clusters of computers using simple programming models. It is designed to scale up from single servers to thousands of machines, each offering local computation and storage.

Google BigQuery can be classified as a tool in the "Big Data as a Service" category, while Hadoop is grouped under "Databases".

"High Performance" is the primary reason why developers consider Google BigQuery over the competitors, whereas "Great ecosystem" was stated as the key factor in picking Hadoop.

Hadoop is an open source tool with 9.18K GitHub stars and 5.74K GitHub forks. Here's a link to Hadoop's open source repository on GitHub.

According to the StackShare community, Hadoop has a broader approval, being mentioned in 237 company stacks & 116 developers stacks; compared to Google BigQuery, which is listed in 156 company stacks and 39 developer stacks.

Decisions about Google BigQuery and Hadoop
Julien Lafont

Cloud Data-warehouse is the centerpiece of modern Data platform. The choice of the most suitable solution is therefore fundamental.

Our benchmark was conducted over BigQuery and Snowflake. These solutions seem to match our goals but they have very different approaches.

BigQuery is notably the only 100% serverless cloud data-warehouse, which requires absolutely NO maintenance: no re-clustering, no compression, no index optimization, no storage management, no performance management. Snowflake requires to set up (paid) reclustering processes, to manage the performance allocated to each profile, etc. We can also mention Redshift, which we have eliminated because this technology requires even more ops operation.

BigQuery can therefore be set up with almost zero cost of human resources. Its on-demand pricing is particularly adapted to small workloads. 0 cost when the solution is not used, only pay for the query you're running. But quickly the use of slots (with monthly or per-minute commitment) will drastically reduce the cost of use. We've reduced by 10 the cost of our nightly batches by using flex slots.

Finally, a major advantage of BigQuery is its almost perfect integration with Google Cloud Platform services: Cloud functions, Dataflow, Data Studio, etc.

BigQuery is still evolving very quickly. The next milestone, BigQuery Omni, will allow to run queries over data stored in an external Cloud platform (Amazon S3 for example). It will be a major breakthrough in the history of cloud data-warehouses. Omni will compensate a weakness of BigQuery: transferring data in near real time from S3 to BQ is not easy today. It was even simpler to implement via Snowflake's Snowpipe solution.

We also plan to use the Machine Learning features built into BigQuery to accelerate our deployment of Data-Science-based projects. An opportunity only offered by the BigQuery solution

See more
Pros of Google BigQuery
Pros of Hadoop
  • 27
    High Performance
  • 23
    Easy to use
  • 21
    Fully managed service
  • 19
    Cheap Pricing
  • 16
    Process hundreds of GB in seconds
  • 11
    Full table scans in seconds, no indexes needed
  • 10
    Big Data
  • 8
    Always on, no per-hour costs
  • 5
    Good combination with fluentd
  • 4
    Machine learning
  • 38
    Great ecosystem
  • 11
    One stack to rule them all
  • 4
    Great load balancer
  • 1
    Java syntax

Sign up to add or upvote prosMake informed product decisions

Cons of Google BigQuery
Cons of Hadoop
  • 1
    You can't unit test changes in BQ data
    Be the first to leave a con

    Sign up to add or upvote consMake informed product decisions

    - No public GitHub repository available -

    What is Google BigQuery?

    Run super-fast, SQL-like queries against terabytes of data in seconds, using the processing power of Google's infrastructure. Load data with ease. Bulk load your data using Google Cloud Storage or stream it in. Easy access. Access BigQuery by using a browser tool, a command-line tool, or by making calls to the BigQuery REST API with client libraries such as Java, PHP or Python.

    What is Hadoop?

    The Apache Hadoop software library is a framework that allows for the distributed processing of large data sets across clusters of computers using simple programming models. It is designed to scale up from single servers to thousands of machines, each offering local computation and storage.

    Need advice about which tool to choose?Ask the StackShare community!

    What companies use Google BigQuery?
    What companies use Hadoop?

    Sign up to get full access to all the companiesMake informed product decisions

    What tools integrate with Google BigQuery?
    What tools integrate with Hadoop?

    Sign up to get full access to all the tool integrationsMake informed product decisions

    Blog Posts

    MySQLKafkaApache Spark+6
    2
    1339
    Aug 28 2019 at 3:10AM
    https://img.stackshare.io/stack/505487/default_e35b8bd5e615e01dc9b420dbd2a444fcbaeff755.png logo

    Segment

    PythonJavaAmazon S3+16
    5
    1892
    Jul 2 2019 at 9:34PM
    https://img.stackshare.io/stack/374725/default_11e3fd22f9363f9e6b2bd10a6a73e7a93b3a221c.png logo

    Segment

    Google AnalyticsAmazon S3New Relic+25
    10
    5210
    GitHubPythonNode.js+47
    43
    67726
    What are some alternatives to Google BigQuery and Hadoop?
    Google Cloud Bigtable
    Google Cloud Bigtable offers you a fast, fully managed, massively scalable NoSQL database service that's ideal for web, mobile, and Internet of Things applications requiring terabytes to petabytes of data. Unlike comparable market offerings, Cloud Bigtable doesn't require you to sacrifice speed, scale, or cost efficiency when your applications grow. Cloud Bigtable has been battle-tested at Google for more than 10 years—it's the database driving major applications such as Google Analytics and Gmail.
    Amazon Redshift
    It is optimized for data sets ranging from a few hundred gigabytes to a petabyte or more and costs less than $1,000 per terabyte per year, a tenth the cost of most traditional data warehousing solutions.
    Snowflake
    Snowflake eliminates the administration and management demands of traditional data warehouses and big data platforms. Snowflake is a true data warehouse as a service running on Amazon Web Services (AWS)—no infrastructure to manage and no knobs to turn.
    Google Analytics
    Google Analytics lets you measure your advertising ROI as well as track your Flash, video, and social networking sites and applications.
    Elasticsearch
    Elasticsearch is a distributed, RESTful search and analytics engine capable of storing data and searching it in near real time. Elasticsearch, Kibana, Beats and Logstash are the Elastic Stack (sometimes called the ELK Stack).
    See all alternatives
    Interest over time
    How much does Google BigQuery cost?
    How much does Hadoop cost?
    Pricing unavailable
    News about Google BigQuery
    More news