Google BigQuery vs Google Cloud Datastore

Overview

Google BigQuery

Stacks1.8K

Followers1.5K

Votes152

Google Cloud Datastore

Stacks290

Followers357

Votes12

Google BigQuery vs Google Cloud Datastore: What are the differences?

Introduction

In this article, we will discuss the key differences between Google BigQuery and Google Cloud Datastore. Both of these services are part of the Google Cloud Platform and provide different functionalities for data storage and analysis.

Scalability: Google BigQuery is a fully managed, serverless data warehouse that can handle large-scale datasets and supports high-concurrency queries. It is designed to handle petabytes of data and allows for fast query execution. On the other hand, Google Cloud Datastore is a NoSQL document database that scales horizontally and is suitable for smaller-scale applications that require rapid development and low-latency access to data.
Querying Abilities: Google BigQuery supports powerful SQL-like queries that can be used for complex data analysis tasks. It has a wide range of built-in functions and supports nested and repeated fields. In contrast, Google Cloud Datastore uses a query language that is based on the Google Datastore API and is optimized for simple lookups and fetching individual entities. It does not support complex queries like joins or aggregations.
Data Structure: Google BigQuery uses a tabular data structure, similar to a traditional relational database, where data is organized into tables with rows and columns. It also supports nested and repeated fields, allowing for more flexible data modeling. On the other hand, Google Cloud Datastore uses an entity-based data model, where each entity can have properties of different types. This enables more flexible schema design but can be less efficient for certain types of queries.
Storage Cost: Google BigQuery charges users based on the amount of data stored and the volume of data processed during queries. It offers different pricing tiers for on-demand and flat-rate usage. In comparison, Google Cloud Datastore charges users for storage usage and operations (reads, writes, deletes) on a per-unit basis. The cost depends on the number of entities stored and the traffic generated by the application.
Transactions and ACID Compliance: Google Cloud Datastore supports strong consistency and provides full ACID compliance for transactional operations. It ensures that all reads and writes within a transaction are isolated and atomic. On the other hand, Google BigQuery is not a fully ACID-compliant database and does not support transactional operations. It is optimized for fast analytics queries and bulk data processing.
Data Lifecycle Management: Google BigQuery provides features for managing data lifecycle, such as table expiration and partitioning, to optimize storage costs and improve query performance. It allows users to automatically delete or move data based on specified criteria. In comparison, Google Cloud Datastore does not have built-in lifecycle management features and users need to implement their own data archiving or deletion strategies.

In summary, Google BigQuery is a powerful data warehousing solution that is optimized for large-scale data analysis, complex queries, and high-concurrency workloads. It provides fast query execution and scalability, but it is not ACID-compliant and may not be suitable for transactional operations. On the other hand, Google Cloud Datastore is a flexible NoSQL database that is suitable for smaller-scale applications, rapid development, and low-latency access to data. It supports strong consistency, ACID compliance, and horizontal scalability. However, it has limitations in terms of complex querying capabilities and data modeling options.

Share your Stack

Help developers discover the tools you use. Get visibility for your team's tech choices and contribute to the community's knowledge.

View Docs

CLI (Node.js)

Manual

Advice on Google BigQuery, Google Cloud Datastore

Julien

CTO at Hawk

Sep 19, 2020

Decided

Cloud Data-warehouse is the centerpiece of modern Data platform. The choice of the most suitable solution is therefore fundamental.

Our benchmark was conducted over BigQuery and Snowflake. These solutions seem to match our goals but they have very different approaches.

BigQuery is notably the only 100% serverless cloud data-warehouse, which requires absolutely NO maintenance: no re-clustering, no compression, no index optimization, no storage management, no performance management. Snowflake requires to set up (paid) reclustering processes, to manage the performance allocated to each profile, etc. We can also mention Redshift, which we have eliminated because this technology requires even more ops operation.

BigQuery can therefore be set up with almost zero cost of human resources. Its on-demand pricing is particularly adapted to small workloads. 0 cost when the solution is not used, only pay for the query you're running. But quickly the use of slots (with monthly or per-minute commitment) will drastically reduce the cost of use. We've reduced by 10 the cost of our nightly batches by using flex slots.

Finally, a major advantage of BigQuery is its almost perfect integration with Google Cloud Platform services: Cloud functions, Dataflow, Data Studio, etc.

BigQuery is still evolving very quickly. The next milestone, BigQuery Omni, will allow to run queries over data stored in an external Cloud platform (Amazon S3 for example). It will be a major breakthrough in the history of cloud data-warehouses. Omni will compensate a weakness of BigQuery: transferring data in near real time from S3 to BQ is not easy today. It was even simpler to implement via Snowflake's Snowpipe solution.

We also plan to use the Machine Learning features built into BigQuery to accelerate our deployment of Data-Science-based projects. An opportunity only offered by the BigQuery solution

193k views193k

Comments

Detailed Comparison

Google BigQuery	Google Cloud Datastore
Run super-fast, SQL-like queries against terabytes of data in seconds, using the processing power of Google's infrastructure. Load data with ease. Bulk load your data using Google Cloud Storage or stream it in. Easy access. Access BigQuery by using a browser tool, a command-line tool, or by making calls to the BigQuery REST API with client libraries such as Java, PHP or Python.	Use a managed, NoSQL, schemaless database for storing non-relational data. Cloud Datastore automatically scales as you need it and supports transactions as well as robust, SQL-like queries.
All behind the scenes- Your queries can execute asynchronously in the background, and can be polled for status.;Import data with ease- Bulk load your data using Google Cloud Storage or stream it in bursts of up to 1,000 rows per second.;Affordable big data- The first Terabyte of data processed each month is free.;The right interface- Separate interfaces for administration and developers will make sure that you have access to the tools you need.	Schemaless access, with SQL-like querying;Managed database;Autoscale with your users;ACID transactions;Built-in redundancy;Local development tools
Statistics
Stacks 1.8K	Stacks 290
Followers 1.5K	Followers 357
Votes 152	Votes 12
Pros & Cons
Pros 28 High Performance 25 Easy to use 22 Fully managed service 19 Cheap Pricing 16 Process hundreds of GB in seconds Cons 1 You can't unit test changes in BQ data 0 Sdas	Pros 7 High scalability 2 Ability to query any property 2 Serverless 1 Pay for what you use
Integrations
Xplenty Fluentd Looker Chartio Treasure Data	No integrations available

What are some alternatives to Google BigQuery, Google Cloud Datastore?

Amazon DynamoDB

With it , you can offload the administrative burden of operating and scaling a highly available distributed database cluster, while paying a low price for only what you use.

Azure Cosmos DB

Azure DocumentDB is a fully managed NoSQL database service built for fast and predictable performance, high availability, elastic scaling, global distribution, and ease of development.

Cloud Firestore

Cloud Firestore is a NoSQL document database that lets you easily store, sync, and query data for your mobile and web apps - at global scale.

Amazon Redshift

It is optimized for data sets ranging from a few hundred gigabytes to a petabyte or more and costs less than $1,000 per terabyte per year, a tenth the cost of most traditional data warehousing solutions.

Qubole

Qubole is a cloud based service that makes big data easy for analysts and data engineers.

Amazon EMR

It is used in a variety of applications, including log analysis, data warehousing, machine learning, financial analysis, scientific simulation, and bioinformatics.

Cloudant

Cloudant’s distributed database as a service (DBaaS) allows developers of fast-growing web and mobile apps to focus on building and improving their products, instead of worrying about scaling and managing databases on their own.

Altiscale

we run Apache Hadoop for you. We not only deploy Hadoop, we monitor, manage, fix, and update it for you. Then we take it a step further: We monitor your jobs, notify you when something’s wrong with them, and can help with tuning.

Snowflake

Snowflake eliminates the administration and management demands of traditional data warehouses and big data platforms. Snowflake is a true data warehouse as a service running on Amazon Web Services (AWS)—no infrastructure to manage and no knobs to turn.

Google Cloud Bigtable

Google Cloud Bigtable offers you a fast, fully managed, massively scalable NoSQL database service that's ideal for web, mobile, and Internet of Things applications requiring terabytes to petabytes of data. Unlike comparable market offerings, Cloud Bigtable doesn't require you to sacrifice speed, scale, or cost efficiency when your applications grow. Cloud Bigtable has been battle-tested at Google for more than 10 years—it's the database driving major applications such as Google Analytics and Gmail.

Related Comparisons

Google BigQuery vs Google Cloud Datastore: What are the differences?

Introduction

Scalability: Google BigQuery is a fully managed, serverless data warehouse that can handle large-scale datasets and supports high-concurrency queries. It is designed to handle petabytes of data and allows for fast query execution. On the other hand, Google Cloud Datastore is a NoSQL document database that scales horizontally and is suitable for smaller-scale applications that require rapid development and low-latency access to data.
Querying Abilities: Google BigQuery supports powerful SQL-like queries that can be used for complex data analysis tasks. It has a wide range of built-in functions and supports nested and repeated fields. In contrast, Google Cloud Datastore uses a query language that is based on the Google Datastore API and is optimized for simple lookups and fetching individual entities. It does not support complex queries like joins or aggregations.
Data Structure: Google BigQuery uses a tabular data structure, similar to a traditional relational database, where data is organized into tables with rows and columns. It also supports nested and repeated fields, allowing for more flexible data modeling. On the other hand, Google Cloud Datastore uses an entity-based data model, where each entity can have properties of different types. This enables more flexible schema design but can be less efficient for certain types of queries.
Storage Cost: Google BigQuery charges users based on the amount of data stored and the volume of data processed during queries. It offers different pricing tiers for on-demand and flat-rate usage. In comparison, Google Cloud Datastore charges users for storage usage and operations (reads, writes, deletes) on a per-unit basis. The cost depends on the number of entities stored and the traffic generated by the application.
Transactions and ACID Compliance: Google Cloud Datastore supports strong consistency and provides full ACID compliance for transactional operations. It ensures that all reads and writes within a transaction are isolated and atomic. On the other hand, Google BigQuery is not a fully ACID-compliant database and does not support transactional operations. It is optimized for fast analytics queries and bulk data processing.
Data Lifecycle Management: Google BigQuery provides features for managing data lifecycle, such as table expiration and partitioning, to optimize storage costs and improve query performance. It allows users to automatically delete or move data based on specified criteria. In comparison, Google Cloud Datastore does not have built-in lifecycle management features and users need to implement their own data archiving or deletion strategies.

Google BigQuery vs Google Cloud Datastore

Overview

Google BigQuery vs Google Cloud Datastore: What are the differences?