Google Cloud Bigtable

Google Cloud Bigtable

Application and Data / Data Stores / NoSQL Database as a Service
Avatar of rorygwozdz
CTO at Harvested Financial·

I'm trying to build a way to read financial data really, really fast, for low cost. We are write/update-light (in this arena) and read-heavy. Google BigQuery being serverless can keep costs beyond low, but query speeds are always a few seconds because, I think, of the lack of indexing and potential to take advantage of the structure of the common queries. I have tried various partitions on BigQuery to speed things up too with some success but nothing extraordinary. I have never used Google Cloud Bigtable but get how it works conceptually. I believe it would make date-range based queries markedly faster. Question is, are there ways to take advantage of date-ranges in BigQuery, or does it makes sense to just shift to BigTable for mega-fast reads? I'd love to get sub-50ms.

READ MORE
2 upvotes·127.8K views
Replies (1)
Avatar of sanjeevsingh
Lead Data Engineer at BharatPe·

As a DataWarehouse Solution Google Bigquery is meant more for Large Data Analysis then real time Write/Update. You can go with BigTable instead of BigQuery but be prepare for the hight cost. Also, in most of the Data solution if you are looking for heavy real time Wrire/Update you have to put some cost on the solution. For more detail you can check this link https://cloud.google.com/blog/products/gcp/in-memory-query-execution-in-google-bigquery

READ MORE
3 upvotes·2.5K views

Context: I wanted to create an end to end IoT data pipeline simulation in Google Cloud IoT Core and other GCP services. I never touched Terraform meaningfully until working on this project, and it's one of the best explorations in my development career. The documentation and syntax is incredibly human-readable and friendly. I'm used to building infrastructure through the google apis via Python , but I'm so glad past Sung did not make that decision. I was tempted to use Google Cloud Deployment Manager, but the templates were a bit convoluted by first impression. I'm glad past Sung did not make this decision either.

Solution: Leveraging Google Cloud Build Google Cloud Run Google Cloud Bigtable Google BigQuery Google Cloud Storage Google Compute Engine along with some other fun tools, I can deploy over 40 GCP resources using Terraform!

Check Out My Architecture: CLICK ME

Check out the GitHub repo attached

READ MORE
GitHub - sungchun12/iot-python-webapp: Live, real-time dashboard in a serverless docker web app, and deployed via terraform with a built-in CICD trigger-See Mock Website (github.com)
33 upvotes·4 comments·2.1M views
Franco Austin
Franco Austin
·
November 22nd 2019 at 10:54AM

Relly like the architectural artefact, what did you use to produce it?

·
Reply
Sung Won Chung
Sung Won Chung
·
November 24th 2019 at 9:57PM

I used draw.io. Free and easy to use!

https://www.draw.io/

·
Reply
SANJEEV TOORA (UK)
SANJEEV TOORA (UK)
·
January 29th 2020 at 8:55AM

Architecture looks great! Just out of curiosity, did you have unit and integration testing components for Terraform e.g. Terratest, Kitchen Terraform, Sentinel?

·
Reply
Sung Won Chung
Sung Won Chung
·
January 29th 2020 at 3:17PM

I did NOT. This was to demo the mechanics. Not something for robust production. Would love if you can make a pull request for some of those tests ;)

·
Reply