I used dbt over manually setting up python wrappers around SQL scripts because it makes managing transformations within Google BigQuery much easier. This saves future Sung dozens of hours maintaining plumbing code to run a couple SQL queries. Check out my tutorial in the link!
I haven't seen any other tool make it as easy to run dependent SQL DAGs directly in a data warehouse.
Context: I wanted to create an end to end IoT data pipeline simulation in Google Cloud IoT Core and other GCP services. I never touched Terraform meaningfully until working on this project, and it's one of the best explorations in my development career. The documentation and syntax is incredibly human-readable and friendly. I'm used to building infrastructure through the google apis via Python , but I'm so glad past Sung did not make that decision. I was tempted to use Google Cloud Deployment Manager, but the templates were a bit convoluted by first impression. I'm glad past Sung did not make this decision either.
Solution: Leveraging Google Cloud Build Google Cloud Run Google Cloud Bigtable Google BigQuery Google Cloud Storage Google Compute Engine along with some other fun tools, I can deploy over 40 GCP resources using Terraform!
Check Out My Architecture: CLICK ME
Check out the GitHub repo attached
I use Tableau because it has a good drag and drop interface for data viz. I usually connect it to databases and warehouses for real-time querying.
I love the dynamic tool tip charting that's pretty easy to create.
However, it's pretty expensive for large organizations, and Tableau server runs pretty slow after awhile. I recommend this for users who don't want to code up data viz.
I use Google Cloud Run because it's like bring your own docker image to Google Cloud Functions.
I use it for building Dash Apps
It creates a nice url for web apps, and I see it being the evolution of serverless if GCP can scale this up.
I use AWS Glue because I thought it was worth all they hype Fall 2018. However, you had to use Python 2.7 with no pandas support, and cold starts lasted as long as 15 minutes. Also, setting up a dev environment for iterative development was near impossible at the time.
It was a terrible experience for me. I recommend using Amazon EMR instead. Even talking with a friend that works at Amazon, they use EMR instead of Glue for internal spark workloads. Just because a company makes something doesn't mean they use that something :/
I use Google Cloud Dataflow because it has great templates for plug and play action.
I haven't invested in the apache beam framework because you need to know Java to take full advantage of it. The Python API is a second class citizen.
I use Amazon Athena because similar to Google BigQuery , you can store and query data easily. Especially since you can define data schema in the Glue data catalog, there's a central way to define data models.
However, I would not recommend for batch jobs. I typically use this to check intermediary datasets in data engineering workloads. It's good for getting a look and feel of the data along its ETL journey.
I use Google Cloud Functions because it's the AWS Lambda equivalent on GCP. It's not as mature compared to lambda because it doesn't have VPC enablement unless done through VPC Service Controls which can be pretty cumbersome.
Although it feels bare bones compared to lambda, it still gets the job done when you want backend tasks done via serverless.
I use Airflow because it's the gold standard for scheduling batch data jobs. It comes with a bit of a learning curve given the extensive UI and working with different connectors. However, it has a lot of great retry features, and the visual DAGS help with a lot of troubleshooting.
I use Google Cloud Pub/Sub because it's good middleware and shock absorber for real-time data streaming.
It's incredibly easy to setup and plugs/plays with a lot of GCP services.
Generalist Engineer • 1 stack
sungchun12 demonstrates balanced experience across multiple technical domains with versatile stack choices.
