Why developers like Google Cloud Dataflow

What is Google Cloud Dataflow?

Google Cloud Dataflow is a unified programming model and a managed service for developing and executing a wide range of data processing patterns including ETL, batch computation, and continuous computation. Cloud Dataflow frees you from operational tasks like resource management and performance optimization.

Google Cloud Dataflow is a tool in the Real-time Data Processing category of a tech stack.

Who uses Google Cloud Dataflow?

Companies

70 companies reportedly use Google Cloud Dataflow in their tech stacks, including Spotify, The New York Times, and BlaBlaCar.

Spotify

The New York Times

BlaBlaCar

PLAID

StreamElements

Teads

Samba Tech

bol.com

Ocado Technology

Developers

144 developers on StackShare have stated that they use Google Cloud Dataflow.

Wishlist

dgirard

My Stack

Bubble Student

gcp-datatech

Smore

My Stack

Personal Stack

My Stack

Google Cloud Dataflow Integrations

Google AI Platform, Google AutoML Tables, Google Cloud Healthcare API, Aviatrix, and Cloud AI Platform Pipelines are some of the popular tools that integrate with Google Cloud Dataflow. Here's a list of all 7 tools that integrate with Google Cloud Dataflow.

Google AI Platform

Google AutoML Tables

Google Cloud Healthcare ...

Aviatrix

Cloud AI Platform ...

WhyLabs

Sematic

Pros of Google Cloud Dataflow

Unified batch and stream processing

Autoscaling

Fully managed

Throughput Transparency

Decisions about Google Cloud Dataflow

Here are some stack decisions, common use cases and reviews by companies and developers who chose Google Cloud Dataflow in their tech stack.

Andrea Latorre

Jan 2, 2023 | 6 upvotes · 212.4K views

Needs advice

Google Cloud Dataflow

and

Google Cloud Data Fusion

I am currently launching 50 pipelines in a Google Cloud Data Fusion version 6.4 instance. These pipelines are launched daily and transport data from a MySQLServer database to Google BigQuery. The cost is becoming very high and I was wondering if the costs with Google Cloud Dataflow decrease for the same rows transported.

Vishal Yadav

Dec 26, 2022 | 6 upvotes · 214.8K views

Needs advice

AWS Glue

Google Cloud Dataflow

and

Google Cloud Data Fusion

Will Dataflow be the right replacement for AWS Glue? Are there any unforeseen exceptions like certain proprietary transformations not supported in Google Cloud Dataflow, connectors ecosystem, Data Quality & Date cleansing not supported in DataFlow. etc?

Also, how about Google Cloud Data Fusion as a replacement? In terms of No Code/Low code .. (Since basic use cases in Glue support UI, in that case, CDF may be the right choice ).

What would be the best choice?

Jithin Eapen

Mar 11, 2020 | 5 upvotes · 158.4K views

Needs advice

Google App Engine

Google Cloud Dataflow

and

Google Cloud Functions

What are the best options to host a Spring Boot application that acts as a receiver and publisher from Google Cloud Pub/Sub. I am using Google App Engine to do that, but there is Google Cloud Dataflow and Google Cloud Run that can be used. Which is the best option that can be used for this purpose and also that can handle the failover scenarios as well. Thanks!

See all decisions

Google Cloud Dataflow's Features

Fully managed
Combines batch and streaming with a single API
High performance with automatic workload rebalancingOpen source SDK

Google Cloud Dataflow Alternatives & Comparisons

What are some alternatives to Google Cloud Dataflow?

Apache Spark

Spark is a fast and general processing engine compatible with Hadoop data. It can run in Hadoop clusters through YARN or Spark's standalone mode, and it can process data in HDFS, HBase, Cassandra, Hive, and any Hadoop InputFormat. It is designed to perform both batch processing (similar to MapReduce) and new workloads like streaming, interactive queries, and machine learning.

Kafka

Kafka is a distributed, partitioned, replicated commit log service. It provides the functionality of a messaging system, but with a unique design.

Hadoop

The Apache Hadoop software library is a framework that allows for the distributed processing of large data sets across clusters of computers using simple programming models. It is designed to scale up from single servers to thousands of machines, each offering local computation and storage.

Akutan

A distributed knowledge graph store. Knowledge graphs are suitable for modeling data that is highly interconnected by many types of relationships, like encyclopedic information about the world.

Apache Beam

It implements batch and streaming data processing jobs that run on any execution engine. It executes pipelines on multiple execution environments.

See all alternatives

Related Comparisons