Google Cloud Dataflow logo
A fully-managed cloud service and programming model for batch and streaming big data processing.
73
50
+ 1
0

What is Google Cloud Dataflow?

Google Cloud Dataflow is a unified programming model and a managed service for developing and executing a wide range of data processing patterns including ETL, batch computation, and continuous computation. Cloud Dataflow frees you from operational tasks like resource management and performance optimization.
Google Cloud Dataflow is a tool in the Real-time Data Processing category of a tech stack.

Who uses Google Cloud Dataflow?

Companies
41 companies reportedly use Google Cloud Dataflow in their tech stacks, including Spotify, RD Station, and Kapten Engineering.

Developers
30 developers on StackShare have stated that they use Google Cloud Dataflow.

Why developers like Google Cloud Dataflow?

Here’s a list of reasons why companies and developers use Google Cloud Dataflow
Top Reasons
Be the first to leave a pro
Google Cloud Dataflow Reviews

Here are some stack decisions, common use cases and reviews by companies and developers who chose Google Cloud Dataflow in their tech stack.

Nick Rockwell
Nick Rockwell
CTO at NY Times · | 5 upvotes · 7.7K views
atThe New York TimesThe New York Times
Amazon DynamoDB
Amazon DynamoDB
Google Cloud Dataflow
Google Cloud Dataflow
Google Cloud Pub/Sub
Google Cloud Pub/Sub
Google BigQuery
Google BigQuery
#AnalyticsPipeline
#Analytics

We really drank the Google Kool-Aid on analytics. So, everything's going into Google BigQuery and almost everything is going straight into Google Cloud Pub/Sub and then doing some processing in Google Cloud Dataflow before ending up in BigQuery. We still do too much processing and augmentation on the front end before it goes into Pub/Sub. And that's using some kind of stuff we pulled together using Amazon DynamoDB and so on. And it's very brittle, actually. Actually, Dynamo throttling is one of our biggest headaches. So, I want all of that to go away and do all our augmentation in BigQuery after the data's been collected. And having it just go straight into Pub/Sub. So, we're working on that. And it'll happen, some time. #Analytics #AnalyticsPipeline

See more
Java
Java
Google Cloud Dataflow
Google Cloud Dataflow

I use Google Cloud Dataflow because it has great templates for plug and play action.

I haven't invested in the apache beam framework because you need to know Java to take full advantage of it. The Python API is a second class citizen.

See more

Google Cloud Dataflow's Features

  • Fully managed
  • Combines batch and streaming with a single API
  • High performance with automatic workload rebalancing Open source SDK

Google Cloud Dataflow Alternatives & Comparisons

What are some alternatives to Google Cloud Dataflow?
Apache Spark
Spark is a fast and general processing engine compatible with Hadoop data. It can run in Hadoop clusters through YARN or Spark's standalone mode, and it can process data in HDFS, HBase, Cassandra, Hive, and any Hadoop InputFormat. It is designed to perform both batch processing (similar to MapReduce) and new workloads like streaming, interactive queries, and machine learning.
Kafka
Kafka is a distributed, partitioned, replicated commit log service. It provides the functionality of a messaging system, but with a unique design.
Hadoop
The Apache Hadoop software library is a framework that allows for the distributed processing of large data sets across clusters of computers using simple programming models. It is designed to scale up from single servers to thousands of machines, each offering local computation and storage.
Beam
A distributed knowledge graph store. Knowledge graphs are suitable for modeling data that is highly interconnected by many types of relationships, like encyclopedic information about the world.
Amazon Kinesis
Amazon Kinesis can collect and process hundreds of gigabytes of data per second from hundreds of thousands of sources, allowing you to easily write applications that process information in real-time, from sources such as web site click-streams, marketing and financial information, manufacturing instrumentation and social media, and operational logs and metering data.
See all alternatives

Google Cloud Dataflow's Followers
50 developers follow Google Cloud Dataflow to keep up with related blogs and decisions.
Danny Robert
Lam Quang
Thiago Chiarato
Ashish Tanwer
Mohamma76685757
Ganesh Chavan
Murat Denizli
Thananjaya Chakravarthy
Vineet M
The-Edgar