I use Google Cloud Dataflow because it has great templates for plug and play action.

I haven't invested in the apache beam framework because you need to know Java to take full advantage of it. The Python API is a second class citizen.