Eugene Ivanchenko

On one side SF could be compared to Airflow: same distributed flow control, but there is a difference: SF is more autonomous: you could specify retry rules and so on, but only before you start execution. It is not possible to have same level of manual control as in Airflow. Also there limitations on number of executed activities. Not small, but still restrictive. I think if you have complex not fully automated process Airflow still good for you. Thinking about your scale, you just need to spawn and control Batch - this fits perfectly for airflow steps. This is my personal opinion.