AWS Step Functions logo

AWS Step Functions

Build Distributed Applications Using Visual Workflows
182
308
+ 1
22

What is AWS Step Functions?

AWS Step Functions makes it easy to coordinate the components of distributed applications and microservices using visual workflows. Building applications from individual components that each perform a discrete function lets you scale and change applications quickly.
AWS Step Functions is a tool in the Cloud Task Management category of a tech stack.

Who uses AWS Step Functions?

Companies
58 companies reportedly use AWS Step Functions in their tech stacks, including Shelf, backend, and FindHotel.

Developers
117 developers on StackShare have stated that they use AWS Step Functions.

AWS Step Functions Integrations

Pros of AWS Step Functions
5
Integration with other services
4
Pricing
4
Easily Accessible via AWS Console
3
Complex workflows
2
Scalability
2
High Availability
2
Workflow Processing
Decisions about AWS Step Functions

Here are some stack decisions, common use cases and reviews by companies and developers who chose AWS Step Functions in their tech stack.

Shared insights
on
AWS Step FunctionsAWS Step FunctionsAirflowAirflow

I am working on a project that grabs a set of input data from AWS S3, pre-processes and divvies it up, spins up 10K batch containers to process the divvied data in parallel on AWS Batch, post-aggregates the data, and pushes it to S3.

I already have software patterns from other projects for Airflow + Batch but have not dealt with the scaling factors of 10k parallel tasks. Airflow is nice since I can look at which tasks failed and retry a task after debugging. But dealing with that many tasks on one Airflow EC2 instance seems like a barrier. Another option would be to have one task that kicks off the 10k containers and monitors it from there.

I have no experience with AWS Step Functions but have heard it's AWS's Airflow. There looks to be plenty of patterns online for Step Functions + Batch. Do Step Functions seem like a good path to check out for my use case? Do you get the same insights on failing jobs / ability to retry tasks as you do with Airflow?

See more

AWS Step Functions Alternatives & Comparisons

What are some alternatives to AWS Step Functions?
AWS Lambda
AWS Lambda is a compute service that runs your code in response to events and automatically manages the underlying compute resources for you. You can use AWS Lambda to extend other AWS services with custom logic, or create your own back-end services that operate at AWS scale, performance, and security.
Airflow
Use Airflow to author workflows as directed acyclic graphs (DAGs) of tasks. The Airflow scheduler executes your tasks on an array of workers while following the specified dependencies. Rich command lines utilities makes performing complex surgeries on DAGs a snap. The rich user interface makes it easy to visualize pipelines running in production, monitor progress and troubleshoot issues when needed.
AWS Batch
It enables developers, scientists, and engineers to easily and efficiently run hundreds of thousands of batch computing jobs on AWS. It dynamically provisions the optimal quantity and type of compute resources (e.g., CPU or memory optimized instances) based on the volume and specific resource requirements of the batch jobs submitted.
AWS Data Pipeline
AWS Data Pipeline is a web service that provides a simple management system for data-driven workflows. Using AWS Data Pipeline, you define a pipeline composed of the “data sources” that contain your data, the “activities” or business logic such as EMR jobs or SQL queries, and the “schedule” on which your business logic executes. For example, you could define a job that, every hour, runs an Amazon Elastic MapReduce (Amazon EMR)–based analysis on that hour’s Amazon Simple Storage Service (Amazon S3) log data, loads the results into a relational database for future lookup, and then automatically sends you a daily summary email.
Batch
Yes, we’re really free. So, how do we keep the lights on? Instead of charging you a monthly fee, we sell ads on your behalf to the top 500 mobile advertisers in the world. With Batch, you earn money each month while accessing great engagement tools for free.
See all alternatives

AWS Step Functions's Followers
308 developers follow AWS Step Functions to keep up with related blogs and decisions.