Alternatives to Amazon SWF logo

Alternatives to Amazon SWF

Celery, Airflow, Amazon SQS, AWS Step Functions, and Google Keep are the most popular alternatives and competitors to Amazon SWF.
24
21
+ 1
0

What is Amazon SWF and what are its top alternatives?

Amazon Simple Workflow allows you to structure the various processing steps in an application that runs across one or more machines as a set of “tasks.” Amazon SWF manages dependencies between the tasks, schedules the tasks for execution, and runs any logic that needs to be executed in parallel. The service also stores the tasks, reliably dispatches them to application components, tracks their progress, and keeps their latest state.
Amazon SWF is a tool in the Cloud Task Management category of a tech stack.

Amazon SWF alternatives & related posts

related Celery posts

James Cunningham
James Cunningham
Operations Engineer at Sentry · | 18 upvotes · 248.7K views
atSentrySentry
Celery
Celery
RabbitMQ
RabbitMQ
#MessageQueue

As Sentry runs throughout the day, there are about 50 different offline tasks that we execute—anything from “process this event, pretty please” to “send all of these cool people some emails.” There are some that we execute once a day and some that execute thousands per second.

Managing this variety requires a reliably high-throughput message-passing technology. We use Celery's RabbitMQ implementation, and we stumbled upon a great feature called Federation that allows us to partition our task queue across any number of RabbitMQ servers and gives us the confidence that, if any single server gets backlogged, others will pitch in and distribute some of the backlogged tasks to their consumers.

#MessageQueue

See more
Michael Mota
Michael Mota
CEO & Founder at AlterEstate · | 4 upvotes · 65.3K views
atAlterEstateAlterEstate
Celery
Celery
RabbitMQ
RabbitMQ
Django
Django

Automations are what makes a CRM powerful. With Celery and RabbitMQ we've been able to make powerful automations that truly works for our clients. Such as for example, automatic daily reports, reminders for their activities, important notifications regarding their client activities and actions on the website and more.

We use Celery basically for everything that needs to be scheduled for the future, and using RabbitMQ as our Queue-broker is amazing since it fully integrates with Django and Celery storing on our database results of the tasks done so we can see if anything fails immediately.

See more
Airflow logo

Airflow

399
378
20
399
378
+ 1
20
A platform to programmaticaly author, schedule and monitor data pipelines, by Airbnb
Airflow logo
Airflow
VS
Amazon SWF logo
Amazon SWF

related Amazon SQS posts

Tim Specht
Tim Specht
‎Co-Founder and CTO at Dubsmash · | 14 upvotes · 154.3K views
atDubsmashDubsmash
Google Analytics
Google Analytics
Amazon Kinesis
Amazon Kinesis
AWS Lambda
AWS Lambda
Amazon SQS
Amazon SQS
Google BigQuery
Google BigQuery
#ServerlessTaskProcessing
#GeneralAnalytics
#RealTimeDataProcessing
#BigDataAsAService

In order to accurately measure & track user behaviour on our platform we moved over quickly from the initial solution using Google Analytics to a custom-built one due to resource & pricing concerns we had.

While this does sound complicated, it’s as easy as clients sending JSON blobs of events to Amazon Kinesis from where we use AWS Lambda & Amazon SQS to batch and process incoming events and then ingest them into Google BigQuery. Once events are stored in BigQuery (which usually only takes a second from the time the client sends the data until it’s available), we can use almost-standard-SQL to simply query for data while Google makes sure that, even with terabytes of data being scanned, query times stay in the range of seconds rather than hours. Before ingesting their data into the pipeline, our mobile clients are aggregating events internally and, once a certain threshold is reached or the app is going to the background, sending the events as a JSON blob into the stream.

In the past we had workers running that continuously read from the stream and would validate and post-process the data and then enqueue them for other workers to write them to BigQuery. We went ahead and implemented the Lambda-based approach in such a way that Lambda functions would automatically be triggered for incoming records, pre-aggregate events, and write them back to SQS, from which we then read them, and persist the events to BigQuery. While this approach had a couple of bumps on the road, like re-triggering functions asynchronously to keep up with the stream and proper batch sizes, we finally managed to get it running in a reliable way and are very happy with this solution today.

#ServerlessTaskProcessing #GeneralAnalytics #RealTimeDataProcessing #BigDataAsAService

See more
Praveen Mooli
Praveen Mooli
Engineering Manager at Taylor and Francis · | 12 upvotes · 382.8K views
MongoDB Atlas
MongoDB Atlas
Java
Java
Spring Boot
Spring Boot
Node.js
Node.js
ExpressJS
ExpressJS
Python
Python
Flask
Flask
Amazon Kinesis
Amazon Kinesis
Amazon Kinesis Firehose
Amazon Kinesis Firehose
Amazon SNS
Amazon SNS
Amazon SQS
Amazon SQS
AWS Lambda
AWS Lambda
Angular 2
Angular 2
RxJS
RxJS
GitHub
GitHub
Travis CI
Travis CI
Terraform
Terraform
Docker
Docker
Serverless
Serverless
Amazon RDS
Amazon RDS
Amazon DynamoDB
Amazon DynamoDB
Amazon S3
Amazon S3
#Backend
#Microservices
#Eventsourcingframework
#Webapps
#Devops
#Data

We are in the process of building a modern content platform to deliver our content through various channels. We decided to go with Microservices architecture as we wanted scale. Microservice architecture style is an approach to developing an application as a suite of small independently deployable services built around specific business capabilities. You can gain modularity, extensive parallelism and cost-effective scaling by deploying services across many distributed servers. Microservices modularity facilitates independent updates/deployments, and helps to avoid single point of failure, which can help prevent large-scale outages. We also decided to use Event Driven Architecture pattern which is a popular distributed asynchronous architecture pattern used to produce highly scalable applications. The event-driven architecture is made up of highly decoupled, single-purpose event processing components that asynchronously receive and process events.

To build our #Backend capabilities we decided to use the following: 1. #Microservices - Java with Spring Boot , Node.js with ExpressJS and Python with Flask 2. #Eventsourcingframework - Amazon Kinesis , Amazon Kinesis Firehose , Amazon SNS , Amazon SQS, AWS Lambda 3. #Data - Amazon RDS , Amazon DynamoDB , Amazon S3 , MongoDB Atlas

To build #Webapps we decided to use Angular 2 with RxJS

#Devops - GitHub , Travis CI , Terraform , Docker , Serverless

See more
AWS Step Functions logo

AWS Step Functions

71
57
0
71
57
+ 1
0
Build Distributed Applications Using Visual Workflows
    Be the first to leave a pro
    AWS Step Functions logo
    AWS Step Functions
    VS
    Amazon SWF logo
    Amazon SWF
    Google Keep logo

    Google Keep

    13
    5
    0
    13
    5
    + 1
    0
    Capture what’s important and get more done
      Be the first to leave a pro
      Google Keep logo
      Google Keep
      VS
      Amazon SWF logo
      Amazon SWF
      Workfront logo

      Workfront

      4
      2
      0
      4
      2
      + 1
      0
      A platform for enterprise work management
        Be the first to leave a pro
        Workfront logo
        Workfront
        VS
        Amazon SWF logo
        Amazon SWF
        Taskworld logo

        Taskworld

        2
        1
        0
        2
        1
        + 1
        0
        Cloud-based task management and communication app
          Be the first to leave a pro
          Taskworld logo
          Taskworld
          VS
          Amazon SWF logo
          Amazon SWF