Airflow vs Azure Functions: What are the differences?
What is Airflow? A platform to programmaticaly author, schedule and monitor data pipelines, by Airbnb. Use Airflow to author workflows as directed acyclic graphs (DAGs) of tasks. The Airflow scheduler executes your tasks on an array of workers while following the specified dependencies. Rich command lines utilities makes performing complex surgeries on DAGs a snap. The rich user interface makes it easy to visualize pipelines running in production, monitor progress and troubleshoot issues when needed.
What is Azure Functions? Listen and react to events across your stack. Azure Functions is an event driven, compute-on-demand experience that extends the existing Azure application platform with capabilities to implement code triggered by events occurring in virtually any Azure or 3rd party service as well as on-premises systems.
Airflow belongs to "Workflow Manager" category of the tech stack, while Azure Functions can be primarily classified under "Serverless / Task Processing".
Some of the features offered by Airflow are:
- Dynamic: Airflow pipelines are configuration as code (Python), allowing for dynamic pipeline generation. This allows for writting code that instantiate pipelines dynamically.
- Extensible: Easily define your own operators, executors and extend the library so that it fits the level of abstraction that suits your environment.
- Elegant: Airflow pipelines are lean and explicit. Parameterizing your scripts is built in the core of Airflow using powerful Jinja templating engine.
On the other hand, Azure Functions provides the following key features:
- Easily schedule event-driven tasks across services
- Expose Functions as HTTP API endpoints
- Scale Functions based on customer demand
Airflow is an open source tool with 12.9K GitHub stars and 4.71K GitHub forks. Here's a link to Airflow's open source repository on GitHub.
Airbnb, Slack, and 9GAG are some of the popular companies that use Airflow, whereas Azure Functions is used by Property With Potential, OneWire, and Veris. Airflow has a broader approval, being mentioned in 72 company stacks & 33 developers stacks; compared to Azure Functions, which is listed in 30 company stacks and 22 developer stacks.
What is Airflow?
What is Azure Functions?
Need advice about which tool to choose?Ask the StackShare community!
Sign up to add, upvote and see more prosMake informed product decisions
What are the cons of using Airflow?
What are the cons of using Azure Functions?
Sign up to get full access to all the companiesMake informed product decisions
Sign up to get full access to all the tool integrationsMake informed product decisions
CodeFactor being a #SAAS product, our goal was to run on a cloud-native infrastructure since day one. We wanted to stay product focused, rather than having to work on the infrastructure that supports the application. We needed a cloud-hosting provider that would be reliable, economical and most efficient for our product.
CodeFactor.io aims to provide an automated and frictionless code review service for software developers. That requires agility, instant provisioning, autoscaling, security, availability and compliance management features. We looked at the top three #IAAS providers that take up the majority of market share: Amazon's Amazon EC2 , Microsoft's Microsoft Azure, and Google Compute Engine.
AWS has been available since 2006 and has developed the most extensive services ant tools variety at a massive scale. Azure and GCP are about half the AWS age, but also satisfied our technical requirements.
It is worth noting that even though all three providers support Docker containerization services, GCP has the most robust offering due to their investments in Kubernetes. Also, if you are a Microsoft shop, and develop in .NET - Visual Studio Azure shines at integration there and all your existing .NET code works seamlessly on Azure. All three providers have serverless computing offerings (AWS Lambda, Azure Functions, and Google Cloud Functions). Additionally, all three providers have machine learning tools, but GCP appears to be the most developer-friendly, intuitive and complete when it comes to #Machinelearning and #AI.
The prices between providers are competitive across the board. For our requirements, AWS would have been the most expensive, GCP the least expensive and Azure was in the middle. Plus, if you #Autoscale frequently with large deltas, note that Azure and GCP have per minute billing, where AWS bills you per hour. We also applied for the #Startup programs with all three providers, and this is where Azure shined. While AWS and GCP for startups would have covered us for about one year of infrastructure costs, Azure Sponsorship would cover about two years of CodeFactor's hosting costs. Moreover, Azure Team was terrific - I felt that they wanted to work with us where for AWS and GCP we were just another startup.
In summary, we were leaning towards GCP. GCP's advantages in containerization, automation toolset, #Devops mindset, and pricing were the driving factors there. Nevertheless, we could not say no to Azure's financial incentives and a strong sense of partnership and support throughout the process.
Bottom line is, IAAS offerings with AWS, Azure, and GCP are evolving fast. At CodeFactor, we aim to be platform agnostic where it is practical and retain the flexibility to cherry-pick the best products across providers.
In a couple of recent projects we had an opportunity to try out the new Serverless approach to building web applications. It wasn't necessarily a question if we should use any particular vendor but rather "if" we can consider serverless a viable option for building apps. Obviously our goal was also to get a feel for this technology and gain some hands-on experience.
We did consider AWS Lambda, Firebase from Google as well as Azure Functions. Eventually we went with AWS Lambdas.PROS
- No servers to manage (obviously!)
- Limited fixed costs – you pay only for used time
- Automated scaling and balancing
- Automatic failover (or, at this level of abstraction, no failover problem at all)
- Security easier to provide and audit
- Low overhead at the start (with the certain level of knowledge)
- Short time to market
- Easy handover - deployment coupled with code
- Perfect choice for lean startups with fast-paced iterations
- Augmentation for the classic cloud, server(full) approach
- Not much know-how and best practices available about structuring the code and projects on the market
- Not suitable for complex business logic due to the risk of producing highly coupled code
- Cost difficult to estimate (helpful tools: serverlesscalc.com)
- Difficulty in migration to other platforms (Vendor lock⚠️)
- Little engineers with experience in serverless on the job market
- Steep learning curve for engineers without any cloud experience
More details are on our blog: https://evojam.com/blog/2018/12/5/should-you-go-serverless-meet-the-benefits-and-flaws-of-new-wave-of-cloud-solutions I hope it helps 🙌 & I'm curious of your experiences.
AWS Lambda Serverless Amazon CloudWatch Azure Functions Google Cloud Functions Node.js
In the last year or so, I moved all Checkly monitoring workloads to AWS Lambda. Here are some stats:
- We run three core functions in all AWS regions. They handle API checks, browser checks and setup / teardown scripts. Check our docs to find out what that means.
- All functions are hooked up to SNS topics but can also be triggered directly through AWS SDK calls.
- The busiest function is a plumbing function that forwards data to our database. It is invoked anywhere between 7000 and 10.000 times per hour with an average duration of about 179 ms.
- We run separate dev and test versions of each function in each region.
Moving all this to AWS Lambda took some work and considerations. The blog post linked below goes into the following topics:
- Why Lambda is an almost perfect match for SaaS. Especially when you're small.
- Why I don't use a "big" framework around it.
- Why distributed background jobs triggered by queues are Lambda's raison d'être.
- Why monitoring & logging is still an issue.
Poor developer experience
I used Azure functions as part of an integration service when creating a bulk insert module in azure.