Need advice about which tool to choose?Ask the StackShare community!
Amazon EMR vs AWS Lambda: What are the differences?
Developers describe Amazon EMR as "Distribute your data and processing across a Amazon EC2 instances using Hadoop". Amazon EMR is used in a variety of applications, including log analysis, web indexing, data warehousing, machine learning, financial analysis, scientific simulation, and bioinformatics. Customers launch millions of Amazon EMR clusters every year. On the other hand, AWS Lambda is detailed as "Automatically run code in response to modifications to objects in Amazon S3 buckets, messages in Kinesis streams, or updates in DynamoDB". AWS Lambda is a compute service that runs your code in response to events and automatically manages the underlying compute resources for you. You can use AWS Lambda to extend other AWS services with custom logic, or create your own back-end services that operate at AWS scale, performance, and security.
Amazon EMR can be classified as a tool in the "Big Data as a Service" category, while AWS Lambda is grouped under "Serverless / Task Processing".
Some of the features offered by Amazon EMR are:
- Elastic- Amazon EMR enables you to quickly and easily provision as much capacity as you need and add or remove capacity at any time. Deploy multiple clusters or resize a running cluster
- Low Cost- Amazon EMR is designed to reduce the cost of processing large amounts of data. Some of the features that make it low cost include low hourly pricing, Amazon EC2 Spot integration, Amazon EC2 Reserved Instance integration, elasticity, and Amazon S3 integration.
- Flexible Data Stores- With Amazon EMR, you can leverage multiple data stores, including Amazon S3, the Hadoop Distributed File System (HDFS), and Amazon DynamoDB.
On the other hand, AWS Lambda provides the following key features:
- Extend other AWS services with custom logic
- Build custom back-end services
- Completely Automated Administration
"On demand processing power" is the primary reason why developers consider Amazon EMR over the competitors, whereas "No infrastructure" was stated as the key factor in picking AWS Lambda.
According to the StackShare community, AWS Lambda has a broader approval, being mentioned in 1023 company stacks & 613 developers stacks; compared to Amazon EMR, which is listed in 95 company stacks and 18 developer stacks.
When adding a new feature to Checkly rearchitecting some older piece, I tend to pick Heroku for rolling it out. But not always, because sometimes I pick AWS Lambda . The short story:
- Developer Experience trumps everything.
- AWS Lambda is cheap. Up to a limit though. This impact not only your wallet.
- If you need geographic spread, AWS is lonely at the top.
Recently, I was doing a brainstorm at a startup here in Berlin on the future of their infrastructure. They were ready to move on from their initial, almost 100% Ec2 + Chef based setup. Everything was on the table. But we crossed out a lot quite quickly:
- Pure, uncut, self hosted Kubernetes — way too much complexity
- Managed Kubernetes in various flavors — still too much complexity
- Zeit — Maybe, but no Docker support
- Elastic Beanstalk — Maybe, bit old but does the job
- Heroku
- Lambda
It became clear a mix of PaaS and FaaS was the way to go. What a surprise! That is exactly what I use for Checkly! But when do you pick which model?
I chopped that question up into the following categories:
- Developer Experience / DX 🤓
- Ops Experience / OX 🐂 (?)
- Cost 💵
- Lock in 🔐
Read the full post linked below for all details
Pros of Amazon EMR
- On demand processing power15
- Don't need to maintain Hadoop Cluster yourself12
- Hadoop Tools7
- Elastic6
- Backed by Amazon4
- Flexible3
- Economic - pay as you go, easy to use CLI and SDKs3
- Don't need a dedicated Ops group2
- Massive data handling1
- Great support1
Pros of AWS Lambda
- No infrastructure129
- Cheap83
- Quick70
- Stateless59
- No deploy, no server, great sleep47
- AWS Lambda went down taking many sites with it12
- Event Driven Governance6
- Extensive API6
- Auto scale and cost effective6
- Easy to deploy6
- VPC Support5
- Integrated with various AWS services3
Sign up to add or upvote prosMake informed product decisions
Cons of Amazon EMR
Cons of AWS Lambda
- Cant execute ruby or go7
- Compute time limited3
- Can't execute PHP w/o significant effort1
Sign up to add or upvote consMake informed product decisions
What is Amazon EMR?
What is AWS Lambda?
Need advice about which tool to choose?Ask the StackShare community!
Sign up to get full access to all the companiesMake informed product decisions
What tools integrate with Amazon EMR?
What tools integrate with AWS Lambda?
Sign up to get full access to all the tool integrationsMake informed product decisions
Blog Posts
+16
+47
+43
+26
+23+42
+44