Need advice about which tool to choose?Ask the StackShare community!

Amazon EMR

441
510
+ 1
54
AWS Lambda

14.7K
10.9K
+ 1
411
Add tool

Amazon EMR vs AWS Lambda: What are the differences?

Developers describe Amazon EMR as "Distribute your data and processing across a Amazon EC2 instances using Hadoop". Amazon EMR is used in a variety of applications, including log analysis, web indexing, data warehousing, machine learning, financial analysis, scientific simulation, and bioinformatics. Customers launch millions of Amazon EMR clusters every year. On the other hand, AWS Lambda is detailed as "Automatically run code in response to modifications to objects in Amazon S3 buckets, messages in Kinesis streams, or updates in DynamoDB". AWS Lambda is a compute service that runs your code in response to events and automatically manages the underlying compute resources for you. You can use AWS Lambda to extend other AWS services with custom logic, or create your own back-end services that operate at AWS scale, performance, and security.

Amazon EMR can be classified as a tool in the "Big Data as a Service" category, while AWS Lambda is grouped under "Serverless / Task Processing".

Some of the features offered by Amazon EMR are:

  • Elastic- Amazon EMR enables you to quickly and easily provision as much capacity as you need and add or remove capacity at any time. Deploy multiple clusters or resize a running cluster
  • Low Cost- Amazon EMR is designed to reduce the cost of processing large amounts of data. Some of the features that make it low cost include low hourly pricing, Amazon EC2 Spot integration, Amazon EC2 Reserved Instance integration, elasticity, and Amazon S3 integration.
  • Flexible Data Stores- With Amazon EMR, you can leverage multiple data stores, including Amazon S3, the Hadoop Distributed File System (HDFS), and Amazon DynamoDB.

On the other hand, AWS Lambda provides the following key features:

  • Extend other AWS services with custom logic
  • Build custom back-end services
  • Completely Automated Administration

"On demand processing power" is the primary reason why developers consider Amazon EMR over the competitors, whereas "No infrastructure" was stated as the key factor in picking AWS Lambda.

According to the StackShare community, AWS Lambda has a broader approval, being mentioned in 1023 company stacks & 613 developers stacks; compared to Amazon EMR, which is listed in 95 company stacks and 18 developer stacks.

Decisions about Amazon EMR and AWS Lambda

When adding a new feature to Checkly rearchitecting some older piece, I tend to pick Heroku for rolling it out. But not always, because sometimes I pick AWS Lambda . The short story:

  • Developer Experience trumps everything.
  • AWS Lambda is cheap. Up to a limit though. This impact not only your wallet.
  • If you need geographic spread, AWS is lonely at the top.
The setup

Recently, I was doing a brainstorm at a startup here in Berlin on the future of their infrastructure. They were ready to move on from their initial, almost 100% Ec2 + Chef based setup. Everything was on the table. But we crossed out a lot quite quickly:

  • Pure, uncut, self hosted Kubernetes — way too much complexity
  • Managed Kubernetes in various flavors — still too much complexity
  • Zeit — Maybe, but no Docker support
  • Elastic Beanstalk — Maybe, bit old but does the job
  • Heroku
  • Lambda

It became clear a mix of PaaS and FaaS was the way to go. What a surprise! That is exactly what I use for Checkly! But when do you pick which model?

I chopped that question up into the following categories:

  • Developer Experience / DX 🤓
  • Ops Experience / OX 🐂 (?)
  • Cost 💵
  • Lock in 🔐

Read the full post linked below for all details

See more
Get Advice from developers at your company using Private StackShare. Sign up for Private StackShare.
Learn More
Pros of Amazon EMR
Pros of AWS Lambda
  • 15
    On demand processing power
  • 12
    Don't need to maintain Hadoop Cluster yourself
  • 7
    Hadoop Tools
  • 6
    Elastic
  • 4
    Backed by Amazon
  • 3
    Flexible
  • 3
    Economic - pay as you go, easy to use CLI and SDKs
  • 2
    Don't need a dedicated Ops group
  • 1
    Great support
  • 1
    Massive data handling
  • 125
    No infrastructure
  • 80
    Cheap
  • 67
    Quick
  • 56
    Stateless
  • 46
    No deploy, no server, great sleep
  • 9
    AWS Lambda went down taking many sites with it
  • 5
    Extensive API
  • 5
    Event Driven Governance
  • 5
    Easy to deploy
  • 4
    Auto scale and cost effective
  • 3
    VPC Support
  • 1
    Integrated with various AWS services

Sign up to add or upvote prosMake informed product decisions

Cons of Amazon EMR
Cons of AWS Lambda
    Be the first to leave a con
    • 5
      Cant execute ruby or go
    • 0
      Can't execute PHP w/o significant effort

    Sign up to add or upvote consMake informed product decisions

    What is Amazon EMR?

    It is used in a variety of applications, including log analysis, data warehousing, machine learning, financial analysis, scientific simulation, and bioinformatics.

    What is AWS Lambda?

    AWS Lambda is a compute service that runs your code in response to events and automatically manages the underlying compute resources for you. You can use AWS Lambda to extend other AWS services with custom logic, or create your own back-end services that operate at AWS scale, performance, and security.

    Need advice about which tool to choose?Ask the StackShare community!

    What companies use Amazon EMR?
    What companies use AWS Lambda?
    See which teams inside your own company are using Amazon EMR or AWS Lambda.
    Sign up for Private StackShareLearn More

    Sign up to get full access to all the companiesMake informed product decisions

    What tools integrate with Amazon EMR?
    What tools integrate with AWS Lambda?

    Sign up to get full access to all the tool integrationsMake informed product decisions

    Blog Posts

    Aug 28 2019 at 3:10AM

    Segment

    +16
    5
    1950
    +47
    45
    68267
    +23
    12
    6315
    +42
    52
    19460
    +44
    109
    49781
    What are some alternatives to Amazon EMR and AWS Lambda?
    Amazon EC2
    It is a web service that provides resizable compute capacity in the cloud. It is designed to make web-scale computing easier for developers.
    Hadoop
    The Apache Hadoop software library is a framework that allows for the distributed processing of large data sets across clusters of computers using simple programming models. It is designed to scale up from single servers to thousands of machines, each offering local computation and storage.
    Amazon DynamoDB
    With it , you can offload the administrative burden of operating and scaling a highly available distributed database cluster, while paying a low price for only what you use.
    Amazon Redshift
    It is optimized for data sets ranging from a few hundred gigabytes to a petabyte or more and costs less than $1,000 per terabyte per year, a tenth the cost of most traditional data warehousing solutions.
    Azure HDInsight
    It is a cloud-based service from Microsoft for big data analytics that helps organizations process large amounts of streaming or historical data.
    See all alternatives
    Reviews of Amazon EMR and AWS Lambda
    Review of
    AWS Lambda

    I switched my auto chatbot to run in lambda and it was peace !

    How developers use Amazon EMR and AWS Lambda
    Nathan Heffley uses
    AWS Lambda

    To use Pusher's presence channel each client must be connected through a backend authentication system. While Pointer doesn't actually have any login based authentication it still needed a backend system to connect users to the proper channel.

    A small function was built that only gets called when a user first joins a session. After the user is authenticated they can communicate directly with other clients on the same channel. This made the authentication code the perfect candidate for a serverless function. Using AWS Lambda through Netlify's Functions feature made it a breeze to host.

    Simple Merchant uses
    AWS Lambda

    We're moving almost the entirety of our backend processes into Lambda. This has given us vast cost savings in terms of pure infrastructure billing - and time worrying about platform and scale. This move has also made our architecture almost entirely event-driven - another huge benefit as our business itself is inherently event-driven.

    Volkan Özçelik uses
    AWS Lambda

    I mostly use AWS Lambda for triggering DevOps-related actions, like triggering an alarm or a deployment, or scheduling a backup.

    I haven’t gone totally “serverless” and I’m not planning to go 100% serverless anytime soon.

    But when I do, AWS Lambda will be an important element in my serverless setup.

    Promethean TV uses
    AWS Lambda

    PrometheanTV uses various Lambda functions to provide back-end capabilities to the platform without the need of deploying servers. Examples include, geo lookup services, and data aggregation services.

    Flux Work uses
    AWS Lambda

    Serverless is the future. And AWS Lambda is the most mature FaaS out there. AWS SAM makes it easy to package Lambda as micro-apps.

    Andrew La Grange uses
    Amazon EMR

    We use Amazon EMR for all our Hadoop workloads.