Need advice about which tool to choose?Ask the StackShare community!
AWS Lambda vs Amazon S3 vs Amazon SQS: What are the differences?
Introduction
In this comparison, we will highlight the key differences between AWS Lambda, Amazon S3, and Amazon SQS.
Functionality: AWS Lambda is a serverless computing service that allows you to run code without provisioning or managing servers. Amazon S3 is a scalable object storage service that allows you to store and retrieve data. Amazon SQS is a fully managed message queuing service that enables you to decouple and scale microservices, distributed systems, and serverless applications.
Use Cases: AWS Lambda is often used for handling event-driven tasks, data processing, real-time file processing, and orchestrating workflows. Amazon S3 is ideal for storing large amounts of data, hosting static websites, securing data backups, and enabling content delivery. Amazon SQS is commonly utilized for implementing distributed systems, decoupling application components, processing tasks asynchronously, and scaling microservices.
Execution: AWS Lambda executes code in response to triggers such as changes in data, HTTP requests, or time-based schedules. Amazon S3 acts as a central repository for data storage and retrieval. Amazon SQS facilitates communication between different components by queuing and processing messages asynchronously.
Billing: AWS Lambda is billed based on the number of requests and the duration of code execution. Amazon S3 charges based on the amount of data stored, data transfer, and requests made. Amazon SQS pricing is based on the number of messages processed and the data volume transferred.
Scalability: AWS Lambda automatically scales based on incoming requests, ensuring high availability and performance. Amazon S3 can handle massive amounts of data and traffic without manual intervention. Amazon SQS is designed to scale horizontally to accommodate varying workloads efficiently.
Integration: AWS Lambda can be integrated with various AWS services, third-party APIs, and custom applications. Amazon S3 seamlessly integrates with AWS services, content delivery networks, and data analytics platforms. Amazon SQS supports integration with different AWS services, message processing systems, and microservices architecture.
In Summary, the key differences between AWS Lambda, Amazon S3, and Amazon SQS lie in their functionality, use cases, execution methods, billing structures, scalability features, and integration capabilities.
I want to schedule a message. Amazon SQS provides a delay of 15 minutes, but I want it in some hours.
Example: Let's say a Message1 is consumed by a consumer A but somehow it failed inside the consumer. I would want to put it in a queue and retry after 4hrs. Can I do this in Amazon MQ? I have seen in some Amazon MQ videos saying scheduling messages can be done. But, I'm not sure how.
Mithiridi, I believe you are talking about two different things. 1. If you need to process messages with delays of more 15m or at specific times, it's not a good idea to use queues, independently of tool SQM, Rabbit or Amazon MQ. you should considerer another approach using a scheduled job. 2. For dead queues and policy retries RabbitMQ, for example, doesn't support your use case. https://medium.com/@kiennguyen88/rabbitmq-delay-retry-schedule-with-dead-letter-exchange-31fb25a440fc I'm not sure if that is possible SNS/SQS support, they have a maximum delay for delivery (maxDelayTarget) in seconds but it's not clear the number. You can check this out: https://docs.aws.amazon.com/sns/latest/dg/sns-message-delivery-retries.html
We offer our customer HIPAA compliant storage. After analyzing the market, we decided to go with Google Storage. The Nodejs API is ok, still not ES6 and can be very confusing to use. For each new customer, we created a different bucket so they can have individual data and not have to worry about data loss. After 1000+ customers we started seeing many problems with the creation of new buckets, with saving or retrieving a new file. Many false positive: the Promise returned ok, but in reality, it failed.
That's why we switched to S3 that just works.
At Joyn we use AWS infrastructures extensively, so we need a wide range of different tools starting from file storage until the health monitoring of our running services. For Smart TV stack we use S3 to mostly store static files (like scripts, styles and html), CloudFront to keep good response times using the cache, Lambda to recognize the TV platform and serve appropriate bundle (except Samsung Tizen platform where application is packaged), CloudWatch to receive alerts in case we have any issues inside of our lambda function.
When adding a new feature to Checkly rearchitecting some older piece, I tend to pick Heroku for rolling it out. But not always, because sometimes I pick AWS Lambda . The short story:
- Developer Experience trumps everything.
- AWS Lambda is cheap. Up to a limit though. This impact not only your wallet.
- If you need geographic spread, AWS is lonely at the top.
Recently, I was doing a brainstorm at a startup here in Berlin on the future of their infrastructure. They were ready to move on from their initial, almost 100% Ec2 + Chef based setup. Everything was on the table. But we crossed out a lot quite quickly:
- Pure, uncut, self hosted Kubernetes — way too much complexity
- Managed Kubernetes in various flavors — still too much complexity
- Zeit — Maybe, but no Docker support
- Elastic Beanstalk — Maybe, bit old but does the job
- Heroku
- Lambda
It became clear a mix of PaaS and FaaS was the way to go. What a surprise! That is exactly what I use for Checkly! But when do you pick which model?
I chopped that question up into the following categories:
- Developer Experience / DX 🤓
- Ops Experience / OX 🐂 (?)
- Cost 💵
- Lock in 🔐
Read the full post linked below for all details
Pros of Amazon S3
- Reliable590
- Scalable492
- Cheap456
- Simple & easy329
- Many sdks83
- Logical30
- Easy Setup13
- REST API11
- 1000+ POPs11
- Secure6
- Easy4
- Plug and play4
- Web UI for uploading files3
- Faster on response2
- Flexible2
- GDPR ready2
- Easy to use1
- Plug-gable1
- Easy integration with CloudFront1
Pros of Amazon SQS
- Easy to use, reliable62
- Low cost40
- Simple28
- Doesn't need to maintain it14
- It is Serverless8
- Has a max message size (currently 256K)4
- Triggers Lambda3
- Easy to configure with Terraform3
- Delayed delivery upto 15 mins only3
- Delayed delivery upto 12 hours3
- JMS compliant1
- Support for retry and dead letter queue1
- D1
Pros of AWS Lambda
- No infrastructure129
- Cheap83
- Quick70
- Stateless59
- No deploy, no server, great sleep47
- AWS Lambda went down taking many sites with it12
- Event Driven Governance6
- Extensive API6
- Auto scale and cost effective6
- Easy to deploy6
- VPC Support5
- Integrated with various AWS services3
Sign up to add or upvote prosMake informed product decisions
Cons of Amazon S3
- Permissions take some time to get right7
- Requires a credit card6
- Takes time/work to organize buckets & folders properly6
- Complex to set up3
Cons of Amazon SQS
- Has a max message size (currently 256K)2
- Proprietary2
- Difficult to configure2
- Has a maximum 15 minutes of delayed messages only1
Cons of AWS Lambda
- Cant execute ruby or go7
- Compute time limited3
- Can't execute PHP w/o significant effort1
Sign up to add or upvote consMake informed product decisions
What is Amazon S3?
What is Amazon SQS?
What is AWS Lambda?
Need advice about which tool to choose?Ask the StackShare community!
Sign up to get full access to all the companiesMake informed product decisions
What tools integrate with Amazon S3?
What tools integrate with Amazon SQS?
What tools integrate with AWS Lambda?
Sign up to get full access to all the tool integrationsMake informed product decisions
Blog Posts
+4
+5
+5
+7
+6
+4
+4
+17