Need advice about which tool to choose?Ask the StackShare community!

AWS Lambda

14.8K
11K
+ 1
411
Kafka

13.6K
12.7K
+ 1
556
Add tool

AWS Lambda vs Kafka: What are the differences?

Developers describe AWS Lambda as "Automatically run code in response to modifications to objects in Amazon S3 buckets, messages in Kinesis streams, or updates in DynamoDB". AWS Lambda is a compute service that runs your code in response to events and automatically manages the underlying compute resources for you. You can use AWS Lambda to extend other AWS services with custom logic, or create your own back-end services that operate at AWS scale, performance, and security. On the other hand, Kafka is detailed as "Distributed, fault tolerant, high throughput pub-sub messaging system". Kafka is a distributed, partitioned, replicated commit log service. It provides the functionality of a messaging system, but with a unique design.

AWS Lambda and Kafka are primarily classified as "Serverless / Task Processing" and "Message Queue" tools respectively.

Some of the features offered by AWS Lambda are:

  • Extend other AWS services with custom logic
  • Build custom back-end services
  • Completely Automated Administration

On the other hand, Kafka provides the following key features:

  • Written at LinkedIn in Scala
  • Used by LinkedIn to offload processing of all page and other views
  • Defaults to using persistence, uses OS disk cache for hot data (has higher throughput then any of the above having persistence enabled)

"No infrastructure", "Cheap" and "Quick" are the key factors why developers consider AWS Lambda; whereas "High-throughput", "Distributed" and "Scalable" are the primary reasons why Kafka is favored.

Kafka is an open source tool with 12.5K GitHub stars and 6.7K GitHub forks. Here's a link to Kafka's open source repository on GitHub.

According to the StackShare community, AWS Lambda has a broader approval, being mentioned in 1002 company stacks & 585 developers stacks; compared to Kafka, which is listed in 501 company stacks and 451 developer stacks.

Advice on AWS Lambda and Kafka

Need advice on what platform, systems and tools to use.

Evaluating whether to start a new digital business for which we will need to build a website that handles all traffic. Website only right now. May add smartphone apps later. No desktop app will ever be added. Website to serve various countries and languages. B2B and B2C type customers. Need to handle heavy traffic, be low cost, and scale well.

We are open to either build it on AWS or on Microsoft Azure.

Apologies if I'm leaving out some info. My first post. :) Thanks in advance!

See more
Replies (3)
Anis Zehani

I recommend this : -Spring reactive for back end : the fact it's reactive (async) it consumes half of the resources that a sync platform needs (so less CPU -> less money). -Angular : Web Front end ; it's gives you the possibility to use PWA which is a cheap replacement for a mobile app (but more less popular). -Docker images. -Kubernetes to orchestrate all the containers. -I Use Jenkins / blueocean, ansible for my CI/CD (with Github of course) -AWS of course : u can run a K8S cluster there, make it multi AZ (availability zones) to be highly available, use a load balancer and an auto scaler and ur good to go. -You can store data by taking any managed DB or u can deploy ur own (cheap but risky).

You pay less money, but u need some technical 2 - 3 guys to make that done.

Good luck

See more

My advice will be Front end: React Backend: Language: Java, Kotlin. Database: SQL: Postgres, MySQL, Aurora NOSQL: Mongo db. Caching: Redis. Public : Spring Webflux for async public facing operation. Admin api: Spring boot, Hibrernate, Rest API. Build Container image. Kuberenetes: AWS EKS, AWS ECS, Google GKE. Use Jenkins for CI/CD pipeline. Buddy works is good for AWS. Static content: Host on AWS S3 bucket, Use Cloudfront or Cloudflare as CDN.

Serverless Solution: Api gateway Lambda, Serveless Aurora (SQL). AWS S3 bucket.

See more
Andrew Boehm
Lead Developer at Elevated · | 2 upvotes · 11.4K views
Recommends

Any relation to Justin Gaethje? He's a friend of mine! UFC champion!

See more
View all (3)
Needs advice
on
Redis
RabbitMQ
and
Kafka

We are going to develop a microservices-based application. It consists of AngularJS, ASP.NET Core, and MSSQL.

We have 3 types of microservices. Emailservice, Filemanagementservice, Filevalidationservice

I am a beginner in microservices. But I have read about RabbitMQ, but come to know that there are Redis and Kafka also in the market. So, I want to know which is best.

See more
Replies (4)
Maheedhar Aluri
Recommends
Kafka

Kafka is an Enterprise Messaging Framework whereas Redis is an Enterprise Cache Broker, in-memory database and high performance database.Both are having their own advantages, but they are different in usage and implementation. Now if you are creating microservices check the user consumption volumes, its generating logs, scalability, systems to be integrated and so on. I feel for your scenario initially you can go with KAFKA bu as the throughput, consumption and other factors are scaling then gradually you can add Redis accordingly.

See more
Recommends
Angular 2

I first recommend that you choose Angular over AngularJS if you are starting something new. AngularJs is no longer getting enhancements, but perhaps you meant Angular. Regarding microservices, I recommend considering microservices when you have different development teams for each service that may want to use different programming languages and backend data stores. If it is all the same team, same code language, and same data store I would not use microservices. I might use a message queue, in which case RabbitMQ is a good one. But you may also be able to simply write your own in which you write a record in a table in MSSQL and one of your services reads the record from the table and processes it. The most challenging part of doing it yourself is writing a service that does a good job of reading the queue without reading the same message multiple times or missing a message; and that is where RabbitMQ can help.

See more
Recommends
NATS

We found that the CNCF landscape is a good advisor when working going into the cloud / microservices space: https://landscape.cncf.io/fullscreen=yes. When choosing a technology one important criteria to me is if it is cloud native or not. Neither Redis, RabbitMQ nor Kafka is cloud native. The try to adapt but will be replaced eventually with technologies that are cloud native.

We have gone with NATS and have never looked back. We haven't spend a single minute on server maintainance in the last year and the setup of a cluster is way too easy. With the new features NATS incorporates now (and the ones still on the roadmap) it is already and will be sooo much mure than Redis, RabbitMQ and Kafka are. It can replace service discovery, load balancing, global multiclusters and failover, etc, etc.

Your thought might be: But I don't need all of that! Well, at the same time it is much more leightweight than Redis, RabbitMQ and especially Kafka.

See more
Amit Mor
Software Architect at Payoneer · | 3 upvotes · 176K views
Recommends
Kafka

I think something is missing here and you should consider answering it to yourself. You are building a couple of services. Why are you considering event-sourcing architecture using Message Brokers such as the above? Won't a simple REST service based arch suffice? Read about CQRS and the problems it entails (state vs command impedance for example). Do you need Pub/Sub or Push/Pull? Is queuing of messages enough or would you need querying or filtering of messages before consumption? Also, someone would have to manage these brokers (unless using managed, cloud provider based solution), automate their deployment, someone would need to take care of backups, clustering if needed, disaster recovery, etc. I have a good past experience in terms of manageability/devops of the above options with Kafka and Redis, not so much with RabbitMQ. Both are very performant. But also note that Redis is not a pure message broker (at time of writing) but more of a general purpose in-memory key-value store. Kafka nowadays is much more than a distributed message broker. Long story short. In my taste, you should go with a minialistic approach and try to avoid either of them if you can, especially if your architecture does not fall nicely into event sourcing. If not I'd examine Kafka. If you need more capabilities than I'd consider Redis and use it for all sorts of other things such as a cache.

See more
View all (4)
Pramod Nikam
Co Founder at Usability Designs · | 2 upvotes · 118.9K views
Needs advice
on
NSQ
Kafka
and
Apache Thrift

I am looking into IoT World Solution where we have MQTT Broker. This MQTT Broker Sits in one of the Data Center. We are doing a lot of Alert and Alarm related processing on that Data, Currently, we are looking into Solution which can do distributed persistence of log/alert primarily on remote Disk.

Our primary need is to use lightweight where operational complexity and maintenance costs can be significantly reduced. We want to do it on-premise so we are not considering cloud solutions.

We looked into the following alternatives:

Apache Kafka - Great choice but operation and maintenance wise very complex. Rabbit MQ - High availability is the issue, Apache Pulsar - Operational Complexity. NATS - Absence of persistence. Akka Streams - Big learning curve and operational streams.

So we are looking into a lightweight library that can do distributed persistence preferably with publisher and subscriber model. Preferable on JVM stack.

See more
Replies (1)
Naresh Kancharla
Staff Engineer at Nutanix · | 4 upvotes · 116.1K views
Recommends
Kafka

Kafka is best fit here. Below are the advantages with Kafka ACLs (Security), Schema (protobuf), Scale, Consumer driven and No single point of failure.

Operational complexity is manageable with open source monitoring tools.

See more
Decisions about AWS Lambda and Kafka

When adding a new feature to Checkly rearchitecting some older piece, I tend to pick Heroku for rolling it out. But not always, because sometimes I pick AWS Lambda . The short story:

  • Developer Experience trumps everything.
  • AWS Lambda is cheap. Up to a limit though. This impact not only your wallet.
  • If you need geographic spread, AWS is lonely at the top.
The setup

Recently, I was doing a brainstorm at a startup here in Berlin on the future of their infrastructure. They were ready to move on from their initial, almost 100% Ec2 + Chef based setup. Everything was on the table. But we crossed out a lot quite quickly:

  • Pure, uncut, self hosted Kubernetes — way too much complexity
  • Managed Kubernetes in various flavors — still too much complexity
  • Zeit — Maybe, but no Docker support
  • Elastic Beanstalk — Maybe, bit old but does the job
  • Heroku
  • Lambda

It became clear a mix of PaaS and FaaS was the way to go. What a surprise! That is exactly what I use for Checkly! But when do you pick which model?

I chopped that question up into the following categories:

  • Developer Experience / DX 🤓
  • Ops Experience / OX 🐂 (?)
  • Cost 💵
  • Lock in 🔐

Read the full post linked below for all details

See more
Get Advice from developers at your company using Private StackShare. Sign up for Private StackShare.
Learn More
Pros of AWS Lambda
Pros of Kafka
  • 126
    No infrastructure
  • 81
    Cheap
  • 68
    Quick
  • 57
    Stateless
  • 47
    No deploy, no server, great sleep
  • 9
    AWS Lambda went down taking many sites with it
  • 5
    Easy to deploy
  • 5
    Event Driven Governance
  • 5
    Extensive API
  • 4
    Auto scale and cost effective
  • 3
    VPC Support
  • 1
    Integrated with various AWS services
  • 118
    High-throughput
  • 113
    Distributed
  • 85
    Scalable
  • 78
    High-Performance
  • 64
    Durable
  • 35
    Publish-Subscribe
  • 17
    Simple-to-use
  • 14
    Open source
  • 10
    Written in Scala and java. Runs on JVM
  • 6
    Message broker + Streaming system
  • 4
    Avro schema integration
  • 2
    Suport Multiple clients
  • 2
    Robust
  • 2
    KSQL
  • 2
    Partioned, replayable log
  • 1
    Fun
  • 1
    Extremely good parallelism constructs
  • 1
    Simple publisher / multi-subscriber model
  • 1
    Flexible

Sign up to add or upvote prosMake informed product decisions

Cons of AWS Lambda
Cons of Kafka
  • 5
    Cant execute ruby or go
  • 0
    Can't execute PHP w/o significant effort
  • 27
    Non-Java clients are second-class citizens
  • 26
    Needs Zookeeper
  • 7
    Operational difficulties
  • 2
    Terrible Packaging

Sign up to add or upvote consMake informed product decisions

What companies use AWS Lambda?
What companies use Kafka?
See which teams inside your own company are using AWS Lambda or Kafka.
Sign up for Private StackShareLearn More

Sign up to get full access to all the companiesMake informed product decisions

What tools integrate with AWS Lambda?
What tools integrate with Kafka?

Sign up to get full access to all the tool integrationsMake informed product decisions

What are some alternatives to AWS Lambda and Kafka?
Serverless
Build applications comprised of microservices that run in response to events, auto-scale for you, and only charge you when they run. This lowers the total cost of maintaining your apps, enabling you to build more logic, faster. The Framework uses new event-driven compute services, like AWS Lambda, Google CloudFunctions, and more.
Azure Functions
Azure Functions is an event driven, compute-on-demand experience that extends the existing Azure application platform with capabilities to implement code triggered by events occurring in virtually any Azure or 3rd party service as well as on-premises systems.
AWS Elastic Beanstalk
Once you upload your application, Elastic Beanstalk automatically handles the deployment details of capacity provisioning, load balancing, auto-scaling, and application health monitoring.
AWS Step Functions
AWS Step Functions makes it easy to coordinate the components of distributed applications and microservices using visual workflows. Building applications from individual components that each perform a discrete function lets you scale and change applications quickly.
Google App Engine
Google has a reputation for highly reliable, high performance infrastructure. With App Engine you can take advantage of the 10 years of knowledge Google has in running massively scalable, performance driven systems. App Engine applications are easy to build, easy to maintain, and easy to scale as your traffic and data storage needs grow.
See all alternatives