Amazon Kinesis logo
Store and process terabytes of data each hour from hundreds of thousands of sources
288
141
0

What is Amazon Kinesis?

Amazon Kinesis can collect and process hundreds of gigabytes of data per second from hundreds of thousands of sources, allowing you to easily write applications that process information in real-time, from sources such as web site click-streams, marketing and financial information, manufacturing instrumentation and social media, and operational logs and metering data.
Amazon Kinesis is a tool in the Real-time Data Processing category of a tech stack.

Who uses Amazon Kinesis?

Companies
164 companies reportedly use Amazon Kinesis in their tech stacks, including Instacart, Lyft, and Zillow.

Developers
107 developers on StackShare have stated that they use Amazon Kinesis.

Amazon Kinesis Integrations

Snowplow, LocalStack, Thundra, Astronomer, and Amazon Kinesis Video Streams are some of the popular tools that integrate with Amazon Kinesis. Here's a list of all 8 tools that integrate with Amazon Kinesis.

Why developers like Amazon Kinesis?

Here’s a list of reasons why companies and developers use Amazon Kinesis
Top Reasons
Be the first to leave a pro
Amazon Kinesis Reviews

Here are some stack decisions, common use cases and reviews by companies and developers who chose Amazon Kinesis in their tech stack.

John Kodumal
John Kodumal
CTO at LaunchDarkly · | 14 upvotes · 51.8K views
atLaunchDarklyLaunchDarkly
Kafka
Amazon Kinesis
Redis
Amazon EC2
Amazon ElastiCache
Consul
Patroni
TimescaleDB
PostgreSQL
Amazon RDS

As we've evolved or added additional infrastructure to our stack, we've biased towards managed services. Most new backing stores are Amazon RDS instances now. We do use self-managed PostgreSQL with TimescaleDB for time-series data—this is made HA with the use of Patroni and Consul.

We also use managed Amazon ElastiCache instances instead of spinning up Amazon EC2 instances to run Redis workloads, as well as shifting to Amazon Kinesis instead of Kafka.

See more
Tim Specht
Tim Specht
‎Co-Founder and CTO at Dubsmash · | 14 upvotes · 16.5K views
atDubsmashDubsmash
Google BigQuery
Amazon SQS
AWS Lambda
Amazon Kinesis
Google Analytics
#BigDataAsAService
#RealTimeDataProcessing
#GeneralAnalytics
#ServerlessTaskProcessing

In order to accurately measure & track user behaviour on our platform we moved over quickly from the initial solution using Google Analytics to a custom-built one due to resource & pricing concerns we had.

While this does sound complicated, it’s as easy as clients sending JSON blobs of events to Amazon Kinesis from where we use AWS Lambda & Amazon SQS to batch and process incoming events and then ingest them into Google BigQuery. Once events are stored in BigQuery (which usually only takes a second from the time the client sends the data until it’s available), we can use almost-standard-SQL to simply query for data while Google makes sure that, even with terabytes of data being scanned, query times stay in the range of seconds rather than hours. Before ingesting their data into the pipeline, our mobile clients are aggregating events internally and, once a certain threshold is reached or the app is going to the background, sending the events as a JSON blob into the stream.

In the past we had workers running that continuously read from the stream and would validate and post-process the data and then enqueue them for other workers to write them to BigQuery. We went ahead and implemented the Lambda-based approach in such a way that Lambda functions would automatically be triggered for incoming records, pre-aggregate events, and write them back to SQS, from which we then read them, and persist the events to BigQuery. While this approach had a couple of bumps on the road, like re-triggering functions asynchronously to keep up with the stream and proper batch sizes, we finally managed to get it running in a reliable way and are very happy with this solution today.

#ServerlessTaskProcessing #GeneralAnalytics #RealTimeDataProcessing #BigDataAsAService

See more
Praveen Mooli
Praveen Mooli
Technical Leader at Taylor and Francis · | 8 upvotes · 12.8K views
RxJS
Angular 2
AWS Lambda
Amazon SQS
Amazon SNS
Amazon Kinesis Firehose
Amazon Kinesis
Flask
Python
ExpressJS
Node.js
Spring Boot
Java
#Webapps
#Eventsourcingframework
#Microservices
#Backend

We are in the process of building a modern content platform to deliver our content through various channels. We decided to go with Microservices architecture as we wanted scale. Microservice architecture style is an approach to developing an application as a suite of small independently deployable services built around specific business capabilities. You can gain modularity, extensive parallelism and cost-effective scaling by deploying services across many distributed servers. Microservices modularity facilitates independent updates/deployments, and helps to avoid single point of failure, which can help prevent large-scale outages. We also decided to use Event Driven Architecture pattern which is a popular distributed asynchronous architecture pattern used to produce highly scalable applications. The event-driven architecture is made up of highly decoupled, single-purpose event processing components that asynchronously receive and process events.

To build our #Backend capabilities we decided to use the following: 1. #Microservices - Java with Spring Boot , Node.js with ExpressJS and Python with Flask 2. #Eventsourcingframework - Amazon Kinesis , Amazon Kinesis Firehose , Amazon SNS , Amazon SQS, AWS Lambda

To build #Webapps we decided to use Angular 2 with RxJS

See more
Mihai Ciureanu
Mihai Ciureanu
Head of Technology at Wecheer · | 1 upvotes · 3.3K views
atWecheerWecheer
Amazon Kinesis

We use Amazon Kinesis because as an IoT business we want to decouple the invocations from the devices from the following complex processing, which involves image classification, fraud detection, analytics processing, and others. We also decouple our core systems from our supporting systems using a publish-subscribe mechanism based on Kinesis streams.

See more

Amazon Kinesis's Features

  • Real-time Processing- Amazon Kinesis enables you to collect and analyze information in real-time, allowing you to answer questions about the current state of your data, from inventory levels to stock trade frequencies, rather than having to wait for an out-of-date report.
  • Easy to use- You can create a new stream, set the throughput requirements, and start streaming data quickly and easily. Amazon Kinesis automatically provisions and manages the storage required to reliably and durably collect your data stream.
  • High throughput. Elastic.- Amazon Kinesis seamlessly scales to match the data throughput rate and volume of your data, from megabytes to terabytes per hour. Amazon Kinesis will scale up or down based on your needs.
  • Integrate with Amazon S3, Amazon Redshift, and Amazon DynamoDB- With Amazon Kinesis, you can reliably collect, process, and transform all of your data in real-time before delivering it to data stores of your choice, where it can be used by existing or new applications. Connectors enable integration with Amazon S3, Amazon Redshift, and Amazon DynamoDB.
  • Build Kinesis Applications- Amazon Kinesis provides developers with client libraries that enable the design and operation of real-time data processing applications. Just add the Amazon Kinesis Client Library to your Java application and it will be notified when new data is available for processing.
  • Low Cost- Amazon Kinesis is cost-efficient for workloads of any scale. You can pay as you go, and you’ll only pay for the resources you use. You can get started by provisioning low throughput streams, and only pay a low hourly rate for the throughput you need.

Amazon Kinesis Alternatives & Comparisons

What are some alternatives to Amazon Kinesis?
Kafka
Kafka is a distributed, partitioned, replicated commit log service. It provides the functionality of a messaging system, but with a unique design.
Apache Spark
Spark is a fast and general processing engine compatible with Hadoop data. It can run in Hadoop clusters through YARN or Spark's standalone mode, and it can process data in HDFS, HBase, Cassandra, Hive, and any Hadoop InputFormat. It is designed to perform both batch processing (similar to MapReduce) and new workloads like streaming, interactive queries, and machine learning.
Amazon SQS
Transmit any volume of data, at any level of throughput, without losing messages or requiring other services to be always available. With SQS, you can offload the administrative burden of operating and scaling a highly available messaging cluster, while paying a low price for only what you use.
Firehose.io
Firehose is both a Rack application and JavaScript library that makes building real-time web applications possible.
Amazon Kinesis Firehose
Amazon Kinesis Firehose is the easiest way to load streaming data into AWS. It can capture and automatically load streaming data into Amazon S3 and Amazon Redshift, enabling near real-time analytics with existing business intelligence tools and dashboards you’re already using today.
See all alternatives

Amazon Kinesis's Followers
141 developers follow Amazon Kinesis to keep up with related blogs and decisions.
Arun Kumar
Vitaliy Dymna
Nishu Goyal
Kumar Raghav
Shay Ben-Haroche
Ashish Tanwer
Mohamma76685757
Muhammet Dilek
Naresh Kancharla
Dien Phan