Need advice about which tool to choose?Ask the StackShare community!

Amazon Athena

512
829
+ 1
49
Kafka

23.3K
21.6K
+ 1
607
Add tool

Amazon Athena vs Kafka: What are the differences?

Key Differences Between Amazon Athena and Kafka

Amazon Athena and Kafka are both widely used technologies in the world of data processing and analytics, but they serve different purposes and have distinct features. Below are six key differences between the two platforms:

  1. Data Processing Model: Amazon Athena is a serverless interactive query service that allows users to analyze data stored in Amazon S3 using standard SQL. It is designed for ad-hoc querying and analysis of data, without the need for infrastructure management. On the other hand, Kafka is a distributed streaming platform that allows for the high-throughput, fault-tolerant publish-subscribe messaging system, enabling real-time data streaming and processing.

  2. Data Source: Amazon Athena is primarily used for querying and analyzing data stored in Amazon S3. It provides the ability to run queries directly against the data without the need to load it into a separate database or data warehouse. In contrast, Kafka acts as a data pipeline and broker that enables the streaming of data from various sources to different applications or systems for processing and consumption.

  3. Real-time vs Batch Processing: While both platforms can handle large volumes of data, Kafka is designed for real-time processing and streaming of data streams, making it ideal for use cases that require low-latency data consumption and real-time analytics. On the other hand, Athena is more suited for batch processing and ad-hoc querying, where the focus is on running SQL queries on stored data in S3 rather than real-time processing.

  4. Scalability and Elasticity: Amazon Athena is a fully managed service that automatically scales up or down the resources required to execute queries based on the amount of data being processed. It offers horizontal scalability and can handle large data sets efficiently. Kafka, on the other hand, can also scale horizontally, allowing users to add more brokers to the cluster to handle increased data throughput. It offers high scalability, fault tolerance, and the ability to handle terabytes of data per day.

  5. Data Format and Schema: Amazon Athena supports various data formats such as CSV, Parquet, JSON, and ORC, and provides the ability to define explicit schemas for structured data. It leverages AWS Glue and the Hive metastore for data cataloging and schema inference. In contrast, Kafka does not enforce any specific data format, allowing users to stream both structured and unstructured data. It provides flexibility in data serialization and deserialization.

  6. Integration and Ecosystem: Amazon Athena integrates seamlessly with other AWS services, such as AWS Glue for data cataloging, AWS Lambda for serverless data transformations, and Amazon QuickSight for data visualization. It benefits from the broader AWS ecosystem and offers tight integration with other AWS analytics tools. Kafka, on the other hand, has a wide ecosystem of connectors and integrations with various data processing frameworks and messaging systems, making it a versatile choice for building real-time data pipelines.

In summary, Amazon Athena is a serverless interactive query service designed for ad-hoc querying and analysis of data stored in S3, while Kafka is a distributed streaming platform designed for real-time data streaming and processing. Athena is focused on batch processing, while Kafka excels in real-time processing and supports various data formats. Athena benefits from tight integration with the AWS ecosystem, while Kafka has a vast ecosystem of connectors and integrations.

Advice on Amazon Athena and Kafka
Needs advice
on
KafkaKafkaRabbitMQRabbitMQ
and
RedisRedis

We are going to develop a microservices-based application. It consists of AngularJS, ASP.NET Core, and MSSQL.

We have 3 types of microservices. Emailservice, Filemanagementservice, Filevalidationservice

I am a beginner in microservices. But I have read about RabbitMQ, but come to know that there are Redis and Kafka also in the market. So, I want to know which is best.

See more
Replies (4)
Maheedhar Aluri
Recommends
on
KafkaKafka

Kafka is an Enterprise Messaging Framework whereas Redis is an Enterprise Cache Broker, in-memory database and high performance database.Both are having their own advantages, but they are different in usage and implementation. Now if you are creating microservices check the user consumption volumes, its generating logs, scalability, systems to be integrated and so on. I feel for your scenario initially you can go with KAFKA bu as the throughput, consumption and other factors are scaling then gradually you can add Redis accordingly.

See more
Recommends
on
AngularAngular

I first recommend that you choose Angular over AngularJS if you are starting something new. AngularJs is no longer getting enhancements, but perhaps you meant Angular. Regarding microservices, I recommend considering microservices when you have different development teams for each service that may want to use different programming languages and backend data stores. If it is all the same team, same code language, and same data store I would not use microservices. I might use a message queue, in which case RabbitMQ is a good one. But you may also be able to simply write your own in which you write a record in a table in MSSQL and one of your services reads the record from the table and processes it. The most challenging part of doing it yourself is writing a service that does a good job of reading the queue without reading the same message multiple times or missing a message; and that is where RabbitMQ can help.

See more
Recommends
on
NATSNATS

We found that the CNCF landscape is a good advisor when working going into the cloud / microservices space: https://landscape.cncf.io/fullscreen=yes. When choosing a technology one important criteria to me is if it is cloud native or not. Neither Redis, RabbitMQ nor Kafka is cloud native. The try to adapt but will be replaced eventually with technologies that are cloud native.

We have gone with NATS and have never looked back. We haven't spend a single minute on server maintainance in the last year and the setup of a cluster is way too easy. With the new features NATS incorporates now (and the ones still on the roadmap) it is already and will be sooo much mure than Redis, RabbitMQ and Kafka are. It can replace service discovery, load balancing, global multiclusters and failover, etc, etc.

Your thought might be: But I don't need all of that! Well, at the same time it is much more leightweight than Redis, RabbitMQ and especially Kafka.

See more
Amit Mor
Software Architect at Payoneer · | 3 upvotes · 765.1K views
Recommends
on
KafkaKafka

I think something is missing here and you should consider answering it to yourself. You are building a couple of services. Why are you considering event-sourcing architecture using Message Brokers such as the above? Won't a simple REST service based arch suffice? Read about CQRS and the problems it entails (state vs command impedance for example). Do you need Pub/Sub or Push/Pull? Is queuing of messages enough or would you need querying or filtering of messages before consumption? Also, someone would have to manage these brokers (unless using managed, cloud provider based solution), automate their deployment, someone would need to take care of backups, clustering if needed, disaster recovery, etc. I have a good past experience in terms of manageability/devops of the above options with Kafka and Redis, not so much with RabbitMQ. Both are very performant. But also note that Redis is not a pure message broker (at time of writing) but more of a general purpose in-memory key-value store. Kafka nowadays is much more than a distributed message broker. Long story short. In my taste, you should go with a minialistic approach and try to avoid either of them if you can, especially if your architecture does not fall nicely into event sourcing. If not I'd examine Kafka. If you need more capabilities than I'd consider Redis and use it for all sorts of other things such as a cache.

See more

Hi all,

Currently, we need to ingest the data from Amazon S3 to DB either Amazon Athena or Amazon Redshift. But the problem with the data is, it is in .PSV (pipe separated values) format and the size is also above 200 GB. The query performance of the timeout in Athena/Redshift is not up to the mark, too slow while compared to Google BigQuery. How would I optimize the performance and query result time? Can anyone please help me out?

See more
Replies (4)

you can use aws glue service to convert you pipe format data to parquet format , and thus you can achieve data compression . Now you should choose Redshift to copy your data as it is very huge. To manage your data, you should partition your data in S3 bucket and also divide your data across the redshift cluster

See more
Carlos Acedo
Data Technologies Manager at SDG Group Iberia · | 5 upvotes · 232.7K views
Recommends
on
Amazon RedshiftAmazon Redshift

First of all you should make your choice upon Redshift or Athena based on your use case since they are two very diferent services - Redshift is an enterprise-grade MPP Data Warehouse while Athena is a SQL layer on top of S3 with limited performance. If performance is a key factor, users are going to execute unpredictable queries and direct and managing costs are not a problem I'd definitely go for Redshift. If performance is not so critical and queries will be predictable somewhat I'd go for Athena.

Once you select the technology you'll need to optimize your data in order to get the queries executed as fast as possible. In both cases you may need to adapt the data model to fit your queries better. In the case you go for Athena you'd also proabably need to change your file format to Parquet or Avro and review your partition strategy depending on your most frequent type of query. If you choose Redshift you'll need to ingest the data from your files into it and maybe carry out some tuning tasks for performance gain.

I'll recommend Redshift for now since it can address a wider range of use cases, but we could give you better advice if you described your use case in depth.

See more
Alexis Blandin
Recommends
on
Amazon AthenaAmazon Athena

It depend of the nature of your data (structured or not?) and of course your queries (ad-hoc or predictible?). For example you can look at partitioning and columnar format to maximize MPP capabilities for both Athena and Redshift

See more
Recommends

you can change your PSV fomat data to parquet file format with AWS GLUE and then your query performance will be improved

See more
Pramod Nikam
Co Founder at Usability Designs · | 2 upvotes · 514K views
Needs advice
on
Apache ThriftApache ThriftKafkaKafka
and
NSQNSQ

I am looking into IoT World Solution where we have MQTT Broker. This MQTT Broker Sits in one of the Data Center. We are doing a lot of Alert and Alarm related processing on that Data, Currently, we are looking into Solution which can do distributed persistence of log/alert primarily on remote Disk.

Our primary need is to use lightweight where operational complexity and maintenance costs can be significantly reduced. We want to do it on-premise so we are not considering cloud solutions.

We looked into the following alternatives:

Apache Kafka - Great choice but operation and maintenance wise very complex. Rabbit MQ - High availability is the issue, Apache Pulsar - Operational Complexity. NATS - Absence of persistence. Akka Streams - Big learning curve and operational streams.

So we are looking into a lightweight library that can do distributed persistence preferably with publisher and subscriber model. Preferable on JVM stack.

See more
Replies (1)
Naresh Kancharla
Staff Engineer at Nutanix · | 4 upvotes · 511.5K views
Recommends
on
KafkaKafka

Kafka is best fit here. Below are the advantages with Kafka ACLs (Security), Schema (protobuf), Scale, Consumer driven and No single point of failure.

Operational complexity is manageable with open source monitoring tools.

See more
Needs advice
on
KafkaKafka
and
RabbitMQRabbitMQ

Our backend application is sending some external messages to a third party application at the end of each backend (CRUD) API call (from UI) and these external messages take too much extra time (message building, processing, then sent to the third party and log success/failure), UI application has no concern to these extra third party messages.

So currently we are sending these third party messages by creating a new child thread at end of each REST API call so UI application doesn't wait for these extra third party API calls.

I want to integrate Apache Kafka for these extra third party API calls, so I can also retry on failover third party API calls in a queue(currently third party messages are sending from multiple threads at the same time which uses too much processing and resources) and logging, etc.

Question 1: Is this a use case of a message broker?

Question 2: If it is then Kafka vs RabitMQ which is the better?

See more
Replies (4)
Tarun Batra
Senior Software Developer at Okta · | 7 upvotes · 715.6K views
Recommends
on
RabbitMQRabbitMQ

RabbitMQ is great for queuing and retrying. You can send the requests to your backend which will further queue these requests in RabbitMQ (or Kafka, too). The consumer on the other end can take care of processing . For a detailed analysis, check this blog about choosing between Kafka and RabbitMQ.

See more
Trevor Rydalch
Software Engineer at InsideSales.com · | 6 upvotes · 715.4K views
Recommends
on
RabbitMQRabbitMQ

Well, first off, it's good practice to do as little non-UI work on the foreground thread as possible, regardless of whether the requests take a long time. You don't want the UI thread blocked.

This sounds like a good use case for RabbitMQ. Primarily because you don't need each message processed by more than one consumer. If you wanted to process a single message more than once (say for different purposes), then Apache Kafka would be a much better fit as you can have multiple consumer groups consuming from the same topics independently.

Have your API publish messages containing the data necessary for the third-party request to a Rabbit queue and have consumers reading off there. If it fails, you can either retry immediately, or publish to a deadletter queue where you can reprocess them whenever you want (shovel them back into the regular queue).

See more
Guillaume Maka
Full Stack Web Developer · | 2 upvotes · 714.7K views
Recommends
on
RabbitMQRabbitMQ

As far as I understand, Kafka is a like a persisted event state manager where you can plugin various source of data and transform/query them as event via a stream API. Regarding your use case I will consider using RabbitMQ if your intent is to implement service inter-communication kind of thing. RabbitMQ is a good choice for one-one publisher/subscriber (or consumer) and I think you can also have multiple consumers by configuring a fanout exchange. RabbitMQ provide also message retries, message cancellation, durable queue, message requeue, message ACK....

See more
Recommends
on
RabbitMQRabbitMQ

In my opinion RabbitMQ fits better in your case because you don’t have order in queue. You can process your messages in any order. You don’t need to store the data what you sent. Kafka is a persistent storage like the blockchain. RabbitMQ is a message broker. Kafka is not a good solution for the system with confirmations of the messages delivery.

See more
Needs advice
on
KafkaKafkaRabbitMQRabbitMQ
and
RedisRedis

Hello! [Client sends live video frames -> Server computes and responds the result] Web clients send video frames from their webcam then on the back we need to run them through some algorithm and send the result back as a response. Since everything will need to work in a live mode, we want something fast and also suitable for our case (as everyone needs). Currently, we are considering RabbitMQ for the purpose, but recently I have noticed that there is Redis and Kafka too. Could you please help us choose among them or anything more suitable beyond these guys. I think something similar to our product would be people using their webcam to get Snapchat masks on their faces, and the calculated face points are responded on from the server, then the client-side draw the mask on the user's face. I hope this helps. Thank you!

See more
Replies (3)
Jordi Martínez
Senior software architect at Bootloader · | 3 upvotes · 664.9K views
Recommends
on
KafkaKafka

For your use case, the tool that fits more is definitely Kafka. RabbitMQ was not invented to handle data streams, but messages. Plenty of them, of course, but individual messages. Redis is an in-memory database, which is what makes it so fast. Redis recently included features to handle data stream, but it cannot best Kafka on this, or at least not yet. Kafka is not also super fast, it also provides lots of features to help create software to handle those streams.

See more
Recommends
on
RabbitMQRabbitMQ

I've used all of them and Kafka is hard to set up and maintain. Mostly is a Java dinosaur that you can set up and. I've used it with Storm but that is another big dinosaur. Redis is mostly for caching. The queue mechanism is not very scalable for multiple processors. Depending on the speed you need to implement on the reliability I would use RabbitMQ. You can store the frames(if they are too big) somewhere else and just have a link to them. Moving data through any of these will increase cost of transportation. With Rabbit, you can always have multiple consumers and check for redundancy. Hope it clears out your thoughts!

See more
Recommends
on
RabbitMQRabbitMQ

For this kind of use case I would recommend either RabbitMQ or Kafka depending on the needs for scaling, redundancy and how you want to design it.

Kafka's true value comes into play when you need to distribute the streaming load over lot's of resources. If you were passing the video frames directly into the queue then you'd probably want to go with Kafka however if you can just pass a pointer to the frames then RabbitMQ should be fine and will be much simpler to run.

Bear in mind too that Kafka is a persistent log, not just a message bus so any data you feed into it is kept available until it expires (which is configurable). This can be useful if you have multiple clients reading from the queue with their own lifecycle but in your case it doesn't sound like that would be necessary. You could also use a RabbitMQ fanout exchange if you need that in the future.

See more
Get Advice from developers at your company using StackShare Enterprise. Sign up for StackShare Enterprise.
Learn More
Pros of Amazon Athena
Pros of Kafka
  • 16
    Use SQL to analyze CSV files
  • 8
    Glue crawlers gives easy Data catalogue
  • 7
    Cheap
  • 6
    Query all my data without running servers 24x7
  • 4
    No data base servers yay
  • 3
    Easy integration with QuickSight
  • 2
    Query and analyse CSV,parquet,json files in sql
  • 2
    Also glue and athena use same data catalog
  • 1
    No configuration required
  • 0
    Ad hoc checks on data made easy
  • 126
    High-throughput
  • 119
    Distributed
  • 92
    Scalable
  • 86
    High-Performance
  • 66
    Durable
  • 38
    Publish-Subscribe
  • 19
    Simple-to-use
  • 18
    Open source
  • 12
    Written in Scala and java. Runs on JVM
  • 9
    Message broker + Streaming system
  • 4
    KSQL
  • 4
    Avro schema integration
  • 4
    Robust
  • 3
    Suport Multiple clients
  • 2
    Extremely good parallelism constructs
  • 2
    Partioned, replayable log
  • 1
    Simple publisher / multi-subscriber model
  • 1
    Fun
  • 1
    Flexible

Sign up to add or upvote prosMake informed product decisions

Cons of Amazon Athena
Cons of Kafka
    Be the first to leave a con
    • 32
      Non-Java clients are second-class citizens
    • 29
      Needs Zookeeper
    • 9
      Operational difficulties
    • 5
      Terrible Packaging

    Sign up to add or upvote consMake informed product decisions

    - No public GitHub repository available -

    What is Amazon Athena?

    Amazon Athena is an interactive query service that makes it easy to analyze data in Amazon S3 using standard SQL. Athena is serverless, so there is no infrastructure to manage, and you pay only for the queries that you run.

    What is Kafka?

    Kafka is a distributed, partitioned, replicated commit log service. It provides the functionality of a messaging system, but with a unique design.

    Need advice about which tool to choose?Ask the StackShare community!

    What companies use Amazon Athena?
    What companies use Kafka?
    See which teams inside your own company are using Amazon Athena or Kafka.
    Sign up for StackShare EnterpriseLearn More

    Sign up to get full access to all the companiesMake informed product decisions

    What tools integrate with Amazon Athena?
    What tools integrate with Kafka?

    Sign up to get full access to all the tool integrationsMake informed product decisions

    Blog Posts

    Dec 22 2021 at 5:41AM

    Pinterest

    MySQLKafkaDruid+3
    3
    565
    Amazon S3KafkaZookeeper+5
    8
    1558
    Mar 24 2021 at 12:57PM

    Pinterest

    GitJenkinsKafka+7
    3
    2127
    What are some alternatives to Amazon Athena and Kafka?
    Presto
    Distributed SQL Query Engine for Big Data
    Amazon Redshift Spectrum
    With Redshift Spectrum, you can extend the analytic power of Amazon Redshift beyond data stored on local disks in your data warehouse to query vast amounts of unstructured data in your Amazon S3 “data lake” -- without having to load or transform any data.
    Amazon Redshift
    It is optimized for data sets ranging from a few hundred gigabytes to a petabyte or more and costs less than $1,000 per terabyte per year, a tenth the cost of most traditional data warehousing solutions.
    Cassandra
    Partitioning means that Cassandra can distribute your data across multiple machines in an application-transparent matter. Cassandra will automatically repartition as machines are added and removed from the cluster. Row store means that like relational databases, Cassandra organizes data by rows and columns. The Cassandra Query Language (CQL) is a close relative of SQL.
    Spectrum
    The community platform for the future.
    See all alternatives