Need advice about which tool to choose?Ask the StackShare community!
InfluxDB vs Presto: What are the differences?
Introduction: InfluxDB and Presto are both popular database systems, but they serve different purposes and have distinct features. Here we will outline the key differences between InfluxDB and Presto.
Data Model: InfluxDB is a time-series database designed for high-performance storage and retrieval of time-series data, making it ideal for use cases like monitoring, IoT, and real-time analytics. On the other hand, Presto is a distributed SQL query engine that enables querying data where it resides, making it more versatile for a wider range of use cases beyond time-series data.
Query Language: InfluxDB uses its own query language called InfluxQL, optimized for time-series data operations like aggregation and downsampling. In contrast, Presto supports standard SQL, making it familiar and accessible to most data analysts and engineers. This flexibility allows Presto to handle complex analytical queries across various data sources easily.
Scalability: InfluxDB is horizontally scalable and can handle large amounts of time-series data using clustering and sharding techniques. Presto, on the other hand, is built for interactive querying on large datasets across distributed storage systems like Hadoop, S3, and more. This architecture makes Presto well-suited for ad-hoc analysis and big data processing.
Use Cases: While InfluxDB is tailored for time-series data management and monitoring applications, Presto is commonly used in data warehousing, business intelligence, and analytics scenarios. InfluxDB excels in IoT sensor data, metrics monitoring, and anomaly detection, while Presto shines in complex queries on structured or semi-structured data sets.
Data Sources: InfluxDB is optimized for handling time-series data from sources like sensors, devices, servers, and applications, storing and analyzing this data efficiently. In contrast, Presto can query data from various sources including relational databases, Hadoop, cloud storage, and more, enabling cross-source analysis and reporting capabilities.
Performance: InfluxDB is optimized for high ingest rates and fast query responses on time-series data, ensuring rapid data retrieval and monitoring capabilities. Presto focuses on distributed query processing, providing parallel execution and scalability for complex analytical workloads on large datasets, ensuring high performance in data exploration and analysis tasks.
In Summary, InfluxDB and Presto differ in their data models, query languages, scalability, use cases, data sources, and performance characteristics, catering to specific requirements for time-series data management and complex querying tasks in different environments.
I have a lot of data that's currently sitting in a MariaDB database, a lot of tables that weigh 200gb with indexes. Most of the large tables have a date column which is always filtered, but there are usually 4-6 additional columns that are filtered and used for statistics. I'm trying to figure out the best tool for storing and analyzing large amounts of data. Preferably self-hosted or a cheap solution. The current problem I'm running into is speed. Even with pretty good indexes, if I'm trying to load a large dataset, it's pretty slow.
Druid Could be an amazing solution for your use case, My understanding, and the assumption is you are looking to export your data from MariaDB for Analytical workload. It can be used for time series database as well as a data warehouse and can be scaled horizontally once your data increases. It's pretty easy to set up on any environment (Cloud, Kubernetes, or Self-hosted nix system). Some important features which make it a perfect solution for your use case. 1. It can do streaming ingestion (Kafka, Kinesis) as well as batch ingestion (Files from Local & Cloud Storage or Databases like MySQL, Postgres). In your case MariaDB (which has the same drivers to MySQL) 2. Columnar Database, So you can query just the fields which are required, and that runs your query faster automatically. 3. Druid intelligently partitions data based on time and time-based queries are significantly faster than traditional databases. 4. Scale up or down by just adding or removing servers, and Druid automatically rebalances. Fault-tolerant architecture routes around server failures 5. Gives ana amazing centralized UI to manage data sources, query, tasks.
We are building an IOT service with heavy write throughput and fewer reads (we need downsampling records). We prefer to have good reliability when comes to data and prefer to have data retention based on policies.
So, we are looking for what is the best underlying DB for ingesting a lot of data and do queries easily
We had a similar challenge. We started with DynamoDB, Timescale, and even InfluxDB and Mongo - to eventually settle with PostgreSQL. Assuming the inbound data pipeline in queued (for example, Kinesis/Kafka -> S3 -> and some Lambda functions), PostgreSQL gave us a We had a similar challenge. We started with DynamoDB, Timescale and even InfluxDB and Mongo - to eventually settle with PostgreSQL. Assuming the inbound data pipeline in queued (for example, Kinesis/Kafka -> S3 -> and some Lambda functions), PostgreSQL gave us better performance by far.
Druid is amazing for this use case and is a cloud-native solution that can be deployed on any cloud infrastructure or on Kubernetes. - Easy to scale horizontally - Column Oriented Database - SQL to query data - Streaming and Batch Ingestion - Native search indexes It has feature to work as TimeSeriesDB, Datawarehouse, and has Time-optimized partitioning.
if you want to find a serverless solution with capability of a lot of storage and SQL kind of capability then google bigquery is the best solution for that.
To provide employees with the critical need of interactive querying, we’ve worked with Presto, an open-source distributed SQL query engine, over the years. Operating Presto at Pinterest’s scale has involved resolving quite a few challenges like, supporting deeply nested and huge thrift schemas, slow/ bad worker detection and remediation, auto-scaling cluster, graceful cluster shutdown and impersonation support for ldap authenticator.
Our infrastructure is built on top of Amazon EC2 and we leverage Amazon S3 for storing our data. This separates compute and storage layers, and allows multiple compute clusters to share the S3 data.
We have hundreds of petabytes of data and tens of thousands of Apache Hive tables. Our Presto clusters are comprised of a fleet of 450 r4.8xl EC2 instances. Presto clusters together have over 100 TBs of memory and 14K vcpu cores. Within Pinterest, we have close to more than 1,000 monthly active users (out of total 1,600+ Pinterest employees) using Presto, who run about 400K queries on these clusters per month.
Each query submitted to Presto cluster is logged to a Kafka topic via Singer. Singer is a logging agent built at Pinterest and we talked about it in a previous post. Each query is logged when it is submitted and when it finishes. When a Presto cluster crashes, we will have query submitted events without corresponding query finished events. These events enable us to capture the effect of cluster crashes over time.
Each Presto cluster at Pinterest has workers on a mix of dedicated AWS EC2 instances and Kubernetes pods. Kubernetes platform provides us with the capability to add and remove workers from a Presto cluster very quickly. The best-case latency on bringing up a new worker on Kubernetes is less than a minute. However, when the Kubernetes cluster itself is out of resources and needs to scale up, it can take up to ten minutes. Some other advantages of deploying on Kubernetes platform is that our Presto deployment becomes agnostic of cloud vendor, instance types, OS, etc.
#BigData #AWS #DataScience #DataEngineering
The platform deals with time series data from sensors aggregated against things( event data that originates at periodic intervals). We use Cassandra as our distributed database to store time series data. Aggregated data insights from Cassandra is delivered as web API for consumption from other applications. Presto as a distributed sql querying engine, can provide a faster execution time provided the queries are tuned for proper distribution across the cluster. Another objective that we had was to combine Cassandra table data with other business data from RDBMS or other big data systems where presto through its connector architecture would have opened up a whole lot of options for us.
I chose TimescaleDB because to be the backend system of our production monitoring system. We needed to be able to keep track of multiple high cardinality dimensions.
The drawbacks of this decision are our monitoring system is a bit more ad hoc than it used to (New Relic Insights)
We are combining this with Grafana for display and Telegraf for data collection
Pros of InfluxDB
- Time-series data analysis59
- Easy setup, no dependencies30
- Fast, scalable & open source24
- Open source21
- Real-time analytics20
- Continuous Query support6
- Easy Query Language5
- HTTP API4
- Out-of-the-box, automatic Retention Policy4
- Offers Enterprise version1
- Free Open Source version1
Pros of Presto
- Works directly on files in s3 (no ETL)18
- Open-source13
- Join multiple databases12
- Scalable10
- Gets ready in minutes7
- MPP6
Sign up to add or upvote prosMake informed product decisions
Cons of InfluxDB
- Instability4
- Proprietary query language1
- HA or Clustering is only in paid version1
Cons of Presto
Sign up to add or upvote consMake informed product decisions
What is InfluxDB?
What is Presto?
Need advice about which tool to choose?Ask the StackShare community!
What companies use Presto?
Sign up to get full access to all the companiesMake informed product decisions
What tools integrate with Presto?
Sign up to get full access to all the tool integrationsMake informed product decisions
Blog Posts
+17
+11
+27