Alternatives to Amazon Kinesis Firehose logo

Alternatives to Amazon Kinesis Firehose

Stream, Kafka, Amazon Kinesis, Postman, and Postman are the most popular alternatives and competitors to Amazon Kinesis Firehose.
236
185
+ 1
0

What is Amazon Kinesis Firehose and what are its top alternatives?

Amazon Kinesis Firehose is a scalable data streaming service that allows you to easily load streaming data into data lakes and analytics services. Key features include automatic scaling, data transformation capabilities, and integration with various AWS services. However, some limitations include the complexity of setting up and managing the service, and the cost associated with data transfer and storage.

  1. Apache Kafka: Apache Kafka is a distributed streaming platform known for its high throughput and fault tolerance. Key features include data replication, partitioning, and message durability. Pros: Scalability, fault tolerance, and high performance. Cons: Requires more configuration and setup compared to Amazon Kinesis Firehose.
  2. Google Cloud Pub/Sub: Google Cloud Pub/Sub is a messaging service that allows you to ingest, transform, and deliver event data. Key features include real-time messaging, event ordering, and reliable message delivery. Pros: Integration with Google Cloud Platform services, horizontal scalability. Cons: Limited support for data transformation.
  3. Azure Stream Analytics: Azure Stream Analytics is a real-time data streaming and analytics service provided by Microsoft Azure. Key features include complex event processing, real-time analytics, and integration with Azure services. Pros: Easy integration with Azure ecosystem, SQL-like querying. Cons: Limited scalability compared to Amazon Kinesis Firehose.
  4. Apache Flink: Apache Flink is a distributed stream processing framework known for its high performance and stateful processing capabilities. Key features include exactly-once processing semantics, event time processing, and support for batch processing. Pros: Advanced processing capabilities, low latency, and fault tolerance. Cons: Steeper learning curve compared to Amazon Kinesis Firehose.
  5. Confluent Platform: Confluent Platform is a distribution of Apache Kafka with additional tools and services for managing and monitoring Kafka clusters. Key features include schema registry, Kafka Connect, and KSQL for stream processing. Pros: Integrated platform for streaming data, rich ecosystem. Cons: Additional cost for enterprise features.
  6. IBM Streams: IBM Streams is a streaming analytics platform that enables real-time processing of data streams. Key features include analytics modeling, visual development tools, and integration with various data sources. Pros: Scalability, visual development interface. Cons: Complexity in deployment and management.
  7. Alibaba Cloud Log Service: Alibaba Cloud Log Service is a fully managed service for collecting, consuming, and analyzing log data in real time. Key features include log collection, indexing, and real-time analytics. Pros: Integration with Alibaba Cloud services, built-in log analysis tools. Cons: Limited scalability options compared to Amazon Kinesis Firehose.
  8. SignalFx: SignalFx is a monitoring and observability platform that provides real-time streaming analytics for metrics, traces, and logs. Key features include advanced analytics, anomaly detection, and data visualization. Pros: Real-time monitoring, integration with various data sources. Cons: Focuses more on monitoring and alerting, less on data transformation.
  9. StreamSets Data Collector: StreamSets Data Collector is an open-source data ingest platform that enables data movement between different sources and destinations. Key features include data drift handling, data quality monitoring, and support for various connectors. Pros: Flexibility, open-source community support. Cons: Requires more technical expertise and configuration.
  10. Elasticsearch Logstash: Elasticsearch Logstash is an open-source data processing pipeline that ingests data from multiple sources, transforms it, and sends it to various destinations. Key features include plugins for different data inputs and outputs, data enrichment capabilities, and scalability. Pros: Easy integration with Elasticsearch and Kibana, custom pipeline configurations. Cons: Requires more manual setup and configuration compared to Amazon Kinesis Firehose.

Top Alternatives to Amazon Kinesis Firehose

  • Stream
    Stream

    Stream allows you to build scalable feeds, activity streams, and chat. Stream’s simple, yet powerful API’s and SDKs are used by some of the largest and most popular applications for feeds and chat. SDKs available for most popular languages. ...

  • Kafka
    Kafka

    Kafka is a distributed, partitioned, replicated commit log service. It provides the functionality of a messaging system, but with a unique design. ...

  • Amazon Kinesis
    Amazon Kinesis

    Amazon Kinesis can collect and process hundreds of gigabytes of data per second from hundreds of thousands of sources, allowing you to easily write applications that process information in real-time, from sources such as web site click-streams, marketing and financial information, manufacturing instrumentation and social media, and operational logs and metering data. ...

  • Postman
    Postman

    It is the only complete API development environment, used by nearly five million developers and more than 100,000 companies worldwide. ...

  • Postman
    Postman

    It is the only complete API development environment, used by nearly five million developers and more than 100,000 companies worldwide. ...

  • Stack Overflow
    Stack Overflow

    Stack Overflow is a question and answer site for professional and enthusiast programmers. It's built and run by you as part of the Stack Exchange network of Q&A sites. With your help, we're working together to build a library of detailed answers to every question about programming. ...

  • Google Maps
    Google Maps

    Create rich applications and stunning visualisations of your data, leveraging the comprehensiveness, accuracy, and usability of Google Maps and a modern web platform that scales as you grow. ...

  • Elasticsearch
    Elasticsearch

    Elasticsearch is a distributed, RESTful search and analytics engine capable of storing data and searching it in near real time. Elasticsearch, Kibana, Beats and Logstash are the Elastic Stack (sometimes called the ELK Stack). ...

Amazon Kinesis Firehose alternatives & related posts

Stream logo

Stream

226
54
Build scalable feeds, activity streams & chat in a few hours instead of months.
226
54
PROS OF STREAM
  • 18
    Up and running in few minutes
  • 18
    Integrates via easy-to-use REST API
  • 18
    It's easy to setup with the minimum coding
CONS OF STREAM
    Be the first to leave a con

    related Stream posts

    Kafka logo

    Kafka

    23.9K
    607
    Distributed, fault tolerant, high throughput pub-sub messaging system
    23.9K
    607
    PROS OF KAFKA
    • 126
      High-throughput
    • 119
      Distributed
    • 92
      Scalable
    • 86
      High-Performance
    • 66
      Durable
    • 38
      Publish-Subscribe
    • 19
      Simple-to-use
    • 18
      Open source
    • 12
      Written in Scala and java. Runs on JVM
    • 9
      Message broker + Streaming system
    • 4
      KSQL
    • 4
      Avro schema integration
    • 4
      Robust
    • 3
      Suport Multiple clients
    • 2
      Extremely good parallelism constructs
    • 2
      Partioned, replayable log
    • 1
      Simple publisher / multi-subscriber model
    • 1
      Flexible
    • 1
      Fun
    CONS OF KAFKA
    • 32
      Non-Java clients are second-class citizens
    • 29
      Needs Zookeeper
    • 9
      Operational difficulties
    • 5
      Terrible Packaging

    related Kafka posts

    Nick Rockwell
    SVP, Engineering at Fastly · | 46 upvotes · 4.4M views

    When I joined NYT there was already broad dissatisfaction with the LAMP (Linux Apache HTTP Server MySQL PHP) Stack and the front end framework, in particular. So, I wasn't passing judgment on it. I mean, LAMP's fine, you can do good work in LAMP. It's a little dated at this point, but it's not ... I didn't want to rip it out for its own sake, but everyone else was like, "We don't like this, it's really inflexible." And I remember from being outside the company when that was called MIT FIVE when it had launched. And been observing it from the outside, and I was like, you guys took so long to do that and you did it so carefully, and yet you're not happy with your decisions. Why is that? That was more the impetus. If we're going to do this again, how are we going to do it in a way that we're gonna get a better result?

    So we're moving quickly away from LAMP, I would say. So, right now, the new front end is React based and using Apollo. And we've been in a long, protracted, gradual rollout of the core experiences.

    React is now talking to GraphQL as a primary API. There's a Node.js back end, to the front end, which is mainly for server-side rendering, as well.

    Behind there, the main repository for the GraphQL server is a big table repository, that we call Bodega because it's a convenience store. And that reads off of a Kafka pipeline.

    See more
    Ashish Singh
    Tech Lead, Big Data Platform at Pinterest · | 38 upvotes · 3.7M views

    To provide employees with the critical need of interactive querying, we’ve worked with Presto, an open-source distributed SQL query engine, over the years. Operating Presto at Pinterest’s scale has involved resolving quite a few challenges like, supporting deeply nested and huge thrift schemas, slow/ bad worker detection and remediation, auto-scaling cluster, graceful cluster shutdown and impersonation support for ldap authenticator.

    Our infrastructure is built on top of Amazon EC2 and we leverage Amazon S3 for storing our data. This separates compute and storage layers, and allows multiple compute clusters to share the S3 data.

    We have hundreds of petabytes of data and tens of thousands of Apache Hive tables. Our Presto clusters are comprised of a fleet of 450 r4.8xl EC2 instances. Presto clusters together have over 100 TBs of memory and 14K vcpu cores. Within Pinterest, we have close to more than 1,000 monthly active users (out of total 1,600+ Pinterest employees) using Presto, who run about 400K queries on these clusters per month.

    Each query submitted to Presto cluster is logged to a Kafka topic via Singer. Singer is a logging agent built at Pinterest and we talked about it in a previous post. Each query is logged when it is submitted and when it finishes. When a Presto cluster crashes, we will have query submitted events without corresponding query finished events. These events enable us to capture the effect of cluster crashes over time.

    Each Presto cluster at Pinterest has workers on a mix of dedicated AWS EC2 instances and Kubernetes pods. Kubernetes platform provides us with the capability to add and remove workers from a Presto cluster very quickly. The best-case latency on bringing up a new worker on Kubernetes is less than a minute. However, when the Kubernetes cluster itself is out of resources and needs to scale up, it can take up to ten minutes. Some other advantages of deploying on Kubernetes platform is that our Presto deployment becomes agnostic of cloud vendor, instance types, OS, etc.

    #BigData #AWS #DataScience #DataEngineering

    See more
    Amazon Kinesis logo

    Amazon Kinesis

    730
    9
    Store and process terabytes of data each hour from hundreds of thousands of sources
    730
    9
    PROS OF AMAZON KINESIS
    • 9
      Scalable
    CONS OF AMAZON KINESIS
    • 3
      Cost

    related Amazon Kinesis posts

    Praveen Mooli
    Engineering Manager at Taylor and Francis · | 19 upvotes · 4.1M views

    We are in the process of building a modern content platform to deliver our content through various channels. We decided to go with Microservices architecture as we wanted scale. Microservice architecture style is an approach to developing an application as a suite of small independently deployable services built around specific business capabilities. You can gain modularity, extensive parallelism and cost-effective scaling by deploying services across many distributed servers. Microservices modularity facilitates independent updates/deployments, and helps to avoid single point of failure, which can help prevent large-scale outages. We also decided to use Event Driven Architecture pattern which is a popular distributed asynchronous architecture pattern used to produce highly scalable applications. The event-driven architecture is made up of highly decoupled, single-purpose event processing components that asynchronously receive and process events.

    To build our #Backend capabilities we decided to use the following: 1. #Microservices - Java with Spring Boot , Node.js with ExpressJS and Python with Flask 2. #Eventsourcingframework - Amazon Kinesis , Amazon Kinesis Firehose , Amazon SNS , Amazon SQS, AWS Lambda 3. #Data - Amazon RDS , Amazon DynamoDB , Amazon S3 , MongoDB Atlas

    To build #Webapps we decided to use Angular 2 with RxJS

    #Devops - GitHub , Travis CI , Terraform , Docker , Serverless

    See more
    John Kodumal

    As we've evolved or added additional infrastructure to our stack, we've biased towards managed services. Most new backing stores are Amazon RDS instances now. We do use self-managed PostgreSQL with TimescaleDB for time-series data—this is made HA with the use of Patroni and Consul.

    We also use managed Amazon ElastiCache instances instead of spinning up Amazon EC2 instances to run Redis workloads, as well as shifting to Amazon Kinesis instead of Kafka.

    See more
    Postman logo

    Postman

    96K
    1.8K
    Only complete API development environment
    96K
    1.8K
    PROS OF POSTMAN
    • 490
      Easy to use
    • 369
      Great tool
    • 276
      Makes developing rest api's easy peasy
    • 156
      Easy setup, looks good
    • 144
      The best api workflow out there
    • 53
      It's the best
    • 53
      History feature
    • 44
      Adds real value to my workflow
    • 43
      Great interface that magically predicts your needs
    • 35
      The best in class app
    • 12
      Can save and share script
    • 10
      Fully featured without looking cluttered
    • 8
      Collections
    • 8
      Option to run scrips
    • 8
      Global/Environment Variables
    • 7
      Shareable Collections
    • 7
      Dead simple and useful. Excellent
    • 7
      Dark theme easy on the eyes
    • 6
      Awesome customer support
    • 6
      Great integration with newman
    • 5
      Documentation
    • 5
      Simple
    • 5
      The test script is useful
    • 4
      Saves responses
    • 4
      This has simplified my testing significantly
    • 4
      Makes testing API's as easy as 1,2,3
    • 4
      Easy as pie
    • 3
      API-network
    • 3
      I'd recommend it to everyone who works with apis
    • 3
      Mocking API calls with predefined response
    • 2
      Now supports GraphQL
    • 2
      Postman Runner CI Integration
    • 2
      Easy to setup, test and provides test storage
    • 2
      Continuous integration using newman
    • 2
      Pre-request Script and Test attributes are invaluable
    • 2
      Runner
    • 2
      Graph
    • 1
      <a href="http://fixbit.com/">useful tool</a>
    CONS OF POSTMAN
    • 10
      Stores credentials in HTTP
    • 9
      Bloated features and UI
    • 8
      Cumbersome to switch authentication tokens
    • 7
      Poor GraphQL support
    • 5
      Expensive
    • 3
      Not free after 5 users
    • 3
      Can't prompt for per-request variables
    • 1
      Import swagger
    • 1
      Support websocket
    • 1
      Import curl

    related Postman posts

    Noah Zoschke
    Engineering Manager at Segment · | 30 upvotes · 3.2M views

    We just launched the Segment Config API (try it out for yourself here) — a set of public REST APIs that enable you to manage your Segment configuration. A public API is only as good as its #documentation. For the API reference doc we are using Postman.

    Postman is an “API development environment”. You download the desktop app, and build API requests by URL and payload. Over time you can build up a set of requests and organize them into a “Postman Collection”. You can generalize a collection with “collection variables”. This allows you to parameterize things like username, password and workspace_name so a user can fill their own values in before making an API call. This makes it possible to use Postman for one-off API tasks instead of writing code.

    Then you can add Markdown content to the entire collection, a folder of related methods, and/or every API method to explain how the APIs work. You can publish a collection and easily share it with a URL.

    This turns Postman from a personal #API utility to full-blown public interactive API documentation. The result is a great looking web page with all the API calls, docs and sample requests and responses in one place. Check out the results here.

    Postman’s powers don’t end here. You can automate Postman with “test scripts” and have it periodically run a collection scripts as “monitors”. We now have #QA around all the APIs in public docs to make sure they are always correct

    Along the way we tried other techniques for documenting APIs like ReadMe.io or Swagger UI. These required a lot of effort to customize.

    Writing and maintaining a Postman collection takes some work, but the resulting documentation site, interactivity and API testing tools are well worth it.

    See more
    Simon Reymann
    Senior Fullstack Developer at QUANTUSflow Software GmbH · | 27 upvotes · 5.7M views

    Our whole Node.js backend stack consists of the following tools:

    • Lerna as a tool for multi package and multi repository management
    • npm as package manager
    • NestJS as Node.js framework
    • TypeScript as programming language
    • ExpressJS as web server
    • Swagger UI for visualizing and interacting with the API’s resources
    • Postman as a tool for API development
    • TypeORM as object relational mapping layer
    • JSON Web Token for access token management

    The main reason we have chosen Node.js over PHP is related to the following artifacts:

    • Made for the web and widely in use: Node.js is a software platform for developing server-side network services. Well-known projects that rely on Node.js include the blogging software Ghost, the project management tool Trello and the operating system WebOS. Node.js requires the JavaScript runtime environment V8, which was specially developed by Google for the popular Chrome browser. This guarantees a very resource-saving architecture, which qualifies Node.js especially for the operation of a web server. Ryan Dahl, the developer of Node.js, released the first stable version on May 27, 2009. He developed Node.js out of dissatisfaction with the possibilities that JavaScript offered at the time. The basic functionality of Node.js has been mapped with JavaScript since the first version, which can be expanded with a large number of different modules. The current package managers (npm or Yarn) for Node.js know more than 1,000,000 of these modules.
    • Fast server-side solutions: Node.js adopts the JavaScript "event-loop" to create non-blocking I/O applications that conveniently serve simultaneous events. With the standard available asynchronous processing within JavaScript/TypeScript, highly scalable, server-side solutions can be realized. The efficient use of the CPU and the RAM is maximized and more simultaneous requests can be processed than with conventional multi-thread servers.
    • A language along the entire stack: Widely used frameworks such as React or AngularJS or Vue.js, which we prefer, are written in JavaScript/TypeScript. If Node.js is now used on the server side, you can use all the advantages of a uniform script language throughout the entire application development. The same language in the back- and frontend simplifies the maintenance of the application and also the coordination within the development team.
    • Flexibility: Node.js sets very few strict dependencies, rules and guidelines and thus grants a high degree of flexibility in application development. There are no strict conventions so that the appropriate architecture, design structures, modules and features can be freely selected for the development.
    See more
    Postman logo

    Postman

    96K
    1.8K
    Only complete API development environment
    96K
    1.8K
    PROS OF POSTMAN
    • 490
      Easy to use
    • 369
      Great tool
    • 276
      Makes developing rest api's easy peasy
    • 156
      Easy setup, looks good
    • 144
      The best api workflow out there
    • 53
      It's the best
    • 53
      History feature
    • 44
      Adds real value to my workflow
    • 43
      Great interface that magically predicts your needs
    • 35
      The best in class app
    • 12
      Can save and share script
    • 10
      Fully featured without looking cluttered
    • 8
      Collections
    • 8
      Option to run scrips
    • 8
      Global/Environment Variables
    • 7
      Shareable Collections
    • 7
      Dead simple and useful. Excellent
    • 7
      Dark theme easy on the eyes
    • 6
      Awesome customer support
    • 6
      Great integration with newman
    • 5
      Documentation
    • 5
      Simple
    • 5
      The test script is useful
    • 4
      Saves responses
    • 4
      This has simplified my testing significantly
    • 4
      Makes testing API's as easy as 1,2,3
    • 4
      Easy as pie
    • 3
      API-network
    • 3
      I'd recommend it to everyone who works with apis
    • 3
      Mocking API calls with predefined response
    • 2
      Now supports GraphQL
    • 2
      Postman Runner CI Integration
    • 2
      Easy to setup, test and provides test storage
    • 2
      Continuous integration using newman
    • 2
      Pre-request Script and Test attributes are invaluable
    • 2
      Runner
    • 2
      Graph
    • 1
      <a href="http://fixbit.com/">useful tool</a>
    CONS OF POSTMAN
    • 10
      Stores credentials in HTTP
    • 9
      Bloated features and UI
    • 8
      Cumbersome to switch authentication tokens
    • 7
      Poor GraphQL support
    • 5
      Expensive
    • 3
      Not free after 5 users
    • 3
      Can't prompt for per-request variables
    • 1
      Import swagger
    • 1
      Support websocket
    • 1
      Import curl

    related Postman posts

    Noah Zoschke
    Engineering Manager at Segment · | 30 upvotes · 3.2M views

    We just launched the Segment Config API (try it out for yourself here) — a set of public REST APIs that enable you to manage your Segment configuration. A public API is only as good as its #documentation. For the API reference doc we are using Postman.

    Postman is an “API development environment”. You download the desktop app, and build API requests by URL and payload. Over time you can build up a set of requests and organize them into a “Postman Collection”. You can generalize a collection with “collection variables”. This allows you to parameterize things like username, password and workspace_name so a user can fill their own values in before making an API call. This makes it possible to use Postman for one-off API tasks instead of writing code.

    Then you can add Markdown content to the entire collection, a folder of related methods, and/or every API method to explain how the APIs work. You can publish a collection and easily share it with a URL.

    This turns Postman from a personal #API utility to full-blown public interactive API documentation. The result is a great looking web page with all the API calls, docs and sample requests and responses in one place. Check out the results here.

    Postman’s powers don’t end here. You can automate Postman with “test scripts” and have it periodically run a collection scripts as “monitors”. We now have #QA around all the APIs in public docs to make sure they are always correct

    Along the way we tried other techniques for documenting APIs like ReadMe.io or Swagger UI. These required a lot of effort to customize.

    Writing and maintaining a Postman collection takes some work, but the resulting documentation site, interactivity and API testing tools are well worth it.

    See more
    Simon Reymann
    Senior Fullstack Developer at QUANTUSflow Software GmbH · | 27 upvotes · 5.7M views

    Our whole Node.js backend stack consists of the following tools:

    • Lerna as a tool for multi package and multi repository management
    • npm as package manager
    • NestJS as Node.js framework
    • TypeScript as programming language
    • ExpressJS as web server
    • Swagger UI for visualizing and interacting with the API’s resources
    • Postman as a tool for API development
    • TypeORM as object relational mapping layer
    • JSON Web Token for access token management

    The main reason we have chosen Node.js over PHP is related to the following artifacts:

    • Made for the web and widely in use: Node.js is a software platform for developing server-side network services. Well-known projects that rely on Node.js include the blogging software Ghost, the project management tool Trello and the operating system WebOS. Node.js requires the JavaScript runtime environment V8, which was specially developed by Google for the popular Chrome browser. This guarantees a very resource-saving architecture, which qualifies Node.js especially for the operation of a web server. Ryan Dahl, the developer of Node.js, released the first stable version on May 27, 2009. He developed Node.js out of dissatisfaction with the possibilities that JavaScript offered at the time. The basic functionality of Node.js has been mapped with JavaScript since the first version, which can be expanded with a large number of different modules. The current package managers (npm or Yarn) for Node.js know more than 1,000,000 of these modules.
    • Fast server-side solutions: Node.js adopts the JavaScript "event-loop" to create non-blocking I/O applications that conveniently serve simultaneous events. With the standard available asynchronous processing within JavaScript/TypeScript, highly scalable, server-side solutions can be realized. The efficient use of the CPU and the RAM is maximized and more simultaneous requests can be processed than with conventional multi-thread servers.
    • A language along the entire stack: Widely used frameworks such as React or AngularJS or Vue.js, which we prefer, are written in JavaScript/TypeScript. If Node.js is now used on the server side, you can use all the advantages of a uniform script language throughout the entire application development. The same language in the back- and frontend simplifies the maintenance of the application and also the coordination within the development team.
    • Flexibility: Node.js sets very few strict dependencies, rules and guidelines and thus grants a high degree of flexibility in application development. There are no strict conventions so that the appropriate architecture, design structures, modules and features can be freely selected for the development.
    See more
    Stack Overflow logo

    Stack Overflow

    69.9K
    893
    Question and answer site for professional and enthusiast programmers
    69.9K
    893
    PROS OF STACK OVERFLOW
    • 257
      Scary smart community
    • 206
      Knows all
    • 142
      Voting system
    • 134
      Good questions
    • 83
      Good SEO
    • 22
      Addictive
    • 14
      Tight focus
    • 10
      Share and gain knowledge
    • 7
      Useful
    • 3
      Fast loading
    • 2
      Gamification
    • 1
      Knows everyone
    • 1
      Experts share experience and answer questions
    • 1
      Stack overflow to developers As google to net surfers
    • 1
      Questions answered quickly
    • 1
      No annoying ads
    • 1
      No spam
    • 1
      Fast community response
    • 1
      Good moderators
    • 1
      Quick answers from users
    • 1
      Good answers
    • 1
      User reputation ranking
    • 1
      Efficient answers
    • 1
      Leading developer community
    CONS OF STACK OVERFLOW
    • 3
      Not welcoming to newbies
    • 3
      Unfair downvoting
    • 3
      Unfriendly moderators
    • 3
      No opinion based questions
    • 3
      Mean users
    • 2
      Limited to types of questions it can accept

    related Stack Overflow posts

    Tom Klein

    Google Analytics is a great tool to analyze your traffic. To debug our software and ask questions, we love to use Postman and Stack Overflow. Google Drive helps our team to share documents. We're able to build our great products through the APIs by Google Maps, CloudFlare, Stripe, PayPal, Twilio, Let's Encrypt, and TensorFlow.

    See more
    Google Maps logo

    Google Maps

    42.3K
    567
    Build highly customisable maps with your own content and imagery
    42.3K
    567
    PROS OF GOOGLE MAPS
    • 253
      Free
    • 136
      Address input through maps api
    • 82
      Sharable Directions
    • 47
      Google Earth
    • 46
      Unique
    • 3
      Custom maps designing
    CONS OF GOOGLE MAPS
    • 5
      Google Attributions and logo
    • 2
      Only map allowed alongside google place autocomplete

    related Google Maps posts

    Tom Klein

    Google Analytics is a great tool to analyze your traffic. To debug our software and ask questions, we love to use Postman and Stack Overflow. Google Drive helps our team to share documents. We're able to build our great products through the APIs by Google Maps, CloudFlare, Stripe, PayPal, Twilio, Let's Encrypt, and TensorFlow.

    See more

    A huge component of our product relies on gathering public data about locations of interest. Google Places API gives us that ability in the most efficient way. Since we are primarily going to be using as google data as a source of information for our MVP, we might as well start integrating the Google Places API in our system. We have worked with Google Maps in the past and we might take some inspiration from our previous projects onto this one.

    See more
    Elasticsearch logo

    Elasticsearch

    34.9K
    1.6K
    Open Source, Distributed, RESTful Search Engine
    34.9K
    1.6K
    PROS OF ELASTICSEARCH
    • 329
      Powerful api
    • 315
      Great search engine
    • 231
      Open source
    • 214
      Restful
    • 200
      Near real-time search
    • 98
      Free
    • 85
      Search everything
    • 54
      Easy to get started
    • 45
      Analytics
    • 26
      Distributed
    • 6
      Fast search
    • 5
      More than a search engine
    • 4
      Awesome, great tool
    • 4
      Great docs
    • 3
      Highly Available
    • 3
      Easy to scale
    • 2
      Nosql DB
    • 2
      Document Store
    • 2
      Great customer support
    • 2
      Intuitive API
    • 2
      Reliable
    • 2
      Potato
    • 2
      Fast
    • 2
      Easy setup
    • 2
      Great piece of software
    • 1
      Open
    • 1
      Scalability
    • 1
      Not stable
    • 1
      Easy to get hot data
    • 1
      Github
    • 1
      Elaticsearch
    • 1
      Actively developing
    • 1
      Responsive maintainers on GitHub
    • 1
      Ecosystem
    • 0
      Community
    CONS OF ELASTICSEARCH
    • 7
      Resource hungry
    • 6
      Diffecult to get started
    • 5
      Expensive
    • 4
      Hard to keep stable at large scale

    related Elasticsearch posts

    Tim Abbott

    We've been using PostgreSQL since the very early days of Zulip, but we actually didn't use it from the beginning. Zulip started out as a MySQL project back in 2012, because we'd heard it was a good choice for a startup with a wide community. However, we found that even though we were using the Django ORM for most of our database access, we spent a lot of time fighting with MySQL. Issues ranged from bad collation defaults, to bad query plans which required a lot of manual query tweaks.

    We ended up getting so frustrated that we tried out PostgresQL, and the results were fantastic. We didn't have to do any real customization (just some tuning settings for how big a server we had), and all of our most important queries were faster out of the box. As a result, we were able to delete a bunch of custom queries escaping the ORM that we'd written to make the MySQL query planner happy (because postgres just did the right thing automatically).

    And then after that, we've just gotten a ton of value out of postgres. We use its excellent built-in full-text search, which has helped us avoid needing to bring in a tool like Elasticsearch, and we've really enjoyed features like its partial indexes, which saved us a lot of work adding unnecessary extra tables to get good performance for things like our "unread messages" and "starred messages" indexes.

    I can't recommend it highly enough.

    See more
    Tymoteusz Paul
    Devops guy at X20X Development LTD · | 23 upvotes · 10.4M views

    Often enough I have to explain my way of going about setting up a CI/CD pipeline with multiple deployment platforms. Since I am a bit tired of yapping the same every single time, I've decided to write it up and share with the world this way, and send people to read it instead ;). I will explain it on "live-example" of how the Rome got built, basing that current methodology exists only of readme.md and wishes of good luck (as it usually is ;)).

    It always starts with an app, whatever it may be and reading the readmes available while Vagrant and VirtualBox is installing and updating. Following that is the first hurdle to go over - convert all the instruction/scripts into Ansible playbook(s), and only stopping when doing a clear vagrant up or vagrant reload we will have a fully working environment. As our Vagrant environment is now functional, it's time to break it! This is the moment to look for how things can be done better (too rigid/too lose versioning? Sloppy environment setup?) and replace them with the right way to do stuff, one that won't bite us in the backside. This is the point, and the best opportunity, to upcycle the existing way of doing dev environment to produce a proper, production-grade product.

    I should probably digress here for a moment and explain why. I firmly believe that the way you deploy production is the same way you should deploy develop, shy of few debugging-friendly setting. This way you avoid the discrepancy between how production work vs how development works, which almost always causes major pains in the back of the neck, and with use of proper tools should mean no more work for the developers. That's why we start with Vagrant as developer boxes should be as easy as vagrant up, but the meat of our product lies in Ansible which will do meat of the work and can be applied to almost anything: AWS, bare metal, docker, LXC, in open net, behind vpn - you name it.

    We must also give proper consideration to monitoring and logging hoovering at this point. My generic answer here is to grab Elasticsearch, Kibana, and Logstash. While for different use cases there may be better solutions, this one is well battle-tested, performs reasonably and is very easy to scale both vertically (within some limits) and horizontally. Logstash rules are easy to write and are well supported in maintenance through Ansible, which as I've mentioned earlier, are at the very core of things, and creating triggers/reports and alerts based on Elastic and Kibana is generally a breeze, including some quite complex aggregations.

    If we are happy with the state of the Ansible it's time to move on and put all those roles and playbooks to work. Namely, we need something to manage our CI/CD pipelines. For me, the choice is obvious: TeamCity. It's modern, robust and unlike most of the light-weight alternatives, it's transparent. What I mean by that is that it doesn't tell you how to do things, doesn't limit your ways to deploy, or test, or package for that matter. Instead, it provides a developer-friendly and rich playground for your pipelines. You can do most the same with Jenkins, but it has a quite dated look and feel to it, while also missing some key functionality that must be brought in via plugins (like quality REST API which comes built-in with TeamCity). It also comes with all the common-handy plugins like Slack or Apache Maven integration.

    The exact flow between CI and CD varies too greatly from one application to another to describe, so I will outline a few rules that guide me in it: 1. Make build steps as small as possible. This way when something breaks, we know exactly where, without needing to dig and root around. 2. All security credentials besides development environment must be sources from individual Vault instances. Keys to those containers should exist only on the CI/CD box and accessible by a few people (the less the better). This is pretty self-explanatory, as anything besides dev may contain sensitive data and, at times, be public-facing. Because of that appropriate security must be present. TeamCity shines in this department with excellent secrets-management. 3. Every part of the build chain shall consume and produce artifacts. If it creates nothing, it likely shouldn't be its own build. This way if any issue shows up with any environment or version, all developer has to do it is grab appropriate artifacts to reproduce the issue locally. 4. Deployment builds should be directly tied to specific Git branches/tags. This enables much easier tracking of what caused an issue, including automated identifying and tagging the author (nothing like automated regression testing!).

    Speaking of deployments, I generally try to keep it simple but also with a close eye on the wallet. Because of that, I am more than happy with AWS or another cloud provider, but also constantly peeking at the loads and do we get the value of what we are paying for. Often enough the pattern of use is not constantly erratic, but rather has a firm baseline which could be migrated away from the cloud and into bare metal boxes. That is another part where this approach strongly triumphs over the common Docker and CircleCI setup, where you are very much tied in to use cloud providers and getting out is expensive. Here to embrace bare-metal hosting all you need is a help of some container-based self-hosting software, my personal preference is with Proxmox and LXC. Following that all you must write are ansible scripts to manage hardware of Proxmox, similar way as you do for Amazon EC2 (ansible supports both greatly) and you are good to go. One does not exclude another, quite the opposite, as they can live in great synergy and cut your costs dramatically (the heavier your base load, the bigger the savings) while providing production-grade resiliency.

    See more