What is Amazon Kinesis Firehose and what are its top alternatives?
Amazon Kinesis Firehose is a scalable data streaming service that allows you to easily load streaming data into data lakes and analytics services. Key features include automatic scaling, data transformation capabilities, and integration with various AWS services. However, some limitations include the complexity of setting up and managing the service, and the cost associated with data transfer and storage.
- Apache Kafka: Apache Kafka is a distributed streaming platform known for its high throughput and fault tolerance. Key features include data replication, partitioning, and message durability. Pros: Scalability, fault tolerance, and high performance. Cons: Requires more configuration and setup compared to Amazon Kinesis Firehose.
- Google Cloud Pub/Sub: Google Cloud Pub/Sub is a messaging service that allows you to ingest, transform, and deliver event data. Key features include real-time messaging, event ordering, and reliable message delivery. Pros: Integration with Google Cloud Platform services, horizontal scalability. Cons: Limited support for data transformation.
- Azure Stream Analytics: Azure Stream Analytics is a real-time data streaming and analytics service provided by Microsoft Azure. Key features include complex event processing, real-time analytics, and integration with Azure services. Pros: Easy integration with Azure ecosystem, SQL-like querying. Cons: Limited scalability compared to Amazon Kinesis Firehose.
- Apache Flink: Apache Flink is a distributed stream processing framework known for its high performance and stateful processing capabilities. Key features include exactly-once processing semantics, event time processing, and support for batch processing. Pros: Advanced processing capabilities, low latency, and fault tolerance. Cons: Steeper learning curve compared to Amazon Kinesis Firehose.
- Confluent Platform: Confluent Platform is a distribution of Apache Kafka with additional tools and services for managing and monitoring Kafka clusters. Key features include schema registry, Kafka Connect, and KSQL for stream processing. Pros: Integrated platform for streaming data, rich ecosystem. Cons: Additional cost for enterprise features.
- IBM Streams: IBM Streams is a streaming analytics platform that enables real-time processing of data streams. Key features include analytics modeling, visual development tools, and integration with various data sources. Pros: Scalability, visual development interface. Cons: Complexity in deployment and management.
- Alibaba Cloud Log Service: Alibaba Cloud Log Service is a fully managed service for collecting, consuming, and analyzing log data in real time. Key features include log collection, indexing, and real-time analytics. Pros: Integration with Alibaba Cloud services, built-in log analysis tools. Cons: Limited scalability options compared to Amazon Kinesis Firehose.
- SignalFx: SignalFx is a monitoring and observability platform that provides real-time streaming analytics for metrics, traces, and logs. Key features include advanced analytics, anomaly detection, and data visualization. Pros: Real-time monitoring, integration with various data sources. Cons: Focuses more on monitoring and alerting, less on data transformation.
- StreamSets Data Collector: StreamSets Data Collector is an open-source data ingest platform that enables data movement between different sources and destinations. Key features include data drift handling, data quality monitoring, and support for various connectors. Pros: Flexibility, open-source community support. Cons: Requires more technical expertise and configuration.
- Elasticsearch Logstash: Elasticsearch Logstash is an open-source data processing pipeline that ingests data from multiple sources, transforms it, and sends it to various destinations. Key features include plugins for different data inputs and outputs, data enrichment capabilities, and scalability. Pros: Easy integration with Elasticsearch and Kibana, custom pipeline configurations. Cons: Requires more manual setup and configuration compared to Amazon Kinesis Firehose.
Top Alternatives to Amazon Kinesis Firehose
- Stream
Stream allows you to build scalable feeds, activity streams, and chat. Stream’s simple, yet powerful API’s and SDKs are used by some of the largest and most popular applications for feeds and chat. SDKs available for most popular languages. ...
- Kafka
Kafka is a distributed, partitioned, replicated commit log service. It provides the functionality of a messaging system, but with a unique design. ...
- Amazon Kinesis
Amazon Kinesis can collect and process hundreds of gigabytes of data per second from hundreds of thousands of sources, allowing you to easily write applications that process information in real-time, from sources such as web site click-streams, marketing and financial information, manufacturing instrumentation and social media, and operational logs and metering data. ...
- Postman
It is the only complete API development environment, used by nearly five million developers and more than 100,000 companies worldwide. ...
- Postman
It is the only complete API development environment, used by nearly five million developers and more than 100,000 companies worldwide. ...
- Stack Overflow
Stack Overflow is a question and answer site for professional and enthusiast programmers. It's built and run by you as part of the Stack Exchange network of Q&A sites. With your help, we're working together to build a library of detailed answers to every question about programming. ...
- Google Maps
Create rich applications and stunning visualisations of your data, leveraging the comprehensiveness, accuracy, and usability of Google Maps and a modern web platform that scales as you grow. ...
- Elasticsearch
Elasticsearch is a distributed, RESTful search and analytics engine capable of storing data and searching it in near real time. Elasticsearch, Kibana, Beats and Logstash are the Elastic Stack (sometimes called the ELK Stack). ...
Amazon Kinesis Firehose alternatives & related posts
- Up and running in few minutes18
- Integrates via easy-to-use REST API18
- It's easy to setup with the minimum coding18
related Stream posts
- High-throughput126
- Distributed119
- Scalable92
- High-Performance86
- Durable66
- Publish-Subscribe38
- Simple-to-use19
- Open source18
- Written in Scala and java. Runs on JVM12
- Message broker + Streaming system9
- KSQL4
- Avro schema integration4
- Robust4
- Suport Multiple clients3
- Extremely good parallelism constructs2
- Partioned, replayable log2
- Simple publisher / multi-subscriber model1
- Flexible1
- Fun1
- Non-Java clients are second-class citizens32
- Needs Zookeeper29
- Operational difficulties9
- Terrible Packaging5
related Kafka posts
When I joined NYT there was already broad dissatisfaction with the LAMP (Linux Apache HTTP Server MySQL PHP) Stack and the front end framework, in particular. So, I wasn't passing judgment on it. I mean, LAMP's fine, you can do good work in LAMP. It's a little dated at this point, but it's not ... I didn't want to rip it out for its own sake, but everyone else was like, "We don't like this, it's really inflexible." And I remember from being outside the company when that was called MIT FIVE when it had launched. And been observing it from the outside, and I was like, you guys took so long to do that and you did it so carefully, and yet you're not happy with your decisions. Why is that? That was more the impetus. If we're going to do this again, how are we going to do it in a way that we're gonna get a better result?
So we're moving quickly away from LAMP, I would say. So, right now, the new front end is React based and using Apollo. And we've been in a long, protracted, gradual rollout of the core experiences.
React is now talking to GraphQL as a primary API. There's a Node.js back end, to the front end, which is mainly for server-side rendering, as well.
Behind there, the main repository for the GraphQL server is a big table repository, that we call Bodega because it's a convenience store. And that reads off of a Kafka pipeline.
To provide employees with the critical need of interactive querying, we’ve worked with Presto, an open-source distributed SQL query engine, over the years. Operating Presto at Pinterest’s scale has involved resolving quite a few challenges like, supporting deeply nested and huge thrift schemas, slow/ bad worker detection and remediation, auto-scaling cluster, graceful cluster shutdown and impersonation support for ldap authenticator.
Our infrastructure is built on top of Amazon EC2 and we leverage Amazon S3 for storing our data. This separates compute and storage layers, and allows multiple compute clusters to share the S3 data.
We have hundreds of petabytes of data and tens of thousands of Apache Hive tables. Our Presto clusters are comprised of a fleet of 450 r4.8xl EC2 instances. Presto clusters together have over 100 TBs of memory and 14K vcpu cores. Within Pinterest, we have close to more than 1,000 monthly active users (out of total 1,600+ Pinterest employees) using Presto, who run about 400K queries on these clusters per month.
Each query submitted to Presto cluster is logged to a Kafka topic via Singer. Singer is a logging agent built at Pinterest and we talked about it in a previous post. Each query is logged when it is submitted and when it finishes. When a Presto cluster crashes, we will have query submitted events without corresponding query finished events. These events enable us to capture the effect of cluster crashes over time.
Each Presto cluster at Pinterest has workers on a mix of dedicated AWS EC2 instances and Kubernetes pods. Kubernetes platform provides us with the capability to add and remove workers from a Presto cluster very quickly. The best-case latency on bringing up a new worker on Kubernetes is less than a minute. However, when the Kubernetes cluster itself is out of resources and needs to scale up, it can take up to ten minutes. Some other advantages of deploying on Kubernetes platform is that our Presto deployment becomes agnostic of cloud vendor, instance types, OS, etc.
#BigData #AWS #DataScience #DataEngineering
Amazon Kinesis
- Scalable9
- Cost3
related Amazon Kinesis posts
We are in the process of building a modern content platform to deliver our content through various channels. We decided to go with Microservices architecture as we wanted scale. Microservice architecture style is an approach to developing an application as a suite of small independently deployable services built around specific business capabilities. You can gain modularity, extensive parallelism and cost-effective scaling by deploying services across many distributed servers. Microservices modularity facilitates independent updates/deployments, and helps to avoid single point of failure, which can help prevent large-scale outages. We also decided to use Event Driven Architecture pattern which is a popular distributed asynchronous architecture pattern used to produce highly scalable applications. The event-driven architecture is made up of highly decoupled, single-purpose event processing components that asynchronously receive and process events.
To build our #Backend capabilities we decided to use the following: 1. #Microservices - Java with Spring Boot , Node.js with ExpressJS and Python with Flask 2. #Eventsourcingframework - Amazon Kinesis , Amazon Kinesis Firehose , Amazon SNS , Amazon SQS, AWS Lambda 3. #Data - Amazon RDS , Amazon DynamoDB , Amazon S3 , MongoDB Atlas
To build #Webapps we decided to use Angular 2 with RxJS
#Devops - GitHub , Travis CI , Terraform , Docker , Serverless
As we've evolved or added additional infrastructure to our stack, we've biased towards managed services. Most new backing stores are Amazon RDS instances now. We do use self-managed PostgreSQL with TimescaleDB for time-series data—this is made HA with the use of Patroni and Consul.
We also use managed Amazon ElastiCache instances instead of spinning up Amazon EC2 instances to run Redis workloads, as well as shifting to Amazon Kinesis instead of Kafka.
- Easy to use490
- Great tool369
- Makes developing rest api's easy peasy276
- Easy setup, looks good156
- The best api workflow out there144
- It's the best53
- History feature53
- Adds real value to my workflow44
- Great interface that magically predicts your needs43
- The best in class app35
- Can save and share script12
- Fully featured without looking cluttered10
- Collections8
- Option to run scrips8
- Global/Environment Variables8
- Shareable Collections7
- Dead simple and useful. Excellent7
- Dark theme easy on the eyes7
- Awesome customer support6
- Great integration with newman6
- Documentation5
- Simple5
- The test script is useful5
- Saves responses4
- This has simplified my testing significantly4
- Makes testing API's as easy as 1,2,34
- Easy as pie4
- API-network3
- I'd recommend it to everyone who works with apis3
- Mocking API calls with predefined response3
- Now supports GraphQL2
- Postman Runner CI Integration2
- Easy to setup, test and provides test storage2
- Continuous integration using newman2
- Pre-request Script and Test attributes are invaluable2
- Runner2
- Graph2
- <a href="http://fixbit.com/">useful tool</a>1
- Stores credentials in HTTP10
- Bloated features and UI9
- Cumbersome to switch authentication tokens8
- Poor GraphQL support7
- Expensive5
- Not free after 5 users3
- Can't prompt for per-request variables3
- Import swagger1
- Support websocket1
- Import curl1
related Postman posts
We just launched the Segment Config API (try it out for yourself here) — a set of public REST APIs that enable you to manage your Segment configuration. A public API is only as good as its #documentation. For the API reference doc we are using Postman.
Postman is an “API development environment”. You download the desktop app, and build API requests by URL and payload. Over time you can build up a set of requests and organize them into a “Postman Collection”. You can generalize a collection with “collection variables”. This allows you to parameterize things like username
, password
and workspace_name
so a user can fill their own values in before making an API call. This makes it possible to use Postman for one-off API tasks instead of writing code.
Then you can add Markdown content to the entire collection, a folder of related methods, and/or every API method to explain how the APIs work. You can publish a collection and easily share it with a URL.
This turns Postman from a personal #API utility to full-blown public interactive API documentation. The result is a great looking web page with all the API calls, docs and sample requests and responses in one place. Check out the results here.
Postman’s powers don’t end here. You can automate Postman with “test scripts” and have it periodically run a collection scripts as “monitors”. We now have #QA around all the APIs in public docs to make sure they are always correct
Along the way we tried other techniques for documenting APIs like ReadMe.io or Swagger UI. These required a lot of effort to customize.
Writing and maintaining a Postman collection takes some work, but the resulting documentation site, interactivity and API testing tools are well worth it.
Our whole Node.js backend stack consists of the following tools:
- Lerna as a tool for multi package and multi repository management
- npm as package manager
- NestJS as Node.js framework
- TypeScript as programming language
- ExpressJS as web server
- Swagger UI for visualizing and interacting with the API’s resources
- Postman as a tool for API development
- TypeORM as object relational mapping layer
- JSON Web Token for access token management
The main reason we have chosen Node.js over PHP is related to the following artifacts:
- Made for the web and widely in use: Node.js is a software platform for developing server-side network services. Well-known projects that rely on Node.js include the blogging software Ghost, the project management tool Trello and the operating system WebOS. Node.js requires the JavaScript runtime environment V8, which was specially developed by Google for the popular Chrome browser. This guarantees a very resource-saving architecture, which qualifies Node.js especially for the operation of a web server. Ryan Dahl, the developer of Node.js, released the first stable version on May 27, 2009. He developed Node.js out of dissatisfaction with the possibilities that JavaScript offered at the time. The basic functionality of Node.js has been mapped with JavaScript since the first version, which can be expanded with a large number of different modules. The current package managers (npm or Yarn) for Node.js know more than 1,000,000 of these modules.
- Fast server-side solutions: Node.js adopts the JavaScript "event-loop" to create non-blocking I/O applications that conveniently serve simultaneous events. With the standard available asynchronous processing within JavaScript/TypeScript, highly scalable, server-side solutions can be realized. The efficient use of the CPU and the RAM is maximized and more simultaneous requests can be processed than with conventional multi-thread servers.
- A language along the entire stack: Widely used frameworks such as React or AngularJS or Vue.js, which we prefer, are written in JavaScript/TypeScript. If Node.js is now used on the server side, you can use all the advantages of a uniform script language throughout the entire application development. The same language in the back- and frontend simplifies the maintenance of the application and also the coordination within the development team.
- Flexibility: Node.js sets very few strict dependencies, rules and guidelines and thus grants a high degree of flexibility in application development. There are no strict conventions so that the appropriate architecture, design structures, modules and features can be freely selected for the development.
- Easy to use490
- Great tool369
- Makes developing rest api's easy peasy276
- Easy setup, looks good156
- The best api workflow out there144
- It's the best53
- History feature53
- Adds real value to my workflow44
- Great interface that magically predicts your needs43
- The best in class app35
- Can save and share script12
- Fully featured without looking cluttered10
- Collections8
- Option to run scrips8
- Global/Environment Variables8
- Shareable Collections7
- Dead simple and useful. Excellent7
- Dark theme easy on the eyes7
- Awesome customer support6
- Great integration with newman6
- Documentation5
- Simple5
- The test script is useful5
- Saves responses4
- This has simplified my testing significantly4
- Makes testing API's as easy as 1,2,34
- Easy as pie4
- API-network3
- I'd recommend it to everyone who works with apis3
- Mocking API calls with predefined response3
- Now supports GraphQL2
- Postman Runner CI Integration2
- Easy to setup, test and provides test storage2
- Continuous integration using newman2
- Pre-request Script and Test attributes are invaluable2
- Runner2
- Graph2
- <a href="http://fixbit.com/">useful tool</a>1
- Stores credentials in HTTP10
- Bloated features and UI9
- Cumbersome to switch authentication tokens8
- Poor GraphQL support7
- Expensive5
- Not free after 5 users3
- Can't prompt for per-request variables3
- Import swagger1
- Support websocket1
- Import curl1
related Postman posts
We just launched the Segment Config API (try it out for yourself here) — a set of public REST APIs that enable you to manage your Segment configuration. A public API is only as good as its #documentation. For the API reference doc we are using Postman.
Postman is an “API development environment”. You download the desktop app, and build API requests by URL and payload. Over time you can build up a set of requests and organize them into a “Postman Collection”. You can generalize a collection with “collection variables”. This allows you to parameterize things like username
, password
and workspace_name
so a user can fill their own values in before making an API call. This makes it possible to use Postman for one-off API tasks instead of writing code.
Then you can add Markdown content to the entire collection, a folder of related methods, and/or every API method to explain how the APIs work. You can publish a collection and easily share it with a URL.
This turns Postman from a personal #API utility to full-blown public interactive API documentation. The result is a great looking web page with all the API calls, docs and sample requests and responses in one place. Check out the results here.
Postman’s powers don’t end here. You can automate Postman with “test scripts” and have it periodically run a collection scripts as “monitors”. We now have #QA around all the APIs in public docs to make sure they are always correct
Along the way we tried other techniques for documenting APIs like ReadMe.io or Swagger UI. These required a lot of effort to customize.
Writing and maintaining a Postman collection takes some work, but the resulting documentation site, interactivity and API testing tools are well worth it.
Our whole Node.js backend stack consists of the following tools:
- Lerna as a tool for multi package and multi repository management
- npm as package manager
- NestJS as Node.js framework
- TypeScript as programming language
- ExpressJS as web server
- Swagger UI for visualizing and interacting with the API’s resources
- Postman as a tool for API development
- TypeORM as object relational mapping layer
- JSON Web Token for access token management
The main reason we have chosen Node.js over PHP is related to the following artifacts:
- Made for the web and widely in use: Node.js is a software platform for developing server-side network services. Well-known projects that rely on Node.js include the blogging software Ghost, the project management tool Trello and the operating system WebOS. Node.js requires the JavaScript runtime environment V8, which was specially developed by Google for the popular Chrome browser. This guarantees a very resource-saving architecture, which qualifies Node.js especially for the operation of a web server. Ryan Dahl, the developer of Node.js, released the first stable version on May 27, 2009. He developed Node.js out of dissatisfaction with the possibilities that JavaScript offered at the time. The basic functionality of Node.js has been mapped with JavaScript since the first version, which can be expanded with a large number of different modules. The current package managers (npm or Yarn) for Node.js know more than 1,000,000 of these modules.
- Fast server-side solutions: Node.js adopts the JavaScript "event-loop" to create non-blocking I/O applications that conveniently serve simultaneous events. With the standard available asynchronous processing within JavaScript/TypeScript, highly scalable, server-side solutions can be realized. The efficient use of the CPU and the RAM is maximized and more simultaneous requests can be processed than with conventional multi-thread servers.
- A language along the entire stack: Widely used frameworks such as React or AngularJS or Vue.js, which we prefer, are written in JavaScript/TypeScript. If Node.js is now used on the server side, you can use all the advantages of a uniform script language throughout the entire application development. The same language in the back- and frontend simplifies the maintenance of the application and also the coordination within the development team.
- Flexibility: Node.js sets very few strict dependencies, rules and guidelines and thus grants a high degree of flexibility in application development. There are no strict conventions so that the appropriate architecture, design structures, modules and features can be freely selected for the development.
- Scary smart community257
- Knows all206
- Voting system142
- Good questions134
- Good SEO83
- Addictive22
- Tight focus14
- Share and gain knowledge10
- Useful7
- Fast loading3
- Gamification2
- Knows everyone1
- Experts share experience and answer questions1
- Stack overflow to developers As google to net surfers1
- Questions answered quickly1
- No annoying ads1
- No spam1
- Fast community response1
- Good moderators1
- Quick answers from users1
- Good answers1
- User reputation ranking1
- Efficient answers1
- Leading developer community1
- Not welcoming to newbies3
- Unfair downvoting3
- Unfriendly moderators3
- No opinion based questions3
- Mean users3
- Limited to types of questions it can accept2
related Stack Overflow posts
Google Analytics is a great tool to analyze your traffic. To debug our software and ask questions, we love to use Postman and Stack Overflow. Google Drive helps our team to share documents. We're able to build our great products through the APIs by Google Maps, CloudFlare, Stripe, PayPal, Twilio, Let's Encrypt, and TensorFlow.
- Free253
- Address input through maps api136
- Sharable Directions82
- Google Earth47
- Unique46
- Custom maps designing3
- Google Attributions and logo5
- Only map allowed alongside google place autocomplete2
related Google Maps posts
Google Analytics is a great tool to analyze your traffic. To debug our software and ask questions, we love to use Postman and Stack Overflow. Google Drive helps our team to share documents. We're able to build our great products through the APIs by Google Maps, CloudFlare, Stripe, PayPal, Twilio, Let's Encrypt, and TensorFlow.
A huge component of our product relies on gathering public data about locations of interest. Google Places API gives us that ability in the most efficient way. Since we are primarily going to be using as google data as a source of information for our MVP, we might as well start integrating the Google Places API in our system. We have worked with Google Maps in the past and we might take some inspiration from our previous projects onto this one.
- Powerful api329
- Great search engine315
- Open source231
- Restful214
- Near real-time search200
- Free98
- Search everything85
- Easy to get started54
- Analytics45
- Distributed26
- Fast search6
- More than a search engine5
- Awesome, great tool4
- Great docs4
- Highly Available3
- Easy to scale3
- Nosql DB2
- Document Store2
- Great customer support2
- Intuitive API2
- Reliable2
- Potato2
- Fast2
- Easy setup2
- Great piece of software2
- Open1
- Scalability1
- Not stable1
- Easy to get hot data1
- Github1
- Elaticsearch1
- Actively developing1
- Responsive maintainers on GitHub1
- Ecosystem1
- Community0
- Resource hungry7
- Diffecult to get started6
- Expensive5
- Hard to keep stable at large scale4
related Elasticsearch posts
We've been using PostgreSQL since the very early days of Zulip, but we actually didn't use it from the beginning. Zulip started out as a MySQL project back in 2012, because we'd heard it was a good choice for a startup with a wide community. However, we found that even though we were using the Django ORM for most of our database access, we spent a lot of time fighting with MySQL. Issues ranged from bad collation defaults, to bad query plans which required a lot of manual query tweaks.
We ended up getting so frustrated that we tried out PostgresQL, and the results were fantastic. We didn't have to do any real customization (just some tuning settings for how big a server we had), and all of our most important queries were faster out of the box. As a result, we were able to delete a bunch of custom queries escaping the ORM that we'd written to make the MySQL query planner happy (because postgres just did the right thing automatically).
And then after that, we've just gotten a ton of value out of postgres. We use its excellent built-in full-text search, which has helped us avoid needing to bring in a tool like Elasticsearch, and we've really enjoyed features like its partial indexes, which saved us a lot of work adding unnecessary extra tables to get good performance for things like our "unread messages" and "starred messages" indexes.
I can't recommend it highly enough.
Often enough I have to explain my way of going about setting up a CI/CD pipeline with multiple deployment platforms. Since I am a bit tired of yapping the same every single time, I've decided to write it up and share with the world this way, and send people to read it instead ;). I will explain it on "live-example" of how the Rome got built, basing that current methodology exists only of readme.md and wishes of good luck (as it usually is ;)).
It always starts with an app, whatever it may be and reading the readmes available while Vagrant and VirtualBox is installing and updating. Following that is the first hurdle to go over - convert all the instruction/scripts into Ansible playbook(s), and only stopping when doing a clear vagrant up
or vagrant reload
we will have a fully working environment. As our Vagrant environment is now functional, it's time to break it! This is the moment to look for how things can be done better (too rigid/too lose versioning? Sloppy environment setup?) and replace them with the right way to do stuff, one that won't bite us in the backside. This is the point, and the best opportunity, to upcycle the existing way of doing dev environment to produce a proper, production-grade product.
I should probably digress here for a moment and explain why. I firmly believe that the way you deploy production is the same way you should deploy develop, shy of few debugging-friendly setting. This way you avoid the discrepancy between how production work vs how development works, which almost always causes major pains in the back of the neck, and with use of proper tools should mean no more work for the developers. That's why we start with Vagrant as developer boxes should be as easy as vagrant up
, but the meat of our product lies in Ansible which will do meat of the work and can be applied to almost anything: AWS, bare metal, docker, LXC, in open net, behind vpn - you name it.
We must also give proper consideration to monitoring and logging hoovering at this point. My generic answer here is to grab Elasticsearch, Kibana, and Logstash. While for different use cases there may be better solutions, this one is well battle-tested, performs reasonably and is very easy to scale both vertically (within some limits) and horizontally. Logstash rules are easy to write and are well supported in maintenance through Ansible, which as I've mentioned earlier, are at the very core of things, and creating triggers/reports and alerts based on Elastic and Kibana is generally a breeze, including some quite complex aggregations.
If we are happy with the state of the Ansible it's time to move on and put all those roles and playbooks to work. Namely, we need something to manage our CI/CD pipelines. For me, the choice is obvious: TeamCity. It's modern, robust and unlike most of the light-weight alternatives, it's transparent. What I mean by that is that it doesn't tell you how to do things, doesn't limit your ways to deploy, or test, or package for that matter. Instead, it provides a developer-friendly and rich playground for your pipelines. You can do most the same with Jenkins, but it has a quite dated look and feel to it, while also missing some key functionality that must be brought in via plugins (like quality REST API which comes built-in with TeamCity). It also comes with all the common-handy plugins like Slack or Apache Maven integration.
The exact flow between CI and CD varies too greatly from one application to another to describe, so I will outline a few rules that guide me in it: 1. Make build steps as small as possible. This way when something breaks, we know exactly where, without needing to dig and root around. 2. All security credentials besides development environment must be sources from individual Vault instances. Keys to those containers should exist only on the CI/CD box and accessible by a few people (the less the better). This is pretty self-explanatory, as anything besides dev may contain sensitive data and, at times, be public-facing. Because of that appropriate security must be present. TeamCity shines in this department with excellent secrets-management. 3. Every part of the build chain shall consume and produce artifacts. If it creates nothing, it likely shouldn't be its own build. This way if any issue shows up with any environment or version, all developer has to do it is grab appropriate artifacts to reproduce the issue locally. 4. Deployment builds should be directly tied to specific Git branches/tags. This enables much easier tracking of what caused an issue, including automated identifying and tagging the author (nothing like automated regression testing!).
Speaking of deployments, I generally try to keep it simple but also with a close eye on the wallet. Because of that, I am more than happy with AWS or another cloud provider, but also constantly peeking at the loads and do we get the value of what we are paying for. Often enough the pattern of use is not constantly erratic, but rather has a firm baseline which could be migrated away from the cloud and into bare metal boxes. That is another part where this approach strongly triumphs over the common Docker and CircleCI setup, where you are very much tied in to use cloud providers and getting out is expensive. Here to embrace bare-metal hosting all you need is a help of some container-based self-hosting software, my personal preference is with Proxmox and LXC. Following that all you must write are ansible scripts to manage hardware of Proxmox, similar way as you do for Amazon EC2 (ansible supports both greatly) and you are good to go. One does not exclude another, quite the opposite, as they can live in great synergy and cut your costs dramatically (the heavier your base load, the bigger the savings) while providing production-grade resiliency.