Need advice about which tool to choose?Ask the StackShare community!
Concourse vs Terraform: What are the differences?
Introduction
In this markdown, we will discuss the key differences between Concourse and Terraform, two popular tools used in the DevOps space.
Installation and Configuration: Concourse is mainly designed as a CI/CD (Continuous Integration/Continuous Deployment) tool, focused on automation pipelines. It requires its own server and workers, and its installation can be more complex compared to Terraform. On the other hand, Terraform is a provisioning and infrastructure-as-code tool, which only requires local installation and configuration.
Infrastructure Management: Terraform is primarily used for managing infrastructure resources such as virtual machines, networks, storage, and other cloud services. It allows users to create, modify, and destroy infrastructure resources using a declarative language. Concourse, on the other hand, is not specifically designed for infrastructure management and focuses more on CI/CD pipelines, automation, and workflow management.
Cloud Provider Support: Terraform provides extensive support for multiple cloud providers, including AWS, Azure, Google Cloud, and many more. It offers a wide range of resource and provider configurations for various cloud services. Concourse, being primarily a CI/CD tool, is not tied to any specific cloud provider but can be used alongside Terraform to deploy and manage infrastructure resources.
Pipeline Definition: Concourse uses a YAML-based pipeline definition file to define the CI/CD workflows and automation tasks. It allows users to define individual jobs, tasks, and resources in a structured and reusable manner. On the other hand, Terraform uses its own declarative language called HashiCorp Configuration Language (HCL) for defining infrastructure resources. The HCL language provides higher-level abstraction and allows users to express infrastructure configurations in a readable and maintainable manner.
Version Control Integration: Both Concourse and Terraform provide integration with version control systems like Git. However, the level of integration and the usage may vary. Concourse pipelines can be defined and version-controlled alongside application code, enabling developers to have proper versioning and history of pipeline changes. Terraform also allows users to store and share infrastructure configurations in version control systems but focuses more on managing the infrastructure state file rather than the pipeline itself.
In summary, Concourse is a powerful CI/CD tool focused on automation and workflow management, while Terraform is an infrastructure-as-code tool used for provisioning and managing infrastructure resources. Each tool has its own strengths and use cases, with Concourse being more suited for CI/CD pipelines and Terraform for infrastructure management.
I'm planning to setup complete CD-CD setup for spark and python application which we are going to deploy in aws lambda and EMR Cluster. Which tool would be best one to choose. Since my company is trying to adopt to concourse i would like to understand what are the lack of capabilities concourse have . Thanks in advance !
I would definetly recommend Concourse to you, as it is one of the most advanced modern methods of making CI/CD while Jenkins is an old monolithic dinosaur. Concourse itself is cloudnative and containerbased which helps you to build simple, high-performance and scalable CI/CD pipelines. In my opinion, the only lack of skills you have with Concourse is your own knowledge of how to build pipelines and automate things. Technincally there is no lack, i would even say you can extend it way more easily. But as a Con it is more easy to interact with Jenkins if you are only used to UIs. Concourse needs someone which is capable of using CLIs.
Ok, so first - AWS Copilot is CloudFormation under the hood, but the way it works results in you not thinking about CFN anymore. AWS found the right balance with Copilot - it's insanely simple to setup production-ready multi-account environment with many services inside, with CI/CD out of the box etc etc. It's pretty new, but even now it was enough to launch Transcripto, which uses may be a dozen of different AWS services, all bound together by Copilot.
Because Pulumi uses real programming languages, you can actually write abstractions for your infrastructure code, which is incredibly empowering. You still 'describe' your desired state, but by having a programming language at your fingers, you can factor out patterns, and package it up for easier consumption.
We use Terraform to manage AWS cloud environment for the project. It is pretty complex, largely static, security-focused, and constantly evolving.
Terraform provides descriptive (declarative) way of defining the target configuration, where it can work out the dependencies between configuration elements and apply differences without re-provisioning the entire cloud stack.
AdvantagesTerraform is vendor-neutral in a way that it is using a common configuration language (HCL) with plugins (providers) for multiple cloud and service providers.
Terraform keeps track of the previous state of the deployment and applies incremental changes, resulting in faster deployment times.
Terraform allows us to share reusable modules between projects. We have built an impressive library of modules internally, which makes it very easy to assemble a new project from pre-fabricated building blocks.
DisadvantagesSoftware is imperfect, and Terraform is no exception. Occasionally we hit annoying bugs that we have to work around. The interaction with any underlying APIs is encapsulated inside 3rd party Terraform providers, and any bug fixes or new features require a provider release. Some providers have very poor coverage of the underlying APIs.
Terraform is not great for managing highly dynamic parts of cloud environments. That part is better delegated to other tools or scripts.
Terraform state may go out of sync with the target environment or with the source configuration, which often results in painful reconciliation.
I personally am not a huge fan of vendor lock in for multiple reasons:
- I've seen cost saving moves to the cloud end up costing a fortune and trapping companies due to over utilization of cloud specific features.
- I've seen S3 failures nearly take down half the internet.
- I've seen companies get stuck in the cloud because they aren't built cloud agnostic.
I choose to use terraform for my cloud provisioning for these reasons:
- It's cloud agnostic so I can use it no matter where I am.
- It isn't difficult to use and uses a relatively easy to read language.
- It tests infrastructure before running it, and enables me to see and keep changes up to date.
- It runs from the same CLI I do most of my CM work from.
Context: I wanted to create an end to end IoT data pipeline simulation in Google Cloud IoT Core and other GCP services. I never touched Terraform meaningfully until working on this project, and it's one of the best explorations in my development career. The documentation and syntax is incredibly human-readable and friendly. I'm used to building infrastructure through the google apis via Python , but I'm so glad past Sung did not make that decision. I was tempted to use Google Cloud Deployment Manager, but the templates were a bit convoluted by first impression. I'm glad past Sung did not make this decision either.
Solution: Leveraging Google Cloud Build Google Cloud Run Google Cloud Bigtable Google BigQuery Google Cloud Storage Google Compute Engine along with some other fun tools, I can deploy over 40 GCP resources using Terraform!
Check Out My Architecture: CLICK ME
Check out the GitHub repo attached
Pros of Concourse
- Real pipelines16
- Containerised builds10
- Flexible engine9
- Fast6
- Open source4
- No Snowflakes3
- Simple configuration management3
- You have to do everything2
- Fancy Visualization1
Pros of Terraform
- Infrastructure as code121
- Declarative syntax73
- Planning45
- Simple28
- Parallelism24
- Well-documented8
- Cloud agnostic8
- It's like coding your infrastructure in simple English6
- Immutable infrastructure6
- Platform agnostic5
- Extendable4
- Automation4
- Automates infrastructure deployments4
- Portability4
- Lightweight2
- Scales to hundreds of hosts2
Sign up to add or upvote prosMake informed product decisions
Cons of Concourse
- Fail forward instead of rollback pattern2
Cons of Terraform
- Doesn't have full support to GKE1