Avatar of Robert Zuber

Robert Zuber

CTO at CircleCI
CTO at CircleCI·

We use MongoDB as our primary #datastore. Mongo's approach to replica sets enables some fantastic patterns for operations like maintenance, backups, and #ETL.

As we pull #microservices from our #monolith, we are taking the opportunity to build them with their own datastores using PostgreSQL. We also use Redis to cache data we’d never store permanently, and to rate-limit our requests to partners’ APIs (like GitHub).

When we’re dealing with large blobs of immutable data (logs, artifacts, and test results), we store them in Amazon S3. We handle any side-effects of S3’s eventual consistency model within our own code. This ensures that we deal with user requests correctly while writes are in process.

READ MORE
Update: How CircleCI Processes Over 30 Million Builds Per Month - CircleCI Tech Stack (stackshare.io)
24 upvotes·2 comments·3.1M views
Manohar Sitapati
Manohar Sitapati
·
August 7th 2019 at 2:36PM

Very cool stack!

·
Reply
David Arcos
David Arcos
·
January 30th 2020 at 1:37PM

Impressive!

·
Reply
CTO at CircleCI·

We are in the process of adopting Next.js as our React framework and using Storybook to help build our React components in isolation. This new part of our frontend is written in TypeScript, and we use Emotion for CSS/styling. For delivering data, we use GraphQL and Apollo. Jest, Percy, and Cypress are used for testing.

READ MORE
Update: How CircleCI Processes Over 30 Million Builds Per Month - CircleCI Tech Stack (stackshare.io)
17 upvotes·5 comments·1.8M views
Guillermo Rauch
Guillermo Rauch
·
July 25th 2019 at 7:08PM

What an incredible endorsement! This makes the ZEIT team very proud! @z00b feel free to reach out to me if you need any help or hit any roadblocks.

·
Reply
Aron Griffis
Aron Griffis
·
July 26th 2019 at 1:14PM

What are you deprecating or migrating away from?

·
Reply
Ron Powell
Ron Powell
·
July 26th 2019 at 8:08PM

Our web app’s UI was written in ClojureScript. We were using the framework Om as our cljs interface to React.

·
Reply
Aron Griffis
Aron Griffis
·
July 26th 2019 at 9:00PM

Thanks Ron. I wondered about that. I'd love to hear more about what factors in particular influenced you to move away from ClojureScript and Om toward TypeScript and React, especially since it sounds like you're sticking with Clojure on the server.

·
Reply
Magnolia Triplett
Magnolia Triplett
·
July 31st 2019 at 3:32PM

Thanks Robert - so glad we could help you and the CircleCI team with your testing workflow! Let us know if you need any assistance along the way.

·
Reply
CTO at CircleCI·

Our primary source of monitoring and alerting is Datadog. We’ve got prebuilt dashboards for every scenario and integration with PagerDuty to manage routing any alerts. We’ve definitely scaled past the point where managing dashboards is easy, but we haven’t had time to invest in using features like Anomaly Detection. We’ve started using Honeycomb for some targeted debugging of complex production issues and we are liking what we’ve seen. We capture any unhandled exceptions with Rollbar and, if we realize one will keep happening, we quickly convert the metrics to point back to Datadog, to keep Rollbar as clean as possible.

We use Segment to consolidate all of our trackers, the most important of which goes to Amplitude to analyze user patterns. However, if we need a more consolidated view, we push all of our data to our own data warehouse running PostgreSQL; this is available for analytics and dashboard creation through Looker.

READ MORE
Update: How CircleCI Processes Over 30 Million Builds Per Month - CircleCI Tech Stack (stackshare.io)
12 upvotes·1 comment·1.5M views
Lucas Rincon
Lucas Rincon
·
July 2nd 2020 at 9:33AM

curious what kind of investigating you and your team did into monitoring tools and what were the things that drew you to Data Dog?

·
Reply
CTO at CircleCI·
Shared insights
on
CircleCICircleCIClojureClojureRailsRails
at

Most of CircleCI is written in Clojure and it has been this way since almost the beginning. Early development included Rails, but by the time that CircleCI was released to the public, it was written entirely in Clojure. Clojure is still at our platform’s core. It helps having a common language across much of our stack to allow for our engineers to move between layers of the stack without much overhead.

READ MORE
Update: How CircleCI Processes Over 30 Million Builds Per Month - CircleCI Tech Stack (stackshare.io)
10 upvotes·47.4K views
CTO at CircleCI·

Our backend consists of two major pools of machines. One pool hosts the systems that run our site, manage jobs, and send notifications. These services are deployed within Docker containers orchestrated in Kubernetes. Due to Kubernetes’ ecosystem and toolchain, it was an obvious choice for our fairly statically-defined processes: the rate of change of job types or how many we may need in our internal stack is relatively low.

The other pool of machines is for running our users’ jobs. Because we cannot dynamically predict demand, what types of jobs our users need to have run, nor the resources required for each of those jobs, we found that Nomad excelled over Kubernetes in this area.

We’re also using Helm to make it easier to deploy new services into Kubernetes. We create a chart (i.e. package) for each service. This lets us easily roll back new software and gives us an audit trail of what was installed or upgraded.

READ MORE
Update: How CircleCI Processes Over 30 Million Builds Per Month - CircleCI Tech Stack (stackshare.io)
7 upvotes·319.3K views
CTO at CircleCI·

When you interact with CircleCI's web application, all of your requests are hitting the #API hosts. We handle the majority of our authentication via #OAuth from GitHub or Bitbucket. We provide programmatic access to everything exposed in the UI through an API token that you can generate once you have authenticated.

READ MORE
Update: How CircleCI Processes Over 30 Million Builds Per Month - CircleCI Tech Stack (stackshare.io)
5 upvotes·12K views
CTO at CircleCI·

We have added very little to the CoffeeScript Hubot application – just enough to allow it to talk to our Hubot workers. The Hubot workers implement our operational management functionality and expose it to Hubot so we can get chat integration for free. We’ve also tailored the authentication and authorization code of Hubot to meet the needs of roles within our team.

For larger tasks, we’ve got an internal #CLI written in Go that talks to the same #API as Hubot, giving access to the same functionality we have in Slack, with the addition of scripting, piping, and all of our favorite #Unix tools. When the Hubot worker recognizes the CLI is in use, it logs the commands to Slack to maintain visibility of operational changes.

READ MORE
Update: How CircleCI Processes Over 30 Million Builds Per Month - CircleCI Tech Stack (stackshare.io)
4 upvotes·22.3K views