Avatar of Glenn Gillen

Glenn Gillen

Glenn Gillen
Glenn Gillen
Recommends
BuildkiteBuildkite

I think I've tried most of the CI tools out there at some point. It took me a while to get around to Buildkite because at first I didn't see much point given it seemed like you had to run the agent yourself. Eventually it dawned on me why this approach was more ingenious than I realised:

Running my app in a production (or production-like) environment was already a solved problem, because everything was already in some form of "everything as code". Having a test environment where the only difference was adding the Buildkite agent was a trivial addition.

It means that dev/test/prod parity is simple to achieve and maintain. It's also proven to be much easier to support than trying to deal with the problems that come with trying to force an app to fit into the nuances and constraints that are imposed by the containers/runtime of a CI service. When you completely control all of the environment the tests are running in you define those constraints too. It's been a great balance between a managed service and the flexibility of running it yourself.

And while none of my needs have hit the scale of Shopify (I saw one of their engineers speak about it at a conference once, I can't find the video now though 馃槥) it's good to know I can scale out my worker nodes to hundreds of thousands of workers to reduce the time it takes for my tests to run.

READ MORE
5 upvotes1 comment72.6K views
Stratos Xakoustos
Stratos Xakoustos
January 21st 2020 at 3:38PM

WOW, Buildkite looks great! I'll throw that into consideration, thanks.

Reply
Glenn Gillen

I'm currently building out a Twitter analysis tool that's using AWS Lambda to stream data into Amazon Kinesis Firehose, which in turns saves the result to Amazon S3. The plan is to have Amazon S3 operate as both a data store and quasi-messaging bus with any post-processing work (e.g., notifications of new tweets going into Slack) fanning out from there. I went with this approach as I can get things up and running quickly and only pay for things on a pay-per-use basis rather than having lots of worker nodes sitting around waiting for work. Amazon Kinesis Firehose also makes it easy to add a different or additional data store in the future.

READ MORE
4 upvotes4.9K views