When creating the web infrastructure for our start-up, I wanted to host our app on a PaaS to get started quickly.
A very popular one for Rails is Heroku, which I love for free hobby side projects, but never used professionally. On the other hand, I was very familiar with the AWS ecosystem, and since I was going to use some of its services anyways, I thought: why not go all in on it?
It turns out that Amazon offers a PaaS called AWS Elastic Beanstalk, which is basically like an “AWS Heroku”. It even comes with a similar command-line utility, called "eb”. While edge-case Rails problems are not as well documented as with Heroku, it was very satisfying to manage all our cloud services under the same AWS account. There are auto-scaling options for web and worker instances, which is a nice touch. Overall, it was reliable, and I would recommend it to anyone planning on heavily using AWS.
Earlier this year we decided to go "all-in" on GraphQL to provide our front-end API. We needed a stable client library to power our React app. We decided to use Apollo Client for a few reasons:
1) Stability 2) Maturity 3) Heaps of features 4) Great documentation (with use cases) 5) Support for server-side rendering 6) Allowed us to stop using Redux and Mobx
Overall we've had great success with this library, along with a few minor hiccups and work arounds, but no show stoppers. If you are coming from Redux.js land, it takes a bit of time to settle into a new way of thinking about how your data is fetched and flows through your React app. This part has been the biggest learning curve of anything to do with GraphQL.
One of the downsides to Apollo Client, once you build a larger application, (past the size of most of the documented use cases and sample apps) the state management tends to get distributed through various places; and not just components. Apollo Client has a state management feature that relies on a normalised local cache. Mastering the knowledge of how this works is key to getting the most out of the library and to architecting your component hierarchy properly.
But customization can only get you so far, and there were little things that I still had to use the mouse for, such as scrolling, repositioning lines on the screen, selecting the line number of a failing test stack trace from a separate plugin pane, etc. After 3 years of wearily moving my arm and hand to perform the same repetitive tasks, I decided to switch to Vim for 3 reasons:
The learning curve is very steep and it took me a year to master it, but investing time to be truly comfortable with my #TextEditor was more than worth it. To me, Vim comes close to being the perfect editor and I probably won’t need to switch ever again. It feels good to ignore new editors that come out every few years, like Atom and Visual Studio Code.
At Heap, we searched for an existing tool that would allow us to express the full range of analyses we needed, index the event definitions that made up the analyses, and was a mature, natively distributed system.
After coming up empty on this search, we decided to compromise on the “maturity” requirement and build our own distributed system around Citus and sharded PostgreSQL. It was at this point that we also introduced Kafka as a queueing layer between the Node.js application servers and Postgres.
If we could go back in time, we probably would have started using Kafka on day one. One of the biggest benefits in adopting Kafka has been the peace of mind that it brings. In an analytics infrastructure, it’s often possible to make data ingestion idempotent.
In Heap’s case, that means that, if anything downstream from Kafka goes down, we won’t lose any data – it’s just going to take a bit longer to get to its destination. We also learned that you want the path between data hitting your servers and your initial persistence layer (in this case, Kafka) to be as short and simple as possible, since that is the surface area where a failure means you can lose customer data. We learned that it’s a very good fit for an analytics tool, since you can handle a huge number of incoming writes with relatively low latency. Kafka also gives you the ability to “replay” the data flow: it’s like a commit log for your whole infrastructure.
#MessageQueue #Databases #FrameworksFullStack
We recently needed to rebuild our documentation site, currently built using Jekyll hosted on GitHub Pages. We wanted to update the content and refresh the style to make it easier to find answers.
We considered hosted services that could accept our markdown content, like ReadMe.io and Read the Docs, however both seemed expensive for essentially hosting the same platform we already had for free.
I also looked at the Gatsby Static Site generator to modernize Jekyll. I don't think this is a fit, as our documentation is relatively simple and relies heavily on Markdown. Jekyll excels at Markdown, while Gatsby seemed to struggle with it.