Heap searched for an existing tool that would allow them to express the full range of analyses they needed, index the event definitions that made up the analyses, and was a mature, natively distributed system.
After coming up empty on this search, they decided to compromise on the “maturity” requirement and build their own distributed system around Citus and sharded PostgreSQL. It was at this point that they also introduced Kafka as a queueing layer between the Node.js application servers and Postgres.
If we could go back in time, we probably would have started using Kafka on day one. One of the biggest benefits in adopting Kafka has been the peace of mind that it brings. In an analytics infrastructure, it’s often possible to make data ingestion idempotent. In Heap’s case, that means that, if anything downstream from Kafka goes down, they won’t lose any data – it’s just going to take a bit longer to get to its destination. He’s also learned that you want the path between data hitting your servers and your initial persistence layer (in this case, Kafka) to be as short and simple as possible, since that is the surface area where a failure means you can lose customer data. We learned that it’s a very good fit for an analytics tool, since you can handle a huge number of incoming writes with relatively low latency. Kafka also gives you the ability to “replay” the data flow: “It’s like a commit log for your whole infrastructure
#MessageQueue #Databases #FrameworksFullStack