Heap

Segment has made it a no-brainer to integrate with third-party scripts and services, and has saved us from doing pointless redeploys just to change the It gives you the granularity to toggle services on different environments without having to make any code changes.

It's also a great platform for discovering SaaS products that you could add to your own – just by browsing their catalog, I've discovered tools we now currently use to augment our main product. Here are a few:

• @{Heap}|tool:588|: We use Heap for our product analytics. Heap's philosophy is to gather events from multiple sources, and then organize and graph segments to form your own business insights. They have a few starter graphs like DAU and retention to help you get started.
• @{Hotjar}|tool:2207|: If a picture's worth a thousand words, than a video is worth 1000 * 30fps = 30k words per second. Hotjar gives us videos of user sessions so we can pinpoint problems that aren't necessarily JS exceptions – say, logical errors in a UX flow – that we'd otherwise miss.
• @{Bugsnag}|tool:150|: Bugsnag has been a big help in catching run-time errors that our users encounter. Their Slack integration pings us when something goes wrong (which we can control if we want to notified on all bugs or just new bugs), and their source map uploader means that we don't have to debug minified code.

At Heap, we searched for an existing tool that would allow us to express the full range of analyses we needed, index the event definitions that made up the analyses, and was a mature, natively distributed system.

After coming up empty on this search, we decided to compromise on the “maturity” requirement and build our own distributed system around Citus and sharded PostgreSQL. It was at this point that we also introduced Kafka as a queueing layer between the Node.js application servers and Postgres.

If we could go back in time, we probably would have started using Kafka on day one. One of the biggest benefits in adopting Kafka has been the peace of mind that it brings. In an analytics infrastructure, it’s often possible to make data ingestion idempotent.

In Heap’s case, that means that, if anything downstream from Kafka goes down, we won’t lose any data – it’s just going to take a bit longer to get to its destination. We also learned that you want the path between data hitting your servers and your initial persistence layer (in this case, Kafka) to be as short and simple as possible, since that is the surface area where a failure means you can lose customer data. We learned that it’s a very good fit for an analytics tool, since you can handle a huge number of incoming writes with relatively low latency. Kafka also gives you the ability to “replay” the data flow: it’s like a commit log for your whole infrastructure.

#MessageQueue #Databases #FrameworksFullStack