How Heap Built an Analytics Platform that Auto-Tracks Every User Event

10,155
Heap
Heap automatically captures every user action in your app and lets you measure it all. Clicks, taps, swipes, form submissions, page views, and more. Track events and segment users instantly. No pushing code. No waiting for data to trickle in.

Written by Paul Jaworski and adapted from an interview with Dan Robinson, CTO of Heap




Dan Robinson, CTO of Heap Dan Robinson, CTO of Heap


When Dan Robinson joined Heap as the company's first engineer, it was unclear whether it was even possible to build the product to scale. And that's exactly why he joined. Most startups, he says, face a significant risk in finding product-market fit. He was absolutely confident that a need for this product existed. The real challenge would be a technical one.

Most analytics platforms require the user to choose the events they want tracked ahead of time. This requires significant developer time and the foresight to know which analytics events you'll care about later. Heap instead tracks everything up front and lets the user define events with a visual tool afterward.


Humble Beginnings

Heap was founded by Matin Movassate, a former Product Manager at Facebook, and Ravi Parikh. They entered the Winter '11 class of YCombinator with a simple MVP of the product: a single Node.js server running on EC2 with PostgreSQL. All of the persistent data had to be mirrored in memory for the queries to be fast. Dan joined soon after, and his first project was to rebuild the infrastructure to be able to handle more than ~200gb of data.

The early version of the product was a JavaScript snippet that customers could install on their site, tracking UI events. This was accompanied by a customer-facing dashboard written in jQuery and D3.js where users could graph their data, create conversion funnels, or do things like click on a recent user and see every single event Heap has tracked for them. These analyses were transformed into SQL queries for Postgres on the backend. Much of the data still had to be stored in memory for the queries to be fast enough to satisfy consumers, though they knew this was not a long-term solution.


Matin Movassate cutting Dan's welcome cake CEO Matin Movassate cutting Dan's "welcome cake"


PostgreSQL was an easy early decision for the founding team. The relational data model fit the types of analyses they would be doing: filtering, grouping, joining, etc., and it was the database they knew best. Shortly after adopting PG, they discovered Citus, which is a tool that makes it easy to distribute queries. Although it was a young project and a fork of Postgres at that point, Dan says the team was very available, highly expert, and it wouldn’t be very difficult to move back to PG if they needed to:

The stuff they forked was in query execution. You could treat the worker nodes like regular PG instances.

Citus also gave them a ton of flexibility to make queries fast, and again, they felt the data model was the best fit for their application.


Growing Pains

In early 2014, Heap released an event visualizer tool that allowed non-technical people to use the product, which Dan believes was the key piece in achieving product-market fit. As a result, the company grew to the point where they had users who were processing millions of events per month. They started to hit the limits of the initial Citus infrastructure. As larger customers began signing up, the large datasets they brought with them became difficult to handle. Eventually, the analyses became too slow to be viable, and simply “throwing more machines” at the problem was cost-prohibitive.

The early version Heap was using didn’t have much distributed systems functionality, so they had rolled their own solutions for things like recovering from a failed node, splitting data into different sharding schemes, and moving data between machines. That homegrown functionality was starting to have issues at scale.

The major breakthrough came when they found a way to cheaply index the event definitions users were creating. These were the only points of data that users were querying, and each event definition represented far less than 1% of the overall data Heap was collecting. It became clear that they could achieve substantial performance gains if they could build an infrastructure around indexing these events.

Heap searched for an existing tool that would allow them to express the full range of analyses they needed, index the event definitions that made up the analyses, and was a mature, natively distributed system. After coming up empty on this search, they decided to compromise on the “maturity” requirement and build their own distributed system around Citus and sharded PostgreSQL. It was at this point that they also introduced Kafka as a queueing layer between the Node.js application servers and Postgres.

The front end had also begun to grow unwieldy. The original jQuery pieces became difficult to maintain and scale, and a decision was made to introduce Backbone, Marionette, and TypeScript. Ultimately this ended up being a “detour” in the search for a scalable and maintainable front-end solution. The system did allow for developers to reuse components efficiently, but adding features was a difficult process, and it eventually became a bottleneck in advancing the product.


Reducing Cost and Improving Performance

Because of the massive amounts of data that Heap is ingesting, it’s taken a great deal of work to get to a cost-viable product. One of the major projects in reducing cost involved switching to ZFS, which allows compression at the file system level. That switch alone allowed them to compress their data by a factor of 2. Dan says they’re currently experimenting with even further improvements that could increase this compression to 3-3.5x. Additional gains have come from doing some low-level CPU profiling” to determine where their resources were being used on the EC2 instances.

Today, they’re doubling query speed each quarter and constantly seeking even more improvements. As Dan points out, the size of the customer they can support is directly correlated to the performance of the application.

If we can make queries 3x faster, we can support a customer who is 3x larger.


DevOps and Organizational Structure

Engineering teams at Heap are broken into 3-5 developers, and about half of them are working on infrastructure. This is mostly related to business priorities, since again, the primary challenge behind the product is not, “What new features should we add?” but “How do we scale to customers who are 100x larger?”.


Dan and engineers Dan with members of the engineering team


All of the code at Heap lives on GitHub, and they use CircleCI, which in turn kicks off Ansible scripts for deployment. They use Salt for managing machine configuration, and Terraform to manage all of their AWS configuration. Currently, everything is running on AWS for Heap, so Terraform was an easy choice, as they loved the modularity and “great dev workflows” it provides.


Lessons Learned

After 5 years of building one of the fastest growing tools for analytics and ingesting billions of events, Dan Robinson has some valuable advice for budding CTOs:

There are decisions you make that are hard to reverse and decisions that are easy to reverse - like one-way doors and two-way doors. Most things are two-way doors and it's better to just go fast and learn something.

He advises that if you’re building a serious distributed system where the performance is critical to the success of your application, one-way doors include things like selecting your data model and data system:

If you want to change from PostgreSQL to MySQL, that's going to be a rewrite.

If he could go back in time, Dan probably would have started using Kafka on day one. He’s learned that it’s a very good fit for an analytics tool, since you can handle a huge number of incoming writes with relatively low latency. Kafka also gives you the ability to “replay” the data flow: “It’s like a commit log for your whole infrastructure.”


Drawing out the current infrastructure


One of the biggest benefits in adopting Kafka has been the peace of mind that it brings. In an analytics infrastructure, it’s often possible to make data ingestion idempotent. In Heap’s case, that means that, if anything downstream from Kafka goes down, they won’t lose any data – it’s just going to take a bit longer to get to its destination. He’s also learned that you want the path between data hitting your servers and your initial persistence layer (in this case, Kafka) to be as short and simple as possible, since that is the surface area where a failure means you can lose customer data.

Dan also says he’s been “continuously shocked” at how often YAGNI has been true:

I remember writing our exports feature in 2014. It was a simple feature that let you get a nightly dump of your Heap data on S3. The code was littered with TODOs that I was completely sure we were going to need to resolve within the next few weeks – minor extensions of the feature, configurability options, operability improvements, or known technical debt items. A lot of those TODOs didn't come up for years, and some of them are still there.

Instead of focusing on writing perfect software, he believes it’s much more important to get something in front of users. This is something you may have heard by now from a product perspective, but it’s equally important for code:

You’ll learn what areas of your technical debt actually matter, and that learning is a lot more important than getting decisions right if the decisions are easily reversible.


The Present and Future of Heap

Today, the Heap product consists primarily of a customer-facing dashboard powered by React, MobX, and TypeScript on the front end. Data is sent up to a Node.js server, passed on to Kafka, and eventually ends up in PostgreSQL. All of the data customers perform analyses on in the dashboard comes, of course, from the JavaScript snippet installed on users’ websites.

Most recently, Heap has released a feature that also pulls in data from third-party providers like MailChimp, Stripe, Optimizely, and Shopify. The team there realized that a large percentage of their customers’ data didn’t actually live on their own platform, but instead was scattered around these various vendors.

Dan points out that the additional data volume hasn’t been a significant challenge - especially since events like payments are very high value from an analytics perspective. The real challenge has been learning the “language” of these providers. Each one has a different API, a different way to format event data, and different semantics for retrieving it.

Once the data is in Heap, they also had to figure out how to correlate that data with their own. How do you attribute an email sent in one system to a button click in another? The answer was building out their own UI for Heap users to tie the events together.

Beyond features, performance improvements continue to be a major focus today and in the future. Heap currently stores 1 petabyte of data, ingests 1 billion events per day, and performs over 250,000 analyses per week.




If your company has a great story behind your tech, email us to be featured!

Heap
Heap automatically captures every user action in your app and lets you measure it all. Clicks, taps, swipes, form submissions, page views, and more. Track events and segment users instantly. No pushing code. No waiting for data to trickle in.
Tools mentioned in article
Open jobs at Heap
Senior Product Manager, Developer Exp...
<div> <div id="content" class="highlighter-context page view" data-inline-comments-target="true" data-testid="page-content-only"> <div class="_1bsb1osq _19pkidpf _2hwx1wug _otyridpf _18u01wug"> <div> <div id="main-content" class="wiki-content css-q12xh8 e5xcnr80" data-testid="pageContentRendererTestId" data-test-appearance="full-page"> <div class="renderer-overrides"> <div class="ak-renderer-wrapper css-2c6ch1"> <div class="css-85nbx0"> <div class="ak-renderer-document"> <p data-renderer-start-pos="35">At Heap, we’re building the next generation of product analytics, for growth-minded teams. Product teams need easy, fast access to a suite of <span id="be277289-2719-4f39-881e-84d197a50f51" data-renderer-mark="true" data-mark-type="annotation" data-mark-annotation-type="inlineComment" data-id="be277289-2719-4f39-881e-84d197a50f51">quantitative and qualitative</span> tools for a complete and accurate understanding of the user experience so they can make the best investment decisions and create delightful experiences.</p> <p data-renderer-start-pos="362">Today, user experiences are across channels, devices, time and space! As Product Manager for the Mobile team you’ll build tools to help users understand the <em data-renderer-mark="true">full</em> user journey. Heap’s built with cross-platform analysis baked in with an event definition layer that’s resilient to constant changes and easy to use across multiple products and platforms. Mobile devices have unique challenges and better understanding these user experiences is critical to our customers.</p> <p data-renderer-start-pos="830">Mobile is part of the Capture group which also drives developer experience. We know that developers are a critical user and champion of Heap and data-driven cultures. We build best-in-class tools for capturing data that make things easier and more delightful for developers.</p> <p data-renderer-start-pos="1106">&nbsp;</p> <p data-renderer-start-pos="1108"><strong data-renderer-mark="true">What you'll do:&nbsp;</strong></p> <p data-renderer-start-pos="1126">Drive impact. Work with Heap’s engineering and design leadership to define measurable long-term objectives that have tangible business impact. Then define a hypothesis-driven roadmap aimed at achieving that impact. Work with engineering and design teams to navigate trade-offs during development, and make sure that we learn from every launch.</p> <p data-renderer-start-pos="1471">Innovate in Mobile Analytics. Help us bring a complete picture of user data to our customers and best practices to the broader market. Understand users (especially developers), using qualitative and quantitative methods, to innovate on and extend our mobile products. You’ll help shape the future of Product Management for mobile with <span id="be277289-2719-4f39-881e-84d197a50f51" data-renderer-mark="true" data-mark-type="annotation" data-mark-annotation-type="inlineComment" data-id="be277289-2719-4f39-881e-84d197a50f51">quantitative and qualitative</span> tools and data-driven methodologies.&nbsp;</p> <p data-renderer-start-pos="1874">Delight Developers. Our capture team builds development tools to help teams capture behavioral data in a robust and secure way while maintaining user privacy. You’ll help us as we extend more of these tools to mobile and delight developers. You’ll drive our efforts around developer experience: documentation, open source strategy, developer community, and more.</p> <p data-renderer-start-pos="2238">&nbsp;</p> <p data-renderer-start-pos="2240"><strong data-renderer-mark="true">What we're looking for:&nbsp;</strong></p> <ul class="ak-ul" data-indent-level="1"> <li> <p data-renderer-start-pos="2268">5+ years of product management experience</p> </li> <li> <p data-renderer-start-pos="2313">Ability to clearly generate and articulate the vision, roadmap, and prioritization for the Mobile area. Key stakeholders include engineering / design and customer-facing teams (e.g. customer success) as well as company leadership.</p> </li> <li> <p data-renderer-start-pos="2547">Experience with a developer-facing product such as an API, SDK, or developer tool.</p> </li> <li> <p data-renderer-start-pos="2633">Ready to drive and collaborate on documentation, marketing to developers, helping customer-facing teams, and more.</p> </li> <li> <p data-renderer-start-pos="2751">Ability to provide clarity on detailed customer needs and facilitate informed, well-reasoned engineering decisions that trade off engineering scope, product quality, and technical debt.</p> </li> <li> <p data-renderer-start-pos="2940">Proven record of driving business impact (e.g. increasing revenue, decreasing cost, decreasing risk) via product changes.</p> </li> </ul> </div> </div> </div> </div> </div> <div id="comment-container" class="css-nrntui e3p9ckn0" data-testid="comment-container"></div> </div> </div> </div> </div> <div class="_19pkys9h _2hwx1wug _otyr1ylp _18u01wug _1bsb1osq _p12fukw8" data-testid="view-page-labels-container"> <div class=" css-1ww96kd-container"><span id="react-select-2-live-region" class="css-7pg0cj-a11yText"></span> <div class="labels__control css-66bq81-control">&nbsp;</div> </div> </div> <hr> <p><span style="font-weight: 400;">For New York City-based candidates, the base pay for this role is anticipated to be between $175,000-$210,000. The anticipated base pay range is based on information as of the time this post was generated. Actual compensation for successful candidates will be carefully determined based on a number of factors, including their skills, qualifications, and experience.</span></p> <p><span style="font-weight: 400;">People are what make Heap awesome. Regardless of age, education, ethnicity, gender, sexual orientation, or any personal characteristics, we want everyone to feel welcome. We are committed to building a diverse and inclusive equal opportunity workplace everyone can call home.</span></p> <p><span style="font-weight: 400;">Heap has raised $205M in funding from NEA, Y Combinator, Menlo Ventures, SVAngel, Sam Altman, Garry Tan, Alexis Ohanian, Harj Taggar, Ram Shriram, and others. We offer plenty of awesome benefits, and we are currently named #17 on</span><a href="https://www.glassdoor.com/Award/Best-Small-and-Medium-Companies-to-Work-For-LST_KQ0,43.htm"><span style="font-weight: 400;"> Glassdoor’s Best Places to Work (SMB)</span></a><span style="font-weight: 400;">. We'd love to hear from you!</span></p> <p><span style="font-weight: 400;">#LI-EC1</span></p>
Engineering Manager, Data Ecosystem
<p data-renderer-start-pos="18">Heap’s mission is to power business decisions with truth. We’re building the next generation of product analytics that helps you surface everything users do on your site and automatically surface unique “unknown unknowns” that remain invisible with other tools. Our goal is to enable everyone to understand their millionth customer as well as they understood their first.</p> <p data-renderer-start-pos="391">You’ll be responsible for technical strategy, execution, and delivery on a core part of our product experience used by thousands of customers, and you’ll coach and develop a diverse team of engineers working cross-functionally to drive technical direction and strategy.</p> <p data-renderer-start-pos="663">You’ll be responsible for Heap’s Connect product and our innovative approach to data pipelines. Data teams spend too much time sanitizing data, which delays answers to business critical questions and tramples team morale. Heap Connect allows easy low-code event definitions on top of raw data. Those definitions are retroactively synced to the warehouse to support seamless and flexible downstream analysis. In addition to warehouses, Heap offers a wide array of other integrations to allow customers to use Heap data to power marketing, business decision making, lead scoring, and more!</p> <p data-renderer-start-pos="1253"><strong data-renderer-mark="true">What you’ll do</strong></p> <ul class="ak-ul" data-indent-level="1"> <li> <p data-renderer-start-pos="1271">Remotely manage a diverse group of 5-8 engineers ranging from junior to senior. Support their growth and development through continuous coaching and feedback.</p> </li> <li> <p data-renderer-start-pos="1433">Drive the technical strategy and partner on product direction for our managed ETL product, <a class="css-tgpl01" href="https://www.heap.io/platform/connect" data-testid="link-with-safety" data-renderer-mark="true">Connect</a>.</p> </li> <li> <p data-renderer-start-pos="1537">Work closely with product management, partnerships team, and 3rd party developers, to maintain and extend our integrations platform.</p> </li> <li> <p data-renderer-start-pos="1673">Own execution and delivery of the eng team, and help the team balance new feature work with technical debt and platform investment.</p> </li> </ul> <p data-renderer-start-pos="1808">We’re a distributed team that operates mostly in US timezones. Because of the high degree of interaction with other teams, for this role, we’re primarily looking for a manager working in the US or a US-adjacent time zone (including all of North and South America).</p> <p data-renderer-start-pos="2074"><strong data-renderer-mark="true">What we’re looking for</strong></p> <ul class="ak-ul" data-indent-level="1"> <li> <p data-renderer-start-pos="2100">A track record of leading and coaching diverse teams of engineers (at least 3+ years managing a team of 4 or more engineers)</p> </li> <li> <p data-renderer-start-pos="2228">A solid engineering background that has kept pace with new technologies, even if it’s been a few years since you coded full time (at least 5+ years of total experience in technical roles)</p> </li> <li> <p data-renderer-start-pos="2419">Experience running team cadences in a fast-moving, agile environment, and working cross-functionally with product management to set roadmap and direction</p> </li> <li> <p data-renderer-start-pos="2576">Some amount of relevant experience in building integrations, building data platforms at scale, working with event streaming architectures, and/or a strong understanding of cloud computing (particularly AWS). What you don’t know from our stack (because nobody knows everything!), you’ll be excited to learn and ramp up on quickly.</p> </li> <li> <p data-renderer-start-pos="2909">Experience working with data warehouse or data engineering technologies a plus (Spark, Snowflake, Redshift, BigQuery)</p> </li> <li> <p data-renderer-start-pos="3030">A genuine commitment to building diverse teams, and promoting equity and inclusion on your team and throughout the organization</p> </li> </ul> <p data-renderer-start-pos="3161">Under the hood, Heap is powered by TypeScript, Golang, Scala, Spark, Kafka, Redis, and PostgreSQL (using Citus). For more about our architecture, check out&nbsp;<a class="css-tgpl01" href="https://heap.io/blog/engineering/virtual-events" data-renderer-mark="true"><u data-renderer-mark="true">Virtual Events: Making Data-Driven Decisions a Reality</u></a>.</p> <hr> <p><span style="font-weight: 400;">For New York City-based candidates, the on-target earnings for this role is anticipated to be between $240,000-$280,000. The anticipated pay is based on information as of the time this post was generated. Actual compensation for successful candidates will be carefully determined based on a number of factors, including their skills, qualifications, and experience.&nbsp;</span></p> <p><span style="font-weight: 400;">People are what make Heap awesome. Regardless of age, education, ethnicity, gender, sexual orientation, or any personal characteristics, we want everyone to feel welcome. We are committed to building a diverse and inclusive equal opportunity workplace everyone can call home.</span></p> <p><span style="font-weight: 400;">Heap has raised $205M in funding from NEA, Y Combinator, Menlo Ventures, SVAngel, Sam Altman, Garry Tan, Alexis Ohanian, Harj Taggar, Ram Shriram, and others. We offer plenty of awesome benefits, and we are currently named #17 on</span><a href="https://www.glassdoor.com/Award/Best-Small-and-Medium-Companies-to-Work-For-LST_KQ0,43.htm"><span style="font-weight: 400;"> Glassdoor’s Best Places to Work (SMB)</span></a><span style="font-weight: 400;">. We'd love to hear from you!</span></p> <p><span style="font-weight: 400;">#LI-EC1</span></p>
Solutions Engineer
India
<p>&nbsp;</p> <p>˜Our strong customer support team is a major reason why companies choose Heap as the source of truth for all their customer data and why our customers love us so much! We’re looking to hire experienced Solutions Engineers (SE) to help our customers understand our product and make the most of their data. Our customers use Heap to make important business decisions, and you'll be on the front line helping them by answering their emails, debugging complex technical issues, and devising strategies to help them get the most out of our product. You’ll work closely with our customer success managers, sales, and product to advocate for our customers.</p> <p>You will be employed by Heap's strategic partner, Scaletrix.AI, with operations in Gurgaon, India. The office address will be Sector 44, Haryana - 122003</p> <ul> <li>INR 8 - 10 LPA</li> <li>Working hours:</li> <ul> <li>12 noon - 5 pm for Office - SE will be required to visit the office twice a week. Hence the other three days individuals will work from home</li> <li>7:30 pm - 11:30 pm from home</li> </ul> </ul> <p>&nbsp;</p> <p><strong>At Heap, this means you will:</strong></p> <ul> <li>Form close relationships with our customers. Our customers will consider you part of their team.</li> <li>Become more knowledgeable about analytics infrastructure than anyone else. You will have complete context on our product and become an expert in the larger analytics space.</li> <li>Communicate technical concepts clearly. You will convey sound data principles to a wide-range of audiences: product managers, marketers, data scientists, engineers and C-level executives.</li> <li>Advocate for the best customer experience. Internally, you’ll serve as the voice of our customers when defining product requirements with engineers or helping marketers educate the world about analytics.</li> <li>Constantly iterate on our processes. You’ll work cross-functionally and manage special projects internally to help Heap scale as fast as possible.</li> </ul> <p><strong>30/60/90 Day Plan</strong></p> <p>Within 30 days, you will:</p> <ul> <li>Answer hundreds of customer questions</li> <li>Triage and prioritize bugs and feature requests</li> <li>Act as internal support for our sales, marketing and engineering teams</li> </ul> <p>Within 60 days, you will:</p> <ul> <li>Understand Heap’s backend infrastructure</li> <li>Use advanced technical tools to debug complex issues</li> <li>Advise customers on the optimal way to install and implement the product for their use-case</li> <li>Understand and aggregate customer use-cases to help our Product and Engineering teams design critical product improvements</li> </ul> <p>Within 90 days, you will:</p> <ul> <li>Know Heap like the back of your hand</li> <li>Come up with clever workarounds to push Heap to the limits</li> <li>Help customers with every question, whether it is joining data in Heap SQL or analyzing data within the UI</li> <li>Use SQL to query our systems and debug any complex issue thrown your way</li> <li>Be able to point to the line of code that is causing bugs and write your own scripts to make the support team more efficient</li> </ul> <p>Most importantly, you’re an empathetic person who is comfortable with ambiguity and is an incredibly fast learner. As a strategic advisor for our customers, you’ll face novel situations and provide expertise every day.</p> <p><strong>What we're looking for:&nbsp;</strong></p> <ul> <li>Experience with HTML, CSS, JS, SQL, and working with various APIs</li> <li>Stellar written communication skills</li> <li>A passion for helping people</li> <li>(Bonus) Knowledge of the analytics market and data infrastructure</li> <li>(Bonus) Experience with Zendesk, Postgres, Kibana, Datadog, AWS Athena</li> </ul> <p>People are what make Heap awesome. Regardless of age, education, ethnicity, gender, sexual orientation, or any personal characteristics, we want everyone to feel welcome. We are committed to building a diverse and inclusive equal opportunity workplace everyone can call home.</p> <p>Heap has raised $95M in funding from NEA, Y Combinator, Menlo Ventures, SVAngel, Sam Altman, Garry Tan, Alexis Ohanian, Harj Taggar, Ram Shriram, and others. We offer plenty of awesome benefits, and we were named #1 on <a href="https://www.glassdoor.com/blog/heap-bptw19/">Glassdoor’s Best Places to Work (SMB)</a>. We'd love to hear from you!</p>
Verified by
You may also like