Feed powered byStream Blue Logo Copy 5Created with Sketch.
Avatar of Dan Robinson

Dan Robinson

Heap, Inc.

Decision at Heap about MobX, React, TypeScript, Marionette, Backbone.js, jQuery, JavascriptMvcFrameworks, JavascriptUiLibraries, TemplatingLanguagesExtensions, Libraries

Avatar of drob
MobXMobX
ReactReact
TypeScriptTypeScript
MarionetteMarionette
Backbone.jsBackbone.js
jQueryjQuery
#JavascriptMvcFrameworks
#JavascriptUiLibraries
#TemplatingLanguagesExtensions
#Libraries

The front end for Heap begun to grow unwieldy. The original jQuery pieces became difficult to maintain and scale, and a decision was made to introduce Backbone.js, Marionette, and TypeScript. Ultimately this ended up being a “detour” in the search for a scalable and maintainable front-end solution. The system did allow for developers to reuse components efficiently, but adding features was a difficult process, and it eventually became a bottleneck in advancing the product.

Today, the Heap product consists primarily of a customer-facing dashboard powered by React, MobX, and TypeScript on the front end. #JavascriptUiLibraries #Libraries #JavascriptMvcFrameworks #TemplatingLanguagesExtensions

17 upvotes·1.9K views

Decision at Heap about Citus, PostgreSQL, Databases, DataStores

Avatar of drob
CitusCitus
PostgreSQLPostgreSQL
#Databases
#DataStores

PostgreSQL was an easy early decision for the founding team. The relational data model fit the types of analyses they would be doing: filtering, grouping, joining, etc., and it was the database they knew best.

Shortly after adopting PG, they discovered Citus, which is a tool that makes it easy to distribute queries. Although it was a young project and a fork of Postgres at that point, Dan says the team was very available, highly expert, and it wouldn’t be very difficult to move back to PG if they needed to.

The stuff they forked was in query execution. You could treat the worker nodes like regular PG instances. Citus also gave them a ton of flexibility to make queries fast, and again, they felt the data model was the best fit for their application.

#DataStores #Databases

16 upvotes·251 views

Decision at Heap about Heap, Node.js, Kafka, PostgreSQL, Citus, FrameworksFullStack, Databases, MessageQueue

Avatar of drob
HeapHeap
Node.jsNode.js
KafkaKafka
PostgreSQLPostgreSQL
CitusCitus
#FrameworksFullStack
#Databases
#MessageQueue

Heap searched for an existing tool that would allow them to express the full range of analyses they needed, index the event definitions that made up the analyses, and was a mature, natively distributed system.

After coming up empty on this search, they decided to compromise on the “maturity” requirement and build their own distributed system around Citus and sharded PostgreSQL. It was at this point that they also introduced Kafka as a queueing layer between the Node.js application servers and Postgres.

If we could go back in time, we probably would have started using Kafka on day one. One of the biggest benefits in adopting Kafka has been the peace of mind that it brings. In an analytics infrastructure, it’s often possible to make data ingestion idempotent. In Heap’s case, that means that, if anything downstream from Kafka goes down, they won’t lose any data – it’s just going to take a bit longer to get to its destination. He’s also learned that you want the path between data hitting your servers and your initial persistence layer (in this case, Kafka) to be as short and simple as possible, since that is the surface area where a failure means you can lose customer data. We learned that it’s a very good fit for an analytics tool, since you can handle a huge number of incoming writes with relatively low latency. Kafka also gives you the ability to “replay” the data flow: “It’s like a commit log for your whole infrastructure #MessageQueue #Databases #FrameworksFullStack

14 upvotes·474 views