---

Hipmob's Tech Stack

Jump to the cloud services

---

LS: So how did the idea for Hipmob first come about?

A: We went through YC building an app that turned your phone into a game controller. And we had several months of rapid adoption and along with that a ton of customer support queries that came in over our web chat widget, (at the time we were using Olark). And we realized we were probably losing a ton of customer engagement because people would discover us through a review or the app store but a subset would come in over the web. And we thought "well if all these people are coming in over the web, how much are we not seeing over mobile?" And there was nothing out there that allowed us to talk to our mobile customers. And it got to the point where we got really frustrated with the whole support process for mobile. We had three apps across both iOS and Android. So when a user discovered us in the app-store, we had no way to talk to them except via email. We'd then ask all these questions; what device they were using, whether they were on wifi, what desktop they had, etc. After a few months of this, we realized we needed something better and we built our own in-app live chat solution for ourselves. Half way through we realized it was something that other developers could use. The point where we decided to switch to focusing on this instead of the gaming app was really a mixture between gut and data. A few of our batchmates were interested in trying it out and we did a bunch of research to see who else it out there and doing this already. And the question was then could we make something better.

If you think about how customer support works on mobile today if you're not using our service, it really happens in one of two ways: you either put the customer into email/mobile Safari or they call you. Both of those take the customer out of your app. In any other scenario, you wouldn't take the customer out of your app. If a customer came into your physical store, would you send them away to get help? Same thing on your website, you don't kick them off to another site. So our belief is that customer communication is better when they're in your experience and captive and you should not send them away when they need help. It's the point when they're most frustrated, and it's an opportunity for you to make them happy, to gain a loyal customer, and in many cases it's an opportunity for you to make a sale. So if you're kicking a customer out of your app, you're doing it at the worst time possible, when they need your help.

The higher the value of your customer, the more you need something like Hipmob. Right now, a bunch of ecommerce apps are using us. And that's helping with things like "do you guys have this in another size?" or "what's the shipping cost?" Any questions around a transaction that helps make a customer more comfortable during that transaction, that's a good use case for us. Another big vertical for us is hardware. The technical debugging associated with physical devices is pretty cumbersome. So a customer bought a device that has a mobile companion app, and they're trying to figure out why it's not working. Any Software as a Service (SaaS) product that has a mobile interface also finds us useful.

For developers, we actually have a lot of plugins and integrations. We have Appcelerator, PhoneGap, etc. It just makes it easy to drop in Hipmob, so that's helpful.

LS: One of the interesting things about Hipmob is that you guys are not only mobile-first, but you're also B2B. What are some of the specific challenges you've seen from a product perspective?

A: At most companies, mobile teams are still really small, or outsourced. Since mobile developers are in such high demand, a lot of companies just have trouble getting mobile talent to stay in-house. This makes it difficult since their in house developers have limited bandwidth. It's harder to have those conversations. Sometimes we'll talk to a really big company that has 2 mobile developers working on an app that does millions in revenue. In these bigger companies the mobile teams are still pretty new. They've rarely worked with outside vendors, the idea of interfacing with mobile services is new to them. And for things like Parse, it's a solution that a developer by themselves can use. For our product, you need to have a developer plus a customer support person involved.

LS: Yup, that makes sense. Let's talk a bit about the tech and your architecture. You have three major platforms that you support: HTML5, Android, and iOS right? How have you dealt with them?

F: So we have about 7 or 8 independent systems that we have to build. We have a management interface, a REST API implemented on different servers, a couple of chat session servers, some XMPP stuff, Android, iOS, HTML5. For mobile we knew we needed native because of the type of solution we have, customers want to be able to customize the app, etc. Our Android libraries are all in Java, iOS in Objective-C. For iPad, you have a much bigger screen and more real estate, you have a chat popover so the person can still be using the application while chatting with support.

In terms of managing versions of platforms, it's interesting. For Apple, as they rev their tools they basically kick off the older versions. The latest version of iOS doesn't support Xcode 4.3. So I literally have a separate computer with Xcode 4.3. But it's not too bad because I don't think we've had any customers actually ask for a build for 4.3. So they're good at pushing users forward.

With Android, our library goes back to 2.1. Android is a little harder just because in iOS-land you can actually get things pixel-perfect, you have 5-10 comparative dimensions that you have to worry about. There's only so many layout structures, so you can just drag and drop assets from Photoshop. But in Android-land, you just don't have that luxury. You end up having to do relative positioning and things like that, so we've been mindful of that since the beginning. We've not shipping video or anything so it's not computationally intensive. So it works fine.

HMTL5 is actually a little messier. Mobile browsers aren't as easy to deal with when compared to desktop browsers where you can sort of get on the automatic update train. For instance, the Android 2.2 native browser is just not that capable, where Safari's version of webkit does a few more things, Chrome's versions on iOS do different things. So there's actually a reasonable bit of work we have to do there. When you think about chat on desktop, about 4 or 5 years ago it was common to have a pop out window for chat and then there was this switch to in-page chat. Getting that done in mobile has been a bit of a challenge given that you're staring at small screen.

LS: So would you walk us through your build and deploy process and talk about some of the tools and services you use?

F: So honestly I'm pretty old school. I use a command line and emacs. For Android dev I use Eclipse, Xcode for iOS. All of our source control is in git. We host our own git instance and we use Gitolite. Our git repositories are mirrored between Linode and EC2. I've used Linode for 7 or 8 years. And for something that's relatively simple, it's easier to start up with, much more classic VPS. EC2, you can get the same experience but there's a little bit more to think about. With Linode you can push a button and get a box. Then with EC2 you have to get this EBS-backed instance, it's more work.

Our front-end components are split between EC2 and Heroku. We do all our dev on servers, I don't actually do any development on my local machine, because we have comparatively more complex environment than we otherwise would. We use XMPP for messaging, we have some components in Node, some components in Django. We have about four independent systems that communicate to deliver the entire experience. I'm a big fan of actually running a system that physically looks like the production system. So I literally have four independent servers sitting between Heroku and EC2 that I do all the development work on. So once I run the code and tests on that integrated platform I'm comparatively certain I won't have any surprises when I actually move the code to production.

LS: So you're basically treating dev as staging.

F: Right. A more formal staging process would be nice but for our purposes, as long as we're actually running all of the instances and we have duplicates of all the EC2 security groups and front-end routing so I can literally take the code from there and put in on the production system then there's effectively no difference. We're deploying to the same availability zones and everything.

Our server components are all in Django and Node. We use PostgreSQL for our database served from Heroku. We have the Xcode pieces for the iOS library side of things. It's a WebSockets connection to a backend and for test purposes we can literally point the WebSocket at staging or dev.

For communication across all the different pieces we use Redis. And that has more or less become the default message bus. We have a number of things that use HTTP APIs internally. But the bulk of our communication between various components has all been through Redis. It's sort of this swiss army knife. And we're hosting that on an EC2 instance.

For queuing we use the RQ python library. It plugs into Django which is what we use for our management console.

LS: From testing perspective what's your process and what do you guys use?

F: We do extensive unit testing. We don't use any structured frameworks for that, it's hand written. It's been on the list of things to do, looking into things like Selenium and much more formal test procedures. For UI testing we've done a bunch of A/B testing using Optimizely for our user-facing web component.

LS: One interesting thing to note is that as a B2B service you don't have to deal with the App Store approval process, right?

F: Well we do in the sense that we can't be the reason that someone's app doesn't pass. And that hasn't happened yet. We don't have to work very hard not to break the build but we have spent some time saying we're very explicitly not doing certain things. Like all the resources such as images are officially created by us. We've had a couple of instances where people had crashes and we figured out that it thankfully wasn't because of our code. But that's one of the dangers of being a third party library. When something goes wrong, it's very easy for people to sort of immediately say you did it. And you have this moment of pause where you're like "wait, was it really me?" And then you find out, actually it was this other thing. But it just comes with the territory.

LS: On the bright side you can push out updates whenever you want.

F: Yeah and when we push out updates we end up having to do a reasonable amount of work for compatibility. Because again once the app is updated, you can't be the reason their app breaks. So our server interfaces essentially have to be reasonably well-versioned. So we have stuff that's a year old that we've made multiple iterations on, for our iOS library and some of our backend. But we still have to maintain support for versions of the libraries that we shipped a year ago because some people still have those apps. People have to be able to say hey, "I'm going to stick your thing in my app and I'm not gonna worry about it." Because if somebody decides not to upgrade the app for any reason and then hits the button a year and a half later, it should still do something meaningful for them.

LS: So how do you guys do monitoring?

F: We use Mixpanel events for server-side analytics. But Heroku provides fairly detailed logging. We looked at a few different log management solutions but just haven't used any of them. So we maintain our own log databases and all the logs from across the entire system are in one spot; we ended up going with Syslog. So we have a Syslog instance and we dump all our logs from all our server components into that. We've started using Graphite as well. So we're still at the stage of figuring out what we want to collect. And we want to have that figured out before get to the question of who should we retain to do this for us. We were on Heroku for a year and half and some things we figured out we can't do on Heroku and some we can. We're getting to the point now where we played around with Graphite, we worked pretty heavily with Mixpanel to actually look at our server behavior and it's one of those things where now we actually understand what it does. We have Google Analytics too. But Mixpanel gives you much more control over what you can do with the data. Less performance monitoring, more behavior analysis. We're comfortable having them do that and we'll probably end up shipping the logging and graphite stuff off to one of the hosted providers, but only after we've actually figured out what to actually do with it.

LS: Is there anything else, open-source or otherwise that has proven to be really useful for you guys and may be useful for other people?

F: So Heroku natively lets you dump the output of any of your Heroku instances into a personal Syslog instance. And so we use that, it's the same technique that all of the Add-On providers use anyway. And we just had that sitting in EC2. And after we did that we essentially realized there were a couple of components that we don't want on Heroku. And it would be helpful to have all of the logging from anywhere pour into that. So the Heroku components that we use are largely in Node. So we built ourselves a little Node server that just stares at the output of the logging process and just ships it over to the syslog server, which is on one of our own servers. So from a debugging perspective, it's been incredibly useful, we can just go to this one place and see everything happens across the entire system. Even though we have four or five separate systems doing the work. So that's been really useful.

LS: So how do you guys prioritize between the different platforms?

F: Honestly, people just use iOS more. The bulk of our traffic is iOS. The core thing we're trying to do here is customer communication. The channel we've picked to start with is chat.

If you were to draw the customer communication spectrum, you have a knowledgebase on one end, where your user is searching your FAQ for help, and along the way you have email which is human to human but asynchronous, call centers and chat for real time communication, and more recently stuff like Amazon's Mayday which is a face-to-face video chat baked into the Kindle/Android OS. So we started at chat because it's what we needed, and we're now filling in the holes. So bringing email responses into the fold. If the core concept is to interact with a customer, they never have to actually leave the app regardless of what the interaction may be. Whether it's them searching for stuff, or them sending a simple query for something they don't necessarily need an answer for right now, or them actually saying I need help at this moment. So any interaction you have with a customer should be in-app, whether asynchronous or fully synchronous, whether you need voice or they actually need to stare at you. All the channels should work together to save your team time and money, which is why we focus so hard on the integrations.

A: Many other channels are pretty well fleshed-out. Like we're not gonna build another PBX that you can build voice on. So what we're saying is, hey there's this new channel and the tools aren't very well developed. We're making two assumptions: Every interaction with your customer is an opportunity to take a step forward for your business. And that means make the customer happy or upsell. The other thing is don't lose the opportunity, solve the problem right then and there. So we help you do that.

When people get kicked out of your app, a lot of times they don't come back. It's not something developers like to have happen.

LS: Right, and it's almost like as end-users we've come to accept being kicked out of the app or having the standard Mail app popover.

A: And we're trying to change that. It doesn't make any sense for it to be that way. You don't kick shoppers out of your shop when they need help. You don't kick visitors from your website when they need help. The reason we're making an in-app knowledgebase, and in-app chat, is so that your mobile users don't have to leave either.