Walk up to just about anybody and tell them you’ve launched a new mapping platform, and prepare for confused stares! WHY would somebody do that? After all, the world has Google Maps, HERE/Nokia, Microsoft Maps, and Mapbox. “Maps have already been done” say the investors.
Well, we’ve got two reasons why:
Our UI platform is used by thousands of marketers, journalists, real estate agents, event organizers and just everyday people. We’ve made it really easy to share on social media and Slack, or embed on a website, or on platforms like Weebly, Shopify, Eventbrite, or Slack (very preliminary Beta version).
Our second reason is that in talking to larger Web sites and Enterprise customers, we found that maps are ridiculously expensive if you achieve real volume. For example, if you’re using Google Maps, you are allotted a generous 750,000 map views a month, with the ability to purchase up to 100,000 a day at a $0.50/thousand rate. Exceed that, and you’ll need to upgrade to Premium, which at your current volumes will generally require a negotiated, 6-figure, multi-year contracts through a reseller! Other vendors offer a “Contact Us” button to begin a multi-year contract negotiation when you approach large volumes as well.
Have an Enterprise application, a site behind a paywall or firewall, or involved in transportation and logistics? Then regardless of volume, you’ll have to step into a Premium or Enterprise license with all vendors.
MapJam’s pricing is simple. Beyond a free 100,000 views/month, we provide maps at a low cost of $0.15 per thousand views vs. the standard $0.50 per thousand pricing of other vendors. We don’t require contracts, we charge for what you use, and we don’t require Enterprise fees for private or transportation applications.
Some have referred to us as “The Dollar Shave Club” of maps, and as you would expect, the pricing is creating a lot of interest in the product. We’re not at a Billion maps served yet, but we fully expect to be very soon, and as we’ll describe, we’ve built our stack to globally scale to a billion...and well beyond!
We’re a small, distributed team with a passion for maps. In fact, My co-founder Jack and I first met in High School when I was an exchange student to Australia (from America). Our first class together: Geography of course! Our teacher was amazingly inspiring, so much so that Jack became a Geography teacher. Our first startup together was a travel blogging site, which did some pretty cool things with mapping, and many features made their way into MapJam.
I think we’re the luckiest startup in the world to have such an amazing, mature, and well-balanced development team!
And that’s a good thing, because we offer two very different services, supported by two very different stacks.
Our base mapping tiles are based on data from OpenStreetMap (OSM). If you’re not familiar with OSM, it’s essentially Wikipedia for maps and grows through millions of data contributions a year. Large companies using OSM include Apple, Uber, Facebook, CraigsList, Wikipedia, and large car companies such as Ford and Toyota power their onboard navigation with it. When we released our tiles a few months ago, Steve Coast, the Founder of OpenStreetMap, reached out to us asking “how can I help?” He’s now a MapJam advisor, helping us with all things OSM.
The raw data behind OSM -- nearly 30 GBs compressed -- is Open Source and free for anyone to download and explore. Once downloaded, we use the amazing Imposm 3 (“import OSM”) to parse, tag, normalize, and store it to a Postgres database with the PostGIS spatial database extension for querying geographic objects. Imposm v3 also offers an import process that efficiently imports OSM incremental files, which we’re gearing up to do daily, or more frequently when required in the case of major natural disasters when teams of OSM contributors mobilize to support relief on the ground (an amazing effort and process that deserves its own blog post!). Currently a full OSM database import requires about 1TB of storage.
Once the data is imported, map tiles are requested using TileStache, a Python-based OpenSource project that queries a dataset representing a small square of the Earth at a specific latitude + longitude at a specific zoom level. TileStache generates the standard GeoJSON required for a client-side vector renderer, and at the same time caches the result to Memcache so the database doesn’t have to be queried on the second request of a tile.
Our server infrastructure is robust, and we are distributing it globally. A typical server site consists of two redundant Postgres servers, each with 32GB of memory and 16 cores, and each in turn feeds a 64GM Memcache so that most of the queryable world (only a small portion of Earth is substantially populated) can be pre-warmed and cached.
Our UI brings a whole new dimension to mapping for non-technical users, and yet we’ve built some pretty cool functionality that I think developers will appreciate as well.
Think of our UI as a map publishing and location content management tool. It allows anyone to drag, style, and annotate markers on a map with images, videos, links, and other helpful information. We provide a “content card” building tool to add even more content and context to the map.
Our UI also offers real-time collaboration (think Google Docs for Maps) which allows a user to work on a map with one or more colleagues watching the changes being made in real-time. We use Pusher to enable this, so when the user edits the map, a websocket message is sent to anyone else who is viewing that map. Soon, we’ll also allow users to give read/write permissions to other users, so two or more users will be able to edit a map together.
Finally, all locations and their geo-coordinates, content and associated metadata, and styling created via the UI can be retrieved with our RESTful API. For example, this beautiful map by the event coordinator at Instants Beaujolais Festival in France contains the locations and descriptions of more than three dozen vineyards, restaurants, and cafes, all of which is retrievable with an API call, allowing this human-curated content to be consumed by other systems for republishing in other forms. Beyond wine festivals, MapJam’s UI (via tablet) is useful for other types of data collection in the field, such as heavy equipment tracking, Internet of Things (IoT) inventory, and curation of any location where there isn’t an address.
We use Docker, Docker Compose and Docker Cloud to manage deployments. We found Docker to be an indispensable tool in a situation when we need a tightly controlled environment for both local development, staging, and production. Docker is ideal for stacks with a large number of so-called "micro-services", when you have different components of your platform isolated, developed, integrated and tested as separate apps.
Docker Cloud runs tests in containers, shows the results, plus it also has hooks into Slack, so we immediately see if a build fails. After tests pass, it deploys the containers. It’s a full-featured CI.
Before that we were using a custom combination of Python scripts with AWS APIs, Ansible, and Codeship. It was flaky, unmaintainable, and unreliable, so we're really happy with how robust Docker-based CI turned out to be.
Docker Compose allows us to set up a whole stack locally, with just one command. Docker Cloud does pretty much the same thing, but for remote servers. It can use anything from AWS EC2 to Digital Ocean to Microsoft Azure as a platform for containers. All we need to do is to define Stackfile, point it to the GitHub repo, and it'll build, test, and deploy the stack (multiple Docker container) on push. It also takes care of load balancing, SSL termination, and redundancy with automatically configured HAProxy under the hood. It allows scaling services up and down, with zero downtime.
In terms of challenges, Docker Cloud relies heavily on its DNS servers to function properly, so if there is a problem there, then the whole stack can go down, which happened multiple times just a couple of weeks ago. To recover we quickly hardcoded IPs to bypass their DNS.
As I mentioned before, MapJam's CI is based on Docker Cloud. It means that it builds a project (with Gulp or Grunt), tests it (with Jasmine/Mocha/Protractor/etc, based on the project), and deploys it to servers based on a set of rules defined in a yml-based Stackfile. Stackfile is very similar to
docker-compose.yml, meaning you can define and configure all dockerized services (which is essentially your app) in one place.
Our environment includes local developer systems, a Dev stack, a Staging stack, and a Production stack - for each of the platform's components. The Dev stack is built on each push to the
master branch, Staging is actively tested and stabilized and then pushed to Production. We're client- and feature-driven and don't have a specific release cycle, but we push bug fixes immediately, and we try to publish new features every 2-3 weeks.
As we mentioned earlier, we launched our developer service a few months ago. Having had a very successful launch, we’re scaling to handle not just a billion maps served per month which is our near-term goal, but also significant spikes in traffic which we’ve experienced frequently with media sites using our service.
For example, last year a newspaper published a MapJam map identifying new parking areas in the northern suburbs. This fairly benign story generated over 100,000 map views in just a few hours, which at the time was a significant spike for us. So it’s 5x-30x sustained spikes that we have to engineer for in cases of breaking news or world events.
We also need to scale globally, and that includes duplicating server infrastructure across continents. This is because a “map” you see on a web page is actually a set of 8-10 individual tiles on a page. Zoom in or out, and pan to the left or right, and you’ve loaded another set of tiles. Each tile requires a request, so rendering a “map” can suffer from latency very quickly.
Our solution is to serve globally, initially in the US, Europe, Asia, and Australia, our most active (and potentially active) markets. For tile serving we’re using SoftLayer and Azure. As a 500 Startups Company we’ve been able to develop valuable support partnerships with these hosts as well as Rackspace and AWS. We continue to host our UI service on AWS, we do pretty significant R&D on Rackspace, and use SoftLayer and Azure as server workhorses for tiles. We use Docker and Docker Cloud, and the beauty of these services is that we can pick the right vendor for the right region and spin up and down sites and additional redundancy with ease. We’ll describe our Docker deployments in more detail below.
Another global challenge we are addressing is language support. Currently we serve tiles with “default language labels” from OpenStreetMap, so cities in the Middle East are presented in Arabic, Asian countries are presented in local languages, Russian in Cyrillic, etc. This is not an ideal experience for anyone expecting countries, cities, and states normalized in their own language. OpenStreetMap itself provides many local translations, which we are utilizing, but to be truly complete we’ll rely on other sources of localized “place names” such as Wikidata and Mapzen’s “Who’s on First” gazetteer of places, which collectively provide the names of millions of places in the world in dozens of languages. The translations are there: for example Wikidata presents “Grand Canyon” in 54 languages. The challenge is to determine which languages and associated fonts to support first, and then consolidate the best sources for each language.
If you're using MapJam, we'd love to hear from you in the comments!