Scaling Wix to 60M Users - From Monolith to Microservices

10,004
Twvyx sp
Wix
Wix.com is a web development platform enabling anyone to build a stunning online presence using simple cloud-based creation and management tools.

By Yoav Abrahami, Chief Architect, at Wix.


Wix Dashboard

Stats

  • Over 61M users - gaining 1.5M users/month
  • 2PB of user media files - 1.5TB of new files/day
  • 3 data centers + 2 clouds (Google, AWS)
  • 1.5B HTTP requests/day
  • 400 engineers

Background

Wix is a cloud-based web development platform that makes it easy for everyone to create beautiful websites. Our drag-and-drop, online editor makes it simple for everyone to get online with a beautiful, professional, and functional web presence. With over 60M sites built in more than 190 countries, Wix is the world's largest "build it yourself" website-building platform.

Wix’s open architecture enables users to add functional services to a site- including email, contact forms, and eCommerce shops with a single click, making it a one-stop shop for managing a small business presence online.

I’ve been working for Wix on and off since the day the company was founded. I interviewed the first backend developer in 2006, then rejoined the company for 2 months in 2008. I’ve been working full-time as Chief Architect since 2010.

As part of my role, I have led the transition from a traditional waterfall development model to Agile continuous delivery, helped introduce DevOps, and initiated the Wix SDK. I am now working on other products, as well as the next-generation infrastructure for Wix.

Engineering Structure

Wix is organized into internal companies and guilds. With around 400 engineers, QA, and product people, organizational structure becomes key for our success. Guilds focus on technical aspects such as scalability, developer productivity, and interservice communication, while companies focus on products. For example, we have an eCommerce company, a backend guild, an Angular guild, etc.

A guild is responsible for hiring people, training them, and building infrastructure and tools to help engineers be more productive. Some guilds take as much as 20% of engineers’ time to work on guild-related tasks (building infrastructure, testing tools, giving lectures, facilitating retrospectives, etc). Companies, on the other hand, are composed of people from different guilds working to build a specific product. Each company has one company head who is the final decision maker. Engineers are members of one guild, and most are assigned to a company. A few select engineers are kept in the guild for cross-company tasks, such as to build frameworks or a knowledge base.

Architecture

As Wix started, we built a single monolith using Java, Hibernate, Ehcache, Tomcat, and MySQL. This monolith enabled us to learn more about our business and for Wix to grow, but by 2008, it started to introduce difficulties.

Our monolith served two functions: serving sites built with Wix, and supporting sites that users were building. Most of the development effort at the time was to support the site-building tools, resulting in a lot of changes in this function. Serving sites, on the other hand, was relatively stable. However, the cost of problems with the first function was considerably higher because it impacted all of the sites built with Wix, compared to only impacting users actively building websites. Because both functions were supported by the same monolith, any change in the site-building function resulted in risk for the site-serving function, even if the site-serving function itself did not change. We experienced pains such as downtimes (planned or unplanned) imposed by changes in the site-building function, but those downtimes affected all Wix users.

So, by 2008, with over 1,000,000 sites, we made our first architecture change based on service-level requirements. Two functions with different sources of risk and different service levels needed to be two distinct software applications. To this day we use Service Level-driven Architecture to separate concerns to different microservices.

Over the years we dropped Hibernate and Ehcache, and broke down the monolith (a 4.5-year process). This was a long, complicated process because we kept Wix running and enhanced Wix with new features while gradually moving features from the monolith to new microservices. Today at Wix we have over 100 microservices. Most are based on the Scala programming language, with Jetty, Spring, and our internal framework.

When Wix started, we had a Flash-based product—both the Editor and the created sites were Flash applications—which turned into a second monolith (especially the Editor application).

The Flash monolith was abandoned in favor of HTML5/JavaScript architecture around 2011. Initially we developed our own JavaScript framework to support the HTML5 sites, but we’ve since transitioned to using React for both our sites and our Editor. For us, React makes a lot of sense because it introduces a simple yet performant development framework ideal for dynamic websites.

In a nutshell, Wix’s current architecture involves 4 main groups of services:

  • Wix Public - A set of microservices that is responsible for hosting and serving published Wix sites. It uses mostly MySQL and Jetty/Spring/Scala applications to serve the HTML of a site from the data the Editor has created. Wix sites are rendered on the browser from JSON using JavaScript (React), or on the Wix Public server for bots.

  • Wix Editor - A set of microservices that supports creating a website. The Wix Editor itself is written in JavaScript and runs in the browser. It saves a JSON representation of the site to one of the Editor services, which in turn stores the JSON in MySQL and then into the WixMP (Wix Media Platform) filesystem. The Editor backend services also use the Jetty/Spring/Scala stack.

  • WixMP - An Internet media filesystem, built and optimized for hosting and delivering images, video, music, and plain files, integrated with CDNs (Akamai, Level3, Fastly), SSL, etc. The platform is running on two clouds (Google and Amazon), utilizing their compute instances and storage (Google Cloud Storage and Amazon S3) for on-the-fly image manipulation and video transcoding. The compute instances software is developed using Python, Go, and C, where applicable.

  • Verticals - A set of applications that adds value to a Wix site, such as eCommerce, Shoutout, or Hotels. The verticals are built using an Angular frontend and the Jetty/Spring/Scala stack for backend. We selected Angular over React for verticals because Angular provides a more complete application framework including dependency injection and services abstraction.

For us, a microservice is a single application deployed as a process with one clear responsibility. A common question is the size of a microservice: Is it a single function? A single class? Or a whole monolith? Our answer is that a microservice is as large as the team of people managing it (a single team can manage a few microservices). That team has to be able to describe the microservice responsibility in one clear sentence. Only a single microservice will write to a specific database (the microservice owns that database). The microservice itself has to be stateless to support frequent deployments and multiple instances, and all persistent states are stored in the database.


Wix Micorservices


One of the other questions people always ask is: why does Wix keep the websites in a JSON representation and not HTML? The reason is that it enables us to address issues with different browsers or mobile devices, fixing just the JS layer without changing the stored site definition. This in turn allows us to respond quickly to various changes and challenges—for example, new releases of browsers. This also enables us to optimize all Wix websites for search engines (SEO) and to update the optimization constantly as search engines evolve.

Wix is a large-scale shop. We have over two petabytes of media (3M files uploaded daily), lots of BI and log data, and some MySQL tables totaling hundreds of gigabytes. We cope with size differently based on the service. For example, WixMP runs on two clouds, utilizing the clouds’ storage and compute services, as well as a CDN for scale.


enter image description here


Most of Wix’s microservices run in our own data centers using stateless Scala services and a MySQL storage engine. Scaling those services involves mostly scaling MySQL, which for us works best by using functional sharding and a NoSQL usage pattern. We found that MySQL is actually a better NoSQL than most, if it’s used as a NoSQL engine (client-generated IDs; lookup only by index or primary key, in which any field that is not indexed is folded into a single JSON field; no joins; etc.). With MySQL working in this way, we get Active - Active replication between data centers, as well as ~1 mSec response time on huge tables for reads by primary keys.

Around 2012 we introduced an applications SDK, allowing third parties to create widgets and applications for a Wix site, such as eCommerce, blogs, contact forms, or CRM systems. Today we’re proud to have over 250 partners selling third-party applications in our App Market.

The SDK utilizes iframes to incorporate third-party content in a secure and seamless way and uses a JavaScript SDK and REST API to overcome the iframe limitations. Today, with so many partners, we’ve found that it’s actually easier to build internally using the same SDK, and most Wix verticals are built as if they were third-party applications.

Workflow & Tools

We build our software using Maven, Grunt, and TeamCity. We use Git repositories hosted on GitHub (over 400 different repositories). For project management we use Jira, to deploy we use Chef, to configure services we use ZooKeeper, and to run A/B tests and experiments we use Petri. For monitoring, we use New Relic, Nagios, Graphite, and BI business alerting. The BI stack is based on Hadoop, Pig, HBase, and Storm, with in-house-built business alerts and exploration tools.

Our services stack is built on the JVM (Scala) running as a standalone application, which uses embedded Jetty, Spring MVC, and our own framework (targeting developer productivity, and providing support for testing and connectivity to other Wix infrastructure services). We use JSON/RPC and ActiveMQ for communication among services, MySQL, MongoDB, and our own WixMP for persistency. Our frontend toolbox, again includes Angular and React, as well as the build and testing ecosystem of both.

What’s Next?

We're now in the process of analyzing what’s holding us back, and we’ve recognized a few issues we want to address. One such issue is the build process and test feedback time that occurs because of dependencies between projects. Changes in some projects require testing a large number of other services, a process taking up to half an hour. We’ve identified considerable inefficiencies in the traditional build pipeline of compile code, compile tests, run tests, compile integration tests, run integration tests, and only then move to the next project down the pipeline. We are considering moving to other build tools, such as Google Bazel, or building our own tool, focusing on parallelizing compilation and testing between different dependent services.

Another issue we are facing is the responsibility boundary between frontend (JavaScript, HTML, CSS) and backend (Scala) developers. We are considering adding Node.js to our stack, moving the responsibility of the first tier Node.js to the frontend developers, who would then communicate with the backend Scala using JSON/RPC.

Another challenge we need to revisit is our cloud strategy: which services run in the cloud vs. which services run in our own data centers. I cannot really say a lot about it at this stage, except that we are working with three different cloud providers to figure out how we can improve developer productivity and scale, while keeping the complexity under control.

People

At Wix we believe that hiring the right people is the key to a successful company. We believe that it is not just hiring the right people, but also getting them to do the right job with the freedom to innovate and impact the business.

We're hiring all the time — backend and frontend developers, JavaScript experts, UX professionals, designers, and product people. If you are looking for a great company and a cool challenge, Wix is probably the place for you. Come join us.


Further reading:

Twvyx sp
Wix
Wix.com is a web development platform enabling anyone to build a stunning online presence using simple cloud-based creation and management tools.
VP Sales Engineering
San francisco
Be a foundational member on a fast-growing sales team at a venture-backed company. We solves the #1 challenge for every CEO: hiring. Recruiting is a hot market, growing 12-15% y/y on its own, and Applicant Tracking Systems are in high demand. At the same time, hiring has never been more competitive and costly, making talent solutions that improve a company’s hiring ROI incredibly valuable. We’ve just brought our product out of beta and we’ve already attracted an elite roster of customers including Box, Quora, Lyft, Coursera, Foursquare, Slack, Reddit, and GitHub. We are the only hiring software that is mobilizing the entire company to collaborate on hiring. We’re the only solution designed to engage managers, who are incredibly crucial stakeholders, more deeply in the process. By supporting a true partnership between recruiters and managers, we’re able to make hiring faster, higher-quality, and more embedded into a company’s core culture. We are seeking a resourceful, driven, results-oriented sales professional to help us set, drive, and execute our revenue goals. As you’ll be joining before we’ve brought on sales leadership, you’ll work closely with our CEO to set priorities and make key strategic decisions. You’ll shape and influence every part of our sales process to roll out a winning strategy that has material impact. Ultimately, when we go to raise our next round, your work in growing our revenue will be the key foundation to our ability to land funding on great terms. You should bring a passion for tech, entrepreneurial ambition, and most importantly, deep sales expertise. It would help if you’ve sold SaaS to mid-market, since you’ll be familiar with our sales cycles. To join a company at our stage, you should be confident in your ability to run high-volume pipelines in dynamic, changing environments. You’ll need to be able to remove your own roadblocks and invent alternative methodologies that open up new pathways forward. At a startup, you’ll face limited resources but have plenty of chances to demonstrate and expand your capabilities. If you’ve felt cornered or restricted on your current team, you may be ready for more: this is a wide-open role where you have lots of growth potential, lots of openings to prove what you’re capable of. This is an opportunity for someone who wants to be part of something big and transformative, not just a job for someone looking to pay out as easily as possible. You’ll see professional development in several dimensions, including the ability to step up and help lead if you prove you’re ready.
Comments
Open jobs at Wix
Android Engineer
New York
As an Android Developer, you will develop applications primarily in Java and using the Android SDK. You will design and implement new products that will delict our customers. Disproportionately impact a product-driven startup that is transforming the way companies grow their teams. We’re building next-generation collaboration software that helps companies to bring more transparency, participation, and engagement to their hiring. Jump into an empowered role doing end-to-end feature development. You’ll be familiar with all parts of our stack—most notably our homegrown, open source web framework, DerbyJS. Not only will you learn how to develop on a modern, real-time framework, you’ll be building enterprise-grade software on top of it. To do so, you’ll exercise judgment in making tradeoffs between design and feasibility. Choose when to hack and when to invest. You’ll engineer your features to be scalable and resilient in a large, single-page app. Lever is an incredibly design-driven company and you’ll be an active voice in shaping our product and user experience, down to the last detail when shipping your features. Our close-knit, cross-functional team is a great chance to grow your knowledge of different domains. Level-up your knowledge of design theory and UX by working closely with our Design and Customer Success teams. Share your knowledge with them so that they’re more empowered to hook up their front-end code to to support users with more internal tools. You’ll join a team where everyone—including you—is knowledgeable about development patterns and opinionated about product development process. We pitch in to help when things are crazy. We build tools and scripts to ease the lives of our fellow developers. And we’re passionate about lots of random things, and share those passions with each other. Core technologies: JavaScript, Node.js, MongoDB, Redis, Solr, Elasticsearch, DerbyJS, ShareJS, IMAP, SMTP, Gmail and Google Calendar, Microsoft Exchange, AWS real time edits

Fully manage our production servers
Be familiar with monitoring tools and know how to triage bugs
Participate in on-call rotation, diagnose and resolve production issues
Design and automate build and test systems
Back-End Engineer
San Francisco, CA
As part of our core technology team, you'll use a variety of technologies and development practices to build pioneering local commerce applications used by hundreds of millions of people. Disproportionately impact a product-driven startup that is transforming the way companies grow their teams. We’re building next-generation collaboration software that helps companies to bring more transparency, participation, and engagement to their hiring. Jump into an empowered role doing end-to-end feature development. You’ll be familiar with all parts of our stack—most notably our homegrown, open source web framework, DerbyJS. Not only will you learn how to develop on a modern, real-time framework, you’ll be building enterprise-grade software on top of it. To do so, you’ll exercise judgment in making tradeoffs between design and feasibility. Choose when to hack and when to invest. You’ll engineer your features to be scalable and resilient in a large, single-page app. Lever is an incredibly design-driven company and you’ll be an active voice in shaping our product and user experience, down to the last detail when shipping your features. Our close-knit, cross-functional team is a great chance to grow your knowledge of different domains. Level-up your knowledge of design theory and UX by working closely with our Design and Customer Success teams. Share your knowledge with them so that they’re more empowered to hook up their front-end code to to support users with more internal tools. You’ll join a team where everyone—including you—is knowledgeable about development patterns and opinionated about product development process. We pitch in to help when things are crazy. We build tools and scripts to ease the lives of our fellow developers. And we’re passionate about lots of random things, and share those passions with each other. Core technologies: JavaScript, Node.js, MongoDB, Redis, Solr, Elasticsearch, DerbyJS, ShareJS, IMAP, SMTP, Gmail and Google Calendar, Microsoft Exchange, AWS

Fully manage our production servers
Be familiar with monitoring tools and know how to triage bugs
Participate in on-call rotation, diagnose and resolve production issues
Design and automate build and test systems
Product Engineer
Paris
Tablet publishing HTML5 mobile first really simple syndication meetups white board walls. User experience iterate algorithm gamification semantic web value add market research stealth. Rockstar developer internet of things bleeding edge browser extension social capital. Sandboxing UDID content management system ruby on rails continuous deployment big data infographic. Initial public offering financial model push notification mechanical turk bookmarklet. Term sheet convertible note colluding bootstrapping. Cloud computing subscription model out of the box proactive solution.

Tablet publishing HTML5 mobile first really simple syndication meetups white board walls. User experience iterate algorithm gamification semantic web value add market research stealth. 
Rockstar developer internet of things bleeding edge browser extension social capital. 
Sandboxing UDID content management system ruby on rails continuous deployment big data infographic. Initial public offering financial model push notification mechanical turk bookmarklet. 
Term sheet convertible note colluding bootstrapping. Cloud computing subscription model out of the box proactive solution.
Software Engineer
San francisco
We build software for teams to find, interview, and hire top talent. We set a new bar for enterprise software with modern, well-designed, realtime apps. After participating in Y-Combinator Summer 2012 and raising a seed round, we have the support of many fantastic advisors and investors. All engineers on the team contribute to full stack feature development as well as improving our open source projects, including Derby (http://derbyjs.com), ShareJS (http://sharejs.org), Tributary (http://tributary.io), and PeerJS (http://peerjs.com/). Learn more about our team, projects, and investors at https://lever.co/about.html. We're looking for an engineer with deep JavaScript expertise and interest to lead framework development. Derby is the only realtime JavaScript framework that does full server and client rendering for much faster page load time and better SEO. It's also the only JavaScript framework fully powered by Operational Transformation for proper conflict resolution in realtime and offline. We are currently rewriting much of Derby to clean up the code and make it many times faster. The next version of Derby uses innovative techniques to render static HTML and live bound DOM nodes with full client and server code sharing.
Verified by
1764161
Head of Frontend Engineering
7656172
Software Engineer & Group Leader
6973218
Group Manager
4093073
VP of Engineering
You may also like
E-Commerce at Scale: Inside Shopify's Tech Stack
How SendGrid Scaled to 40 Billion Emails Per Month
How Stream Built a Modern RSS Reader With JavaScript
How Heap Built an Analytics Platform that Auto-Tracks Every User Event