Improving the Quality of Recommended Pins with Lightweight Ranking

796
Pinterest
Pinterest is a social bookmarking site where users collect and share photos of their favorite events, interests and hobbies. One of the fastest growing social networks online, Pinterest is the third-largest such network behind only Facebook and Twitter.

By Poorvi Bhargava | Software Engineer, Homefeed Recommendations, Sen Wang | Software Engineer, Homefeed Recommendations, Andrew Liu | Tech Lead, Homefeed Recommendations, Duo Zhang | Engineering Manager, Homefeed Recommendations


The Pinterest corpus is composed of billions of Pins, however, each Pinner only sees a small subset based on their interests when browsing their home feed or other recommendation surfaces. How do we provide these recommendations to each person?

Introduction to Pixie: a key recommendation system at Pinterest

Pixie is one of Pinterest’s major recommendation systems used for fetching relevant Pins. Pixie is composed of a bipartite graph of all Pins and boards on Pinterest. Starting at a Pin that has recently been interacted with, we perform multiple random walks along the graph to generate thousands of similar Pins for that Pinner. These Pins are sorted by “visit count”, or the number of times it was “visited” during the random walks. After Pins are fetched from Pixie, they’re sent to Pixie’s clients for further personalized ranking.

As a major recommendation source, Pixie generates over 75 million Pins per second and powers multiple downstream clients, including important surfaces like the home feed, recommendations feeds, email notifications, search, and ads.

How does Pixie fit into the Pinterest recommendation pipeline?

Like other recommendation systems, we use a two-step approach to narrow down billions of Pins to select the best few for the Pinner.

Figure 1. Number of Pins at each stage of the current Recommendation Funnel

The first step, “Candidate Generation”, is a recall-driven step that aims to efficiently fetch a set of broadly relevant Pins. Pixie is one such candidate generator. This step uses recent user engagement to formulate an input representative of the Pinner’s interests. The input Pin is used to fetch similar Pins, which are quickly scored based on simple heuristics (in Pixie’s case, this is the visit count score), with additional boosting logic applied for specific business needs. The Pins with the highest score are then passed to the next step of the funnel.

The second step, or the “Full Ranking” layer, aggregates and ranks all of the recommendations fetched from multiple candidate generators. It consists of a precise, yet complex, neural network that uses user, Pin, and context features to accurately predict how likely a Pinner is to engage with the candidates. Largely due to model complexity, this step is usually quite costly and time-consuming, so we rank a limited number at this step. The highest ranked Pins are then shown to the user.

Making Pixie recommendations even more personalized

Despite its great success across multiple major product surfaces at Pinterest, Pixie currently faces several challenges that limit it from generating even more relevant content:

  1. Pixie’s “visit count” score sorts the generated pins purely on graph structure; it does not take any user preferences into account. At Pinterest, we’ve built a bunch of features that represent user interest and pin information that may help improve personalization.
  2. Ad-hoc business needs (such as promotion of local content) are not incorporated easily and require adding to the existing web of boosting logic.

Both of these challenges can be elegantly solved by replacing the existing visit count scoring and boosting layer within Pixie with a machine learning model.

Machine learning models, however, can be expensive and time-consuming to run in production. Since Pixie recommends more than 75 million Pins per second and lies on the critical path of multiple recommendation surfaces, it is crucial that adding a model would not significantly increase Pixie’s latency. Additionally, since Pixie candidate generation is followed by a more thorough full ranking step on the client side, we can afford to use a simpler model and trade off some precision for efficiency. This type of “lightweight” ranking (in contrast to the “heavyweight” full ranker) is often instituted in industry to improve personalization earlier on in the recommendation funnel. Lastly, since Pixie powers a wide range of major surfaces at Pinterest, it is important to keep in mind that the design should be flexible, scalable and extensible to support multiple client needs across the company. Thus, the main goals for this lightweight ranking model for Pixie are to:

  1. Recommend Pins with better relevance and personalization to the user, especially compared to those from the existing “visit count” solution.
  2. Build an efficient machine learning model to support scoring 75 million Pins per second.
  3. Build a scalable and extensible multi-tenant framework that can be easily applied to support different Pixie clients.
  4. Add flexibility to be able to stress certain types of pins based on the client surface’s ever-changing business needs.

Figure 2. Recommendation Funnel after adding a Lightweight Ranking Step

Building a multi-tenant lightweight ranking system

In practice, machine learning-based recommendation systems consist of multiple key components, such as a training data pipeline and model training strategy. Here, we will focus on the components that required major design decisions which allowed us to build a multi-tenant lightweight ranking system for Pixie.

Creating a training dataset extensible to multiple clients

The success of a machine learning model heavily relies on the training data. To generate this data, we joined the labels from Pinners’ explicit engagement with Pin and user context features.

Figure 3. Logging Pipeline Design: logging at the front end stage versus at the serving stage.

One common practice for logging training data is to log at the “front end stage”. This means that we pass feature data to the front end and only store data for pins seen by users. The main benefit of logging at the front end stage is savings on storage space, since we do not need to store any data unseen (and therefore, unlabeled) by the user. The vast majority of data, especially at the lightweight scoring level, is unseen by the user.

However, since one of the project’s major design goals was to build an extensible pipeline for multiple clients, we decided to use another approach in which we log directly at the “serving stage”. This involves storing feature data for all candidate pins immediately generated by Pixie, including that of unseen Pins. This avoids each client having to pass features to the front end and set up their own logging infrastructure. Additionally, the unseen data helps us define training optimization strategies unique to lightweight ranking, as described below.

How to train and optimize a lightweight ranking model

Since the goal of lightweight ranking is to efficiently score a large number of Pins, we started by training a low-complexity XGBoost GBDT model. We included the most important user and Pin features from each of the client full rankers, but also included features unique to Pixie, such as graph and input pin features. The feature set is shared across all Pixie client surfaces.

Since our lightweight model is followed by a more precise full ranking step downstream, the major design choice during model training came in choosing the model optimization strategy. A full ranker usually optimizes for predicting user engagement. However, because there is an additional layer of ranking following lightweight ranking and unseen data is available to us, we could also choose to optimize our model for (a) trying to explicitly mimic the results of the full ranker through a technique known as Model Distillation or (b) passing through the full ranker by improving the “funnel efficiency”.

Model distillation refers to the process of training a smaller “student” model to reproduce the behavior of a larger “teacher” model as accurately as possible, but using fewer parameters. This involves “distilling” knowledge from the teacher model to the student model. Funnel efficiency optimization is a less explored, but related concept. It aims to maximize the number of candidates that pass through downstream ranking layers.

We’ll explore model distillation in a future blog post, and for now focus on comparing engagement and funnel efficiency optimization strategies. How did we implement these strategies? Because our logging pipeline captures data at the serving stage, we have access to examples that did pass (impressed Pins) and did not pass (unimpressed Pins) the full ranking layer. We designed our training pipeline to allow us to easily define the positive and negative labels depending on our optimization strategy:

In practice, we saw that both types of funnel efficiency approaches did indeed increase the number of Pixie pins being passed through the recommendation funnel. However, the “pure” funnel efficiency approach was less effective at predicting which Pins a user was going to take action on. This is because, even with significantly higher training weights for action labels, the large number of impression labels and low complexity of the model made it hard for the model to distinguish between the two types of positive labels. We saw that the “blended” funnel efficiency approach showed the most favorable results, even compared with those of the models optimized for engagement.

Additionally, each client has different business needs. For example, the home feed may want to maximize saved Pins for a given person whereas notifications may want to maximize clicks within an email. To address these concerns, we trained a different model for each client and emphasized the training weights relevant to the particular surface. In practice, we saw this led to much finer control and major metrics gains on important labels as compared with the original, rigid heuristic-based sorting.

Lastly, ad-hoc boosting, such as “localness boosting”, was previously applied to the final rank for each Pin. To match the performance of this type of boosting, we not only added locale features to the model, but also assigned “local labels” for those examples in our training set where the locale of the pin matched that of the user. In practice, we saw that these labels significantly boosted local recommendations, allowing us to replace the previous boosting layer.

Wins

Impact to Pixie and its clients

We saw great wins for the models instituted for each of Pixie’s major clients. On the home feed we saw an increase in saves by 1–2% and on Related Pins we saw a 1% increase in both time spent on the site and CTR. For e-mail notifications powered by Pixie, we saw a 6% increase in CTR with significant gains in weekly active users as a result. Having a scalable pipeline with the flexibility to define new labels and optimization strategies allows us to cleanly cater to each client’s dynamic business needs.

Impact to the user: improved quality of recommendations

On the user side, lightweight ranking allows us to fetch more relevant and personalized recommendations earlier on in the funnel. Below is one such example of this.

Figure 4. Comparing recommendations from old heuristic-based sorting versus those with lightweight ranking.

On the left side of Figure 4, you can see that for a user looking at tips on hiking Rainbow Mountain in Peru, the recommendations without lightweight ranking are dominated by wallpaper images of general travel scenes, unrelated to tips, hiking, or Peru. On the right side, you see that all of the recommended pins are about either Peru, hiking tips, or other mountainous travel destinations, and are, therefore, much more relevant.

Future directions

In the development of this model, we faced a few recurring challenges:

  1. Infrastructure limitations: Due to the scale of Pixie, the high workload limited our capacity to add new features and to use more complex machine learning models to improve performance.
  2. Model fragmentation: On some client surfaces, such as the Home Feed, we have several different lightweight ranking models. Having multiple models leads to a waste of computational resources and redundant development efforts across a single surface.

To address these challenges, we plan to unify the serving framework and model design used across all candidate generators (not just limited to Pixie) on a particular surface. By moving the lightweight ranking away from an individual candidate generator, we are able to share features more broadly, unify the logging pipeline, and easily iterate on all models in unison.

We plan to further investigate the difference in performance between different optimization strategies, such as model distillation, and to experiment with more powerful, but less costly, model architectures, such as a Two Tower Embedding-Based DNN (inspired by YouTube).

Acknowledgements

Tao Cheng, Angela Sheu, Chen Chen, Se Won Jang, Jay Adams, Nadia Fawaz

Pinterest
Pinterest is a social bookmarking site where users collect and share photos of their favorite events, interests and hobbies. One of the fastest growing social networks online, Pinterest is the third-largest such network behind only Facebook and Twitter.
Tools mentioned in article
Open jobs at Pinterest
Software Engineer, Continuous Delivery
San Francisco, CA, US

About Pinterest:  

Millions of people across the world come to Pinterest to find new ideas every day. It’s where they get inspiration, dream about new possibilities and plan for what matters most. Our mission is to help those people find their inspiration and create a life they love. In your role, you’ll be challenged to take on work that upholds this mission and pushes Pinterest forward. You’ll grow as a person and leader in your field, all the while helping Pinners make their lives better in the positive corner of the internet.

The Engineering Productivity team is a small and diverse group of experienced engineers who build products, tools and systems enabling thousands of Pinterest engineers to be productive across the development lifecycle (code, build, deploy). Our vision is to build a platform that inspires developers to do their best work, and our mission is to provide a fast and safe path from prototype to production without worrying about the underlying infrastructure. We believe in contributing to open source work and collaboration as much as possible. 

What you’ll do:

  • Work on our next-generation, continuous delivery platform for multiple compute backends (VM, K8S, Yarn)
  • Build partnerships with various teams (our customers) by learning their use-cases and pain points; champion key efforts across teams.
  • Have an opportunity to impact key open source systems such as Teletraan (Deploy/Compute Engine), Kubernetes (Compute Engine), Spinnaker (Workflow Orchestration).

What we’re looking for:

  • 4+ years experience as a software engineer and have a penchant to build and scale critical distributed systems.
  • Proficient in coding and scripting languages such as Go, Java, NodeJS or Python.
  • You enjoy articulating technical details on key problem areas by writing RFCs and design documents.

#LI-SG1

Engineering Manager, Creator Platform
San Francisco, CA, US

About Pinterest:  

Millions of people across the world come to Pinterest to find new ideas every day. It’s where they get inspiration, dream about new possibilities and plan for what matters most. Our mission is to help those people find their inspiration and create a life they love. In your role, you’ll be challenged to take on work that upholds this mission and pushes Pinterest forward. You’ll grow as a person and leader in your field, all the while helping Pinners make their lives better in the positive corner of the internet.

The Core Eng - Create Team’s mission is to empower people to create content and build communities that inspire others to create a life they love.

Our focus is to:

  • Grow our quality native content corpus, so we can enable more immersive and engaging experiences on Pinterest.
  • Build delightful tools for anyone who wants to create inspiring content, then build sustained value in our publishing ecosystem - whether through a positive, growing community for the creator or monetization.
  • Protect our pinners and creators by maintaining a positive community and the trust and safety of our pinners.

What you'll do:

  • Partner closely with Design, Product Management, and other Engineering leaders within Core Eng and across Pinterest to identify the highest-impact opportunities to accelerate the product and user experience.
  • Take ownership of critical outcomes. Manage a cross-platform (Android/iOS/Web/Backend) engineering team through the process of planning and executing against those outcomes, delivering software experiences and solutions of the highest quality.
  • Grow the team's capabilities through career development of the current engineers and strategic hiring.

What we're looking for:

  • Passion for developing software solutions for the people using Pinterest.
  • Ability to deliver on immediate goals and form long-term strategies around technology, processes, and people.
  • Strong people development skills and software project management skills.
  • Proven success launching complex software development projects that deliver high-impact outcomes.

#LI-AG1

Design Technologist - Figma Plugin De...
, US

About Pinterest:  

Millions of people across the world come to Pinterest to find new ideas every day. It’s where they get inspiration, dream about new possibilities and plan for what matters most. Our mission is to help those people find their inspiration and create a life they love. In your role, you’ll be challenged to take on work that upholds this mission and pushes Pinterest forward. You’ll grow as a person and leader in your field, all the while helping Pinners make their lives better in the positive corner of the internet.

Pinterest is looking for a Javascript developer to help build out our Figma tooling solutions to better support the company’s design team. You will be creating the tools which help our designers improve Pinterest so it can live up to its mission to bring inspiration and a positive impact to people’s lives.
 
What you’ll do:
  • Talk with product designers to gain a direct understanding of how they work and how our plugins can better support their workflow.
  • Work directly with the product design team to craft and ship tools that will help our team work better and with greater velocity.
  • Develop a strong baseline/framework for all future Figma plugin solutions within Pinterest.
What we’re looking for:
  • 5+ years of experience building on the web platform.
  • Strong background in current web app development practices as well as a strong familiarity with Javascript, Typescript and Webpack.
  • Solid experience with HTML and CSS fundamentals.
  • Background and familiarity with modern design processes and tools is a big plus.
  • Experience with React and using Figma’s plugin/REST APIs a big plus.

More about contract roles at Pinterest: 

  • This is a contract position at Pinterest. As such, the contractor who fills this role will be employed either by our staffing partner (ProUnlimited) or by an agency partner, and not an employee of Pinterest
  • All interviews will be scheduled and/or conducted by the Pinterest assignment manager. When a finalist has been selected, ProUnlimited or the agency partner will extend the offer and provide assignment details including duration, benefits options and onboarding details

#LI-AZ415

#LI-REMOTE

Engineering Manager, Stream Processin...
San Francisco, CA, US

About Pinterest:  

Millions of people across the world come to Pinterest to find new ideas every day. It’s where they get inspiration, dream about new possibilities and plan for what matters most. Our mission is to help those people find their inspiration and create a life they love. In your role, you’ll be challenged to take on work that upholds this mission and pushes Pinterest forward. You’ll grow as a person and leader in your field, all the while helping Pinners make their lives better in the positive corner of the internet.

As the manager for the Stream Processing Platform team at Pinterest, you will lead a team of talented engineers to work on the large scale platform that powers real-time stream processing applications that process trillions of messages every day. You will have the opportunity to work with the team on the latest cutting edge real-time stream processing challenges, defining the future strategy and taking the platform to the next level. You will also have the opportunity to work closely with product teams that develop exciting new applications on our platform every day, from machine learning, analytics, trust safety to shopping. 

What you’ll do:

  • Lead the team that owns the entire stack of stream processing platform
  • Drive Pinterest’s stream processing strategy and vision
  • Collaborate with customers to understand requirements and incorporate them in the roadmap and work closely with partners to align on strategic directions
  • Hire and further build up the team to support more use cases

What we’re looking for:

  • 7+ years of experience, including 2+ years of management experience
  • Solid expertise in big data or other types of large scale distributed systems
  • Platform development and operational experience
  • Enjoying working in an agile environment

#LI-MJ1

Verified by
Security Software Engineer
Tech Lead, Big Data Platform
Software Engineer
Talent Brand Manager
Sourcer
Software Engineer
You may also like