LinkedIn

Career stories: Influencing engineering growth at LinkedIn

Since learning frontend and backend skills, Rishika’s passion for engineering has expanded beyond her team at LinkedIn to grow into her own digital community. As she develops as an engineer, giving back has become the most rewarding part of her role.

Career Stories: Learning and growing through mentorship and community

Lekshmy has always been interested in a role in a company that would allow her to use her people skills and engineering background to help others. Working as a software engineer at various companies led her to hear about the company culture at LinkedIn. After some focused networking, Lekshmy landed her position at LinkedIn and has been continuing to excel ever since.

Solving Espresso’s scalability and performance challenges to support our member base

Espresso is the database that we designed to power our member profiles, feed, recommendations, and hundreds of other Linkedin applications that handle large amounts of data and need both high performance and reliability. As Espresso continued to expand in support of our 950M+ member base, the number of network connections that it needed began to drive scalability and resiliency challenges. To address these challenges, we migrated to HTTP/2. With the initial Netty based implementation, we observed a 45% degradation in throughput which we needed to analyze then correct.

Enhancing Security and Developer Productivity: LinkedIn's Journey with Implementing Content Security Policy

Co-authors: Covenant Goo, Matthew Lemons, Mira Thambireddy, and Roman Shafigullin

Open Sourcing iris-message-processor

One measure of a successful network is uptime - providing consistent, reliable service for members and customers. If there are frequent connection errors or downtime notifications, it becomes difficult to deliver an experience where people can connect and interact with ease. When faced with uptime challenges, being able to quickly escalate issues to network engineers helps ensure that people can work the way that they want to.

Enhancing homepage feed relevance by harnessing the power of large corpus sparse ID embeddings

Co-authors: Jason (Siyu) Zhu, Amol Ghoting, Birjodh Tiwanna, Maneesh Varshney

Career Stories: Building community and connections

As Keerthana explored her passion for iOS engineering, she also found mentorship and new working styles. But the community she’s discovered as she began connecting and nurturing the next generation of female engineers has truly been the most rewarding part of her LinkedIn journey.

Announcing Our LinkedIn-Cornell 2023 Grant Recipients

LinkedIn and Cornell Ann S. Bowers College of Computing and Information Science (Bowers CIS) embarked on a partnership, bringing together our collective research power to make technological advances that will further our goal to connect professionals with opportunities at scale. Through this partnership, we support Ph.D. students and faculty members on their research in areas in Computer Science, AI, Information Science including Diversity and Equity. Our 2022 recipients showcased the strong value of our partnership through incredible research in diverse topics including Allison Koenecke’s research on early stoppage of randomized controlled trials on heterogeneous populations, David S. Matteson’s research on deep generative models to enable fairer and more responsible AI, and Ruihan Wu’s research in privacy in machine learning. Riding on the back of such tremendous success, we are excited to announce the 2023 grant recipients.

Costwiz: Saving cost for LinkedIn enterprise on Azure

Authors: Deven Walia, Vivek Subramaniam, Simon Desowza, and Karthik Subramanian

Taking Charge of Tables: Introducing OpenHouse for Big Data Management

Co-Authors: Sumedh Sakdeo, Lei Sun, Sushant Raikar, Stanislav Pak, and Abhishek Nath

Career stories: The power of an impactful mentor

Initially a Chicago-based data analyst, Jelanah had her heart set on a more meaningful career in frontend (UI) engineering. She chronicles for us how a life-changing engineering mentor at LinkedIn unlocked her frontend technical skills, supported a relocation to Atlanta, and championed a promotion to software engineer.

Declarative Data Pipelines with Hoptimator

For the last several years, internal infrastructure at LinkedIn has been built around a self-service model, enabling developers to onboard themselves with minimal support. We have various user experiences that let application teams provision their own resources and infrastructure, generally by filling out forms or using command-line tools. For example, developers can provision Kafka topics, Espresso tables, Venice stores and more via Nuage, our internal cloud-like infra management platform. These self-service integrations are typically owned by the teams that build and support the underlying infrastructure.

Cybersecurity Professionals: The Unsung Superheroes of the Digital World

In a world where superheroes captivate our imaginations, it's sometimes hard to recognize the real-life superheroes among us like intelligence analysts, forensic scientists, and cybersecurity professionals. Yes, cybersecurity professionals! Though we may not wear capes or possess extraordinary powers, our role, especially here at LinkedIn, is crucial in safeguarding our members, customers, and employees from the ever-present threat of cyberattacks. In this post, I want to take the opportunity to explain a bit about what we do as cybersecurity professionals, share a couple of statistics and insights for the cybersecurity job market, and highlight why I believe women should consider pursuing a cybersecurity engineering career.

New Approaches For Detecting AI-Generated Profile Photos

Co-authors: Shivansh Mundra, Gonzalo Aniano Porcile, Smit Marvaniya, Hany Farid

Open Sourcing AvroTensorDataset: A Performant TensorFlow Dataset For Processing Avro Data

Co-authors: Jonathan Hung, Pei-Lun Liao, Lijuan Zhang, Abin Shahab, Keqiu Hu

From the Economic Graph to Economic Insights: Building the Infrastructure for Delivering Labor Market Insights from LinkedIn Data

Authors: Dr. Patrick Driscoll and Akash Kaura

LinkedIn Bug Bounty Program - One Year Anniversary of Public Launch

Authors: Ameen Maali, Rohit Pitke, Surbhi Jain, and Mira Thambireddy

Metal as a Service (MaaS): DIY server-management at scale

Guaranteeing that our servers are continually upgraded to secure and vetted operating systems is one major step that we take to ensure our members and customers can access LinkedIn to look for new roles, access new learning programs, or exchange knowledge with other professionals. LinkedIn has quite a large fleet of servers on-premise that depend on internal tooling to ensure they stay on the latest operating systems. This post will introduce an internal tool that serves as an interface for managing servers' lifecycles at the LinkedIn scale. We will emphasize the rationale behind this tool's existence, the path to making it available for our major consumers (i.e., site reliability engineers), and how we rearchitected and scaled this service from only being able to accept a maximum of 72,000 server submissions per day to having no limitations on acceptance rate.

Upscaling LinkedIn's Profile Datastore While Reducing Costs

Co-Authors: Estella Pham and Guanlin Lu

How LIquid Connects Everything So Our Members Can Do Anything

Imagine a tool that can store and connect all the information you need to make decisions and solve problems. Most people would say it’s nice to think about, but not yet possible. The good news is this tool already exists - and it’s called a graph database. At LinkedIn, technologies like graph databases are essential to powering today's platform, while being flexible enough to scale for our future needs. In earlier blog posts, we defined what a graph database is and also shared how to index graph data for fast, constant-time access. One of the biggest (and most important) questions that we haven’t answered is how we utilize graph databases to deliver tangible benefits for our members.

How LinkedIn Adopted A GraphQL Architecture for Product Development

With the widespread adoption of Rest.li since its inception in 2013, LinkedIn has built thousands of microservices to enable the exchange of data with our engineers and our external partners. Though this microservice architecture has worked out really well for our API engineers, when our clients need to fetch data they find themselves talking to several of these microservices. Over time, this has become a challenge as every client is faced with a few key issues:

Viral spam content detection at LinkedIn

On the LinkedIn platform, members from around the world share their knowledge, perspectives, and discuss topics important to them. Our goal at LinkedIn is to enable them to do so in a safe, trusted, and professional environment.

Scaling Salt for Remote Execution to support LinkedIn Infra growth

At LinkedIn, site engineers like to automate operational tasks at various infrastructure layers to minimize manual interventions, which can scale well and be easy to operate. Certain automations are performed via onDemand job executions.

Our Responsible AI Principles in Practice

We recently shared our Responsible AI principles which summarized how we build using AI at LinkedIn. These principles guide our work and ensure we are consistent in how we use AI to (1) Advance Economic Opportunity, (2) Uphold Trust, (3) Promote Fairness and Inclusion, (4) Provide Transparency, and (5) Embrace Accountability. We also recently announced three initial products that leverage generative AI to help our members and customers be more productive and successful. Today, we’ll share lessons we’ve learned from our journey applying our Responsible AI principles as we developed these early generative AI powered tools for our members and customers.

LinkedIn Integrates Protocol Buffers With Rest.li for Improved Microservices Performance

Authors: Karthik Ramgopal and Aman Gupta

Our Learnings from the Early Days of Generative AI

It’s been an exciting few months at LinkedIn, as our engineering and product teams have been working hard to build some new and advanced AI-powered experiences for our members and customers. I have the opportunity to sit at such a unique vantage point where I get to see first hand the work that went into setting the technology foundations - from the technical resources, tools, engineering playgrounds and guidelines - to make it all possible.

Inside Look: Measuring Developer Productivity and Happiness at LinkedIn

Authors:Viktoras Truchanovicius and Selina Zhang

Career stories: Military commander turned Trust & Safety manager

Avery's career in military IT took an unexpected turn when he caught wind of a LinkedIn Trust & Safety (QA) manager opportunity in his hometown of Omaha, Nebraska. Now charged with keeping our LinkedIn platform safe, he shares his career transition into tech, and how his team has supported him as a dad and U.S. Army National Guard cyber-protection commander.

How LinkedIn automates cherry-picking commits to improve developer productivity

Our developers at LinkedIn are constantly exploring ways to enhance and strengthen our platform, aiming to provide our members and customers with the greatest possible access to knowledge and connections. With approximately 15,000 code repositories, our developers work tirelessly to make thousands of code changes each day, improving functionality and resolving any issues that may arise. Most of these repositories practice trunk-based development, which means developers push code changes frequently to the trunk (main branch) and avoid any other long-lived branches. Code changes are peer reviewed using a pull request (PR) workflow on GitHub and merged into the main branch.

Evolving the LinkedIn Business Platform

At LinkedIn, one of our most important goals is to offer a top-notch experience for all of our customers. Purchasing is a big part of that customer experience, and it is not as straightforward as many people would think. It can be complex and difficult to navigate, which makes the LinkedIn Business Platform such an important tool for delivering high-quality experiences to LinkedIn sales representatives (our internal consumers) and also customers of our products and services.

Unified Streaming And Batch Pipelines At LinkedIn: Reducing Processing time by 94% with Apache Beam

Co-Authors: Yuhong Cheng, Shangjin Zhang, Xinyu Liu, and Yi Pan

Building and maintaining the skills taxonomy that powers LinkedIn's Skills Graph

Co-authors: Sofus Macskássy, Carol Jin, Shiyong Lin, Xiaomin Wei, and Michael O’Neill

Scaling AutoBuild: Our Journey Towards Delivering An Enhanced Customer Experience

Ensuring operating system upgrades are done quickly and efficiently is just one way we ensure that the underlying applications and our website are running smoothly for LinkedIn members and customers. AutoBuild is LinkedIn’s end-to-end automated server lifecycle management system. Besides building new servers and reimaging the existing server fleet, Autobuild supports rebooting, powering on/off, wiping, and decommissioning bare-metal servers. It’s also a critical product because it’s in a coveted spot along with Metal As A Service (MaaS), a self-service API for performing OS upgrades, server reboots, etc.

Career stories: Going mobile in NYC

Christine’s journey to LinkedIn has not been linear. A graphic designer by training, Christine initially worked in hospitality before she found her true passion in engineering. After working remotely in Arizona as a LinkedIn frontend (UI) engineer, she shares how her supportive supervisor and our Mobile Academy training/mentorship program catalyzed a Next Play internal mobility opportunity to transition into iOS mobile engineering back in her hometown of NYC:

Career stories: Taking LinkedIn Engineering to Tel Aviv

A mom of three, military veteran, and former white-hat hacker, Eynav joined LinkedIn through the acquisition of the marketing analytics startup, Oribi. Now as the engineering leader behind our new LinkedIn Tel Aviv office, she shares more about the transition, building our engineering team in Israel, and creating opportunities for women in engineering:

Unifying Messaging Experiences across LinkedIn

Co-authors: Michele Ursino and Joe Xue

Improving the customer’s experience via ML-driven payment routing

Co-Authors: Xianyun Mao, Stan Xu, Rachit Kumar, Vikas R, Xia Hong, and Divyakumar Menghani

Career stories: Next-gen systems, servers, and SREs

Saira joined our Bangalore site reliability engineering (SRE) team to tackle large-scale, site engineering challenges and grow. She highlights for us the impactful work she found here — from ushering in LinkedIn’s next-generation, server query system that runs over a fleet of 350,000 servers, to mentoring the next generation of female engineers:

Accelerating Code Delivery By 97% With Yarn Workspaces

As teams and applications experience growth, it’s critical to adopt architectures that optimize for clear code ownership, build isolation, and provide efficient delivery of code. While many projects start small with just one or two repositories (for example, frontend and backend), this approach often becomes difficult to maintain as the codebases expand. At LinkedIn, we develop many applications that receive regular contributions from a multitude of teams, with each team owning distinct products or features. Our infrastructure teams enable developers to work effectively within these large applications without being impacted by the sheer scale of each codebase. In the face of challenging productivity problems, our LinkedIn Talent Solutions (LTS) teams recently adopted yarn workspaces, unlocking a 97% improvement in lead time for delivering commits to our deployment pipeline, reduced from 39 hours to 125 mins.

Ocelot: Scaling observational causal inference at LinkedIn

Co-authors: Kenneth Tay and Xiaofeng Wang

Hosted Search: LinkedIn Search as a managed service

Search functionality is a core part of most data-driven products, and is used widely at LinkedIn. We have long provided a central platform for search functionalities; however, it was not fully managed in the sense that the application teams needed to own and operate the corresponding resources. As data needs grow and an increasingly high number of products want to integrate search, we discovered a need for a fully managed self-service platform to completely democratize search for all of our product teams. In this post, we will talk about Hosted Search, our new search solution that allows product developers to integrate search functionality with minimal onboarding and no maintenance or operational overhead, allowing them to instead focus on their products and what they care about most: creating value for our members.

Our Approach to Research and A/B Testing

We are constantly striving to improve the experience on LinkedIn for our members and customers, with research and experimentation, such as A/B Testing, playing a key role in that work.

Operating System Snapshot Automation

Co-authors: Rohit Jamuar, Tianxin Zhou

Building LinkedIn's Skills Graph to Power a Skills-First World

Co-authors: Sofus Macskássy, Yi Pan, Ji Yan, Yanen Li, Di Zhou, Shiyong Lin

TopicGC: How LinkedIn cleans up unused metadata for its Kafka clusters

Introduction Apache Kafka is an open-sourced event streaming platform where users can create Kafka topics as data transmission units, and then publish or subscribe to the topic with producers and consumers. While most of the Kafka topics are actively used, some are not needed anymore because business needs changed or the topics themselves are ephemeral. Kafka itself doesn’t have a mechanism to automatically detect unused topics and delete them. It is usually not a big concern, since a Kafka cluster can hold a considerable amount of topics, hundreds to thousands. However, if the topic number keeps growing, it will eventually hit some bottleneck and have disruptive effects on the entire Kafka cluster. The TopicGC service was born to solve this exact problem. It was proven to reduce Kafka pressure by deleting ~20% of topics, and improved Kafka’s produce and consume performance by at least 30%.

Render Models at LinkedIn

Co-Authors: Mahesh Vishwanath, Eric Babyak, Sonali Bhadra, Umair Saeed

(Re)building Threat Detection and Incident Response at LinkedIn

Co-authors: Sagar Shah and Jeff Bollinger

Career stories: From Hollywood videographer to frontend engineer

Originally an LA-based videographer and self-taught developer, Kiope wanted to turn her hobby into a career. Following a coding bootcamp and a stint at a startup, she joined LinkedIn as a frontend (UI) engineer. Based in San Francisco, Kiope speaks about her meaningful accessibility work, impact at scale, and career growth as an aspiring manager.

How LinkedIn Ditched the "One Size Fits All" Hiring Approach for InfoSec and Won

What drew me to LinkedIn was the chance to take on the challenge of scaling and adapting the security framework for the world’s largest professional network. Today, our Information Security (InfoSec) team is responsible for protecting LinkedIn’s infrastructure and data for our members, customers, and employees. As our platform has scaled, our security programs and strategies have grown in tandem to keep up with the company’s trajectory. But we haven’t stopped there. We want to be more than just protectors of our members, customers, and all the data we process internally and externally; we also want to expand our efforts to proactively support the goals of our business and the people who join our team. I’m a firm believer that cybersecurity is as much a business function as a technical one, and that means we have a crucial role in contributing to our company’s ongoing growth.

Career stories: Four engineering careers. One LinkedIn.

LinkedIn’s Next Play culture celebrates transformational growth and internal mobility, and engineering leader Shalini is its biggest champion. Based in Silicon Valley, this mom of three walks us through her impactful career journey from our LinkedIn consumer and data teams, then to working on the LinkedIn Sales Solutions team, to now working for LinkedIn Talent Solutions.

Career stories: Mobilizing learners worldwide

After stints at a Madrid startup and a Chicago financial-services company, Android mobile engineer Jose was looking for his ideal role in the Bay Area. Now based in San Francisco, he dives into the rewarding technical challenges his team is collaborating on, and how he’s having a meaningful impact on millions of learners worldwide as a core member of the LinkedIn Learning mobile team.

LinkedIn’s GraphQL journey for integrations and partnerships: How we accelerated development by 90%

Co-authors: Mimi Chen, Calvin Lei, and Amit Yadav

Skyfall: eBPF agent for infrastructure observability

Currently, LinkedIn infrastructure is composed of hundreds of thousands of hosts across multiple data centers. Observability into our infrastructure makes it possible for us to focus on the health and performance of our critical services to provide the best experience to our members. With LinkedIn's large infrastructure growth over the past few years, observability has become more critical to pinpoint the potential root causes for any infrastructure failure or anomaly. There are a few elegant in-house monitoring systems at LinkedIn that provide network switch level metrics, logs, and even flow-level visibility by sampling packets going through our network. However, all of these rely on sampling or some kind of periodic polling of data, which for any meaningful sampling rate generates a very large volume of data to be processed and analyzed.

Real-time analytics on network flow data with Apache Pinot

The LinkedIn infrastructure has thousands of services serving millions of queries per second. At this scale, having tools that provide observability into the LinkedIn infrastructure is imperative to ensure that issues in our infrastructure are quickly detected, diagnosed, and remediated. This level of visibility helps prevent the occurrence of outages so we can deliver the best experience for our members. To provide observability, there are various data points that need to be collected, such as metrics, events, logs, and flows. Once collected, the data points can then be processed and made available, in real-time, for engineers to use for alerting, troubleshooting, capacity planning, and other operations.

Feathr joins LF AI & Data Foundation

Co-authors: Hangfei Lin, Jinghui Mo

Career stories: Rejoining LinkedIn to scale our media infrastructure

Originally from Argentina, systems & infrastructure engineering leader Federico was a founding member of the Media Infrastructure team in 2015. Now based in Bellevue, Wash., Federico shares how his supportive mentor, LinkedIn’s “sweet spot” scale, and the distinctive engineering challenges here ultimately brought him back to LinkedIn in 2019.

Operating system upgrades at LinkedIn’s scale

Co-authors: Hengyang Hu, Dinesh Dhakal, Kalyanasundaram Somasundaram

Challenges and practical lessons from building a deep-learning-based ads CTR prediction model

Co-authors: Ruoyan Wang, Sirou Zhu, Chengming Jiang

Improving Post Search at LinkedIn

Co-authors: Amol Chandla, Jagadeesan Sundaresan, Siddharth Pratap Singh, Saipriya Kumar, and Anand Kishore

REACH turns five - Celebrating the power of apprenticeships

Every tech company wants to hire great talent, but many have historically hired based on pedigree -- where candidates went to school, where they worked and who they know. However, there's a pool of qualified talent who have followed non-traditional paths but may have been overlooked in traditional recruiting pipelines. Five years ago, we piloted our REACH program to help solve this problem by providing those with non-traditional software engineering backgrounds the opportunity to get their foot in the door and develop their coding skills. What began as a small six-month program across three engineering focus areas (UI, Apps, Mobile) has grown into a multi-year apprenticeship model that has helped over 170 people transform their careers and become a core part of our hiring rhythm. Today, we are celebrating REACH’s five year anniversary, which includes more than 90 active program apprentices working on more than 10 focus areas throughout North America, with plans to expand in the coming years.

Applying multitask learning to AI models at LinkedIn

Co-authors: Ji Yan, Sen Zhou, Dansong Zhang, Anastasiya Karpovich

Career stories: Breaking barriers with LinkedIn

After interning with us, Beatrix resonated with the culture and community she found at LinkedIn, and rejoined us post-undergrad. As she continues exploring her passion for frontend (UI) and accessibility engineering, she shares why launching her career with LinkedIn is one of the best decisions she has made.

Measuring downstream impact on social networks by using an attribution framework

Co-authors: Qiannan Yin, Derek Koh, and Jenny Wu

Measuring marketing incremental impacts beyond last click attribution

Co-authors: Maggie Zhang, Joyce Chen, and Ming Wu

Career stories: Next plays, jungle gyms, and Python

Since she was a child, Deepti has been motivated to help people. This drive led her on a career journey with many pivots and moves — akin to navigating a children’s jungle gym — between industries and around the world. Based in Bangalore, this biomedical engineer turned data scientist shares how LinkedIn helped her gain new technical skills, dive into meaningful work, and grow.

Welcome to the Inaugural LinkedIn-Cornell grant recipients

At LinkedIn, we're always looking for ways to improve our platform for members and customers and believe collaborating with academia can advance these efforts. That is why we are excited to announce the first group of grant recipients from our partnership with the Cornell Ann S. Bowers College of Computing and Information Science (Cornell Bowers CIS). These grants provide financial resources to Ph.D. students and faculty for research and amplify efforts in diversity, equity, and inclusion within Cornell Bowers CIS.

Migration madness: How to navigate the chaos of large cross-team initiatives towards a common goal

Co-authors: Lily Wittle and Cathy Ji

INWED2022: How Skills Can Help Shape Your Future in Engineering

Today is an exciting day for female engineers! June 23 marks the ninth anniversary of International Women in Engineering Day (INWED). Developed by the Women’s Engineering Society (WES), this annual day of recognition plays a vital role in encouraging more young women to explore the benefits of an engineering career and showing the progress made in the field.

LinkedIn’s journey to Java 11

Introduction At LinkedIn, we are committed to deliver a best-in-class platform experience for our members. One of the technologies that we use to do that is Java, an object-oriented programming language that produces software for multiple platforms. We are a huge consumer of Java, running over 1000 Java applications on 320,000+ hosts. Based on the promised performance improvements of the G1 garbage collector (G1GC) and other well-documented benefits captured in this article, we made the decision to begin our journey from Java 8 to Java 11.

Under the hood: How we built API versioning for LinkedIn Marketing APIs

Co-authors: Nihit Purwar, Demian Lessa, and Shruthi Jeganathan

Towards data quality management at LinkedIn

Co-authors: Liangzhao Zeng, Ting Yu (Cliff) Leung, Jimmy Hong, and Kevin Lau

Overcoming challenges with Linux cgroups memory accounting

Introduction LinkedIn's de facto search solution, Galene, is a Search-as-a-Service infrastructure that powers a multitude of search products at LinkedIn, from member-facing searches (such as searching for jobs or other members) to internal index searches. Galene’s responsiveness and reliability are paramount as it caters to many critical features.

Entity Routing Service (ERS) load test evolution

LinkedIn is operating today at a scale of nearly 1.6M graph queries per second (QPS), with 830M+ members. At this kind of scale (and seeing roughly 34% year-over-year growth), it can be hard to predict and manage capacity. Beyond being prepared for everyday operations, we also want to ensure LinkedIn could handle a catastrophic issue, such as one data center experiencing critical issues and going offline, which would in turn affect our customers’ and members' experience on the site. This is called High Availability/Disaster Recovery (HA/DR). To address both these challenges, we do load tests every day by moving end user traffic (referred to as a “TrafficShift”) to a specific target data center and then simulating the load scenarios with a specific QPS target. For context, LinkedIn operates as active-active segmented traffic across multiple data centers, and each member has a “home” in a particular data center. During this kind of TrafficShift, we move those members’ homebase from one data center to another. This is referred to as Member Sticky Routing (MSR), and you can read more about it in this blog post. To maintain member segmentation and session consistency, MSR is used as first-level routing at the Apache Traffic Server (ATS level - L1 proxy).

Career stories: Taking LinkedIn to Bellevue, and mentorship global

From starting our LinkedIn Bellevue office and launching a successful e-newsletter to driving our efforts on equity and making mentorship more accessible, Bef represents LinkedIn’s aspiration of transformational growth through and through. This dad and engineering leader shares with us how LinkedIn has helped him foster his personal and professional passions:

LinkedIn’s simple, unified, and discoverable Job Postings API

At LinkedIn, our engineering teams build what’s needed to connect the world’s professionals to economic opportunity. For the Partner Solutions Engineering (PSE) team, this includes the technology that powers our talent and hiring solutions and support for how jobs get posted on the LinkedIn platform.

One-stop MLOps portal at LinkedIn

Co-authors: Eing Ong, Shannon Bain, and Daniel Qiu

Supporting large fanout use cases at scale in Venice

image-of-venice-read-path

Expanding our Bug Bounty Program

Co-authors: Ameen Maali and Surbhi Jain

Shifting left on governance: DataHub and schema annotations

Co-authors: Joshua Shinavier and Shirshanka Das

Introducing the LinkedIn Scholars Program: Partnering with academia to solve real-world business challenges

image-that-says-introducing-the-linkedin-scholars-program-partnering-with-academia-to-solve-real-world-business-challenges

Improving job matching with machine-learned activity features

image-of-model-for-machine-learned-activity-embedding

Scaling storage in the datacenter with RoCE

image-of-sonic buffer-scheduler-configuration

Community building recommender for group chats in LinkedIn Messaging

Co-authors: Adam Leon and David Golland

Matching external companies to LinkedIn’s Economic Graph at scale

Co-authors: Remi Mir, Tianhao Lu, Xiaoqiang Luo, Yunpeng Xu

Addressing the last mile problem with MySQL high availability

image-of-my-sql-cluster

Client Zero Protocol: Taking intelligent risks to go beyond the traditional SDLC control framework

Introduction Traditional IT processes supporting change management rely on a clear separation of the environments involved in the software development lifecycle (SDLC). Developments and Unit Tests are performed in the “Development” environments, Quality Assurance and User Tests are conducted in the User Acceptance Testing (UAT) environments, and the “Production” environment is the sacred place where only real-life transactions and activities are performed.

Open sourcing Feathr – LinkedIn’s feature store for productive machine learning

We are open sourcing Feathr – the feature store we built to simplify machine learning (ML) feature management and improve developer productivity. At LinkedIn, dozens of applications use Feathr to define features, compute them for training, deploy them in production, and share them across teams. With Feathr, users reported significantly reduced time required to add new features to model training workflows and improved runtime performance compared to previous application-specific feature pipeline solutions.

Load-balanced Brooklin Mirror Maker: Replicating large-scale Kafka clusters at LinkedIn

At LinkedIn, Apache Kafka is used heavily to store all kinds of data, such as member activity, log storage, metrics storage, and a multitude of inter-service messaging. LinkedIn maintains multiple data centers with multiple Kafka clusters per data center, each of which contains an independent set of data. Mirroring (i.e., replicating) Kafka topics across the clusters and data centers not only enables easy accessibility and analytics by aggregation of data from multiple data centers, but also fault tolerance and recovery during cluster or data-center wide outages. Additionally, mirroring enables the ability to isolate the data access for performance or security reasons and limits the data availability based on security domains. Streaming infrastructure at Linkedin mirrors more than seven trillion messages per day between the clusters using Brooklin (Github repo).

The journey to build an explainable AI-driven recommendation system to help scale sales efficiency across LinkedIn

Co-authors: Jilei Yang, Parvez Ahammmad, Fangfang Tan, Rodrigo Aramayo, Suvendu Jena, Jessica Li

Leveraging configurable components to scale LinkedIn's Profile experience

Background and motivation The Profile lies at the center of LinkedIn’s ecosystem and is the primary tool members use to showcase their professional identity. To meet the growing representational needs of our members, we had to acknowledge some of the challenges that come with building at scale:

Building a next-generation forms framework at LinkedIn

Co-authors: Madhura Deo, DJ Kim, and Ning Jin

Supporting innovation in AI research: Partnering with the Cornell Ann S. Bowers College of Computing and Information Science (Bowers CIS)

image-that-says-supporting-innovation-in-ai-research-partnering-with-the-cornell-ann-s-bowers-college-of-computing-and-information-science-bowers-CIS)

Opal: Building a mutable dataset in data lake

opal-data-flow

FastTreeSHAP: Accelerating SHAP value computation for trees

graph-of-TreeSHAP-vs-FastTreeSHAPv1-vs-FastTreeSHAPv2-Adult

Performance-Adaptive Sampling Strategy (PASS) for GNNs: Open sourcing PASS

Co-authors: Jaewon Yang, Minji Yoon, Sufeng Niu, Dash Shi, and Qi He

Near real-time features for near real-time personalization

Co-authors: Rupesh Gupta, Sasha Ovsankin, Qing Li, Seunghyun Lee, Benjamin Le, and Sunil Khanal

Why am I seeing this ad?

Co-authors: Dhruv Bansal, Aanchal Somani, Sneha Dewan, and Vikrant Mahajan

Hodor: Detecting and addressing overload in LinkedIn microservices

figure-of-framework

Accelerating the LinkedIn Experience with Azure Front Door

Co-authors: Samir Jafferali, Viranch Mehta, and Thanglalson Gangte

Career stories: Dream big and work hard

Originally from South India, Mounika relocated from our parent company in Seattle, Washington to Sunnyvale, California, to help improve LinkedIn’s business platform as a backend engineer. She shares how her female engineering mentors here have been an invaluable support system amidst her transition, and how she’s paying it forward to the next generation of female technologists.

Using Bayesian optimization for balancing metrics in recommendation systems

Co-authors: Yunbo Ouyang,Viral Gupta, Kinjal Basu, Cyrus Diciccio, Brendan Gavin, and Lin Guo.

How our content abuse defense systems work to keep members safe

To create a safe and trusted experience for our members, our Trust & Safety (TnS) team strives to keep content that violates our Professional Community Policies off of LinkedIn. In this blog post, we’ll provide insight into how we try to ensure conversations remain respectful and professional on our platform.

DARWIN: Data Science and Artificial Intelligence Workbench at LinkedIn

Co-authors: Varun Saxena, Harikumar Velayutham, and Balamurugan Gangadharan

A closer look at how LinkedIn integrates fairness into its AI products

Co-authors: Heloise Logan, Preetam Nandy, Kinjal Basu, and Sakshi Jain

Career stories: Blending analytics and design thinking

An experienced quality assurance analyst and credentialed graphic designer, Michaela was seeking in her next role added flexibility, and the opportunity to blend both her analytical and creative skills. She found it in LinkedIn Engineering’s Trust & Safety team in her hometown of Omaha, Nebraska.

Spike detection in Alert Correlation

Introduction LinkedIn’s stack consists of thousands of different microservices and the associated complex dependencies among them. When a production outage happens due to an issue with misbehaving services, finding the exact service responsible for the outage is challenging and time-consuming. Although each service has multiple alerts configured in a distributed infrastructure, finding the real root cause of the issue during an outage is like finding a needle in a haystack, even with all the right instrumentation. This is because every service in the critical path of a client request could have multiple active alerts. The lack of a proper mechanism to derive meaningful information from these disjointed alerts often leads to false escalations, causing increased issue resolution time. On top of all this, imagine being woken up in the middle of the night by a NOC engineer about a site outage that they believe is caused by your service, only to realize that it’s a false escalation that is not caused by your service.

2021 in review: New frontiers in innovation and scale for our Engineering teams

It’s been an incredible journey this year for our LinkedIn Engineering organization, full of new technology milestones, industry-leading innovations, and a continued focus on designing useful, responsible features for our members and customers. After the dramatic shifts that 2020 brought to the ways we live, work, and play, in 2021 we focused on learning to adapt—and thrive—in the new world we’re all living in. With so many exciting moments throughout the year, I wanted to take the time to reflect on some of our highlights. Of course, it would be impossible to fit all the achievements of the year into a single blog post, but it’s my hope that this post will give everyone on our team a sense of pride, and that it will provide a glimpse for the outside world into what we work on every day at LinkedIn.

Career stories: Making a difference with LinkedIn Engineering, globally

After launching his engineering career with LinkedIn in Argentina, Juan jumped at the chance to open a new office across the world in Dublin. In his nine years as an engineer with the company, Juan has found his leadership role in building global engineering teams across LinkedIn to be the most rewarding part of his career yet.

Evolving LinkedIn’s analytics tech stack

Co-authors: Steven Chuang, Qinyu Yue, Aravind Rao, and Srihari Duddukuru

Completing a member knowledge graph with Graph Neural Networks

Co-authors: Jaewon Yang, Jiatong Chen, and Yanen Li

Career stories: Welcoming LinkedIn’s largest engineering class

As we welcome our largest class of experienced engineers to date, we sat down with LinkedIn’s head of global engineering recruitment, Chris Jenkins, to hear more about his own career journey to LinkedIn, engineering’s tremendous growth, and tips for engineering recruits.

Career stories: A cross-country, family move

After years of hopping between coasts, Charles found a home for his family and his ambitions in Silicon Valley. LinkedIn made the transition seamless, setting him up for success from day one, even through the challenges of the pandemic.

Building in the cloud with remote development

Co-authors: Shivani Pai Kasturi and Swati Gambhir

How our AI is helping members navigate the Great Reshuffle

Co-authors: Liangjie Hong, Alexandre Patry, Benjamin Le, and Fei Chen

Distributed Firewall (DFW): Network security at the host level at LinkedIn

Co-authors: Matthew Davidson, Walter Marchuk, Andreas Zaugg, Michael Garate, and William Buenzle

Project Magnet, providing push-based shuffle, now available in Apache Spark 3.2

Co-authors: Venkata Krishnan Sowrirajan and Min Shen

Our approach to building transparent and explainable AI systems

We’ve also showcased our process for building more inclusive and equitable products and programs to ensure they empower individuals regardless of their background or social status. In this post, we will describe how transparent and explainable AI systems can help verify that these needs have been met.

Distributed tier merge: How LinkedIn tackles stragglers in search index build

Co-authors: Andy Li and Hongbin Wu

Scaling LinkedIn's Hadoop YARN cluster beyond 10,000 nodes

Co-authors: Keqiu Hu, Jonathan Hung, Haibo Chen, and Sriram Rao

Using deep learning to detect abusive sequences of member activity

Co-authors: James Verbus and Beibei Wang

HTTP/2 in infrastructure: Ambry network stack refactoring

Co-authors: Ze Mao, Matt Wise, Casey Getz, Justin Lin, Ashish Singhai, and Rob Block

Lambda Learner: Nearline learning on data streams

Co-authors: Kirill Talanine, Jeffrey D Gee, Rohan Ramanath, Konstantin Salomatin, Gungor Polatkan, Onkar Dalal, and Deepak Kumar

Building Microsoft-powered native video meetings on LinkedIn

Co-authors: Crystal Hsieh, Kyle Huynh, Sameera Padhye, Kalyan Punukollu, Federico Hlawaczek, Pavan Mellamputi, and Christian Byza

DuaLip: Solving extreme-scale linear programs for web applications

Co-authors: Kinjal Basu, Yao Pan, Rohan Ramanath, Konstantin Salomatin, Amol Ghoting, and S. Sathiya Keerthi

Optimizing People You May Know (PYMK) for equity in network creation

Co-authors: Qiannan Yin, Yan Wang, Divya Venugopalan, Cyrus Diciccio, Heloise Logan, Preetam Nandy, Kinjal Basu, and Albert Cui

New Recruiter & Jobs: The largest enterprise data migration at LinkedIn

Co-authors: Xiaoyang Gu, Xie Lu, and Xiaoguang Wang

From daily dashboards to enterprise grade data pipelines

Within a matter of hours of each day beginning, we ingest tens of billions of records from online sources to HDFS, aggregated across 50+ offline data flows, and visualize 40+ metrics across our business lines to support a dashboard providing company leaders with timely and critical business insights. Dubbed the Daily Executive Dashboard (DED), this dashboard contains critical growth, engagement, and bookings metrics that indicate the health of LinkedIn’s business. In this blog post, we’ll spotlight the challenges to DED’s delivery and performance posed by the growing complexity of computing and aggregating these metrics, and how this dashboard transformed our approach to data pipelines across the company.

TonY joins LF AI & Data Foundation

Co-authors: Keqiu Hu, Jonathan Hung, and Junfan Zhang

Model health assurance at LinkedIn

Co-authors: Rajeev Kumar, Dhritiman Das, Saatwik Nagpal, Shubham Gupta, and Vikram Singh

Helping members connect to opportunity through AI

Co-authors: Rupesh Gupta and Lingjie Weng

Text analytics on LinkedIn Talent Insights using Apache Pinot

Co-authors: Siddharth Teotia and Tim Santos

Releasing Bluepill to the Mobile Native Foundation

In March, LinkedIn joined the recently announced Mobile Native Foundation (MNF), a Linux Foundation project launched with the support of Airbnb, Corellium, Elotl, Flare.build, GitHub, GogoApps, Haystack, Line, Lyft, Microsoft, screenplay.dev, Slack, Solid Software, and Spotify. The goal of the Mobile Native Foundation is to disseminate knowledge and tooling needed for building mobile applications at scale. We plan to help foster a diverse community that encourages collaboration and contributes libraries and tools to move the industry forward.

Homepage feed multi-task learning using TensorFlow

Co-authors: Ian Ackerman and Saurabh Kataria

The exabyte club: LinkedIn’s journey of scaling the Hadoop Distributed File System

Co-authors: Konstantin V. Shvachko, Chen Liang, and Simbarashe Dzinamarira

An update on Responsible AI at LinkedIn

Co-authors: Parvez Ahammad, Kinjal Basu, Shaunak Chatterjee, Sakshi Jain, Ryan Rogers, and Guillaume Saint-Jacques

Greykite: A flexible, intuitive, and fast forecasting library

Co-authors: Reza Hosseini, Albert Chen, Kaixu Yang, Sayan Patra, Rachit Arora, and Parvez Ahammad

How to build an effective professional network on LinkedIn: Some data-driven insights

Co-authors: YinYin Yu, Guillaume Saint Jacques, and Paul Matsiras

Building a more intuitive and streamlined search experience

Co-authors: Rashmi Jain and Sonali Bhadra

Solving for the cardinality of set intersection at scale with Pinot and Theta Sketches

Co-authors: Vincent Wang, Siddharth Teotia, Manoj Thakur, and Mayank Shrivastava

Rethinking site capacity projections with Capacity Analyzer

While site outages are inevitable, it’s our job to minimize both the duration of outages and the likelihood for an outage to occur. One of our preemptive measures is in the way we determine overall site capacity and health on an everyday basis, in that we load-test in production. There’s an elegant system to bucket and route members to specific data centers from our edge nodes, which also provides an override capability in case we need to fail out of a data center—or, in our case for testing, fail out parts of almost all data centers and direct desired traffic to the target data center instead.

Solving the data integration variety problem at scale, with Gobblin

Co-authors: Chris Li, Kevin Lau, and Subbu Sanka

LinkedIn Sales Insights: Quality data foundations for smarter sales planning

Co-authors: Sabeer Thajudeen, Dan Melamed, Sashikanth Damaraju, Jean Baptiste Chery, Tai Bendit, and Ajay Singh

Leveraging behavior analytic computation for anti-abuse defenses

LinkedIn is committed to providing a trusted environment to keep our nearly 740 million members safe. Part of that commitment involves protecting against various types of abuse on the platform. Fighting abuse presents many challenges; one example is when bad actors use bots for large-scale attacks, while another is when attack signals constantly evolve to adapt to existing defenses. In these scenarios, the adaptability of the adversary can limit the effectiveness of our existing anti-abuse models over time, which is why it's important to have both high-quality features and a faster response time to enhance our defense resilience.

Using the LinkedIn Fairness Toolkit in large-scale AI systems

Co-authors: Preetam Nandy, Yunsong Meng, Cyrus DiCiccio, Heloise Logan, Amir Sepehri, Divya Venugopalan, Kinjal Basu, and Noureddine El Karoui

Open source update: School of SRE

Co-authors: Akbar KM and Kalyanasundaram Somasundaram

Taming memory fragmentation in Venice with Jemalloc

Sometimes, an engineering problem arises that might make us feel like maybe we don't know what we're doing, or at the very least, forces us out of the comfort zone of our area of expertise. That day came for the Venice team at Linkedin when we began to notice that some Venice processes would consume all available memory and crash if given enough time to run. This ended up being due to memory fragmentation. In this blog post, we'll explain the different ways we tried to diagnose the problem, and how we solved it.

Creating a secure and trusted Jobs ecosystem on LinkedIn

Co-authors: Sakshi Jain, Grace Tang, Gaurav Vashist, Yu Wang, John Lu, Ravish Chhabra, Shruti Sharma, Dana Tom, and Ranjeet Ranjan

Smart Argument Suite: Seamlessly connecting Python jobs

Co-authors: Jun Jia and Alice Wu

Budget-split testing: A trustworthy and powerful approach to marketplace A/B testing

Co-authors: Min Liu, Vangelis Dimopoulos, Elise Georis, Jialiang Mao, Di Luo, and Kang Kang

How LinkedIn turned to real-time feedback for developer tooling

Over the last year, we have been using real-time feedback to evolve our tooling and provide a more productive experience for LinkedIn's developers. It’s helped us double our feedback participation, and more importantly, better tailor our recommendations and improvements.

FastIngest: Low-latency Gobblin with Apache Iceberg and ORC format

Co-authors: Zihan Li, Sudarshan Vasudevan, Lei Sun, and Shirshanka Das

A/B testing at LinkedIn: Assigning variants at scale

Co-authors: Alexander Ivaniuk and Weitao Duan

Coral: A SQL translation, analysis, and rewrite engine for modern data lakehouses

Co-authors: Walaa Eldin Moustafa, Wenye Zhang, Sushant Raikar, Raymond Lam, Ron Hu, Shardul Mahadik, Laura Chen, Khai Tran, Chris Chen, and Nagarathnam Muthusamy

DataHub: Popular metadata architectures explained

When I started my journey at LinkedIn ten years ago, the company was just beginning to experience extreme growth in the volume, variety, and velocity of our data. Over the next few years, my colleagues and I in LinkedIn’s data infrastructure team built out foundational technology like Espresso, Databus, and Kafka, among others, to ensure that LinkedIn would survive and thrive through the next wave of growth. A few years later, I became the tech lead for what was then a pretty small “data analytics infrastructure” team that ran and supported LinkedIn’s Hadoop usage, and also maintained a hybrid data warehouse spanning Hadoop and Teradata.

From Lambda to Lambda-less: Lessons learned

Co-authors: Xiang Zhang and Jingyu Zhu

Pegasus Data Language: Evolving schema definitions for data modeling

Pegasus Data Schema (PDSC) is a Pegasus schema definition language that has been used for data modeling with Rest.li services for years. It's the underlying language that helps define data models, describe the data returned by REST endpoints, and generate derivative schemas for other uses, such as XML schemas and various database schemas. However, writing PDSC files is hard and often error-prone because they can be verbose and lack integrated developer environment (IDE) support.

Building a smooth Stories experience on iOS

Co-authors: Chen Yuan, Lauren Caponong, and Derrick Showers

Dagli: Faster and easier machine learning on the JVM, without the tech debt

In recent years, we’ve been fortunate to see a growing number of excellent machine learning tools, such as TensorFlow, PyTorch, DeepLearning4J, and CNTK for neural networks, Spark and Kubeflow for very-large-scale pipelines, and scikit-learn, ML.NET, and the recent Tribuo for a wide variety of common models. However, models are typically part of an integrated pipeline (including feature transformers), and constructing, training, and deploying these pipelines to production remains more cumbersome than it should be. Duplicated or extraneous work is often required to accommodate both training and inference, engendering brittle “glue” code that complicates future evolution and maintenance of the model and creating long-term technical debt.

Behind the scenes: The new LinkedIn Jobs experience

Co-authors: Hang Li, David Huang, Shuangfei Li, and Shannon Zhang

How LinkedIn scales compatibility testing

Co-authors: Nima Dini and Dan Sully

Under the hood: Updating LinkedIn's UI

Co-author: Jitesh Gandhi and Eric Babyak

Magnet: A scalable and performant shuffle architecture for Apache Spark

Co-authors: Min Shen, Chandni Singh, Ye Zhou, and Sunitha Beeram

Fixing Linux filesystem performance regressions

As companies grow, adapt, morph, and mature, one item remains the same: the need for reinvention. Technical infrastructure is no exception. As our member community grew, our priorities were to keep up with that growth, or as we say, ensure continuous “site up.” (Read: adding servers to scale from hundreds to hundreds of thousands.) We ran into challenges about how to plan for this type of scaling—in particular, in keeping the platform images and kernels installed on our servers up to date. We moved forward in fits and starts, reimaging our entire physical server fleet in ad-hoc all-hands efforts in order to respond to various extrinsic factors, such as publicly disclosed CPU bugs.

Pensieve: An embedding feature platform

Co-authors: Benjamin Le, Daniel Gmach, Aman Grover, Roshan Lal, Jerry Lin, Austin Lu, Qingyun Wan, and Leighton Zhang

Building a heterogeneous social network recommendation system

Co-authors: Parag Agrawal, Ankan Saha, Yafei Wang, Aastha Nigam, and Eric Lawrence

Building a better and faster Beam Samza runner

Co-authors: Yixing Zhang, Bingfeng Xia, Ke Wu, and Xinyu Liu

Celebrating Innovation, Success, and the Future at Data Science Week 2020

Last week, the global LinkedIn Data Science team joined together for our third-annual Data Science Week. This virtual event included workshops, conference talks, networking events, an awards ceremony, and a fireside chat with Mohak Shroff, LinkedIn’s Senior Vice President of Engineering. It was a celebration of our team and our successes over the past year. A chance to take a moment as a team to celebrate our accomplishments and reconnect.

GDMix: A deep ranking personalization framework

Our logo is inspired by the chameleon: You can enable personalization on your ranking model with GDMix, bringing a personalized experience to every user, like a chameleon that can match its surroundings.

Our evolution towards T-REX: The prehistory of experimentation infrastructure at LinkedIn

Editor’s note: This blog post is the first in a series providing an overview and history of LinkedIn’s experimentation platform.

Helping members discover communities around interests

Co-authors: Chiachi Lo, Bohong Zhao, and Elina Lin

LIquid: The soul of a new graph database, Part 2

Co-authors: Scott Meyer, Andrew Carter, and Andrew Rodriguez

Production testing with dark canaries

The internet software industry has moved away from long development cycles and dedicated quality assurance (QA) stages, toward a fast-paced continuous-integration/continuous-delivery (CI/CD) pipeline, where new code is quickly written, committed, and pushed to user-facing applications and services. Doing so has dramatically increased iteration speeds, and at LinkedIn, it is not uncommon for new code to be committed and pushed to production within an hour, multiple times a day. While this has been incredibly successful—it has improved developer productivity by allowing us to make changes more quickly—it has also caused problems such as site or service outages when bad code, configuration, or AI models have been pushed to production. Even with complete unit and integration tests, sometimes there is no substitute for production-level testing using both production data and production volume, especially with regards to system performance metrics, such as memory consumption, CPU/GPU usage, concurrency, and latency.

Jhubbub on Helix: Stateless and elastic made easy

Co-authors: Hunter Lee and Dru Pollini

Addressing bias in large-scale AI applications: The LinkedIn Fairness Toolkit

Co-authors: Sriram Vasudevan, Cyrus DiCiccio, and Kinjal Basu

Rebuilding messaging: How we built for extensibility

In the previous blog posts of our “Rebuilding messaging” series, we shared the process of how we designed the system from high-level product and engineering requirements, and how we bootstrapped the data. In this post, we’ll explore why we made extensibility a core aspect of our messaging platform, what that meant for our partner teams, and how we got it done.

The technology behind fighting harassment on LinkedIn

Co-authors: Grace Tang, Montinee Khunvirojpanich, and Ting Chen

Improving Espresso availability with preemptive Helix-managed traffic shift

Co-authors: Gaurav Mishra, Song Lu, Antony Curtis, Shuangyang Yang

Theory vs. Practice: Learnings from a recent Hadoop incident

Co-authors: Sandhya Ramu and Vasanth Rajamani

Quality matches via personalized AI for hirer and seeker preferences

Co-authors: Konstantin Salomatin, Kirill Talanine, Barış Özmen, Gungor Polatkan, Linda Fayad, Arjun Kulothungun, and Deepak Kumar

DeText: A deep NLP framework for intelligent text understanding

Co-authors: Weiwei Guo, Xiaowei Liu, Sida Wang, Huiji Gao, and Bo Long

LIquid: The soul of a new graph database, Part 1

Co-authors: Scott Meyer, Andrew Carter, and Andrew Rodriguez

Rebuilding messaging: How we bootstrapped our platform

Co-authors: Pradhan Cadabam and Jingxuan (Rex) Zhang

Bridging batch and stream processing for the Recruiter usage statistics dashboard

Co-authors: Khai Tran and Steve Weiss

A closer look at the AI behind course recommendations on LinkedIn Learning, Part 2

Co-authors: Sneha Chaudhari, Mahesh Joshi, and Gungor Polatkan

Building LinkedIn Talent Insights to democratize data-driven decision making

Co-authors: Timothy Santos and Jeremy Lwanga

Faster testing on Android with Mobile Test Orchestrator

Introduction When testing Android applications (for example, the LinkedIn mobile app for Android), thorough testing practices can easily lead to a growth in the number of overall tests being performed—from hundreds to thousands to even tens of thousands. Clearly, there is a need to scale significantly, while at the same time keeping test run times reasonably short. This is particularly true in the context of a continuous automated pipeline for development to release, because large applications can have multiple commits in a short time frame, and a pipeline that is fast prevents backups.

Monitoring business performance data with ThirdEye smart alerts

Co-authors: Madhumita Mantri and Tyler James

The impact of slow NFS on data systems

Espresso is LinkedIn's defacto NoSQL database solution. It is an online, distributed, fault-tolerant database that powers most of LinkedIn’s applications including member profiles, InMail (LinkedIn's member-to-member messaging system), sections of the main LinkedIn homepage, our mobile applications, and more. Since Espresso caters to many critical features, its responsiveness and reliability are paramount. Espresso stores about 2PB of data (before replication) across its clusters and its busiest cluster serves around 3M QPS.

A closer look at the AI behind course recommendations on LinkedIn Learning, Part 1

Co-authors: Sneha Chaudhari, Mahesh Joshi, and Gungor Polatkan

Scaling LinkedIn’s Edge with Azure Front Door

Co-authors: Viranch Mehta, Jon Sorenson, Samir Jafferali

Rebuilding messaging: How we designed our new system

Co-authors: Tyler Grant, Armen Hamstra, Cliff Snyder

Open sourcing Kube2Hadoop: Secure access to HDFS from Kubernetes

Co-authors: Cong Gu, Abin Shahab, Chen Qiang, Keqiu Hu

Bringing Project Every Member to life: Open sourcing our Spark inequality A/B testing library

Co-authors: Guillaume Saint-Jacques, Stuart Ambler

LinkedIn’s approach to automated accessibility (A11y) testing

Co-authors: Oliver Tse, Andrew Lee, Melanie Sumner, Renato Iwashima

Understanding dwell time to improve LinkedIn feed ranking

Co-authors: Siddharth Dangi, Johnson Jia, Manas Somaiya, and Ying Xuan

Spark-TFRecord: Toward full support of TFRecord in Spark

Co-authors: Jun Shi, Mingzhou Zhou

How LinkedIn handles merging code in high-velocity repositories

The product ecosystem at LinkedIn is vast, and managing its infrastructure can be daunting. Nearly 10 years ago, we transitioned away from a single monolithic codebase toward a microservice architecture, allowing teams to manage their own repositories. Today, we have thousands of developers working on more than ten thousand active repositories, some of which have upwards of 300 commits a day flowing through them.

Introducing Apache Pinot 0.3.0

Built at LinkedIn, Pinot is an open source, distributed, and scalable OLAP data store that we use as our de-facto near-real-time analytics service. We’ve previously discussed how and why we built Pinot to power a wide spectrum of use cases, including internal business intelligence dashboards to analyze highly-dimensional data and “Who Viewed My Profile” to deliver member-facing analytics at high throughput and low latency.

Building with agility for our members in times of need

The needs of our members and customers have evolved rapidly in recent weeks, and we’ve been focused on evolving alongside them to meet their needs to the best of our ability. Recently, we introduced the ability for healthcare and essential organizations that have urgent hiring needs to post jobs for free on LinkedIn, and unlocked free learning paths on LinkedIn Learning to help people navigate the shift to remote work. Both updates were in service of providing our members access to tools and skills to help them navigate these uncertain macroeconomic times. 

How we reduced latency and cost-to-serve by merging two systems

Co-authors: Xiang Zhang, Estella Pham, and Ke Wu

Feature spotlight: Building a horizontal carousel on Android

LinkedIn recently launched a new feature on Profile called Featured Items. It allows members to pin a post, article, or uploaded media to their profiles to showcase their work. Because the range of what members can choose to highlight is so broad, it was important that the feature was built to accommodate them all. We were excited to release Featured Items into the world, but before we did that, we had to make sure that Featured Items would have a consistent viewing experience. As we built out this experience, we came across a problem of how to deliver a horizontal carousel on Android that was flexible to accommodate different types of content our members would want to feature.

Rapid experimentation through standardization: Typed AI features for LinkedIn’s feed

Serving the most relevant information for LinkedIn members in the homepage feed requires a massive effort—hundreds of features are used to personalize content for hundreds of millions of members. For each homepage visit, our machine learning models have to find and surface the best activity across a member's whole network, and they have to source that content from multiple online systems. For our members with over 5,000 connections, the amount of data processed quickly balloons, as we want to consider the likes, comments, and posts from each person in their network. Operating at this scale can make innovation challenging.

Building a platform: Learnings from our pursuit for leverage

Co-authors: Jefferson Lai, Vinyas Maddi, and Vijay Ramamurthy

Building inclusive products through A/B testing

Co-authors: Guillaume Saint-Jacques, Amir Sepehri, Nicole Li, and Igor Perisic

Advanced schema management for Spark applications at scale

Co-authors: Walaa Eldin Moustafa, Wenye Zhang, Adwait Tumbde, Ratandeep Ratti

InSearch: LinkedIn’s new message search platform

Co-authors: Suruchi Shah and Hari Shankar

Preparing your infrastructure for a WFH shift

Title card

Data Sentinel: Automating data validation

Co-authors: Arun Swami, Sriram Vasudevan, Sailesh Mittal, Jiefu Zheng, Joojay Huyn, Audrey Alpizar, Changling Huang, Maneesh Varshney, Adrian Fernandez

How we improved latency through projection in Espresso

Co-authors: Xiang Zhang and Chuck Jerian

Analyzing anomalies with ThirdEye

Co-authors: Yen-Jung Chang, Yang Yang, Xiaohui Sun, and Tie Wang

Open sourcing DataHub: LinkedIn’s metadata search and discovery platform

Co-authors: Kerem Sahin, Mars Lan, and Shirshanka Das Finding the right data quickly is critical for any company that relies on big data insights to make data-driven decisions. Not only does this impact the productivity of data users (including analysts, machine learning developers, data scientists, and data engineers), but it also has a direct impact on end products that rely on a quality machine learning (ML) pipeline. Additionally, the trend towards adopting or building ML platforms naturally begs the question: what is your method for internal discovery of ML features, models, metrics, datasets, etc.?

Coding Conversations: Four teams, three tracks, two offices

Editor's Note: LinkedIn Engineering is dedicated to solving complex problems at scale to create economic opportunity for every member of the global workforce. This challenge provides our engineers with the opportunity to build their technical skills as they make this mission a reality. In this series, we’ll talk about the career development of the people that comprise our Engineering team. 

How we retired Python 2 and improved developer happiness

Nearly 20 years after the first release of Python 2 and 11 years after the first release of Python 3, the Python development community has retired Python 2.7, the last of the Python 2 series. This marks the end of all upstream support for Python 2, including bug and security fixes, and allows developers to devote their time fully to Python 3, which is faster, delivers more consistency, and brings lots of great features for developers to take advantage of (i.e., asyncio, type hinting).

Keeping LinkedIn professional by detecting and removing inappropriate profiles

Our members place their trust in us, and expect and deserve a safe and trusted community where they can express themselves professionally. We are constantly investing in a variety of systems (Automated Fake Account Detection at LinkedIn, Defending Against Abuse at LinkedIn’s Scale, and How We’re Protecting Members From Fake Profiles) to help automatically detect and remediate behaviors that violate our Terms of Service, such as spam, harassment, scams, and account takeovers. In this post, we’ll detail our approach for handling one particular type of bad behavior: profiles that contain inappropriate content. This can range from profanity to advertisements for illegal services.

Upgrading to RHEL7 with minimal interruptions

Pinot, an open source scalable distributed columnar OLAP data store, is used extensively across almost all areas of LinkedIn. It drives member-facing use cases such as Feed Item Popularity and Who Viewed My Profile, supports recruiter tools such as Talent Intelligence, and holds the data for our internal unified metrics platform. Pinot adoption has grown dramatically in the last few years and we now have several thousand servers working continuously to serve real-time queries.

Building a blazingly fast Android app, Part 2

Monitor, profile, optimize, and ramp. This is the cycle of performance operations that we rely on to improve our Android app. Part 1 of this blog series covered monitoring and profiling — monitoring gives us the “true north” of optimizations to make data-driven decisions, while profiling helps us pinpoint the areas of opportunity. In part 2, we will look at specific optimizations that have been applied to our Android app.

Making the LinkedIn experimentation engine 20x faster

Co-authors: Alexander Ivaniuk, Jingbang Liu

Building a blazingly fast Android app, Part 1

Defined as the speed at which an app loads and responds to a member, app performance is critical to an app’s success. When an app responds slowly to a member interaction, it’s an unsatisfactory experience. In order to maintain a reliable and consistent member experience, we have a dedicated performance engineering team to monitor and troubleshoot performance issues. However, the process for identifying negative trends and the underlying causes of poor performance has not always been as sophisticated and nimble as it is today.

LiTr: A Lightweight Video/Audio Transcoder for Android

If a picture’s worth a thousand words, then what about a video? 

On LinkedIn Learning: Career advice from leadership in tech

One of the most important traits we look for when hiring engineering talent is a growth mindset: the ability to learn and adapt to changing conditions and technologies over time. This is partly because, as engineers, we work at the forefront of tech, which means we constantly seek new and exciting ways to unlock the potential of technology. Aside from our own explorations, a growth mindset is also important because technology is changing rapidly every day around us. 

How we mapped the “skills genome” of emerging jobs

In today’s newly-published Emerging Jobs Report, we introduce the “skills genome methodology” to highlight the unique skills for emerging jobs—occupations that have experienced tremendous growth in hiring, but may not have had a correspondingly large or established workforce. Often, these jobs are relatively new, such as the top job globally on this year’s report: Artificial Intelligence Specialist. Highlighting the unique skills associated with each emerging job provides insights about what aptitude or knowledge is valued when employers are hiring for a particular role. It also can show how one job can be differentiated from others, even if the titles or roles appear to be very similar. 

The Top 2019 LinkedIn Engineering Blogs

As the year draws to a close, we’re taking a look back at ten of our most popular 2019 articles on the LinkedIn Engineering Blog. Examining the list, it’s clear that topics pertaining to open source and artificial intelligence are some of the most popular, as are posts that look at how we tackle technical challenges at scale. We’re excited to share new progress and updates in all of these areas in the new year, but in the meantime, take a look at what you might have missed in 2019. 

Eliminating toil with fully automated load testing

Introduction In 2013, when LinkedIn moved to multiple data centers across the globe, we needed a way to redirect traffic from one data center to another in order to mitigate potential member impact in the event of a disturbance to our services. This need led to the birth of one of the most important pieces of engineering at LinkedIn, called TrafficShift. It provides the ability to move live production traffic from one data center to another in an effortless manner. 

Empowering our developers with the Nuage SDK

LinkedIn delivers value to more than 660 million members via thousands of microservices, most of which depend on data infrastructure or mid-tier infrastructure platforms. This means that in order to launch a new application, developers traditionally had to request and set up an online database as a business data source of truth or stream data via Kafka topics, as well as configure the alerting mechanisms and monitor system health and traffic patterns. Such tasks used to take up to several days or weeks to complete, requiring support from platform developers or SREs. This all changed when we introduced Nuage as application developers are now able to complete these tasks via self-service in minutes.

A look at our biggest SRE[in]con yet

Co-authors: Todd Palino, Samir Jafferali, Kurt Andersen, and Carolyn Blood

Getting to Know Natasha Michel

LinkedIn wouldn't be the company it is today without the engineers who built it and the talented individuals in technical roles across the company. They are the ones who create, build, and maintain our platform, tools, and features—as well as write posts for this blog. In this series, we feature some of the people and personalities that make LinkedIn great.  

Feature spotlight: Advancing language settings for content on LinkedIn

Co-authors: Nanna Ericson and Tetyana Bruevich  

An inside look at LinkedIn’s data pipeline monitoring system

Co-authors: Krishnan Raman and Joey Salacup  

Getting to Know Chiranjeevi Devi

LinkedIn wouldn't be the company it is today without the engineers who built it. We have no shortage of talented individuals in technical roles across the company. They are the ones who create, build, and maintain our platform, tools, and features—as well as write posts for this blog. In this series, we feature some of the people and personalities that make LinkedIn great. 

Learnings from the journey to continuous deployment

As an engineer, your goal is for every commit to seamlessly land in production and provide a delightful experience for your customers. While frequent releases give you the ability to iterate and apply feedback quickly, they also require significant time, effort, and cost to achieve. In this blog post, we’ll explore the components of our development ecosystem that help support continuous deployment, a release strategy where each code commit goes through the build pipeline and deploys in the production environment automatically.

Making LinkedIn Media More Inclusive with Alternative Text Descriptions

Co-authors: Vipin Gupta, Ananth Sankar and Jyotsna Thapliyal

How LinkedIn customizes Apache Kafka for 7 trillion messages per day

Co-authors: Jon Lee and Wesley Wu

Productivity at scale: How we improved build time with Gradle build cache

Editor's Note: This is the second in a series of posts describing how we improved productivity at scale—both in terms of lines of code and number of engineers—at LinkedIn. In our first post of the #ProductivityAtScale series, we shared details on how we improved build time by 400%. This post covers how we continue to improve productivity with Gradle build cache.  

The building blocks of LinkedIn Skill Assessments

Co-authors: Christian Mathiesen and Jie Zhang

Solving manageability challenges at scale with Nuage

Introduction LinkedIn is committed to providing economic opportunities for every member of the global workforce, and we’re growing at a rapid pace. Our platform is built on a collection of large-scale multi-cluster services functioning in harmony to offer a unified product experience to members. 

Talent Spotlight: REACH at LinkedIn

Co-authors: Nicole Brunnett, Shalini Agarwal

Recap: Engineering Excellence Meetup

We regularly play host to a series of meetups here at the LinkedIn office in the Empire State Building. Open to the community, these events cover a range of topics—from distributed systems, web and mobile development, to machine learning—and are a great way for engineers to meet, share notes, and learn from each other on various technical topics.

Auditing Content Features in FollowFeed

The LinkedIn feed relies on a ranked list of the most relevant content for a member. More than 80% of the feed is organic content created by people, companies, or groups that a member is following; the rest consists of recommendations such as jobs, articles, or ads. All organic content in the Linkedin feed is powered by FollowFeed. FollowFeed has two main responsibilities: the efficient indexing of content and the optimal storage of the attributes and metadata of the content (also known as features in this blog) to be used for ranking in proximity to the data for efficient relevance scoring.

Evaluating Language Adoption At Scale

When adopting a new language at scale, there are many different factors to consider because things can change dramatically. For many, choosing a language can arguably rely on personal preference, but at LinkedIn, we have a Foundation team tasked with evaluating the impact of such fundamental technical decisions. Recently, we underwent the process of evaluating and assessing languages for Android development. From the perspective of the mobile infrastructure team, I would like to share the steps we took in assessing languages that eventually led us to land on Kotlin. 

Evaluating Connection Coalescing For Static and Media Asset Downloads

One of the core aspirations of the engineering teams at LinkedIn is to delight our members with instantaneous page load experiences. To achieve this, we strive for optimal configurations and settings across the different layers of our software stack. Recently, we completed a large-scale experiment in reusing TCP/IP connections and found a perceivable drop in the 90th percentile page load time. While the production rollout of this strategy is still in the planning steps, we wanted to share the initial findings, explain the details of connection coalescing, and walk through how we conducted the experiment.

Data Hub: A Generalized Metadata Search & Discovery Tool

Co-authors: Mars Lan, Seyi Adebajo, Shirshanka Das

Detecting and Preventing Abuse on LinkedIn Using Isolation Forests

The Anti-Abuse AI Team at LinkedIn creates, deploys, and maintains models that detect and prevent various types of abuse, including the creation of fake accounts, member profile scraping, automated spam, and account takeovers. There are several unique challenges we face when using machine learning to prevent abuse on a large professional network, including:

Who Depends On Me? Serving Dependency Queries at Scale

Co-authors: Yu Li, Szymon Gizecki, and Chinmaya Dattathri

Fairness, Privacy, and Transparency by Design in AI/ML Systems

Editor’s note: A shorter version of this article was originally posted by Krishnaram Kenthapadi on LinkedIn.

Building the next version of our infrastructure

The pursuit of our mission to connect the world’s professionals to make them more productive and successful is deeply dependent on the technology and infrastructure we build and maintain. Ten years ago, we had 50 million members. Fast forward five years and that number jumped to 300 million. Today, we have more than 645 million members, 20 million jobs and someone is hired using LinkedIn every eight seconds. Our focus on building a platform that created opportunities for people and businesses has remained steadfast, and our capabilities have evolved to match that. 

Open Sourcing Brooklin: Near Real-Time Data Streaming at Scale

Brooklin - a stateless and distributed service for streaming data in near real-time and at scale - has been running in production at LinkedIn since 2016, powering thousands of data streams and over 2 trillion messages per day. Today, we are pleased to announce the open-sourcing of Brooklin and that the source code is available in our Github repo! 

Auto-Tuning Pinot Real-Time Consumption

Pinot, a scalable distributed columnar OLAP data store developed at LinkedIn, delivers real-time analytics for site-facing use cases such as LinkedIn's Who viewed my profile, Talent insights, and more. Pinot uses Apache Helix for managing cluster resources and Apache Zookeeper to store metadata. Pinot has wide adoption at LinkedIn, ranging from internal dashboards to site-facing applications.

Expediting Data Fixes and Data Migrations

With over 630 million members, the LinkedIn platform delivers thousands of features that individually serve and store large amounts of diverse data. Protecting, maintaining, and serving data has always been of paramount importance for enriching the member experience and ensuring service reliability. In this blog post, we’ll address a critical part of data management involving ad hoc operations through data migrations or data fixes. More specifically, we’ll detail how we developed a centralized scaleable self-service platform for implementing and executing both data fixes and migrations.

Driving Change with Authenticity and Allyship at WomenConnect NYC

WomenConnect is a series of events focused on bringing together women in tech to build meaningful relationships. Over the past four years, we’ve shared countless stories, learned a ton of lessons, and made lifelong connections at the 16 WomenConnect events in four different cities.

Improving Performance and Capacity for Espresso with New Netty Framework

In this blog post, we’ll share how we migrated Espresso, LinkedIn’s distributed data store, to a new Netty4-based framework and achieved a large performance and capacity gain throughout the Espresso system as a result. In the larger scheme, this is particularly important since Espresso is a master data hub that serves many important applications across LinkedIn, and every improvement we implement has an impact on the user experience in one way or another.