Apache Mesos vs Yarn

Overview

Apache Mesos

Stacks306

Followers418

Votes31

GitHub Stars5.3K

Forks1.7K

Yarn

Stacks28.2K

Followers13.5K

Votes151

GitHub Stars41.5K

Forks2.7K

Apache Mesos vs Yarn: What are the differences?

Resource Allocation: In Apache Mesos, resources are allocated to individual frameworks based on the needs and constraints of tasks, allowing for fine-grained resource sharing. On the other hand, Yarn uses a static partitioning model where resources are divided among different queues at the cluster level, leading to less efficient resource utilization.
Task Scheduling: Apache Mesos employs a two-level scheduling mechanism, separating resource offers from task scheduling, which provides better scalability and fault tolerance. In contrast, Yarn uses a centralized ResourceManager for scheduling all applications, leading to potential bottlenecks in large-scale deployments.
Scaling: Apache Mesos is designed to scale to tens of thousands of nodes with efficient resource sharing and fault tolerance mechanisms built-in. Yarn can also scale to large clusters but may have limitations in efficiently handling dynamic workloads and resource sharing at scale.
API Compatibility: Apache Mesos provides a more flexible API for interacting with the cluster, offering support for both Mesos-native frameworks and popular Hadoop ecosystem tools. Yarn, on the other hand, primarily caters to the Hadoop ecosystem and may not be as versatile in supporting different types of workloads and frameworks.
Fault Tolerance: Apache Mesos has built-in fault tolerance features such as checkpointing and fast recovery from failures, ensuring high availability for critical workloads. Yarn also provides fault tolerance mechanisms but may not have the same level of resiliency and recovery capabilities as Mesos.
Community Support: Apache Mesos has a diverse community supporting its development and maintenance, with contributions from various organizations and individuals. Yarn is primarily backed by the Apache Software Foundation and the Hadoop ecosystem community, with a focus on integrating well with Hadoop-related projects.

In Summary, Apache Mesos and Yarn differ in resource allocation strategies, task scheduling mechanisms, scalability, API compatibility, fault tolerance features, and community support, making them suitable for different use cases based on specific requirements.

Share your Stack

Help developers discover the tools you use. Get visibility for your team's tech choices and contribute to the community's knowledge.

View Docs

CLI (Node.js)

Manual

Advice on Apache Mesos, Yarn

StackShare

Apr 23, 2019

Needs adviceon

Node.js

npm

Yarn

From a StackShare Community member: “I’m a freelance web developer (I mostly use Node.js) and for future projects I’m debating between npm or Yarn as my default package manager. I’m a minimalist so I hate installing software if I don’t need to- in this case that would be Yarn. For those who made the switch from npm to Yarn, what benefits have you noticed? For those who stuck with npm, are you happy you with it?"

294k views294k

Comments

zen-li

Apr 24, 2019

Reviewon

Yarn

p.s.

I am not sure about the performance of the latest version of npm, whether it is different from my understanding of it below. Because I use npm very rarely when I had the following knowledge.

------⏬

I use Yarn because, first, yarn is the first tool to lock the version. Second, although npm also supports the lock version, when you use npm to lock the version, and then use package-lock.json on other systems, package-lock.json Will be modified. You understand what I mean, when you deploy projects based on Git...

250k views250k

Comments

Oleksandr

Senior Software Engineer at joyn

Dec 7, 2019

Decided

As we have to build the application for many different TV platforms we want to split the application logic from the device/platform specific code. Previously we had different repositories and it was very hard to keep the development process when changes were done in multiple repositories, as we had to synchronize code reviews as well as merging and then updating the dependencies of projects. This issues would be even more critical when building the project from scratch what we did at Joyn. Therefor to keep all code in one place, at the same time keeping in separated in different modules we decided to give a try to monorepo. First we tried out lerna which was fine at the beginning, but later along the way we had issues with adding new dependencies which came out of the blue and were not easy to fix. Next round of evolution was yarn workspaces, we are still using it and are pretty happy with dev experience it provides. And one more advantage we got when switched to yarn workspaces that we also switched from npm to yarn what improved the state of the lock file a lot, because with npm package-lock file was updated every time you run npm install, frequent updates of package-lock file were causing very often merge conflicts. So right now we not just having faster dependencies installation time but also no conflicts coming from lock file.

310k views310k

Comments

Detailed Comparison

Apache Mesos	Yarn
Apache Mesos is a cluster manager that simplifies the complexity of running applications on a shared pool of servers.	Yarn caches every package it downloads so it never needs to again. It also parallelizes operations to maximize resource utilization so install times are faster than ever.
Fault-tolerant replicated master using ZooKeeper;Scalability to 10,000s of nodes;Isolation between tasks with Linux Containers;Multi-resource scheduling (memory and CPU aware);Java, Python and C++ APIs for developing new parallel applications;Web UI for viewing cluster state	-
Statistics
GitHub Stars 5.3K	GitHub Stars 41.5K
GitHub Forks 1.7K	GitHub Forks 2.7K
Stacks 306	Stacks 28.2K
Followers 418	Followers 13.5K
Votes 31	Votes 151
Pros & Cons
Pros 21 Easy scaling 6 Web UI 2 Fault-Tolerant 1 Elastic Distributed System 1 High-Available Cons 1 Depends on Zookeeper 1 Not for long term	Pros 85 Incredibly fast 22 Easy to use 13 Open Source 11 Can install any npm package 8 Works where npm fails Cons 16 Facebook 7 Sends data to facebook 4 Should be installed separately 3 Cannot publish to registry other than npm
Integrations
Apache Aurora	JavaScript npm

What are some alternatives to Apache Mesos, Yarn?

npm

npm is the command-line interface to the npm ecosystem. It is battle-tested, surprisingly flexible, and used by hundreds of thousands of JavaScript developers every day.

RequireJS

RequireJS loads plain JavaScript files as well as more defined modules. It is optimized for in-browser use, including in a Web Worker, but it can be used in other JavaScript environments, like Rhino and Node. It implements the Asynchronous Module API. Using a modular script loader like RequireJS will improve the speed and quality of your code.

Browserify

Browserify lets you require('modules') in the browser by bundling up all of your dependencies.

Nomad

Nomad is a cluster manager, designed for both long lived services and short lived batch processing workloads. Developers use a declarative job specification to submit work, and Nomad ensures constraints are satisfied and resource utilization is optimized by efficient task packing. Nomad supports all major operating systems and virtualized, containerized, or standalone applications.

Component

Component's philosophy is the UNIX philosophy of the web - to create a platform for small, reusable components that consist of JS, CSS, HTML, images, fonts, etc. With its well-defined specs, using Component means not worrying about most frontend problems such as package management, publishing components to a registry, or creating a custom build process for every single app.

DC/OS

Unlike traditional operating systems, DC/OS spans multiple machines within a network, aggregating their resources to maximize utilization by distributed applications.

Mesosphere

Mesosphere offers a layer of software that organizes your machines, VMs, and cloud instances and lets applications draw from a single pool of intelligently- and dynamically-allocated resources, increasing efficiency and reducing operational complexity.

Verdaccio

A simple, zero-config-required local private npm registry. Comes out of the box with its own tiny database, and the ability to proxy other registries (eg. npmjs.org), caching the downloaded modules along the way.

pip

It is the package installer for Python. You can use pip to install packages from the Python Package Index and other indexes.

Gardener

Many Open Source tools exist which help in creating and updating single Kubernetes clusters. However, the more clusters you need the harder it becomes to operate, monitor, manage and keep all of them alive and up-to-date. And that is exactly what project Gardener focuses on.

Related Comparisons

Apache Mesos vs Yarn: What are the differences?

Resource Allocation: In Apache Mesos, resources are allocated to individual frameworks based on the needs and constraints of tasks, allowing for fine-grained resource sharing. On the other hand, Yarn uses a static partitioning model where resources are divided among different queues at the cluster level, leading to less efficient resource utilization.
Task Scheduling: Apache Mesos employs a two-level scheduling mechanism, separating resource offers from task scheduling, which provides better scalability and fault tolerance. In contrast, Yarn uses a centralized ResourceManager for scheduling all applications, leading to potential bottlenecks in large-scale deployments.
Scaling: Apache Mesos is designed to scale to tens of thousands of nodes with efficient resource sharing and fault tolerance mechanisms built-in. Yarn can also scale to large clusters but may have limitations in efficiently handling dynamic workloads and resource sharing at scale.
API Compatibility: Apache Mesos provides a more flexible API for interacting with the cluster, offering support for both Mesos-native frameworks and popular Hadoop ecosystem tools. Yarn, on the other hand, primarily caters to the Hadoop ecosystem and may not be as versatile in supporting different types of workloads and frameworks.
Fault Tolerance: Apache Mesos has built-in fault tolerance features such as checkpointing and fast recovery from failures, ensuring high availability for critical workloads. Yarn also provides fault tolerance mechanisms but may not have the same level of resiliency and recovery capabilities as Mesos.
Community Support: Apache Mesos has a diverse community supporting its development and maintenance, with contributions from various organizations and individuals. Yarn is primarily backed by the Apache Software Foundation and the Hadoop ecosystem community, with a focus on integrating well with Hadoop-related projects.

Apache Mesos vs Yarn

Overview