mesos

MotivationMesos

ImplementationEvaluationConclusion

Mesos: A Platform for Fine-Grained ResourceSharing in the Data Center

Muhammad Anis uddin Nasir

KTH Royal Institute of Technology

November 20, 2012

Muhammad Anis uddin Nasir Mesos 1/32

MotivationMesos


1 Motivation

2 MesosOverviewArchitecture

3 Implementation

4 Evaluation

5 Conclusion


MotivationMesos


Motivation

Diverse cluster computer framework


MotivationMesos


Motivation

No optimal framework

Run multiple frameworks

Higher UtilizationData sharing betweenclustersReduce Cost


MotivationMesos


Motivation


Run multiple frameworksHigher Utilization

Data sharing betweenclustersReduce Cost


MotivationMesos


Motivation


Run multiple frameworksHigher UtilizationData sharing betweenclusters

Reduce Cost


MotivationMesos


Motivation


Run multiple frameworksHigher UtilizationData sharing betweenclustersReduce Cost


MotivationMesos


Existing Solutions

Static Partitioning

Virtual Machines


MotivationMesos


Fine-grained

Hadoop and Dryad

SlotsTasks

Benefits

Data LocalityUtilization


MotivationMesos


Fine-grained

Hadoop and DryadSlots

Tasks

Benefits



MotivationMesos


Fine-grained

Hadoop and DryadSlotsTasks

Benefits



MotivationMesos


Fine-grained


BenefitsData Locality

Utilization


MotivationMesos


Fine-grained


BenefitsData LocalityUtilization


MotivationMesos


OverviewArchitecture

1 Motivation


3 Implementation

4 Evaluation

5 Conclusion


MotivationMesos



Mesos

Common resource sharing layer

Fine grained sharing

Across diverse cluster computing frameworks


MotivationMesos



Goals

High Utilization

Data sharing among frameworks


MotivationMesos



Challenges

Scalable

Support diverse frameworks

Efficient

Fault tolerant

Highly available


MotivationMesos



Other Benefits

Run Multiple instances of same framework

Isolate production and experimental jobsRun multiple versions of a framework

Build special framework targeting particular problemdomain


MotivationMesos



Other Benefits

Run Multiple instances of same frameworkIsolate production and experimental jobs

Run multiple versions of a framework



MotivationMesos



Other Benefits

Run Multiple instances of same frameworkIsolate production and experimental jobsRun multiple versions of a framework



MotivationMesos



Design Elements

Fine-grained sharing

Allocation at the level of tasks within a jobImprove utilization, latency and data locality

Resource offers

Simple and ScalableApplication-controlled scheduling mechanism


MotivationMesos



Design Elements

Fine-grained sharingAllocation at the level of tasks within a job

Improve utilization, latency and data locality

Resource offers



MotivationMesos



Design Elements

Fine-grained sharingAllocation at the level of tasks within a jobImprove utilization, latency and data locality

Resource offers



MotivationMesos



Design Elements


Resource offersSimple and Scalable

Application-controlled scheduling mechanism


MotivationMesos



Design Elements


Resource offersSimple and ScalableApplication-controlled scheduling mechanism


MotivationMesos



Fine-Grained Sharing


MotivationMesos



Resource Offers

Offers resources to framework

Framework choose resources according to needs

Benefits

gives freedom to framework for implementationkeep Mesos simple and scalable

Drawback

decentralized decision might not be optimal


MotivationMesos



Resource Offers



Benefitsgives freedom to framework for implementation

keep Mesos simple and scalable

Drawback



MotivationMesos



Resource Offers



Benefitsgives freedom to framework for implementationkeep Mesos simple and scalable

Drawback



MotivationMesos



Resource Offers



Benefitsgives freedom to framework for implementationkeep Mesos simple and scalable

Drawbackdecentralized decision might not be optimal


MotivationMesos



Architecture


MotivationMesos



More Features

Resource Allocation

Fair sharingStrict prioritiesDelay sharing

Isolation

Linux ContainersSolaris Project

Scalability andRobustness

FiltersRe-offering resourcesIncentives

Fault Tolerance

Soft state masterReport node failure toframeworkMultiple schedulers


MotivationMesos



More Features

Resource AllocationFair sharing

Strict prioritiesDelay sharing

Isolation




Fault Tolerance



MotivationMesos



More Features

Resource AllocationFair sharingStrict priorities

Delay sharing

Isolation




Fault Tolerance



MotivationMesos



More Features

Resource AllocationFair sharingStrict prioritiesDelay sharing

Isolation




Fault Tolerance



MotivationMesos



More Features


IsolationLinux Containers

Solaris Project



Fault Tolerance



MotivationMesos



More Features


IsolationLinux ContainersSolaris Project



Fault Tolerance



MotivationMesos



More Features




Filters

Re-offering resourcesIncentives

Fault Tolerance



MotivationMesos



More Features




FiltersRe-offering resources

Incentives

Fault Tolerance



MotivationMesos



More Features





Fault Tolerance



MotivationMesos



More Features





Fault ToleranceSoft state master

Report node failure toframeworkMultiple schedulers


MotivationMesos



More Features





Fault ToleranceSoft state masterReport node failure toframework

Multiple schedulers


MotivationMesos



More Features





Fault ToleranceSoft state masterReport node failure toframeworkMultiple schedulers


MotivationMesos


1 Motivation


3 Implementation

4 Evaluation

5 Conclusion


MotivationMesos


Implementation

10,000 lines of code in C++

Linux, Solaris, OS X

support framework written in Java, C++, Python

ZooKeeper

Frameworks

HadoopTorque and MPISpark


MotivationMesos


Implementation




ZooKeeper

FrameworksHadoop

Torque and MPISpark


MotivationMesos


Implementation




ZooKeeper

FrameworksHadoopTorque and MPI

Spark


MotivationMesos


Implementation




ZooKeeper

FrameworksHadoopTorque and MPISpark


MotivationMesos


Spark

Data flow for logistic regression


MotivationMesos


1 Motivation


3 Implementation

4 Evaluation

5 Conclusion


MotivationMesos


Dynamic Resource Sharing

96 Node Mesos Cluster

4 CPU Cores and 15GB Ram


MotivationMesos



96 Node Mesos Cluster4 CPU Cores and 15GB Ram


MotivationMesos




MotivationMesos


Data Locality with Resource Offers

16 instance of Hadoop using 93 EC2 nodes

1.7x speed up with Mesos

97% data locality with 5sec data scheduling


MotivationMesos


Scalability

99 Amazon EC2 nodes

Scaled to 50,000 emulated slaves, 200 frameworks, 100K tasks

unable to scale beyond 50,000 slaves as Amazon EC2 clusterwas the bottleneck


MotivationMesos


Further Experiments

Fault Tolerance

Mesos masters connected to a 5-node ZooKeeper quorumfault detection and recovery in 10 sec

Overhead

LINPACK for MPI and Wordcount Hadoop BecnhmarkOverhead of Mesos was less than 4%


MotivationMesos


Further Experiments

Fault ToleranceMesos masters connected to a 5-node ZooKeeper quorum

fault detection and recovery in 10 sec

Overhead



MotivationMesos


Further Experiments

Fault ToleranceMesos masters connected to a 5-node ZooKeeper quorumfault detection and recovery in 10 sec

Overhead



MotivationMesos


Further Experiments


OverheadLINPACK for MPI and Wordcount Hadoop Becnhmark

Overhead of Mesos was less than 4%


MotivationMesos


Further Experiments


OverheadLINPACK for MPI and Wordcount Hadoop BecnhmarkOverhead of Mesos was less than 4%


MotivationMesos


1 Motivation


3 Implementation

4 Evaluation

5 Conclusion


MotivationMesos


Conclusion

Mesos shares clusters efficiently among diverseframeworks

Fine-grained sharing at the level of tasksResource Sharing, a scalable mechanism forapplication-controlled scheduling

Enabales co-existence of current frameworks anddevelopment of new specialized frameworks


MotivationMesos


Conclusion


Fine-grained sharing at the level of tasks

Resource Sharing, a scalable mechanism forapplication-controlled scheduling



MotivationMesos


Conclusion


Fine-grained sharing at the level of tasksResource Sharing, a scalable mechanism forapplication-controlled scheduling



MotivationMesos


References

Hindman, B., Konwinski, A., Zaharia, M., Ghodsi, A., Joseph,A. D., Katz, R., Shenker, S., et al. (n.d.). Mesos : A Platformfor Fine-Grained Resource Sharing in the Data Center.

http://incubator.apache.org/mesos/

http://datainthecloud.blogspot.se/2011/10/

mesos-platform-for-fine-grained.html

https://www.usenix.org/conference/nsdi11/

mesos-platform-fine-grained-resource-sharing-data-center

http://static.usenix.org/event/nsdi11/tech/

slides/hindman.pdf


http://incubator.apache.org/mesos/

http://datainthecloud.blogspot.se/2011/10/mesos-platform-for-fine-grained.html

http://datainthecloud.blogspot.se/2011/10/mesos-platform-for-fine-grained.html

https://www.usenix.org/conference/nsdi11/mesos-platform-fine-grained-resource-sharing-data-center

https://www.usenix.org/conference/nsdi11/mesos-platform-fine-grained-resource-sharing-data-center

http://static.usenix.org/event/nsdi11/tech/slides/hindman.pdf

http://static.usenix.org/event/nsdi11/tech/slides/hindman.pdf

MotivationMesos


Mesos: A Platform for Fine-Grained ResourceSharing in the Data Center

Muhammad Anis uddin Nasir

KTH Royal Institute of Technology

November 20, 2012


MotivationMesos


Different workloads

Homogeneous Tasks

elastic frameworks with constant task durationrigid frameworks with exponential task duration

Placement Preferences

weighted fair allocation policylottery scheduling

Heterogeneous Tasks

random task assignmentreserve some resources on each node for small tasksmaximum task duration

Framework incentives

short taskselastic tasksdo not accept unknown resources


MotivationMesos


Different workloads

Homogeneous Taskselastic frameworks with constant task duration

rigid frameworks with exponential task duration



Heterogeneous Tasks





MotivationMesos


Different workloads

Homogeneous Taskselastic frameworks with constant task durationrigid frameworks with exponential task duration



Heterogeneous Tasks





MotivationMesos


Different workloads


Placement Preferencesweighted fair allocation policy

lottery scheduling

Heterogeneous Tasks





MotivationMesos


Different workloads


Placement Preferencesweighted fair allocation policylottery scheduling

Heterogeneous Tasks





MotivationMesos


Different workloads



Heterogeneous Tasksrandom task assignment

reserve some resources on each node for small tasksmaximum task duration




MotivationMesos


Different workloads



Heterogeneous Tasksrandom task assignmentreserve some resources on each node for small tasks

maximum task duration




MotivationMesos


Different workloads



Heterogeneous Tasksrandom task assignmentreserve some resources on each node for small tasksmaximum task duration




MotivationMesos


Different workloads




Framework incentivesshort tasks

elastic tasksdo not accept unknown resources


MotivationMesos


Different workloads




Framework incentivesshort taskselastic tasks

do not accept unknown resources


MotivationMesos


Different workloads




Framework incentivesshort taskselastic tasksdo not accept unknown resources


MotivationMesos


Limitations

Fragmentation

not optimal bin packinglarge jobs may starveminimum offer size on each slave

Interdependent framework constraints

scenarios where only one task can be accommodated

Framework Complexity

framework scheduling is complexframework has a choicefailures are easy to handle with resource offers


MotivationMesos


Limitations

Fragmentationnot optimal bin packing

large jobs may starveminimum offer size on each slave






MotivationMesos


Limitations

Fragmentationnot optimal bin packinglarge jobs may starve

minimum offer size on each slave






MotivationMesos


Limitations

Fragmentationnot optimal bin packinglarge jobs may starveminimum offer size on each slave






MotivationMesos


Limitations


Interdependent framework constraintsscenarios where only one task can be accommodated




MotivationMesos


Limitations



Framework Complexityframework scheduling is complex

framework has a choicefailures are easy to handle with resource offers


MotivationMesos


Limitations



Framework Complexityframework scheduling is complexframework has a choice

failures are easy to handle with resource offers


MotivationMesos


Limitations



Framework Complexityframework scheduling is complexframework has a choicefailures are easy to handle with resource offers


mesos

Education