introduction to mesos
Post on 17-Aug-2015
69 Views
Preview:
TRANSCRIPT
- 1. marc@sloppy.io Introduction to Apache Mesos
- 2. marc@sloppy.io Overview Introduction Architecture Security Container High availability
- 3. marc@sloppy.io Introduction First release in 2009 at the Berkley University Framework to use datacenter resources efficiently Combine Cpu, storage, memory etc. to one big shared virtual resource A distributed systems kernel 10.000 lines C++ code
- 4. marc@sloppy.io Introduction - Definitions Master - Scheduler Slaves Working Nodes Frameworks Application running on Mesos Executors Run tasks on the slaves Executor-Task - Running job on the slave Resource Offer - Slave resources which could be used by the frameworks
- 5. marc@sloppy.io Architecture
- 6. marc@sloppy.io Introduction - Frameworks https://docs.mesosphere.com/frameworks/
- 7. marc@sloppy.io Resource Allocation
- 8. marc@sloppy.io Resource Allocation 1) Slave 1 reports to the master that it has 4 CPUs and 4 GB of memory free. The master then invokes the allocation policy module, which tells it that framework 1 should be offered all available resources. 2) The master sends a resource offer describing what is available on slave 1 to framework 1.
- 9. marc@sloppy.io Resource Allocation 3) The frameworks scheduler replies to the master with information about two tasks to run on the slave, using for the first task, and for the second task. 2) Finally, the master sends the tasks to the slave, which allocates appropriate resources to the frameworks executor, which in turn launches the two tasks
- 10. marc@sloppy.io Resource Allocation - DRF Resource offer decision are made by the Resource Allocation Modul in the master In a heterogeneous environment resource allocation is difficult What is a fair share, when: User a require 1 CPU, 4GB RAM User b require 3 CPUs, 1 GB RAM Mesos: Dominant Resource Fairness
- 11. marc@sloppy.io DRF A modified fair share algorithm The goal is that each framework receives a fair share of the the resources most needed by the framework Dominant resource: Resource most demand by the framework Dominant Share: The highest percentage of shares owned across all resources of a framework
- 12. marc@sloppy.io DRF - Example Resource offer: 9 Cpu, 18GB RAM Tasks User A: 1CPU, 4 GB RAM - RAM=DR Tasks User B: 3CPUs, 1GB RAM CPU=DR Each Framework has 2/3 of its DS
- 13. marc@sloppy.io DRF - Example Framework1: 1CPU, 4GB RAM Framework2: 3CPU, 1GB RAM Buggy tasks could be killed by mesos Framework can have guaranteed allocation, non of its tasks should be killed
- 14. marc@sloppy.io RA Master Configuration Name Default Example allocation_interval 1s framework_sorter drf user_sorter drf offer_timeout 5 minutes roles - marathon,jenkins weights - marathon=2,jenkins=1
- 15. marc@sloppy.io RA Slave Configuration Name Default Example attributes ssd:true,rack:2 default_role * resources cpus(jenkins):1;disk(jenkins):10000; cpus(marathon):3;mem(marathon):2000
- 16. marc@sloppy.io Mesos Security Default configuration = No security Name Example Master authenticate_slaves true credentials /etc/mesos.pw authenticators crammd5 authenticate true Slave credential /etc/mesos.pw
- 17. marc@sloppy.io Framework Security 1) Framework to (re-)register with authorized roles 2)Framework to launch task/executors as authorized users 3)Authorized principals to shutdown frameworks through /shutdown HTTP endpoint
- 18. marc@sloppy.io Security ACLs Subjects Action Object principals register_framework roles usernames run_tasks users shutdown_frameworks framework_principals A set of subjects can perform an action on a set of objects
- 19. marc@sloppy.io Security ACLs Example
- 20. marc@sloppy.io Extract of the mesos api URL Function master:5050/help REST Documentation master:5050/metrics/snapshot Metrics of the cluster master:5050/master/tasks.json List mesos tasks master:5050/master/redirect 307 to the leading master master:5050/master/shutdown Shutdown Framework master:5050/registrar(1)/registry Content of the current registry slave:5051/files/browse.json?path=pathOnSlave Browse files in sandbox slave:5051/files/read.json?path=stdoutOnSlave Read stdout from sandbox slave:5051/system/stats.json Local system metrics
- 21. marc@sloppy.io Resource Isolation Mesos supports Docker - and Mesos Container Resource isolation with cgroups or posix
- 22. marc@sloppy.io Mesos and Docker
- 23. marc@sloppy.io Mesos HA
- 24. marc@sloppy.io Mesos Tasks States TaskState Int Description TASK_STARTING 0 TASK_RUNNING 1 Task TASK_FINISHED 2 TERMINAL: The task finished successfully TASK_FAILED 3 TERMINAL: The task failed to finished TASK_KILLED 4 TERMINAL: The task was killed by executor TASK_LOST 5 TERMINAL: The task was failed but can rescheduled TASK_STAGING 6 Initial State TASK_ERROR 7 TERMINAL: Task description contains an error
- 25. marc@sloppy.io References http://mesos.apache.org/documentation/latest/ Mesos: A Platform for Fine-Grained Resource Sharing in the Data Center Dominant Resource Fairness: Fair Allocation of Multiple Resource Types playing-traffic-cop-resource-allocation-in-apache-mesos https://mesosphere.com/
- 26. marc@sloppy.io Thank you
top related