apache mesos distributed computing talk
TRANSCRIPT
Apache MesosA romantic stroll into Distributed Computing
Brandon Gulla - Praxis Engineering
●Beer.●Why distributed-computing is hard? ***●How Mesos helps.●Mesos in the wild.●DCOS-Demo.●Questions?●Repeat Step 1
This is a shortened version of the talk I normally give, feel free to contact me offline for a deeper dive.
Agenda
Mesosphere
● Commercial company supporting Apache Mesos based in SF and Hamburg, Germany.
● Founded by the Apache Mesos Founder Ben Hindman, Former AirBnB CTO Tobi Knapp and previous Twitter Engineer Florian Leibert.
● Provides full stack Mesos solutions running Apache Mesos as well as their corporate product: DCOS (Datacenter Operating System).
● Keith McClellan will be presenting a demo as soon as you’re tired of me.
MesosFrameworks
●Servers, Memory and Disks have gotten cheaper so we buy more. These often go under-utilized.
●Traditionally, clusters have been provisioned statically.
●Puppet, SaltStack, Ansible, **pdsh** :(●Not Multi-tenant friendly.●DevOps Nightmare!!!●Doesn’t Scale :(
“Just Network them together!”
Personal Gateway To Mesos
●Customer had a 100+ node Hadoop cluster with legacy dependency requirements.
●Wanted to utilize current MapReduce libraries without breaking ingest requirements.
●More importantly: Customer wanted a way to manage their resources on demand...
Solution:
●Open-source two-level Resource Manager.
●Written in C++ wrapping around libprocess.
●Scalable - Runs on 10,000s of servers.●Fault-Tolerant! Multiple masters. Load
Balancing!●Solve Bigger Problems.●I/O Isolation via Linux cgroups.●Distributed Resource Manager:
●RAM●Disks●Ports●GPUs●Network (Yes- Software Defined Networking)
Welcome to Apache Mesos
Mesos: Turn Your Datacenter into One Large Computer● Where did we install MySQL? Who cares! Mesos +
Service Discovery = Dynamic Awesomeness. ● Separates dependencies between Ops and
Development. Ops wants to take a rack down, who cares, just move the processes from that rack somewhere else. Ops can do this as part of their work. Developers shouldn't know/care.
● Fault Tolerant, if something dies, Mesos will automatically spin another one up.
History
Shared Resources = Multi-Tenancy
Mesos Explained
Resource Offers and Launching A Task
Writing your own Mesos Framework
●Feature Complete API using:●JVM Languages (Java, Scala, Groovy)●C++●Python●Go
●Write a framework in minutes using Mesos-Compose.
●YML interface ●Similar to Docker-Compose
●Unit Testing with Mini-Mesos
Don’t want to write your own Mesos Framework? No problemo, Amigo.
0 to Mesos in seconds with Marathon.
Containers: VMs but awesomer.●Shares Kernel with Host (no hypervisor)●Boots in milliseconds.●Portable. docker pull python:27
●Run CentOS on Debian, Run Gentoo on Alpine.
●Dependency Friendly
●If you’re replacing a once statically installed, non-distributed application: Use Marathon.
●Apache, Nginx, Jenkins..etc
●If your application requires fine grain control and orchestration of tasks: Write your own Framework.
●Apache Accumulo●Hadoop Yarn (Shameless Apache Myriad Plug)●Spark
Framework Development: My Experiences
Mesos in the Wild●Netflix: Mantis+Fenzo●Apple: Jarvis (Siri)●Microsoft Azure Container Service.
Mesos: Full Stack Replacement
Current Mesos Equivalent
Systemd Marathon
Cron Chronos
DNS Mesos-DNS
Chroot / BSD Jail Docker
NFS Gluster, CEF, HDFS
Redefined ThinkingBefore:
After:
Processes
MesosFrameworks
Demo TimeTo Infinity and Beyond.