emc world 2016 - deep dive with mesos and persistent storage for applications

2© Copyright 2016 EMC Corporation. All rights reserved. 2© Copyright 2016 EMC Corporation. All rights reserved.

DEEP DIVE WITH MESOS & PERSISTENT STORAGE FOR APPLICATIONSDAVID VONTHENEN, EMC {code}


• David vonThenen, Developer Advocate - EMC {code}– Twitter: @dvonthenen– Blog: http://dvonthenen.com– Github: https://github.com/dvonthenen

PRESENTER

http://dvonthenen.com/

https://github.com/dvonthenen

https://github.com/dvonthenen


• Quick Review of Mesos• Frameworks and Two Layer Scheduling• Production Applications• Demo• Q&A

AGENDA


QUICK REVIEW OF MESOS


• From a resource perspective, it’s a cluster manager:– Pools resources to be centrally managed as a single unit

• From an application perspective, it’s a scheduler:– Dispatches workloads to consume pooled resources

• Often described as a Data Center Operating System (DCOS)– Static versus Dynamic Partition of Resource– Efficient utilization of CPU, Memory, Disk

• Proven resource scheduler for production workloads

QUICK REVIEW OF MESOS


SCHEDULERS ARE DIFFICULT TO BUILD


• Master– Manages Mesos Agents– Enables fine-grained sharing of resources (CPU, RAM, …)

across frameworks in the form of resource offers.• Zookeeper

– Enables highly reliable distributed coordination (cluster state)

• Agent (aka Slave)– The entity that will be running the task

MESOS COMPONENTS


MESOS ARCHITECTURE


• Generic– Anything that can be run on a standard Linux shell– Provides a mechanism for deploying applications packages– Supports Docker workloads

• Framework– A plug-in to Mesos that allows you to implement a sub-

scheduler– specialize a workload by providing custom placement and

resource allocation

APPLICATION SUPPORT


GENERIC APPLICATIONS


FRAMEWORKS ANDTWO LAYER SCHEDULING


• On the surface…• Allows specialization of

workload by providing custom placement and resource allocation• Provides monitoring of

these workloads to protect and react when things fail

MESOS FRAMEWORK


• Scheduler– a scheduler that registers with the master to be offered

resources• Executor

– an executor process that is launched on slave nodes to run the framework’s tasks

FRAMEWORK COMPONENTS


FRAMEWORK ARCHITECTURE


• Odds are you have been using at least 1 framework without realizing it…

AS IT TURNS OUT…

• Marathon is a framework– A framework for performing container

orchestration for long running tasks• Chronos is a framework

– A framework for performing batch or scheduled tasks


• Zooming into the “Marathon” box…• Marathon is a Scheduler• Launching a Marathon task, the “generic”

applications are created inside a “generic” (or default) executor• Dogfooding!

MARATHON DISSECTED


MESOS FRAMEWORKS


• 2 layer scheduling or sub-scheduling of resources• Manages the lifecycle of your

workload through application specific monitoring• What does this mean

specifically?

DIGGING DEEPER


BUILDING HADOOP CLUSTERS


MANAGING HADOOP CLUSTERS


APPLICATIONS USING HADOOP


THIS SEEMS SILLY


• Allows you to subsume the default scheduler such that:• You can install and configure your application• Resources can be allocated in a way that optimizes

your applications• Allows operation real-time automated improvements

for your applications– Scaling up for load balancing peak times– Scaling back to free up resources

2 LAYER SCHEDULING


• The details of your applications are hidden from the end user!– Deployment– Operations– Growth– Disaster recovery– Performance

• Just a consumer of the application as a service!

THE NON-TECHNICAL VERSION


• CouchBase Framework (doesn’t exist, let’s imagine it does)– CouchBase is a NoSQL DB

• Monitor the CouchBase instances, when nearing full– Dynamically add an node using “server-add”– Then call “rebalance”

• Reactive and Proactive but fully automated• I think we had all seen this movie…

FOR EXAMPLE


SKYNET


PRODUCTION APPLICATIONS


• Containers are ephemeral therefore stateless• Quickly found the need for state

– Configuration– Data– Logging / Auditing

• Containerizers like Mesos and Docker introduced exporting to local storage• Failure of a node? hardware? A better way?

EVOLUTION OF STATEFUL


• Persistent– Containers can come and go,

but continue where they left off– Application Data, Logs, Audit

Trail, etc• External

– Storage lives outside the Mesos Agent node

– High Availability– Tolerates node failure

PERSISTENT EXTERNAL STORAGE


• REX-Ray– Provides a vendor agnostic storage orchestration engine– AWS, GCE, ScaleIO, XtremIO, many many more– Github: https://github.com/emccode/rexray

• mesos-module-dvdi– Provides a hook to Mesos agent nodes to provision, attach,

and detach external storage– Github: https://github.com/emccode/mesos-module-dvdi

HOW WE ACHIEVE THIS?

https://github.com/emccode/rexray

https://github.com/emccode/rexray

https://github.com/emccode/mesos-module-dvdi

https://github.com/emccode/mesos-module-dvdi


• We can run “real world” applications• Persistence – DBs• Highly Available, Fault Tolerant• Audit trail – Transaction Logs• Not worry about the “2am call”

STATELESS TO STATEFUL


LET’S FACE IT…


• Trouble is… this still represents the “old way” of thinking• PREVENT THE CRASH OR FAILURE!• Problem is you can’t• What is the alternative?

STATEFUL 2.0?


• Mesos frameworks can proactively monitor for thing going sideways• Leveraging external persistent storage and

frameworks fail fast and recover quicker!• Skynet!

FRAMEWORKS + STATEFUL 2.0


DEMO


• Running on Amazon EC2– Mesos– Marathon– Mesos DNS

• Production HA Cluster– 3 x Master + Zookeeper Nodes (t2.micros)– 5 x Slave Nodes with Docker Support (t2.medium)

DEMO CONFIGURATION


• Elastic Search Framework– Create and mount external volume– Pour some data into Elastic Search– Github: https://github.com/mesos/elasticsearch

• Elastic Search Instances backed by Persistent External Storage– Using REX-Ray– Using mesos-module-dvdi

DEMO: FRAMEWORK + EXTERNAL STORAGE

https://github.com/mesos/elasticsearch

https://github.com/mesos/elasticsearch


• Deploy Elastic Search Framework using REST API– Observe Scheduler creation– Using browser, log into the Elastic Search Framework UI– Observe Executor nodes get created and attaching to EBS

volumes• Simulate Failure

– Kill EC2 Instance– Watch executor recreation and reattach to EBS volumes

DEMO: FRAMEWORK + EXTERNAL STORAGE


DEMO: AUTO DEPLOY ELASTIC SEARCH


Data Persistence in the New Container WorldWednesday 3PM

Joshua BernsteinVP of Technology for ETD

Tobi Knaup CEO & Co-Founder of Mesosphere

Guru Session


@EMCcode@dvonthenen

emccode.comcommunity.emccode.com

Come visit us at Booth #1044 or in the vLab

Questions?