DOING BIG DATA FORREAL WITH DOCKER

MESOSPHERE DCOSElizabeth Lingg

elizabeth@mesosphere.io

AGENDA1. Intro2. Mesosphere, Docker, and DCOS Overview3. Big Data Container Orchestration using DCOS and Docker4. Demo5. Q & A

INTROEngineering Manager @ MesosphereM.S. Computer Science with a Specialization in ArtificialIntelligence from StanfordB.S. Computer Science with a Minor in Math, B.S. Policyand Management from Carnegie MellonExperience in AI, Big Data, and SystemsEnjoys applying Distributed Systems to Manage andReason Over Large Amounts of Data

MESOSProvides primitives to author datacenter-native apps.

PRIMITIVES

Resources (cpu, mem, disk, ports)Asset fetchingTask state trackingAPI for the datacenter

STATUS QUO IS STATICPARTITIONING

AND USE OF VIRTUAL MACHINES

MESOS LET US TREAT A CLUSTER OFNODES...

AS ONE BIG COMPUTER

Not as individual

machines

Not as VMs

BUT AS COMPUTATIONALRESOURCES LIKE CORES, MEMORY,

DISKS, ETC.

WE LOVE CONTAINERS

MOST MODERN APPLICATIONS ARE A WEB OFCONTAINERS

A CONTAINER ORCHESTRATION PLATFORM

Containerization in Mesos, a brief history

MESOSPHERE DCOSSoftware to provide a complete OS: init, cron, apt-get,discovery, routingBeautiful web UI and CLISupportEcosystem of DCOS ServicesMesos Master and Mesos Workers Running in DockerContainers

DCOS UI

DCOS CLI$ dcos

Command line utility for the Mesosphere Datacenter OperatingSystem (DCOS). The Mesosphere DCOS is a distributed operatingsystem built around Apache Mesos. This utility provides toolsfor easy management of a DCOS installation.

Available DCOS commands:

config Get and set DCOS CLI configuration properties help Display command line usage information marathon Deploy and manage applications on the DCOS node Manage DCOS nodes package Install and manage DCOS software packages service Manage DCOS services task Manage DCOS tasks

BIG DATA DISTRIBUTEDAPPLICATIONS ON DCOS

Mesos Master and Mesos Workers Running in DockerContainersDistributed Applications Running in Containers on theMesos WorkersContainer Orchestration done by Apache MesosResource Allocation and Scaling Managed by ApacheMesos

BIG DATA DISTRIBUTEDAPPLICATIONS ON DCOS

Popular Distributed Apps easily deployed on a singleDCOS ClusterKafka, Cassandra, HDFS, Spark, and other Big DataServicesHealth checks and failure recovery are automated

APPLICATION NETWORKINGInteract with the CLI or REST API's to interact with theservicesMesos DNS ResolutionDocker Networking mainly done through host modenetworking, works seamlessly

DATA SECURITYServices storing secure data run on private worker nodesin the clusterPrivate nodes can only be accessed through VPNAs needed, services choose what is exposed through aproxy running on a public nodeDistributed Application can authenticate with the Masterusing Framework Authentication (Kerberos Option)

EXAMPLE: SIMPLE DOCKER APP ONDCOS

{ "id": "/mesosphere/cd-demo-app", "instances": 1, "cpus": 1, "mem": 512, "container": { "type": "DOCKER", "docker": { "image": "mesosphere/cd-demo-app:$tag", "network": "BRIDGE", "portMappings": [ { "servicePort": 28080, "containerPort": 80, "hostPort": 0, "protocol": "tcp" }<

EXAMPLE: CASSANDRA DCOSSERVICEFEATURES

Managed node configurationHealth MonitoringRest APIDNS Names for nodesMultiple Rings in one cluster

INSTALL$ dcos package install cassandra

CUSTOMIZABLE INSTALL OPTIONS{ "cassandra": { "cluster-name": "dev", "resources": { "cpus": 3.0, "mem": 6144, "disk": 30720 } }}

$ dcos package install cassandra --options=options.json

INSTALLING

HEALTHY

REST APIGET /node/all

GET /health/cluster/report

POST /node/{node}/replace

POST /cluster/repair/start

POST /scale/nodes?nodeCount={count}

DEMO!

Q & A

THANKS!LET'S CHAT!WE'RE HIRING!

DCOS: Join:

mesosphere.commesosphere.com/careers/