container orchestration: which conductor?

ContainerOrchestration:WhichConductor?

ContainerConEurope,Berlin,Oct2016

MikeBright, @mjbright

HaikelGuemar, @hguemar

MarioLoriedo, @mariolet

First...Alittlebitofhistory

Solet'sfirstlookatrecentcontainerhistory...

Unix

con

tain

ers,

Unik

erne

lsLi

nux

cont

aine

rs(L

XC),

Mes

osLX

C(C

loud

Foun

dry,

Dot

Clou

dPa

aS)

Dock

er

Cont

aine

rOrc

hest

ratio

nOp

tions

PaaS

ado

ptio

nso

fDoc

ker

Dock

er"S

war

mm

ode"

,OCI

D...

Mic

ro-O

Ses,

Swar

m,R

kt,L

XD

Dock

erb

uysU

nike

rnel

Sys

tem

s

...2009 2013 2014 2016...2015

@hguemar@mjbright@mariolet

History μ-OSesManyvendorsaredevelopingμ-OSes,smallOS(mainlyLinux-based)tobethebasisforcontainerenginehostswhethertheybebare-metalorvirtualhostmachines.

They'resmall,withfaststartup,usefewresourcesandhaveasmallattacksurfaceandoften"atomic"softwareupdates.

OS VendorCoreOS - (CoreOS)ProjectAtomic - (RedHat)RancherOS - (RancherLabs)Photon - (VMWare)NanoServerOS - (Microsoft)UbuntuSnappyCore - (Canonical)

...Unikernels

μ-Services

μ-services Frommonolithstoμ-services

Rememberwhenhighavailabilitymeantthis...?

Active StandbyAppApp

ServersrunningmonolithicapplicationsinActive-Standbymodes,as1+1,N+1,orN+Morsplitacross3tiers.

Scalingmeantto"scaleup"byaddingCPU,RAM,disk.Butthere'salimittothis...thenyouhaveto"scaleout"@hguemar@mjbright@mariolet


Thencameμ-services..

Astheindustrymovedtovirtualizedmicro-servicesthisallowedtoobtaingreaterefficiencies(higherutilisationofresources)andtheredesignofapplicationsallowstoscaleoutandachievehighavailability.

Containersfacilitatethismove,allowingfasterscalingandevengreaterefficiencieswithlessredundancy(noOStoreproduce).


μ-services Howcontainershelp?

ContainersolutionssuchasDockergobeyondtheisolationcapabilitiesofLXCbyprovidingsimpletousetoolstoenablepackagingofappswiththeirdependenciesallowingportableapplicationsbetweensystems.

Containersarelightweight

Versionedimagescontainingalldependanciescanbeshared

Containersallowtousethesameapplicationbinariesondevelopment,testandproductionsystemswhetherthatbeonalaptop,serverorinthecloud.

It'sanobrainerfordevelopers,whocanbuildandsharetheirownimages



But1000'sofnodesareunmanageable...aren'tthey?

Wecan'ttakecareofour

,

sowehavetotreatthemlike

that'scloudnative!


Soweneedcontainerorchestration

OrchestrationWhatwasContainerOrchestrationagain?

Architecture-Composition&StitchingWorkflows&Policiesto

Scalein/out(maybeautomatically)Placeworkloadsfor

loadbalancing,faulttolerance,resourceoptimization

Adapttofaults

....

....

Master

Worker Worker Worker

....App1 AppNDistrib

Cluster

Config

(quorum)

Master Master

WorkerTasks:

OrchestrationGettingto"DesiredState"Tomanage100's,1000's,10,000'sofnodesweneedtoexpress"desiredstate"ratherthan"dothis".

Imperative DeclarativeTellsystem Dothis desiredstate

"startanewnode" "3mysqlnodes".

Intelligence Operator OrchestrationEngine.

Flexibility Best Least.

Itisnotlongerfeasibleforanoperatorto

knowtheresourcesavailable(e.g.SSD/HDD,GPU,...)reacttofailure,knowwhentoscale...


Choiceisgreat-whenyouknowwhatyouwant...

Orchestration

TheBig3-MainOrchestrationChoices

DockerSwarm("SwarmMode")

ApacheMesos

Kubernetes

Orchestration

TheBig3-MainOrchestrationChoices

DockerSwarm("SwarmMode")

ApacheMesos

Kubernetes

...moreChoices...

Rancher(RancherLabs)

Fleet(CoreOS)

Nomad(HashiCorp)

Kontena

OpenStackMagnum@hguemar@mjbright@mariolet

OrchestrationTheBig3-WhatdoesGoogleTrendssay?

ClearlyKuberneteshasaleadinGoogle"searchtrends"

Butwecanexpect"DockerSwarm"tomakequickprogressthankstothenew"swarmmode"

DockerSwarm

DockerSwarm

Dec2014 ...DockerSwarmisannounced

OrchestrationusingDockerCompose

Jun2016 ... SwarmToolkitreleasedOpenSourceOrchestrationToolkit

Jun2016 ... SwarmModeannouncedOrchestrationintegratedintoDockerEngine

Docker1.12isthefirstreleasetointegrate"SwarmMode"TheoriginalDockerSwarmismaintainedforlegacyuse.

"SwarmMode"isarevolutionbringing:

OrchestrationdirectlyintheDockerEngineAdvancednetworkingfeatures

meshnetwork,vxlanLoadbalancingServiceDiscoveryDockertraditionaleaseofuse

docker.com @hguemar@mjbright@mariolet

https://docker.com/

DockerSwarm

Architecture

....

....

Master


....App1 AppN DistribClusterConfig(quorum)

Master Master

WorkerTasks:

GossipNetwork

DockerEngine DockerEngine DockerEngine DockerEngine

gRPC


https://docker.com/

DockerSwarm

UsingDocker"SwarmMode"Createanewswarmbycreatingthemasternode:

$dockerswarminit--advertise-addr192.168.2.100Swarminitialized:currentnode(dxn1zf6l61qsb1josjja83ngz)isnowamanager.

JoinanewWorkernodetotheswarm:

$dockerswarmjoin--tokenTOKEN192.168.2.100:2377

JoinanewMasternodetotheswarm:

$dockerswarmjoin-tokenmanager


https://docker.com/

DockerSwarmDemo

DockerSwarm

DockerSwarmDemoCreationofa3nodecluster

Runaserviceontheclusterandscaleitto3replicas

Makearollingupdateoftheservice

Drainanodeofthecluster


https://docker.com/

DockerSwarm

Gettingstarted

BrunoCornecs'Docker101LabNewtoDocker?Cometohttp://sched.co/7oHf

Docker101Lab,9am-Friday7thOctober

https://github.com/bcornec/Labs/tree/master/Docker

Followedby..

JeromePetazzoni'sOrchestrationWorkshopormoreadvanced?Cometohttp://sched.co/7oHx

OrchestratingContainersinProductionatScalewithDockerSwarm,Friday7thOctober

https://github.com/jpetazzo/orchestration-workshop


http://sched.co/7oHf

https://github.com/bcornec/Labs/tree/master/Docker

http://sched.co/7oHx

https://github.com/jpetazzo/orchestration-workshop

https://docker.com/

Kubernetes

FromtheGreek:"Steersman,helmsman,sailingmaster"

Kubernetes

Googlecreatedbasedonextensiveexperiencerunningcontainersinternally~billionsofcontainersayear

StartedOct2014,reachedv1.0inJuly2015,nowatv1.4

ManagedbytheCloudNativeComputingFoundationhttps://cncf.io/

CommercialofferingsfromCoreOS(Tectonic)andCanonical

Integratedin:

GKE(GoogleContainerEngine)OpenStackaboveKubernetes

Stackanetes(CoreOS,usesTectonic)SelfhealingOpenStackdemo

Mirantis(OpenStackCI/CDbasedonKolla)VariousPaaS:

RedHatOpenShiftCPHPEStackatov.40Deis

kubernetes.io @hguemar@mjbright@mariolet

https://cncf.io/

https://kubernetes.io/

Kubernetes

Architecture

....

....

Master


....App1 AppN etcd

Master Master

WorkerPods: 192.168.100.10

ServicesLabels,Selectors



ApacheMesos

ApacheMesos

Themostprovenorchestratortoday,existssince2009.

Canscaleto~10,000nodes.

Usedinproductionby:

UberTwitterPaypalHubspotAirbnbeBayGrouponNetflix

SupportsContainerizerstoisolatetasks

mesos.apache.org

http://mesos.apache.org/

ApacheMesos

Architecture

....

....

Master

Worker(agent)

....Spark AppNDistrib

Cluster

Config

(quorum)

Master Master

WorkerTasks:

ResourcesAvailable

Scheduler

SparkExecutor SparkExecutor Executor

Offer

AcceptRegister

Worker(agent) Worker(agent)

mesos.apache.org


ApacheMesos

MesosisusedinconjunctionwithFrameworkssuchas

Forlongrunningtasks:

Marathon(Mesosphere),AuroraorSingularity

Forjoborchestration:

Chronos"cron",Jenkins

ForBigDataProcessing:

Hadoop,Spark,StormCassandra,ElasticSearch,...

mesos.apache.org@hguemar@mjbright@mariolet


Soisn'tittimewetoldyouwhattochoose?

...let'sjustcomparethem...

What'scommon

DockerSwarmandKubernetesarecreatingrichOrchestrationstackswithintegratedruntimes.

They'removingincrediblyquicky...

Theyareaddingfeaturessuchasnetworkingcapabilities,loadbalancing,services,labels.

Theyhaveamore'declarative'approach

Theysupportorarelookingtosupportdifferentruntimeengines(*)


Whatadvantages?Docker"Swarm

Mode"Simpletouse(despiteunderlyingcomplexity)

All-in-onecontainerengineplusorchestration

UsesDockerAPIandfamiliardockercommands

Advancednetworking

-meshnetworking-LoadBalancingandServiceDiscovery

Replication


Whatadvantages?Kubernetes

Richconceptualmodel

Podsasgroupingsofcontainers

LabelsandSelectors(forallcomponents)

Largeecosystem

Networking

-LoadBalancingandServiceDiscovery

Replication


Whatadvantages?ApacheMesos

Mostmature

Battletestedbymanyserviceproviders

Scalesto10,000nodes

DataCenterOS-appearsas1resource

Notjustcontainers

Manyframeworksavailable

Difficultramp-up


Handson...


Hands-on Comealong

Thisafternoon'stutorialsessionledbyMario:Tuesday,October4-15:30-16:20

5Containersfor5Languages:PatternsforSoftwareDevelopmentUsingContainers-MarioLoriedo,RedHat

Tomorrow'slabsessionledbyHaikel:Wednesday,October5-11:00-12:50

ContainerOrchestrationLab:Swarm,Mesos,Kubernetes-HaïkelGuémar,FedoraProject

Labsetupinstructionshere

DockerSwarmKubernetesApacheMesos


https://linuxconcontainerconeurope2016.sched.org/event/7oHE/5-containers-for-5-languages-patterns-for-software-development-using-containers-mario-loriedo-red-hat

https://linuxconcontainerconeurope2016.sched.org/event/7oHC/container-orchestration-lab-swarm-mesos-kubernetes-haikel-guemar-fedora-project

http://bit.ly/2674h5J

Questions?Thankyou


Resources


ResourcesBooksPublisher Title AuthorOReilly DockerCookbook SébastienGoasguen

OReilly DockerUp&Running

KarlMatthias,SeanP.Kane

OReilly UsingDocker AdrianMouat

OReilly[EarlyAccess]KubernetesUp&Running

KelseyHightower

Manning [MEAP]CoreOSinAction MattBailey

Manning [MEAP]KubernetesinAction MarkoLukša


http://shop.oreilly.com/product/0636920036791.do




https://www.manning.com/books/coreos-in-action

https://www.manning.com/books/kubernetes-in-action

ResourcesArticles/OrganismsCloudNativeComputingFoundation-Kubernetes,Prometheushttps://cncf.io/

"KubernetestheHardWay,KelseyHightower"-https://github.com/kelseyhightower/kubernetes-the-hard-way

"KubernetesUserGuide,Walkthrought"-http://kubernetes.io/docs/user-guide/walkthrough/


https://cncf.io/

https://github.com/kelseyhightower/kubernetes-the-hard-way

http://kubernetes.io/docs/user-guide/walkthrough/

ResourcesVideosJune2016-ContainerOrchestrationWars,KarlIsenberg,Mesosphere

Mar2016-ContainerOrchestrationwithKubernetes,DockerSwarm&Mesos-Marathon-AdrianMouat,ContainerSolutions

Jan2016-Docker,Kubernetes,andMesos:Compared.,,AdrianOtto,SouthernCaliforniaLinuxExpo

Repos


https://www.youtube.com/watch?v=C_u4_l84ED8

https://www.youtube.com/watch?v=_uw1ISM_uRU

https://www.youtube.com/watch?v=QJ-3nZvvrhE

Kubernetes

DocumentationGettingstartedguides

CreatingaKubernetesClusterportKubernetestoanewenvironment

inGettingStartedfromScratchUserdocumentation

torunprogramsonanexistingKubernetescluster

KubernetesUserGuide:ManagingApplications

theKubectlCommandLineInterfaceisadetailedreferenceonthekubectlCLIUserFAQ


http://localhost:9000/docs/getting-started-guides/README.md

http://localhost:9000/docs/getting-started-guides/scratch.md

http://localhost:9000/docs/user-guide/README.md

http://localhost:9000/docs/user-guide/kubectl/kubectl.md

https://github.com/kubernetes/kubernetes/wiki/User-FAQ


Kubernetes

Documentation-2

ClusteradministratordocumentationforpeoplewhowanttocreateaKubernetesclusterandadministeritintheKubernetesClusterAdminGuide

DeveloperandAPIdocumentationtowriteprogramsusingtheKubernetesAPI,writepluginsorextensions,ormodifycorecodeKubernetesDeveloperGuidenotesontheAPIAPIobjectdocumentation,adetaileddescriptionofallfieldsfoundinthecoreAPIobjects

Walkthroughsandexampleshands-onintroductionandexampleconfigfilesintheuserguideinthedocs/examplesdirectory

ContributionsfromtheKubernetescommunity

inthedocs/contribdirectory


http://localhost:9000/docs/admin/README.md

http://localhost:9000/docs/devel/README.md

http://localhost:9000/docs/api.md

http://localhost:9000/docs/api-reference/README.md

http://localhost:9000/docs/user-guide/README.md#quick-walkthrough

http://localhost:9000/examples/

http://localhost:9000/contrib/


Kubernetes

Documentation3Designdocumentationanddesignproposals

tounderstandthedesignofKubernetes,andfeatureproposalsKubernetesDesignOverviewandthedocs/designdirectorydocs/proposalsdirectory

Wiki/FAQthewikitroubleshootingguide

Community,discussion,contribution,andsupport

ConsiderjoiningtheCloudNativeComputingFoundation.Fordetailsaboutwho'sinvolvedandhowKubernetesplaysarole,readtheirannouncement.


http://localhost:9000/docs/design/README.md

http://localhost:9000/docs/design/

http://localhost:9000/docs/proposals/

https://github.com/kubernetes/kubernetes/wiki

http://localhost:9000/docs/troubleshooting.md

https://cncf.io/about

https://cncf.io/news/announcement/2015/07/new-cloud-native-computing-foundation-drive-alignment-among-container


container orchestration: which conductor?

Documents