characterizing and contrasting container orchestrators
TRANSCRIPT
CharacterizingandContrastingContainer
Orchestrators
LeeCalcote
http://calcotestudios.com/ccco
LeeCalcoteclouds,containers,infrastructure,applications
andtheirmanagement
linkedin.com/in/leecalcote
@lcalcote
blog.gingergeek.com
[kuhn-tey-ner][awr-kuh-streyt-or]
Definition:
FleetNomadSwarm
KubernetesMesos+Marathon
(Staytunedforupdatestopresentation)
CategoricallySpeaking
CategoricallySpeakingGenesis&Purpose
CategoricallySpeakingGenesis&PurposeSupport&Momentum
CategoricallySpeakingGenesis&PurposeSupport&MomentumHost&ServiceDiscovery
CategoricallySpeaking
Scheduling
Genesis&PurposeSupport&MomentumHost&ServiceDiscovery
CategoricallySpeaking
Scheduling
Genesis&PurposeSupport&MomentumHost&ServiceDiscovery
Modularity&Extensibility
CategoricallySpeaking
Scheduling
Genesis&PurposeSupport&MomentumHost&ServiceDiscovery
Modularity&ExtensibilityUpdates&Maintenance
CategoricallySpeaking
Scheduling
Genesis&PurposeSupport&MomentumHost&ServiceDiscovery
Modularity&ExtensibilityUpdates&MaintenanceHealthMonitoring
CategoricallySpeaking
Scheduling
Genesis&PurposeSupport&MomentumHost&ServiceDiscovery
Modularity&ExtensibilityUpdates&MaintenanceHealthMonitoringNetworking&Load-Balancing
CategoricallySpeaking
Scheduling
Genesis&PurposeSupport&MomentumHost&ServiceDiscovery
Modularity&ExtensibilityUpdates&MaintenanceHealthMonitoringNetworking&Load-BalancingHighAvailability&Scale
HypervisorManagerElements
ComputeNetworkStorage
ContainerOrchestratorElements
Host(Node)ContainerServiceVolumeApplications
CoreCapabilities
Scheduling
ClusterManagement
HostDiscovery
HostHealthMonitoring
OrchestratorUpdatesandHost
Maintenance
ServiceDiscovery
NetworkingandLoad-Balancing
AdditionalKeyCapabilitiesApplicationHealthMonitoring
ApplicationDeployments
ApplicationPerformanceMonitoring
DockerSwarm
Genesis&PurposeSwarmissimpleandeasytosetup.Animperativesystem,Swarmisresponsiblefortheclusteringandschedulingaspectsoforchestration.Swarm’sarchitectureisnotcomplexasthoseofKubernetesandMesos
hasManager(s)andAgent(s)
WritteninGolang,Swarmislightweight,modular,portable,andextensible
Support&Momentum~3,000commitsby12coremaintainers(130contributorsinall),andforDockerCompose,thenumbersareapproximatelythesameSwarmleveragesthesameAPIforDockerenginewhichover23,000commitsand1,350contributors
~250DockermeetupsworldwideAnnouncedasproduction-ready5monthsago(Nov2015)
image:digitaltrends
Host&ServiceDiscoveryHostDiscovery
usedintheformationofclustersbytheManagertodiscoverforNodes(hosts).
ServiceDiscovery
SwarmhasaconceptofservicesthroughnetworkaliasesandroundrobinDNS
image:iStock
SchedulingSwarm’sschedulerispluggableSwarmschedulingisacombinationofstrategiesandfilters/constraint:Strategies
RandomBinpackSpread*
Filterscontainerconstraints(affinity,dependency,port)aredefinedas
environmentvariablesinthespecificationfile
nodeconstraints(health,constraint)mustbespecifiedwhen
startingthedockerdaemonanddefinewhichnodesacontainer
maybescheduledon.image:pickywallpapers
Modularity&ExtensibilityAbilitytoremovebatteriesisastrengthforSwarm:
PluggableschedulerPluggablenetworkdriverPluggabledistributedK/VstoreDockercontainerengineruntime-only
image: AlanChia
Updates&Maintenance
Nodes
Manualswarmmanagerandagentupdates
Applications
Nofacilitiesforupdatingapplication
image:123RF
HealthMonitoringNodes
Swarmmonitorstheavailabilityandresourceusageofnodeswithinthecluster
Applications
NofacilitiesformonitoringapplicationsUsethird-partysoftware
Networking&Load-Balancing
DockerSwarmisfullycompatiblewithDocker’snetworkingDockermulti-hostnetworking(requiresaK/Vstore)providesforuser-definedoverlaynetworksthataremicro-segmentable
usesagossipprotocolforquickconvergenceofneighbortable
facilitatescontainernameresolutionviaembeddedDNSserver
(previouslyviaetc/hosts)
YoumaybringyourownnetworkdriverLoad-balancingisnewin1.11.0
-nodocumentationavailable
HighAvailability&ScaleManagersmaybedeployedinahighly-availableconfiguration(outofexperimentalin1.11.0)
Active/Standby-anactivemanagerandreplicasconfiguration
Reschedulinguponnodefailure(experimentalin1.11.0)andsimple.Rebalancingisnotavailable
Rescheduledcontainersloseaccesstovolumesmountedontheformerhost.
usevolumepluginslikeFlockertoavoidthis
Doesnotsupportmultiplefailureisolationregionsorfederation(although,withcaveats, ).
thisispossible
Scalingswarmto1,000AWSnodesand50,000containers
Suitablefororchestratingacombinationofinfrastructurecontainers
Lessbuilt-infunctionalityforapplicationcontainers
Swarmisayoungprojectandlacksadvancedfeatures.
HighAvailabilityoutofexperimentalasoflatestrelease(1.11.0)
Reschedulingisexperimentalin1.11.0(noRebalancing,yet)
Load-balancingin1.11.0?Swarmneedstobeusedwiththird-partysoftware.
OnlyschedulesDockercontainers,notcontainersusingotherspecifications.
Whiledependencyandaffinityfiltersareavailable,Swarmdoesnotprovide
theabilitytoenforceschedulingoftwocontainersontothesamehostornot
atall.
Currentfiltersdonotfacilitatesidecarpattern.No“pod”concept.
Swarmworks.Swarmissimpleandeasyto
deploy.
IfyoualreadyknowDockerCLI,usingSwarm
isstraight-forward
facilitatesearlierstagesofadoptionby
organizationsviewingcontainersasfasterVMs
Lessbuilt-infunctionalityforapplications
Swarmiseasytoextend,ifcanalready
knowDockerAPIs,youcancustomizeSwarm
Highlymodular:
Pluggablescheduler
PluggableK/Vstoreforbothnodeandmulti-
hostnetworking
Kubernetes
Genesis&Purposeanopinionatedframeworkforbuildingdistributedsystems
orasitstaglinestates"anopensourcesystemforautomating
deployment,scaling,andoperationsofapplications."
WritteninGolang,Kubernetesislightweight,modular,portable,andextensibleconsideredathirdgenerationcontainerorchestratorledbyGoogle,RedHatandothers.
bakesinload-balancing,scale,volumes,deploymentsandsecret
managementamongotherfeatures.
Declaratively,opinionatedwithmanykeyfeaturesincluded
Support&MomentumKubernetesisyoung(abouttwoyearsold)
Announcedasproduction-ready10monthsago(July2015)
Projectcurrentlyhasover1,000commitspermonth(~23,000total)
madebyabout100(650total)Kubernauts(Kubernetesenthusiasts).~5,000commitsmadeinthelatestrelease-1.2.
UnderthegovernanceoftheCloudNativeComputingFoundationRobustsetofdocumentationand~90meetups
Host&ServiceDiscoveryHostDiscovery
bydefault,thenodeagent(kubelet)isconfiguredtoregisteritselfwiththemaster(APIserver)
automatingthejoiningofnewhoststothecluster.
ServiceDiscoveryTwoprimarymodesoffindingaService
environmentvariables
environmentvariablesareusedasasimplewayofprovidingcompatibilitywithDockerlinks-style
networking
DNS
whenenabledisdeployedasaclusteradd-on image:iStock
Scheduling
Bydefault,schedulingishandledbykube-scheduler.PluggableSelectioncriteriausedbykube-schedulertoidentifythebest-fitnodeisdefinedbypolicy:
Predicates(noderesourcesandcharacteristics):
PodFitPorts,PodFitsResources,NoDiskConflict
,MatchNodeSelector,HostName,ServiceAffinit,LabelsPresence
Priorities(weightedstrategiesusedtoidentify“bestfit”node):
LeastRequestedPriority,BalancedResourceAllocation,ServiceSpreadingPriority,
EqualPriority
Modularity&Extensibility
OneofKubernetesstrengthsitspluggablearchitectureChoiceof:
databaseforservicediscoveryornetworkdrivercontainerruntime
usersmaychoosetorunDockerwithRocketcontainers
Clusteradd-onsoptionalsystemcomponentsthatimplementaclusterfeature(e.g.DNS,logging,etc.)shippedwiththeKubernetesbinariesandareconsideredaninherentpartoftheKubernetesclusters
Updates&MaintenanceApplications
Deploymentobjectsautomatedeployingandrollingupdatingapplications.
KubernetesComponents
UpgradingtheKubernetescomponentsandhostsisdoneviashellscriptHostmaintenance-markthenodeasunschedulable.
existingpodsarenotvacatedfromthenodepreventsnewpodsfrombeingscheduledonthenode
image:123RF
HealthMonitoringNodes
Failures-activelymonitorsthehealthofnodeswithinthecluster
viaNodeController
Resources-usagemonitoringleveragesacombinationofopensourcecomponents:
cAdvisor,Heapster,InfluxDB,Grafana
Applications
threetypesofuser-definedapplicationhealth-checksandusestheKubeletagentasthethehealthcheckmonitor
HTTPHealthChecks,ContainerExec,TCPSocket
Cluster-levelLogging
collectlogswhichpersistbeyondthelifetimeofthepod’scontainerimagesorthelifetimeofthepodorevencluster
standardoutputandstandarderroroutputofeachcontainercanbeingestedusinga
Networking&Load-Balancing
…enterthePod
atomicunitofschedulingflatnetworkingwitheachpodreceivinganIPaddressnoNATrequired,portconflictslocalizedintra-podcommunicationvialocalhost
Load-Balancing
Servicesprovideinherentload-balancingviakube-proxy:
runsoneachnodeofaKubernetescluster
reflectsservicesasdefinedintheKubernetesAPI
supportssimpleTCP/UDPforwardingandround-robinandDocker-links-
basedserviceIP:PORTmapping.
HighAvailability&Scale
Eachmastercomponentmaybedeployedinahighly-availableconfiguration.
Active/Standbyconfiguration
Intermsofscale,v1.2bringssupportfor1,000nodeclustersandasteptowardfully-federatedclusters(Ubernetes)Application-levelauto-scalingissupportedwithinKubernetesviaReplicationControllers
Onlyrunscloud-nativeapplications
ForthosefamiliarwithDocker-only,
Kubernetesrequiresunderstandingofnew
concepts
Powerfulframeworkswithmoremovingpiecesbeget
complicatedclusterdeploymentandmanagement.
Lightweightgraphicaluserinterface
Doesnotprovideassophisticatedtechniques
forresourceutilizationasMesos
Kubernetescanscheduledockerorrkt
containers
Inherentlyopinionatedwithfunctionalitybuilt-
in.
nothird-partysoftwareneededtorunservices,load-
balancing,
buildsinmanyapplication-levelconceptsandservices
(secrets,petsets,jobsets,daemonsets,rollingupdates,etc.)
Kubernetesarguablymovingthequickest
Relativelythoroughprojectdocumentation
Multi-master,Robustlogging&metrics
aggregation
Mesos+
Marathon
Genesis&PurposeMesosisadistributedsystemskernel
stitchestogethermanydifferentmachinesintoalogicalcomputer
Mesoshasbeenaroundthelongest(launchedin2009)andisarguablythemoststable,withhighest(proven)scalecurrently
MessiswritteninC++withJava,PythonandC++APIs
FrameworksMarathonisoneofanumberofframeworks(ChronosandAuroraother
examples)thatmayberunontopofMesos
Frameworkshaveaschedulerandexecutor.Schedulersgetresource
offers.Executorsruntasks.
MarathoniswritteninScala
Support&Momentum
MesosCon2015inSeattlehad700attendeesupfrom262attendeesin2014
78contributorsinthelastyear
UnderthegovernanceofApacheFoundationUsedbyTwitter,AirBnb,eBay,Apple,Cisco,Yodle
Host&ServiceDiscovery
Mesos-DNSgeneratesanSRVrecordforeachMesostask
includingMarathonapplicationinstances
MarathonwillensurethatalldynamicallyassignedserviceportsareuniqueMesos-DNSisparticularlyusefulwhen:
appsarelaunchedthroughmultipleframeworks(notjustMarathon)
youareusinganIP-per-containersolutionlike
youuserandomhostportassignmentsinMarathon
ProjectCalico
image:iStock
Scheduling
TwolevelschedulerFirstlevelschedulinghappensatmesosmasterbasedonallocationpolicy,whichdecideswhichframeworkgetresourcesSecondlevelschedulinghappensatFrameworkscheduler,whichdecideswhattaskstoexecute.
Providereservationsandoversubscriptions
Modularity&ExtensibilityFrameworks
multipleavailablemayrunmultipleframeworks
Modules
hasthreetypesofModulesReplacement,Hook,Anonymous
Updates&Maintenance
Nodes-MesoshasmaintenancemodeApplications
Marathoncanbeinstructedtodeploycontainersbasedonthatcomponentusingablue/greenstrategy
whereoldandnewversionsco-existfora
time.
image:123RF
HealthMonitoringNodes
Mastertracksasetofstatisticsandmetricstomonitorresourceusage
CountersandGauges
Applications
supportforhealthchecks(HTTPandTCP)aneventstreamthatcanbeintegratedwithload-balancersorforanalyzingmetrics
Networking&Load-Balancing
Networking
AnIPperContainerNolongersharethenode'sIP
Helpsremoveportconflicts
Enables3rdpartynetworkdrivers
Load-Balancing
MarathonofferstwoTCP/HTTPproxy.Asimpleshellscriptandamorecomplexonecalledmarathon-lbthat
hasmorefeatures.
Pluggable(e.g.Traeficforload-balancing)
HighAvailability&ScaleAstrengthofMesos’sarchitecture
requiresmasterstoformaquorumusingZooKeeper
onlyoneActive(Leader)Marathonmasterat-a-time
ScaleisastrongsuitforMesos.UsedatTwitter,AirBnB...TBDforMarathon
Greatatasynchronousjobs.Highavailabilitybuilt-in.Referredtoasthe“goldenstandard”bySolomonHykes,DockerCTO.
UniversalContainerizer
abstractawayfromDockerEngine,(runc,appc)
Canrunmultipleframeworks,includingKubernetesandSwarm.
Onlyofthecontainerorchestratorsthatsupportsmulti-tenancy
GoodforBigDatahouseandjob-orientedortask-orientedworkloads.
Goodformixedworkloadsandwithdata-localitypolicies
Powerfulandscalable,Battle-tested
Goodformultiplelargethingsyouneedtodo10,000+nodeclustersystem
MarathonUIisyoung,butpromising
Stillneeds3rdpartytools
MarathoninterfacecouldbemoreDockerfriendly
(hardtogetatvolumesandregistry)
MayneedadedicatedinfrastructureITteam
anoverlycomplexsolutionforsmalldeployments
Summary
Ahigh-levelperspectiveofthecontainerorchestratorspectrum.
LeeCalcote
linkedin.com/in/leecalcote
@lcalcote
blog.gingergeek.com
Thankyou.Questions?