using containers for building and testing: docker, kubernetes and mesos. fosdem 2017

60
USING CONTAINERS FOR BUILDING AND TESTING DOCKER, KUBERNETES AND MESOS Carlos Sanchez / csanchez.org @csanchez Watch online at carlossg.github.io/presentations

Upload: carlos-sanchez

Post on 21-Feb-2017

102 views

Category:

Software


1 download

TRANSCRIPT

Page 1: Using Containers for Building and Testing: Docker, Kubernetes and Mesos. FOSDEM 2017

USING CONTAINERS FOR BUILDING ANDTESTING

DOCKER, KUBERNETES AND MESOSCarlos Sanchez

/ csanchez.org @csanchez

Watch online at carlossg.github.io/presentations

Page 2: Using Containers for Building and Testing: Docker, Kubernetes and Mesos. FOSDEM 2017

ABOUT MEEngineer @ CloudBees, Private SaaS Edition Team

Contributor to Jenkins Mesos plugin & Jenkins and Mavenofficial Docker images

Author of Jenkins Kubernetes plugin

Long time OSS contributor at Apache, Eclipse, Puppet,…

Google Cloud Platform Expert

Page 3: Using Containers for Building and Testing: Docker, Kubernetes and Mesos. FOSDEM 2017

DOCKER DOCKERDOCKER

Page 4: Using Containers for Building and Testing: Docker, Kubernetes and Mesos. FOSDEM 2017
Page 5: Using Containers for Building and Testing: Docker, Kubernetes and Mesos. FOSDEM 2017

BUT IT IS NOT TRIVIAL

Page 6: Using Containers for Building and Testing: Docker, Kubernetes and Mesos. FOSDEM 2017

CLUSTER SCHEDULINGRunning in public cloud, private cloud, VMs or bare metalHA and fault tolerantWith Docker support of course

Page 7: Using Containers for Building and Testing: Docker, Kubernetes and Mesos. FOSDEM 2017
Page 8: Using Containers for Building and Testing: Docker, Kubernetes and Mesos. FOSDEM 2017

A distributed systems kernel

Page 9: Using Containers for Building and Testing: Docker, Kubernetes and Mesos. FOSDEM 2017

APACHE MESOSStarted before 2011Runs tasks, any binary or Docker, rkt, appc imagesFrameworks run on top of Mesos

Mesosphere Marathon: long running servicesApache Aurora: long running servicesChronos: distributed cron-like system

Used in Twitter, Airbnb, eBay, Apple, Verizon, Yelp,...

Page 10: Using Containers for Building and Testing: Docker, Kubernetes and Mesos. FOSDEM 2017

DOCKER SWARM

Page 11: Using Containers for Building and Testing: Docker, Kubernetes and Mesos. FOSDEM 2017

DOCKER SWARMBy Docker Inc.Uses the same Docker APINo need to modify existing tooling

Page 12: Using Containers for Building and Testing: Docker, Kubernetes and Mesos. FOSDEM 2017

DOCKER ENGINE SWARM MODENew in Docker 1.12No need to install extra so�ware, each daemon can run asa Swarm memberNew service object to describe distributed containers

Existing tooling needs to be updated

Swarm mode

Page 13: Using Containers for Building and Testing: Docker, Kubernetes and Mesos. FOSDEM 2017
Page 14: Using Containers for Building and Testing: Docker, Kubernetes and Mesos. FOSDEM 2017

KUBERNETESBased on Google BorgRun in local machine, virtual, cloudGoogle provides Google Container Engine (GKE)Other services run by stackpoint.io, CoreOS Tectonic,Azure,...Minikube for local testing

Page 15: Using Containers for Building and Testing: Docker, Kubernetes and Mesos. FOSDEM 2017

SCALING JENKINSTwo options:

More build agents per masterMore masters

Page 16: Using Containers for Building and Testing: Docker, Kubernetes and Mesos. FOSDEM 2017

SCALING JENKINS: MORE BUILDAGENTS

Pros

Multiple plugins to add more agents, even dynamically

Cons

The master is still a SPOFHandling multiple configurations, plugin versions,...There is a limit on how many build agents can beattached

Page 17: Using Containers for Building and Testing: Docker, Kubernetes and Mesos. FOSDEM 2017

SCALING JENKINS: MORE MASTERSPros

Different sub-organizations can self service and operateindependently

Cons

Single Sign-OnCentralized configuration and operation

Covered by CloudBees Jenkins Operations Center andCloudBees Jenkins Platform Private SaaS Edition

Page 18: Using Containers for Building and Testing: Docker, Kubernetes and Mesos. FOSDEM 2017
Page 19: Using Containers for Building and Testing: Docker, Kubernetes and Mesos. FOSDEM 2017

If you haven't automatically destroyedsomething by mistake, you are not

automating enough

Page 20: Using Containers for Building and Testing: Docker, Kubernetes and Mesos. FOSDEM 2017

RUNNING IN DOCKER

Page 21: Using Containers for Building and Testing: Docker, Kubernetes and Mesos. FOSDEM 2017
Page 22: Using Containers for Building and Testing: Docker, Kubernetes and Mesos. FOSDEM 2017
Page 23: Using Containers for Building and Testing: Docker, Kubernetes and Mesos. FOSDEM 2017

CLUSTER SCHEDULINGIsolated build agents and jobs

Using Docker

Capabilities can be dropped

Page 24: Using Containers for Building and Testing: Docker, Kubernetes and Mesos. FOSDEM 2017

GROUPING CONTAINERSExample:

Jenkins agentMaven buildSelenium testing in

FirefoxChromeSafari

5 containers

Page 25: Using Containers for Building and Testing: Docker, Kubernetes and Mesos. FOSDEM 2017

GROUPING CONTAINERSMesos Experimental in 1.1.0

Swarm Supports grouping through DockerCompose Can force execution in the same host

Kubernetes Supports the concept of Pods natively All running in the same host

MESOS-2449

Page 26: Using Containers for Building and Testing: Docker, Kubernetes and Mesos. FOSDEM 2017

MEMORY LIMITSScheduler needs to account for container memory

requirements and host available memory

Prevent containers for using more memory than allowed

Mesos required

Swarm optional

Kubernetes optional (plus namespaces)

Memory constrains translate to Docker --memory

Page 27: Using Containers for Building and Testing: Docker, Kubernetes and Mesos. FOSDEM 2017

WHAT DO YOU THINK HAPPENS WHEN?Your container goes over memory quota?

Page 28: Using Containers for Building and Testing: Docker, Kubernetes and Mesos. FOSDEM 2017
Page 29: Using Containers for Building and Testing: Docker, Kubernetes and Mesos. FOSDEM 2017

WHAT ABOUT THE JVM?WHAT ABOUT THE CHILD PROCESSES?

Page 30: Using Containers for Building and Testing: Docker, Kubernetes and Mesos. FOSDEM 2017

CPU LIMITSScheduler needs to account for container CPU requirements

and host available CPUs

Mesos required

Swarm optional

Kubernetes optional (plus namespaces)

CPU translates into Docker --cpu-shares

Page 31: Using Containers for Building and Testing: Docker, Kubernetes and Mesos. FOSDEM 2017

WHAT DO YOU THINK HAPPENS WHEN?Your container tries to access more than one CPU

Your container goes over CPU limits

Page 32: Using Containers for Building and Testing: Docker, Kubernetes and Mesos. FOSDEM 2017

Totally different from memory

Page 33: Using Containers for Building and Testing: Docker, Kubernetes and Mesos. FOSDEM 2017

STORAGEHandling distributed storage

Jenkins masters need persistent storage, agents (typically)don't

Mesos in 1.0+

Swarm Docker volume plugins: RexRay, Convoy,Flocker,...

Kubernetes

Docker volume support

Persistent volumes

Page 34: Using Containers for Building and Testing: Docker, Kubernetes and Mesos. FOSDEM 2017

PERMISSIONSContainers should not run as root

Container user id != host user id

i.e. jenkins user in container is always 1000 but matchesubuntu user in host

Page 35: Using Containers for Building and Testing: Docker, Kubernetes and Mesos. FOSDEM 2017

CAVEATSOnly a limited number of EBS volumes can be mounted

Docs say /dev/sd[f-p], but /dev/sd[q-z] seem towork too

NFS users must be centralized and match in cluster and NFSserver

Page 36: Using Containers for Building and Testing: Docker, Kubernetes and Mesos. FOSDEM 2017

NETWORKINGJenkins masters open several ports

HTTPJNLP Build agentSSH server (Jenkins CLI type operations)

Jenkins agents connect to master:

inbound (SSH)outbound (JNLP)

Page 37: Using Containers for Building and Testing: Docker, Kubernetes and Mesos. FOSDEM 2017

Allows getting one IP per container

Mesos : Calico, Weave

Swarm , and others from plugins

Kubernetes Multiple : GCE, Weave,Calico,...

Network Isolator Modules

Docker overlay

networking options

Page 38: Using Containers for Building and Testing: Docker, Kubernetes and Mesos. FOSDEM 2017

JENKINS PLUGINS

Page 39: Using Containers for Building and Testing: Docker, Kubernetes and Mesos. FOSDEM 2017

JENKINS DOCKER PLUGINSDynamic Jenkins agents with Docker plugin or Yet AnotherDocker Plugin

No support yet for Docker 1.12 Swarm modeAgent image needs to include Java, downloads slave jarfrom Jenkins masterMultiple plugins for different tasks

Docker build and publishDocker build step pluginCloudBees Docker Hub/Registry NotificationCloudBees Docker Traceability

Great pipeline support

Page 40: Using Containers for Building and Testing: Docker, Kubernetes and Mesos. FOSDEM 2017
Page 41: Using Containers for Building and Testing: Docker, Kubernetes and Mesos. FOSDEM 2017
Page 42: Using Containers for Building and Testing: Docker, Kubernetes and Mesos. FOSDEM 2017
Page 43: Using Containers for Building and Testing: Docker, Kubernetes and Mesos. FOSDEM 2017
Page 44: Using Containers for Building and Testing: Docker, Kubernetes and Mesos. FOSDEM 2017

JENKINS DOCKER PIPELINEdef maven = docker.image('maven:3.3.9-jdk-8');

stage 'Mirror' maven.pull() docker.withRegistry('https://secure-registry/', 'docker-registry-login'

stage 'Build' maven.inside { sh "mvn -B clean package" }

stage 'Bake Docker image' def pcImg = docker.build("examplecorp/spring-petclinic:${env.BUILD_TAG}"

pcImg.push(); }

Page 45: Using Containers for Building and Testing: Docker, Kubernetes and Mesos. FOSDEM 2017

JENKINS DOCKER SLAVES PLUGINUse any Docker image, no need for Java

Definition in pipeline

Can have side containers

Page 46: Using Containers for Building and Testing: Docker, Kubernetes and Mesos. FOSDEM 2017
Page 47: Using Containers for Building and Testing: Docker, Kubernetes and Mesos. FOSDEM 2017

Building MavendockerNode("maven:3.3.3-jdk-8") { sh "mvn -version" }

Page 48: Using Containers for Building and Testing: Docker, Kubernetes and Mesos. FOSDEM 2017

JENKINS MESOS PLUGINDynamic Jenkins agents, both Docker and isolatedprocessesAgent image needs to include Java, grabs slave jar fromMesos sandboxCan run Docker commands on the host, outside of Mesos

Page 49: Using Containers for Building and Testing: Docker, Kubernetes and Mesos. FOSDEM 2017
Page 50: Using Containers for Building and Testing: Docker, Kubernetes and Mesos. FOSDEM 2017
Page 51: Using Containers for Building and Testing: Docker, Kubernetes and Mesos. FOSDEM 2017
Page 52: Using Containers for Building and Testing: Docker, Kubernetes and Mesos. FOSDEM 2017
Page 53: Using Containers for Building and Testing: Docker, Kubernetes and Mesos. FOSDEM 2017
Page 54: Using Containers for Building and Testing: Docker, Kubernetes and Mesos. FOSDEM 2017

JENKINS MESOS PLUGINCan use Docker pipelines with some tricks

Need Docker client installedShared docker.sock from hostMount the workspace in the host, visible under same dir

Page 55: Using Containers for Building and Testing: Docker, Kubernetes and Mesos. FOSDEM 2017

MESOS PLUGIN AND PIPELINEnode('docker') { docker.image('golang:1.6').inside {

stage 'Get sources' git url: 'https://github.com/hashicorp/terraform.git', tag: "v0.6.15"

stage 'Build' sh """#!/bin/bash -e mkdir -p /go/src/github.com/hashicorp ln -s `pwd` /go/src/github.com/hashicorp/terraform pushd /go/src/github.com/hashicorp/terraform make core-dev plugin-dev PLUGIN=provider-aws popd cp /go/bin/terraform-provider-aws . """

stage 'Archive' archive "terraform-provider-aws" } }

Page 56: Using Containers for Building and Testing: Docker, Kubernetes and Mesos. FOSDEM 2017

JENKINS KUBERNETES PLUGINDynamic Jenkins agents, running as PodsMultiple container support

One jnlp image, others customPipeline support for both agent Pod definition andexecutionPersistent workspace in the next version

Page 57: Using Containers for Building and Testing: Docker, Kubernetes and Mesos. FOSDEM 2017

JENKINS KUBERNETES PIPELINEpodTemplate(label: 'mypod', containers: [ [name: 'maven', image: 'maven:3-jdk-8', ttyEnabled: true, command: [name: 'golang', image: 'golang:1.6', ttyEnabled: true, command: ]) {

node ('mypod') { stage 'Get a Maven project' git 'https://github.com/jenkinsci/kubernetes-plugin.git' container('maven') { stage 'Build a Maven project' sh 'mvn clean install' }

stage 'Get a Golang project' git url: 'https://github.com/hashicorp/terraform.git' container('golang') { stage 'Build a Go project' sh """ mkdir -p /go/src/github.com/hashicorp ln -s `pwd` /go/src/github.com/hashicorp/terraform cd /go/src/github.com/hashicorp/terraform && make core-dev """ } } }

Page 58: Using Containers for Building and Testing: Docker, Kubernetes and Mesos. FOSDEM 2017

JENKINS PLUGINS RECAPDynamic Jenkins agent creationUsing JNLP slave jar

In complex environments need to use the tunneloption to connect internally

Using the Cloud APINot ideal for containerized workloadAgents take > 1 min to start provision and are keptaroundAgents can provide more than one executor

Page 59: Using Containers for Building and Testing: Docker, Kubernetes and Mesos. FOSDEM 2017

JENKINS ONE SHOT EXECUTORImproved API to handle one off agents

Optimized for containerized agents

Plugins need to support it