OVERCOMING 5 COMMON

DOCKER CHALLENGES: HOW WE DID IT AT RIGHTSCALE

Panelists

• Ryan O’Leary: Moderator

• Senior Director, Product

• Tony Spataro

• Senior Systems Architect

• Mark Dotson

• Principal System Administrator

• The State of Docker

• RightScale’s Docker Journey

• Development Challenges

• Managing Docker images

• Multi-host Docker development

• Ops Challenges

• Orchestrating Docker in production:

• service discovery, configuration, and auditability

• Dynamic monitoring and alerting

• Increasing container density per host

Agenda

2

RightScale Vision: Manage Any Resource Pool

3

Public

Clouds

Private

Clouds

Virtual

Environments

Bare Metal

Servers

Containers Containers Containers Containers

Orchestration, Management, and Governance for Any Environment

2%

3%

3%

4%

6%

7%

9%

20%

27%

32%

32%

8%

10%

12%

11%

16%

18%

12%

14%

35%

18%

19%

Rancher

Rocket

Docker Tutum

Mesosphere

Docker Swarm

Kubernetes

Salt

Ansible

Docker

Puppet

Chef

Respondents Using DevOps Tools

Use today

Plan to use

2016 DevOps Tools – All Respondents

Source: RightScale 2016 State of the Cloud Report

3%

6%

10%

13%

28%

24%

3%

9%

20%

27%

32%

32%

Rocket

Salt

Ansible

Docker

Chef

Puppet

Respondents Using DevOps Tools

2016

2015

DevOps Tools YoY – All Respondents

Source: RightScale 2016 State of the Cloud Report

RIGHTSCALE’S DOCKER

JOURNEY

• Architectural role of VMs

didn’t change

• Services other then app

lived outside container

• Deployed (generally) 1

app container onto each

VM, using nginx on VM to

route to app container.

• Static Mapping

Step 1: Containerize Code

7

syslog smtp

my-awesome-app

Application Server 1..n

Container

No container

nginx

• Deploy N containers in

a non-cloud VM

• Integration test in real

time

• No persistence, HA, etc

Step 2: Composing in dev-and-test

8

Developer Laptop

nginx

my-awesome-app

smtp

syslog

• Deploy N ‘good

neighbor’ containers

onto a VM

• Host-local balancer

• Service discovery

• Supports

microservices

• Supports “traditional”

services

Step 3: Bay of Containers

9

Application Logical Grouping 1..n

balancer

A

smtp syslog srv.

dsc.

B C D

Sidecar B

balancer

E

smtp syslog srv.

dsc.

F

G H Z

DEVELOPMENT AND

DOCKER

CHALLENGE 1

MAKING (USEFUL) IMAGES

Images: Build Args

# In my Dockerfile

ARG gitref=unknown

LABEL git.ref=${gitref}

# During my build

commit=`git rev-parse --verify HEAD`

docker build --build-arg gitref=${commit}

1

2

# In my built image‘s metadata

Id: "sha256:11bb...",

RepoTags: [

"rightscale/right_api:latest”

],

Labels: {

git.ref: "f131ac4047...”

}

3

Metadata-rich images

Conditional content (e.g. debug support)

Your idea here! (Careful…)

Fetch dependencies before the build; install them during the build.

# Add fetched dependencies to

# the image

COPY vendor/cache vendor/cache/

# Build them inside the container

# for matched shared libraries &c

RUN bundle install

# No key material in my image!

Images: The Golden Rule

# Continuous integration script

# Use an SSH private key to

# fetch private dependencies

bundle package

docker build

Secure - no embedded private keys

Efficient layer-cached image builds

Repeatable dependency resolution

• Developers want to integration test without using production-

grade orchestration tools

• Docker images that know how to dance

• Two (or more) ways to dance

• Fail fast

• Eventual consistency

Images: Choreography

# I fail at startup if my database

# is not reachable, or if it isn’t

# in the ideal state for me.

CMD bin/webapp --database=$DB_HOST

Brittle containers Resilient compositions

# docker-compose will relaunch me

# every time I fail.

# Not suitable for real deployment!

webapp:

image: rightscale/webapp:latest

restart: always

env:

DB_HOST: postgres

links:

- postgres

Choreography: Fail Fast

+ Implicit failure is less likely

- Causes log spam at startup

# Wrapper script runs my actual

# service

CMD bin/entrypoint.sh

# Wrapper converges on an ideal state

# before passing control to my app

while [ “$?” != “0” ]; do

psql –qc "select now()" $DB_HOST

sleep 1

done

Resilient containers Simple compositions

# Failures are an error, not an

# expected outcome

webapp:

image: rightscale/webapp:latest

restart: never

env:

DB_HOST: postgres

links:

- postgres

Choreography: Eventual consistency

+ Allows powerful and sophisticated setup logic

- Developer tooling built into images; may ship to production!

CHALLENGE 2

MULTI-HOST DOCKER DEVELOPMENT

Mixed containers/processes Pure-container

Which Runtime Model?

Split Connected

Which Networking Model?

Mixed runtime; split networks

Pure-container runtime; connected network

• Many options

• Experiment & play

• As simple as possible

• …but no simpler!

Multi-Host Docker

OPERATIONS AND

DOCKER

CHALLENGE 3

ORCHESTRATION

• Existing, proven architecture

• Orchestration tools already in place

• SSAE-16 certified key management & auditing practices

• Managing a mixture of resources

• VMs

• Containers-on-VMs

• Databases, load balancers, networks, other non-compute resources

• Need for cross-cutting solutions

• Service discovery that spans VMs and containers

• Key/value store that works for with VM-hosted services

Orchestration: No Clean Slate

Orchestration

26

• Haproxy

• Host-local load balancer.

• Consul

• Service Discovery

• Key-Value Store

• Consul-Template

haproxy

A

smtp syslog consul

B C D

Sidecar B

Orchestration: HAProxy

27

• Acts as host-level balancer between boundry balancers and

containers.

• Allows for enforcement of an container-level maximum connections

allowed to a given service

• Dynamically creates or destroys app listener pools with upstream

hosts (containers) using service discovery/consul + consul-template.

Orchestration: Consul

28

• Service Discovery

• Registers service offering along with randomly assigned

local host port map

• Key-Value Store

• Universal port mapping

• Application configuration (“inputs”) information

• Audit trail

• Consul-template

• Consul + HAProxy = Win

CHALLENGE 4

MONITORING & ALERTING

Monitoring & alerting

30

• Application specific

monitoring for services in

containers

• Deployment information in

consul key-value

• Collectd + docker exec

• Application specific alerts for

services in containers

• Alert definations stored in consul

key-value

• Creation and destruction calls to

add to vm at container runtime

# alert stored as json hash and into kv

{

“deployment”: {

“alerts”: {

"container - High Activity Delay": {

"name": "container - High Activity Delay",

"description": "This graph represents the delay before an

activity completing a request",

"file": "cwf_stats/gauge-median_activity_delay",

"variable": "value",

"condition": ">=",

"threshold": "60",

"duration": 10,

"escalation_name": "critical"

},

…

CHALLENGE 5

INCREASED HOST DENSITY

Increased host density

32

• Old model was 1 properly-sized

vm to 1 container.

• At the end of the day, bringing up a

new service still meant… launching

a new instance.

• New model is 1 properly-sized

vm to many “good neighbor”

containers.

• Have some excess capacity due to

your usage patterns on a host/set of

hosts? Add that new service right

on in!

haproxy

A

smtp syslog consul

B C D

Sidecar B

WHAT COMES NEXT?

Sea of Containers

34

VM VM VM

A A A

A A

A

C C

B B

B B

VM VM VM

A A A

C C

A A A

C C

B B B

B B

Container Management

B • N(×M) containers

• 0..N VMs

• Elastic mesh network

• Declarative

everything

• Resource scheduling

A A

ECS Kubernetes Rancher Swarm

Docker CLI X X X

Orchestration API,

compose API compose compose

Constraints replication placement placement

Shared Networking layer-3 overlay (IPSec) overlay (VXLAN)

Shared Storage X X D.I.Y.

Load Balancer X X X D.I.Y.

Self-Healing replace replace restart restart

Portability low medium medium high

Overview of Solutions

• Using Docker and RightScale

• www.rightscale.com/docker

Q&A

36