the cloud is my laboratory

The cloud is my laboratoryExperimenting with Docker

Adam Friedman

Dimension Data (ITaaS R&D)

[email protected]

What this talk is about»

What this talk is about» Experimenting with Docker

What this talk is about» Experimenting with Docker (der)


» Using virtualisation to reduce cycle time when trying out ideas



» Using the cloud to achieve scale



» Using the cloud to achieve scale» Without breaking the bank

What I'd like to leave you with1.

What I'd like to leave you with1. We learn more from our failures than our

successes


successes 2. Occam's razor


successes 2. Occam's razor (prefer to minimise variables)


successes 2. Occam's razor

» Aim for repeatability where possible


successes 2. Occam's razor3. Virtualisation + automation make it cheap to

experiment



experiment4. If experimentation is cheap enough, you can try

out almost any idea you can come up with




out almost any idea you can come up with» Think it's a silly idea? Prove it.




out almost any idea you can come up with» Don’t be afraid to experiment!

Why I love Virtualisation Virtualisation is the bedrock on which most things cloud are built.

Why I love Virtualisation Virtualisation is the bedrock on which most things cloud are built. It is the key to:»

Why I love Virtualisation Virtualisation is the bedrock on which most things cloud are built. It is the key to:» Economies of scale

Why I love Virtualisation Virtualisation is the bedrock on which most things cloud are built. It is the key to:» Economies of scale» Speed of provisioning

Why I love Virtualisation Virtualisation is the bedrock on which most things cloud are built. It is the key to:» Economies of scale» Speed of provisioning» Repeatability

Why I love Virtualisation Virtualisation is the bedrock on which most things cloud are built. It is the key to:» Economies of scale» Speed of provisioning» Repeatability

» The ability to readily and reliably duplicate an environment and its resources

Why I love the concept of CloudFor all its potential downsides, the cloud has vastly improved the process of experimentation:

Why I love the concept of CloudFor all its potential downsides, the cloud has vastly improved the process of experimentation:» By selectively abstracting away levels of

infrastructure, it enables us to focus on what we're trying to do


infrastructure, it enables us to focus on what we're trying to do, rather than how to do it



» Economies of scale also come into play:



» Economies of scale also come into play:» You can’t fit 20 servers under your desk



» Economies of scale also come into play:» You can’t fit 20 servers under your desk, but

your cloud provider has plenty to spare (for as long as you need them)



» Economies of scale also come into play:» Most providers now charge only for what you

actually use



» Economies of scale also come into play» Lower costs make it easier to explore a problem

space, which helps to mitigate risk

Experimenting in the cloudThe trick to being productive when experimenting in the cloud:» Get set up

Experimenting in the cloudThe trick to being productive when experimenting in the cloud:» Get set up» Do your experiment

Experimenting in the cloudThe trick to being productive when experimenting in the cloud:» Get set up» Do your experiment» Clean up the evidence

Without breaking the bankThe trick to being productive when experimenting in the cloud:» Get set up» Do your experiment» Clean up the evidence» Before the mob arrives with pitchforks and

flaming torches

Docker is greatfor experimenting

with new technologies

If you've heard of it, there's probably a Docker image of

it.


it.

Example: What’s Neo4J?


it.

Example: What’s Neo4J?

docker run -p 7474:7474 neo4j


it.

Ok, how about RabbitMQ?


it.

Ok, how about RabbitMQ?

docker run -p 5672:5672 rabbitmq


it.

Ok, smartarse


it.

Ok, smartarse, but I bet there isn't one for

cowsay


it.

Ok, smartarse, but I bet there isn't one for

cowsay

docker run mwendler/cowsay "Sorry."

________

< Sorry. >

--------

\ ^__^

\ (oo)\_______

(__)\ )\/\

||----w |

|| ||

That's super useful, but what if you're experimenting with docker itself?


docker-machine create docker1

eval $(docker-machine env docker1)


docker-machine create docker1

eval $(docker-machine env docker1)

Ok, but not very cloudy.

That's super useful, but what if you're experimenting with docker itself?How about:

That's super useful, but what if you're experimenting with docker itself?How about:

docker-machine create --driver amazonec2 docker1

docker-machine create --driver amazonec2 docker2

That's super useful, but what if you're experimenting with docker itself?Or:

That's super useful, but what if you're experimenting with docker itself?Or:

docker-machine create --driver digitalocean docker1

docker-machine create --driver digitalocean docker2

Docker Machine is nice

Docker Machine is nice, but it's opaque

Docker Machine is nice, but it's opaqueSometimes that’s what you want.

Docker Machine is nice, but it's opaqueSometimes that’s what you want (again, the focus may be on what you're trying to do).

Docker Machine is nice, but it's opaqueSometimes that’s what you want (again, the focus may be on what you're trying to do, not how).

Docker Machine is nice, but it's opaqueSometimes that’s what you want (again, the focus may be on what you're trying to do, not how).Other times, not so much.

Docker Machine is nice, but it's opaqueSometimes that’s what you want (again, the focus may be on what you're trying to do, not how).Other times, not so much.So what other options are there?

Do it by hand

You could do this at least once - a useful learning experience (especially for when things go wrong).

Automate it!

I'm probably preaching to the converted here :)

Infrastructure with Terraform

Declarative configuration for infrastructure» Multi-cloud / multi-provider» Repeatable» Easy to consistently create / destroy / re-create

infrastructure But Terraform is only half the story. Once the infrastructure has been created, how do you get Docker itself deployed and configured?

Software with Ansible

» Hardly the only option out there, but if you're experimenting with throw-away systems then it's a good choice because it's just SSH (no master / agents to deploy) » Chef Solo (or Chef Zero) is an alternative, but may

still require more work to bootstrap each node» Quick to get started (just run commands or modules

on target machines) » Scales up to Roles + Playbooks for repeatability

» Not great for managing large numbers of machines

Software with Ansible

» Can be run from Terraform (via a plug-in provisioner) but it's a lot easier to run it separately while you're experimenting.

» There are Ansible inventory plugins that can read a Terraform state file (so it knows server roles, host names, IP addresses, etc).

» Ansible Galaxy has modules for everything, including Docker (hint, hint)

» Beware if you are deploying an OS with only Python 3.x (Ansible needs 2.x)

A quick detour:Docker from simple to complex» Stand-alone

» Docker» Orchestrated

» Local» Docker Compose

» Clustered» Docker Swarm

» Clustered with GUI» Cattle / Rancher» Kubernetes» Mesos / Marathon

» Kitchen Sink» Mantl

Note that simple and easy-to-use are not the same thing :)

Cisco Mantl

» Mantl is a microservices platform based on Docker, Mesos / Marathon, Consul, Traefik, Kubernetes, Calico, Contiv, etc (pretty much everything but the kitchen sink).

» Because it already includes several popular systems that extend / orchestrate Docker, it’s a useful starting point if you’re stuck trying to work out how to deploy or integrate a particular component

Cisco Mantl

» Mantl is deployed using Terraform with Ansible, and can therefore be deployed on a variety of clouds by swapping out Terraform modules as required.

» Their Ansible inventory plugin understands which cloud provider was used to create the infrastructure and so Ansible playbooks can adjust their behaviour to suit (if required).

Build your own lab

Build your own (throw-away) lab

Build your own (throw-away) lab

» Digital Ocean is fastAdd Terraform and you can repeatedly create and destroy environments in seconds, rather than minutes.» Obviously you can use AWS / Azure, too (or

Dimension Data Cloud Control) but for quick-and-dirty experiments I prefer the simplest thing that works).

» Create an Ansible playbook to install Docker» Start small - put everything in a single playbook

and then move stuff out to separate reusable roles when needed

Build your own lab

4 servers (1 master, 3 workers).

lab.tf:

resource "digitalocean_droplet" "master" {

count = 1

image = "ubuntu-14-04-x64"

name = "master-${count.index + 1}"

region = "nyc2"

size = "1024mb"

}

resource "digitalocean_droplet" "worker" {

count = 4

image = "ubuntu-14-04-x64"

name = "worker-${count.index + 1}"

region = "nyc2"

size = "1024mb"

}

Build your own lab

» terraform apply

Build your own lab

» terraform apply» …

Build your own lab

» terraform apply» …» Profit!

Build your own lab

» terraform apply» …» Profit!» terraform destroy

Build your own lab

» terraform apply» …» Profit!» terraform destroy» No pitchforks

Build your own lab

» terraform apply» …» Profit!» terraform destroy» No pitchforks (probably)

Why not use Rancher / Docker Cloud?

» If all you want to do is deploy containers then these are a much better choice


» If all you want to do is deploy containers then these are a much better choice» Rancher uses docker-machine to create nodes

for you



for you » But this is a good choice if you want to learn

about:




about:» How to deploy docker




about:» How to deploy docker» Non-standard docker configurations




about:» How to deploy docker» Non-standard docker configurations

» It's useful to have a feeling for what's behind the UI

Links

» github.com/tintoy» github.com/DimensionDataResearch» mantl.io» terraform.io» ansible.com» github.com/angstwad/docker.ubuntu

http://github.com/tintoy

http://github.com/DimensionDataResearch

http://mantl.io/

http://terraform.io/

http://ansible.com/

http://github.com/angstwad/docker.ubuntu

Questions?