containing the world with docker

Containing the World with Docker

Giuseppe Piccolo

Summary

• The evolution of IT• The Solutions• Containers!• Docker• Use Cases• Examples– Deploy an application with Docker

The Past

• Monolithic software• Monolithic environments– Specific OS– Specific Middleware– Specific Runtime

• One application can run in more devices– PC– Smartphone– Tablets– …

• One service for each needs

This involves

More scalable than this!

More reuse

More testable components (services)

Software engineers are happy J

Great things, but…

Static website

Web frontend

User DB Queue

Analytics DB

Background workers

API endpoint

nginx 1.5 + modsecurity + openssl + bootstrap 2postgresql + pgv8 + v8

hadoop + hive + thrift + OpenJDKRuby + Rails + sass + Unicorn

Redis + redis-sentinel

Python 3.0 + celery + pyredis + libcurl + ffmpeg + libopencv + nodejs + phantomjs

Python 2.7 + Flask + pyredis + celery + psycopg + postgresql-client

Development VM

QA serverPublic Cloud

Disaster recovery

Contributor’s laptop

Production Servers

Multip

licity

of Stacks

Production Cluster

Customer Data Center

Do services and apps interact

appropriately?

Can I migrate

smoothly and quickly?M

ultip

licity

of

hardware

environm

ents

Static website

Web frontend

Background workers

User DB

Analytics DB

Queue

Development VM QA Server Single Prod

ServerOnsite Cluster Public Cloud Contributor’

s laptopCustomer

Servers

? ? ? ? ? ? ?

? ? ? ? ? ? ?

? ? ? ? ? ? ?

? ? ? ? ? ? ?

? ? ? ? ? ? ?

? ? ? ? ? ? ?

Hell of configurations!

and the worst is yet to come…

Dependency Hell!

“Dependency hell is a colloquial term for the frustration of some software users who have

installed software packages which have dependencies on specific versions of other

software packages”

Wikipedia

As a consequence

Development environment and Production environment are different!

Challenges

• How to avoid the Hell of configuration?• How to avoid the Dependency hell?• How to migrate and scale quickly ensuring compatibility?

• How to replicate my services quickly?

Solutions

1. Virtual Machines

• Software abstraction of physical machine

• Virtualize anything, hardware included

• Provide a complete isolation, thanks to Hypervisor

• Pro– Fully virtualized OS– Total isolation

• Cons– Needs to take a snapshot entire VM in order to replicate them

– Use a lot of space– Slow to move around– Poor performance

Hypervisors trend

• 2011– XEN : Default choice given Rackspace and Amazon use

– KVM: Bleeding edge users

• 2012– KVM: Emerges as lead– XEN: Loses momentum

Hypervisors trend

• 2013– KVM: Maintains lead (around 90% for Mirantis)– Vmware: Emerges as surpising second choice– Containers (LXC, Parallels, Docker): Web Hosting and SAS focused

– Xen and HyperV: infrequent requests (XenServer.org)

• 2014 – 2015– Stay tuned J

Vagrant

• Open source VM manager • Released in 2010• It allows you to script and package VM configand the provisioning setup via a Vagrant File

• It is designed to run on top of almost VM tool: VirtualBox, VMWare, AWS, OpenStack

• Automating operations with tools such as shell scripts, Chef and Puppet

Vagrant idea:A file to define a service!

In a Vagrant File you define:– An operating system

– Required library and software

– Programs and processes of your final application

Vagrant: Feature

• Command-‐Line interface• Vagrant Share• Boxes• Provisioning• Networking• Synced Folders• Multi-‐Machine• Providers• Plugins

Vagrant: CLI

• It allow us to interact with Vagrant• It offers the following commands: box,connect,destroy,halt,init,login,package a vm, rdp, …

https://docs.vagrantup.com/v2/cli/index.html

Vagrant Example

Install Vagrant and Virtual Box then init Vagrant

1. Install a box (in this example ubuntu)

2. Edit Vagrant previously created by init command

Other box can be found at https://vagrantcloud.net

Start a Vagrant Machine

1. Start the box

2. Login into the vm

3. You can destroy the vm by

Synced Folders

• By default, it shares your project directory to the /vagrant directory in the guest machine

• If you create, in your host os the file will be on the vagrant vm

Provisioning

• Let’s install Apache via a bootstrap.sh file

• If you create a file on your host os the file will be on the vagrant vm (use vagrant reload -‐-‐provision)

Networking

• Port forwarding : allows you to specify port on the guest system to share via a port on the host machine

• By running vagrant reload or vagrant up we can see on http://127.0.0.1:8080 our apache

• It supports also bridge configurations and other configurations (https://docs.vagrantup.com/v2/networking)

Provider and Share

Providers

• Vagrant ships out of the box with support for VirtualBox

• If you want to change this behavior (e.g. use VMWare or AWS) you can change the provider

• How?

AWS Vagrant file example

Share

• It is possible to share a Vagrant box with anyone in the world in many ways:– HTTP – SSH– Others (AWS, …)

Containers

How many people have heard of Containers before this Talk?

You've always used him but did not know

• Google services runs into a container (Gmail, Search, …)

• Spotify runs into containers• Ebay runs into container• Baidu runs into containers• ...

Why Containers?

I try to explain with an analogy J

Cargo Transport – Pre 1960Multip

licity

of G

oods Do I w

orry about how

goods interact (e.g. coffee beans next to spices)

Multip

ilicity

of

metho

ds fo

r tran

sportin

g/storing Can I transport quickly

and smoothly

(e.g. from boat to train

to truck)

Also a matrix from hell? ? ? ? ? ? ?

? ? ? ? ? ? ?

? ? ? ? ? ? ?

? ? ? ? ? ? ?

? ? ? ? ? ? ?

? ? ? ? ? ? ?

As you can see not only computer scientists have problems to solve

How they solved them?

Standardized shipment with containers!Multip

licity of G

oods Do I w

orry about how

goods interact (e.g. coffee beans next to spices)

Can I transport quickly and sm

oothly(e.g. from

boat to train to truck)

…in between, can be loaded and unloaded, stacked, transported efficiently over long distances, and transferred from one mode of transport to another

A standard container that is loaded with virtually any goods, and stays sealed until it reaches final delivery.

Farewell, Matrix from hell!

Farewell, Matrix from hell!?

?

?

?

?

?

Farewell, Matrix from hell!

Sounds Good!

Why don’t use the same principle in Computer Science?

Static website Web frontend User DB Queue Analytics DB

Development VM QA server

Public Cloud

Contributor’s laptop

Multip

licity of Stacks

Multip

licity of

hardware

environm

ents

Production Cluster

Customer Data Center

Do services and apps interact

appropriately?

Can I migrate

smoothly and quickly

…that can be manipulated using standard operations and run consistently on virtually any hardware platform

An engine that enables any payload to be encapsulated as a lightweight, portable, self-sufficient container…

Static website

Web frontend

Background workers

User DB

Analytics DB

Queue

Development VM QA Server Single Prod

ServerOnsite Cluster Public Cloud Contributor’

s laptopCustomer

Servers

Problem Solved!

This Solution already exists…

Linux Containers!

Linux Containers (LXC)

• Units of software delivery• Run everywhere

– Don’t worry about kernel version– Don’t worry about host distro– (but container and host architecture must match*)

• Run anything– If it can run on Linux Kernel, it can run J

*Unless you emulate CPU with quemu and binfmt

What is a Container?

• It’s a lightweight VM– Own his process space– Own his network interface– Can run stuff as root– Can have it’s own /sbin/init (different from the host)

What is a Container?

• At low level– Container run his processes isolated from the other containers

– Share the kernel with host– No device emulation

Separation of Concerns at environment level!

Developer Point of View

Inside my container

• My code• My libraries• My package manager• My app• My data

Ops Point of View

Outside my container

• Logging• Remote access• Network configuration• Monitoring

All containers start, stop, copy, attach, migrate, etc. the same

way!

How it works?

Isolation with namespaces

• pid• mnt• net• uts• ipc• user

Isolation with cgroups

• memory• cpu• blkio• devices

• AppArmor and SELinux Capability• Seccomp (very specific use case); seccomp-‐bpf• Beware of full-‐scale kernel exploits!• Least but not last: think twice before granting root!

And security?

• Unioning file system (AUFS, overlayfs)

• Snapshotting filesystems(BTRFS, ZFS)

Copy-‐on-‐write storage

Efficiency

Almost no overhead!

• Processes are isolated but run straight on the host

• Native CPU performance (no emulation)• Good memory performance • A little overhead in network performance

Storage efficiency

Docker

Quick Survey

• How many people have heard of Docker before this Talk?

• How many people have tried Docker?• How many people are using Docker in production?

What is Docker?

“With Docker, developers can build any app in any language using any toolchain. “Dockerized” apps are completely portable and can run anywhere – collegue’s OSX and Windows Laptops, QA servers running Ubuntu in the cloud, and production data center VMs running Red Hat.”

Docker.io

In a nutshell …

• Rewrite of dotCloud internal container engine– Original version: Python, closed source, internal stuff only

– Release version: Go, legacy-‐free

• Docker is an open source engine to commoditize LXC

• Allows to create and share standard images, using copy-‐on-‐write mechanism (e.g. AUFS)

Better than VMsAppA

Hypervisor (Type 2)

Host OS

Server

GuestOS

Bins/Libs

AppA’

GuestOS

Bins/Libs

AppB

GuestOS

Bins/Libs

App A’

Docker

Host OS

Server

Bins/Libs

App A

Bins/Libs

App B

App B’

App B’

App B’

VM

Container

Containers are isolated,but share OS and, whereappropriate, bins/libraries

GuestOS

GuestOS

…result is significantly faster deployment, much less overhead, easier migration, faster restart

Features

• VE (Virtual Environments) based on LXC• Portable deploy across machines• Versioning: docker include git-‐like capabilities for tracking versions of a container

• Component reuse: it allows building or stacking already created packages. You can create “base images” and then running more machine based on the image

• Shared: libraries there is a public repository with several images (http://registry.hub.docker.com)

Why Docker and not only LXC?

• Fast delivery of your applications• Deploy and scale more easily• Get higher density and run more workload• Fast deployment makes for easier management• Provide standard operation to handle containers• Because is a commodity to easily handle containers

How it Works

Source Code

Repository

DockerfileFor A

Docker Engine

DockerContainerImage Registry

Build

Docker

Host 2 OS (Linux)

Container A

Container B

Container C

Container A

Push

Search Pull

Run

Host 1 OS (Linux)

Changes and updates

Docker Engine

DockerContainerImage Registry

Docker Engine

Push

Update

Bins/Libs

AppA

App Δ

Bins/

Base ContainerImage

Host is now running A’’

Container Mod A’’

App Δ

Bins/

Bins/Libs

AppA

Bins/

Bins/Libs

AppA’’

Host running A wants to upgrade to A’’. Requests update. Gets only diffs

Container Mod A’

Docker Ecosystem

• Operating systems– Virtually any distribution with a 2.6.32+ kernel– Red Hat/Docker collaboration to make work across RHEL 6.4+, Fedora, and other

members of the family (2.6.32 +)– CoreOS—Small core OS purpose built with Docker

• OpenStack– Docker integration into NOVA (& compatibility with Glance, Horizon, etc.) accepted

for Havana release• Private PaaS

– OpenShift– Solum (Rackspace, OpenStack) – Other TBA

• Public PaaS– Deis, Voxoz, Cocaine (Yandex), Baidu PaaS

• DevOps Tools– Integrations with Chef, Puppet, Jenkins, Travis, Salt, Ansible +++

• Applications– 1000’s of Dockerized applications available at index.docker.io

Docker examples

Run a simple container

• Using the run command

• if the image does not exist in the local repository, you get it from the remote repository

• After the download we run into container

• If we want to exit from container just press CTRL + P + Q• If we want to see the status of running container just use the

‘ps’ command

• If we want to execute a specific command on a running container just use the ‘exec’ command

Start and Stop containers

• To run a container from an existing image just use the ‘run’ command

• To stop a container just use the ‘stop’ command pass the container’s name or ID

Tip: If you don’t remember container’s ID or name get it by a ‘docker ps’ command

Remove a containerIf you want to remove a container you can use the ‘rm’ command

Network Setup

• You can specify one or more port on the host to map with one or more port on the container newtwork interface

• If you see the status of container, you can see the network interfaces of container

Share data

In Docker we can specify one or more folder to share between container and host, while run a container

• If we create a file into host

• we see the same file into container and vice-‐versa

Dockerfile

• A Dockerfile is a file that tell to docker how to build custom images

• It starts from a base image• All the added data from this point on are added as an additional layer starting from base image

https://docs.docker.com/reference/builder/

Dockerfile example: mongodb

https://github.com/docker-‐library/mongo/blob/1d641659a75cf2f8ce1b517c7fc2a0ebfd033eed/2.6/Dockerfile

Build a Dockerfile

Docker assume that a file named Dockerfile exists in specified folder!

After build we can see the image into images list

We can run the created image with docker run comand

Deploy a web application

Deploy a web application -‐ Wordpress

• Wordpress Application (1 Container)– Apache webserver / PHP 5.6–Wordpress 4

• MySQL (1 Container)– MySQL server Community 5.6

Run the containers

• Run MySQL container

Tips: -‐e flag permit send environment variable to container

• And the wordpress container

References

• http://www.slideshare.net/dotCloud/docker-‐intro-‐november?related=2

• http://www.slideshare.net/fabiofumarola1/develop-‐with-‐linux-‐containers-‐and-‐docker

• http://www.slideshare.net/jpetazzo/introduction-‐docker-‐linux-‐containers-‐lxc

• https://www.docker.com/• https://docs.docker.com• https://github.com/docker/docker• https://docs.docker.com/examples/mongodb/