the challenge of application distribution - introduction to docker (2014 dec 03)
TRANSCRIPT
The challenge of application distributionIntroduction to Docker
Credits & references
Intros slides borrowed from • dotCloud’s « Docker introduction » slideshow• AWS re:Invent 2014 session APP303, « Lighting Fast Deploys with
Docker Containers and AWS »
Agenda
• The application distribution challenge• The current solutions• Introduction to Docker, Containers, and the Matrix from Hell• Why people care: Separation of Concerns• Technical Discussion• Ecosystem, momentum• How to build Docker images• How to make containers talk to each other, how to handle data
persistence• Demo 1: isolation• Demo 2: real case - installing Go Math!, tail –f containers, unit
tests
The challenge
The challenge: apps have fundamentally changed
~2000 2014
Long lived Development is iterative and constant
Monolithic and built on a single stack
Built from loosely coupled components
Deployed to a single server Deployed to a multitude of servers
Everybody already heard/said it:
Static website
Web frontend
User DB
Queue Analytics DB
Background workers
API endpoint
nginx 1.5 + modsecurity + openssl + bootstrap 2
postgresql + pgv8 + v8
hadoop + hive + thrift + OpenJDK
Ruby + Rails + sass + Unicorn
Redis + redis-sentinel
Python 3.0 + celery + pyredis + libcurl + ffmpeg + libopencv + nodejs + phantomjs
Python 2.7 + Flask + pyredis + celery + psycopg + postgresql-client
Development VM
QA server
Public Cloud
Disaster recovery
Contributor’s laptop
Production Servers
The challenge: apps have fundamentally changedM
ult
iplicit
y o
f Sta
cks
Mult
iplicit
y o
f hard
ware
envir
onm
ents
Production Cluster
Customer Data Center
Do s
erv
ices a
nd
ap
ps in
tera
ct
ap
pro
pria
tely
?
Can I m
igra
te
sm
ooth
ly a
nd
q
uic
kly
?
The Matrix From Hell
Static website
Web frontend
Background workers
User DB
Analytics DB
Queue
Dev VM QA ServerSingle Prod
ServerOnsite Cluster
Public CloudContributor’s
laptopCustomer Servers
? ? ? ? ? ? ?
? ? ? ? ? ? ?
? ? ? ? ? ? ?
? ? ? ? ? ? ?
? ? ? ? ? ? ?
? ? ? ? ? ? ?
The current solutions
Current solutions
• ‘Configuration’ tools like Ansible, Chef, Puppet, …
• VM tools : Vagrant
• Container tools: Google’s LMCTFY
Mult
iplicit
y o
f G
oods
Mult
ipilic
ity
of
meth
ods f
or
transp
ort
ing/sto
ring
Do I w
orry
about
how
goods
inte
ract (e
.g.
coffe
e b
eans n
ext
to s
pic
es)
Can I tra
nsport
quic
kly
and
sm
ooth
ly(e
.g. fro
m b
oat to
tra
in to
truck)
Cargo Transport Pre-1960
? ? ? ? ? ? ?
? ? ? ? ? ? ?
? ? ? ? ? ? ?
? ? ? ? ? ? ?
? ? ? ? ? ? ?
? ? ? ? ? ? ?
Also a matrix from hell
Mult
iplicit
y o
f G
oods
Mult
iplicit
y o
f m
eth
ods f
or
transport
ing/sto
rin
g
Do I w
orry
about
how
goods
inte
ract (e
.g.
coffe
e b
eans n
ext
to s
pic
es)
Can I tra
nsport
quic
kly
and
sm
ooth
ly(e
.g. fro
m b
oat to
tra
in to
truck)
Solution: Intermodal Shipping Container
…in between, can be loaded and unloaded, stacked, transported efficiently over long distances, and transferred from one mode of transport to another
A standard container that is loaded with virtually any goods, and stays sealed until it reaches final delivery.
Containers applied to software
Static website Web frontend User DB Queue Analytics DB
Dev VMQA server Public Cloud
Contributor’s laptop
Docker is a shipping container system for code M
ult
iplicit
y o
f Sta
cks
Mult
iplicit
y o
f hard
ware
envir
onm
ents
Production Cluster
Customer Data Center
Do s
erv
ices a
nd
apps in
tera
ct
appro
pria
tely
?
Can I m
igra
te
sm
ooth
ly a
nd
quic
kly
…that can be manipulated using standard operations and run consistently on virtually any hardware platform
An engine that enables any payload to be encapsulated as a lightweight, portable, self-sufficient container…
Static website
Web frontend
Background workers
User DB
Analytics DB
Queue
Dev VM QA ServerSingle Prod
Server
Onsite
ClusterPublic Cloud
Contributor’s
laptop
Customer
Servers
Docker eliminates the matrix from Hell
Good for developers
• Build once…(finally) run anywhere*• A clean, safe, hygienic and portable runtime environment for your app.• No worries about missing dependencies, packages and other pain points during
subsequent deployments.• Run each app in its own isolated container, so you can run various versions of
libraries and other dependencies for each app without worrying• Automate testing, integration, packaging…anything you can script • Reduce/eliminate concerns about compatibility on different platforms, either your
own or your customers. • Cheap, zero-penalty containers to deploy services? A VM without the overhead of a
VM? Instant replay and reset of image snapshots? That’s the power of Docker
* Anywhere = any x86 server running a modern Linux kernel (3.2+ generally. 2.6.32+ for RHEL 6.5+, Fedora, & related)
Good for (Dev)Ops
• Configure once…run anything• Make the entire lifecycle more efficient, consistent, and repeatable• Increase the quality of code produced by developers. • Eliminate inconsistencies between development, test, production, and customer
environments• Support segregation of duties• Significantly improves the speed and reliability of continuous deployment and
continuous integration systems• Because the containers are so lightweight, address significant performance, costs,
deployment, and portability issues normally associated with VMs
Why it works—separation of concerns
• Dan the Developer– Worries about what’s “inside” the
container• His code• His Libraries• His Package Manager• His Apps• His Data
– All Linux servers look the same
• Oscar the Ops Guy• Worries about what’s
“outside” the container• Logging
• Remote access
• Monitoring
• Network config
• All containers start, stop, copy, attach, migrate, etc. the same way
Under the hood
• To learn more about how it works, some background:
–Linux namespaces• PID• IPTables• UTS (~hostname)• Mount• IPC (inter-process)
–CGroups–LXC Containers
AppA
Containers vs. VMs
Hypervisor (Type 2)
Host OS
Server
GuestOS
Bins/
Libs
AppA’
Guest
OS
Bins/
Libs
AppB
Guest
OS
Bins/
Libs
App A
’
Docker
Host OS
Server
Bins/Libs
App A
Bins/Libs
App B
App B
’
App B
’
App B
’VM
Container
Containers are isolated,but share OS and, whereappropriate, bins/libraries
GuestOS
GuestOS
…result is significantly faster deployment, much less overhead, easier migration, faster restart
Why are Docker containers lightweight?
Bins/
Libs
AppA
Original App(No OS to takeup space, resources,or require restart)
App
ΔBin
s/
AppA
Bins/
Libs
AppA’
Guest
OS
Bins/
Libs
Modified App
Copy on write capabilities allow us to only save the diffs between container A and container A’
VMsEvery app, every copy of anapp, and every slight modificationof the app requires a new virtual server
AppA
GuestOS
Bins/
Libs
Copy of AppNo OS. CanShare bins/libs
AppA
GuestOS
GuestOS
VMs Containers
Containers, images, …
Terminology
Docker concept OOP analogy
Dockerfile Source code
Image Class
Container Instance
What are the basics of the Docker system?
Source Code
Repository
DockerfileFor
A
Docker Engine
DockerContainer
Image Registry
Build
Docker
Host 2 OS (Linux)
Conta
iner A
Conta
iner B
Conta
iner C
Conta
iner A
Push
SearchPull
Run
Host 1 OS (Linux)
Changes and Updates
Docker Engine
DockerContainer
Image Registry
Docker Engine
Push
Update
Bins/
Libs
AppA
App
Δ
Bin
s/
Base Container
Image
Host is now running A’’
Container Mod A’’
App
ΔBin
s/
Bins/
Libs
AppA
Bin
s/
Bins/
Libs
AppA’’
Host running A wants to upgrade to A’’. Requests update. Gets only diffs
Container Mod A’
A new open source project… or the next big thing?
• Compatible with Jenkins, Travis, Chef, Puppet, Vagrant and OpenStack…
• Ecosystem: lots of tools, even by serious contributors (Google –Kubernetes, cAdvisor - NetFlix …), new IAAS provider emerge (like Tutum),
• Supported by Amazon, Google, Microsoft, Rackspace, Digital Ocean, Joyent, RedHat, …
• Momentum... See pull requests numbers and frequency to get an idea
• Used in production… even by big players (HP, Ebay, New Relic, Etsy, Yelp, Spotify, …)
Ecosystem Support
• Operating systems– Virtually any distribution with a 2.6.32+ kernel– Red Hat/Docker collaboration to make work across RHEL 6.4+, Fedora, and other members of the family (2.6.32
+)– CoreOS—Small core OS purpose built with Docker
• OpenStack– Docker integration into NOVA (& compatibility with Glance, Horizon, etc.) accepted for Havana release
• Private PaaS– OpenShift– Solum (Rackspace, OpenStack) – Other TBA
• Public PaaS– Deis, Voxoz, Cocaine (Yandex), Baidu PaaS
• Public IaaS– Native support in Rackspace, Digital Ocean,+++– AMI (or equivalent) available for AWS & other
• DevOps Tools– Integrations with Chef, Puppet, Jenkins, Travis, Salt, Ansible +++
• Orchestration tools– Mesos, Heat, ++– Shipyard & others purpose built for Docker
• Applications– 1000’s of Dockerized applications available at index.docker.io
Use Cases
• Ted Dziuba on the Use of Docker for Continuous Integration at Ebay Now– https://speakerdeck.com/teddziuba/docker-at-ebay– http://www.youtube.com/watch?feature=player_embedded&v=0Hi0W4gX--4
• Sasha Klizhentas on use of Docker at Mailgun/Rackspace
• http://www.youtube.com/watch?feature=player_embedded&v=CMC3xdAo9RI
• Sebastien Pahl on use of Docker at CloudFlare
• http://www.youtube.com/watch?feature=player_embedded&v=-Lj3jt_-3r0
• Cambridge HealthCare• http://blog.howareyou.com/post/62157486858/continuous-delivery-with-docker-
and-jenkins-part-i
• Red Hat Openshift and Docker
• https://www.openshift.com/blogs/technical-thoughts-on-openshift-and-docker
Use Cases—From the Community
Use Case Examples LinkClusters Building a MongoDB cluster using docker http://bit.ly/1acbjZf
Production Quality MongoDB Setup with Docker http://bit.ly/15CaiHbWildfly cluster using Docker on Fedora http://bit.ly/1bClX0O
Build your own PaaS OpenSource PaaS built on Docker, Chef, and Heroku Buildpacks http://deis.io
Web Based Environment for Instruction
JiffyLab – web based environment for the instruction, or lightweight use of, Python and UNIX shell
http://bit.ly/12oaj2K
Easy Application Deployment
Deploy Java Apps With Docker = Awesome http://bit.ly/11BCvvuHow to put your development environment on docker http://bit.ly/1b4XtJ3Running Drupal on Docker http://bit.ly/15MJS6B
Installing Redis on Docker http://bit.ly/16EWOKh
Create Secure Sandboxes
Docker makes creating secure sandboxes easier than ever http://bit.ly/13mZGJH
Create your own SaaS Memcached as a Service http://bit.ly/11nL8vh
Automated Application Deployment
Multi-cloud Deployment with Docker http://bit.ly/1bF3CN6
Continuous Integration and Deployment
Next Generation Continuous Integration & Deployment with dotCloud’sDocker and Strider
http://bit.ly/ZwTfoy
Testing Salt States Rapidly With Docker http://bit.ly/1eFBtcm
Lightweight Desktop Virtualization
Docker Desktop: Your Desktop Over SSH Running Inside Of A Docker Container
http://bit.ly/14RYL6x
How to build an image?
Two ways to create an image:
• You work in a container, then commit it for exploratory work
• You build a new image from a Dockerfile for automated, repeatable process
1 container ?= (1process|1task|other) ?
Best practices appear
• Do not force you to 1 process per container• Ideal case: 1 ‘task’ per container, with separation of
concerns.
• The task might require 1 single process•Or 0 (for data-only containers)•Or several (ex: nginx+php-fpm, monitoring, …)
Containers are isolated… then how to make them talk to each other?
• Private network between containers• Linking to make them easy to talk to a friend container
• Expose ports•Mount volumes (rw or ro)
• Advanced/not so trivial other solutions–Service discovery–Enter into the container’s namespace
How to handle data persistence
• Data only containers, sharing data volumes•Mounting this data volumes in the running containers
How to handle data persistence
How data volume works
How to handle data persistence
How data volume works
Demo time!
•Quick look at the Docker commands
• Isolation demo
•Multi containers demo: installing and launching Go Math! Academy in seconds.
• Running unit tests in a container
Warning
•Docker is great–It reduces dependencies, provides isolated and controlled context
•But it brings new tools, and new technologies, hence a potential complexity increase–libswarm, systemd, etcd, ambassadord, fleet, etc
Some references
• Start by the online tutorial and documentation, they are great!
• Here’s some links kept while learning… but so much available!–Eight Docker Development Patterns–Panamax and Docker and Some Best Practices–Tutum blog–The 5 Most Important Things I’ve Learned From Using Docker
–Understanding Docker Volumes–Docker's Youtube videos
More references
First steps• dotScale 2013 - Solomon Hykes - Why we built Docker (video)• Twitter University – Solomon Hykes – Introduction to Docker (video)
Going further• Docker layers cost (blog), on how to optimize your images layer size.• Twitter University - Docker at Spotify (video)• Top 10 open-source Docker developer tools (blog, incomplete but gives ideas)• Docker In-depth: Volumes (blog)
Food for thought / Stimulate your imagination• AWS re:Invent 2014 - Lightning fast deploys with Docker containers and AWS (video,
slides)• FutureStack14 - Docker and the Future of Modern Software (New Relic blog, video) • AWS re:Invent 2014 - Infrastructure as Code (video, more related to Cloudformation
than to Docker but very inspiring if you consider the two technologies)• Alternative? Rocket by CoreOS… started a debate (like Joyent answer)
Tools for dev, day-to-day
• For everyone
• On Mac– Kitematic UI to manage your images/containers
• On Windows– Try the new XShell5 beta to easily deal with all your current containers
• So many others… and new ones every week
• Fig (Fast isolated environments, by Docker) repo, site• A Better Development Environment with Docker and Fig (blog) • Lots of tools to manage multi containers apps: Kubernetes, Shipyard,
Panamax,
Thank you