Tricks of the Captains

Adrian Mouat

Chief Scientist @ Container Solutions

Tricks of the Captains● A hodgepodge of tricks from

members of the Docker Captains Programme

● And other luminaries of microservices and containers

Daily Development

Configure docker ps OutputDefault output of docker ps (or docker container ls)

$ docker psCONTAINER ID IMAGE COMMAND ... 0f1f72c9aac0 nginx "nginx -g 'daemon ...

Takes up too much width, difficult to read

Configure docker ps OutputSolution is to use the --format argument

$ docker ps --format \ "table {{.Names}}\\t{{.Image}}\\t{{.Status}}"NAMES IMAGE STATUSweb nginx Up 25 minutes

https://docs.docker.com/engine/reference/commandline/ps/#formatting

Configure docker ps OutputMake it a permanent default by adding it to config.json

$ cat ~/.docker/config.json {... "psFormat": "table {{.ID}}\\t{{.Names}}\\t{{.Image}}\\t{{.Status}}"}

https://docs.docker.com/engine/reference/commandline/cli/#configuration-files

File Mounting GotchaBeware when mounting a single file as a volume$ cat index.htmlMoby Rules!$ docker run -d -p 8000:80 \ -v $PWD/index.html:/usr/share/nginx/html/index.html nginx0cdacef2cbaea960f710d90900b23c57550aaf626ccd2752f3a9287b7e5$ curl localhost:8000Moby Rules!

File Mounting Gotcha$ vi index.html...$ cat index.htmlGordon Rules!$ curl localhost:8000Moby Rules!

?

File Mounting Gotcha Volumes are mounted at the inode levelText editors save to a new inodeSolutions:● mount parent directory

○ -v $PWD:/usr/share/nginx/html● copy modified file

○ cp new.html index.html● overwrite with >

○ echo “bla” > index.htmlThanks Antonis Kalipetis!

Cleaning UpDelete “dangling” images (<none> images)$ docker image pruneWARNING! This will remove all dangling images.Are you sure you want to continue? [y/N] yDeleted Images:deleted: sha256:708624719836212ccb681d5898a64ebfcc4569f37460537609f6db6…Total reclaimed space: 3.677 GB

Cleaning UpDelete stopped containers$ docker container pruneWARNING! This will remove all stopped containers.Are you sure you want to continue? [y/N] yDeleted Containers:6e5033be3e106d04912fb91b966abc693b77ae47d85946190bdbe73c48112959…Total reclaimed space: 304.6 MB

$ docker container pruneWARNING! This will remove all stopped containers.Are you sure you want to continue? [y/N] yDeleted Containers:6e5033be3e106d04912fb91b966abc693b77ae47d85946190bdbe73c48112959…Total reclaimed space: 304.6 MB

Cleaning Up$ docker volume pruneWARNING! This will remove all volumes not used by at least one container.…Total reclaimed space: 3.494 GB$ docker network pruneWARNING! This will remove all networks not used by at least one container.Are you sure you want to continue? [y/N] yDeleted Networks: ...

Cleaning UpAll in one$ docker container pruneWARNING! This will remove all stopped containers.Are you sure you want to continue? [y/N] yDeleted Containers:6e5033be3e106d04912fb91b966abc693b77ae47d85946190bdbe73c48112959…Total reclaimed space: 304.6 MB

$ docker system pruneWARNING! This will remove:

- all stopped containers- all volumes not used by at least one container- all networks not used by at least one container- all dangling images

Building ImagesSidenote: Also see Abby Fuller “Creating Effective Docker Images”

The Build ContextThe . in

$ docker build -t myimage .

● Tarballed and sent to the Docker Daemon● Don’t run a build from ~/ or Downloads!● Use .dockerignore to exclude large directories

Don’t Bust the Build CacheTo keep builds fast, add dependencies before source code in Dockerfiles

...COPY ./ /usr/src/RUN npm install...

...COPY package.json /usr/src/RUN npm installCOPY ./ /usr/src/...

Minimal Images● Good for security

○ Less software that can be exploited● Good for distribution

○ Faster updates, less network costs

Minimal Images● Alpine

○ Only ~ 5MB!○ Uses musl, smaller package manager

● Debian Slim○ Around 55MB

Minimal Images● Scratch and static binariesFROM rust:1.20 as builder…RUN cargo build --release --target x86_64-unknown-linux-musl

FROM scratchCOPY --from=builder /.../release/mybin /mybinUSER 65534CMD ["/mybin"]

Beware of “latest”● Nothing special about tag

○ Not guaranteed to be “new”○ Not guaranteed to exist

Default used when no tag specified docker push/pull/build myimage == docker push/pull/build myimage:latest

Use Meaningful Tags● Semantic versioning

○ docker build -t myimage:1.2.1 .● Git hash

○ docker tag myimage:1.2.1 myimage:$(git rev-parse --short HEAD)

Make it obvious what is running in production!

And Labels for the Rest$ docker build --label org.opencontainers.image.created=\ "$(date --rfc-3339=s)" -t myimage ....$ docker inspect \ -f "{{json .ContainerConfig.Labels}}" myimage{"org.opencontainers.image.created":"2017-10-05 16:21:00+01:00"}

See Annotations in the OCI image spec and Gareth Rushgrove on why standard metadata is important

Container Lifecycle

Start-up Dependably● Do not require containers to start in sequence● If a container depends on another service:

○ It should wait for that service○ Do not crash - back off

● Do this in application code○ or start-up script if you can’t

See 12 Fractured Apps by Kelsey Hightower

Shutdown GracefullyWhen Docker stops a container, it will:

● Send the container a SIGTERM signal● Wait 10s for the container to stop● Hard kill the container with a SIGKILL

Shutdown GracefullyProper handling of SIGTERM will mean:

● The application gets a chance to “tidy up”○ close network connections, sockets, handles○ write data to file or database ○ output to log

● Faster shutdown of the container

Sreenivas Makam on Docker Features for Handling Container Death

Shutdown GracefullyTo ensure your application receives signals either:

● Run it as PID 1○ Use exec in any start-up scripts

● Or forward signals to it○ tini can help https://github.com/krallin/tini

● Prefer node to npm for starting node.js apps○ Bret Fisher Node and Docker Good Defaults

HealthchecksUsed by Docker to determine “health” of containerFROM nginxRUN apt-get update && apt-get install -y curlHEALTHCHECK --interval=10s --timeout=3s \ CMD curl -f http://localhost/ || exit 1

Thanks Laura Frank!

HealthchecksUsed by Docker to determine “health” of containerFROM nginxRUN apt-get update && apt-get install -y curlHEALTHCHECK --interval=10s --timeout=3s \ CMD curl -f http://localhost/ || exit 1

HealthchecksGive better status output$ docker psCONTAINER ID ... STATUS79616fdd4308 Up 3 seconds (health: starting)

$ docker psCONTAINER ID ... STATUS79616fdd4308 Up 16 seconds (healthy)

Healthchecks● Docker Swarm Mode will only route to healthy

containers○ Essential for zero-downtime deploys

● Healthcheck event for integration with other services

Healthchecks● Note command runs in the container

○ Not on the host● Using curl is a good start but

○ Extra dependency○ Can be limiting○ Consider writing bespoke tool

Elton Stoneman on Docker Healthchecks

Thanks Michael Irwin!

SecuritySidenote: Also see pretty much anything by Diogo Monica & Nathan McCauley

Read Only FilesystemAn easy way to improve security$ docker run -d --name n1 --read-only -p 8000:80 \ --tmpfs /var/run --tmpfs /var/cache/nginx nginxc1da395bec73ef7933fecb6d8d821140ce203c426c433e5102d25e46cdb66

$ docker exec n1 /bin/bash -c \ 'echo "HACKED" > /usr/share/nginx/html/index.html'/bin/bash: /usr/share/nginx/html/index.html: Read-only file system

Don’t Run as RootSet a USER in Dockerfiles e.g:

FROM debianRUN groupadd -r mygroup && useradd -r -g mygroup myuser…USER myuser

Or use the nobody user

Don’t Run as RootSometimes need to change user at run-time

sudo works, but has a drawback:

$ docker run debian-with-sudo sudo -u nobody ps ax PID TTY STAT TIME COMMAND 1 ? Rs 0:00 sudo -u nobody ps ax 7 ? R 0:00 ps ax

Don’t Run as RootInstead use gosu by Tianon Gravi

$ docker run debian-with-gosu gosu nobody ps ax PID TTY STAT TIME COMMAND 1 ? Rs 0:00 ps ax

https://github.com/tianon/gosu

Other stuff

Docker-in-Docker● A lot of people want to run Docker in Docker

○ Often for CI/CD● Normally this is a bad ideaTM

○ Issues with filesystems○ Also caching, image stores

Jérôme Petazzoni Do Not Use DinD For CI

Docker-in-DockerInstead, mount the Docker socket:$ docker run \ -v /var/run/docker.sock:/var/run/docker.sock \ docker \ docker psCONTAINER ID IMAGE COMMAND ...8bdba5bc5c7a docker "docker-entrypoint.sh" ...

Docker-in-DockerIf you really need true DinD

$ docker run --privileged --name dind -d docker:dind4b78ae49d77dcf3c2e169c9e4440ace0813676f76e998f0aea2ef065a4b$ docker run --link dind:docker docker docker run -d nginx Unable to find image 'nginx:latest' locallylatest: Pulling from library/nginx...$ docker run --rm --link dind:docker docker docker psCONTAINER ID IMAGE COMMAND ...983cd6cb5a82 nginx "nginx -g 'daemon off" ...

Docker and GUIsYou can run GUI apps in containers!

$ docker run -d \-v /tmp/.X11-unix:/tmp/.X11-unix \-e DISPLAY=unix$DISPLAY \--device /dev/snd:/dev/snd \--name spotify \jess/spotify

Jessie Frazelle Docker Containers on the Desktop

Thanks for listening!

@adrianmouat

ReferencesGood Defaults for Node and Docker - Bret Fisher

12 Fractured Apps - Kelsey Hightower

Least Privilege Containers - Nathan McCauley and Diogo Monica

Gosu - sudo for containers by Tianon Gravi

tini - minimal init system for containers by Thomas Orozco

Docker Containers on the Desktop - Jessie Frazelle

ReferencesDocker Features for Handling Container Death and Resurrection by Sreenivas Makam

Creating Effective Docker Images - Abby Fuller, DockerConEU 2017

Multi-stage builds - Alex Ellis

Do Not Use DinD For CI - Jérôme Petazzoni

Docker Healthchecks - Elton Stoneman

Annotations in the OCI image spec

Thanks to all the captains for discussions!