Docker securitySOFTWARE FREEDOM DAY, 2019, SZEGED

Janos SUTO, sj@acts.hu

Isn't docker secure?

Ars Technica: Infected images mined digital coins

"17 images posted by a single account over 10 months may have generated $90,000."

"For ordinary users, just pulling a Docker image from Docker Hub is like pulling arbitrary binary data from somewhere, executing it, and hoping for the best without really knowing what’s in it.”

https://arstechnica.com/information-technology/2018/06/backdoored-images-downloaded-5-million-times-finally-removed-from-docker-hub/

CVE-2019-5736

"runc through 1.0-rc6, as used in Docker before 18.09.2 and other products, allows attackers to overwrite the host runc binary (and consequently obtain host root access) by leveraging the ability to execute a command as root within one of these types of containers: (1) a new container with an attacker-controlled image, or (2) an existing container, to which the attacker previously had write access, that can be attached with docker exec. This occurs because of file-descriptor mishandling, related to /proc/self/exe.

Important stuff I won't talk this time

Physical security

Host security (patched OS, only necessary packages, OS hardening, ...)

Network security (open ports, firewalls, strict SSH access, …)

Educating users

...

Securing images

Official images

Essential base OS repositories as the starting point for users

Lead examples of Dockerfile best practices

Security updates are applied in a timely manner

Scanned for vulnerabilities

https://docs.docker.com/docker-hub/official_images/

Docker Content Trust (DCT)

Use digital signatures for data sent to and received from remote Docker registries.

These signatures allow client-side or runtime verification of the integrity and publisher of specific image tags.

Through DCT, image publishers can sign their images and image consumers can ensure that the images they use are signed.

DCT #2

export DOCKER_CONTENT_TRUST="1"

docker pull user/someimage

Error: remote trust data does not exist for docker.io/user/someimage: notary.docker.io does not have trust data for docker.io/user/someimage

https://docs.docker.com/engine/security/trust/content_trust/

Runtime Enforcement with DCT

Applies to Docker Enterprise only

Personal Access Tokens for Docker HUB

Use your own registry

Docker registry

Harbor

Quay (automatic security scanning)

JFrog Artifactory (PRO edition)

Don't use insecure registries

By default it's not enabled

Build your own images

Start from official images

Use a reasonable distro (eg. alpine)

Include only what's really required (eg. --no-install-recommends)

USER someuser

No sudo

No sshd

Don't bake any secrets to the image

ENV MYSQL_PASSWORD "aaaa"

Scan your images

Microscanner: https://github.com/aquasecurity/microscanner

FROM debian:jessie-slim

RUN apt-get update && apt-get -y install ca-certificates

ADD https://get.aquasec.com/microscanner /

ARG token

RUN chmod +x /microscanner && /microscanner ${token}

"vulnerabilities": [

{

"name": "CVE-2017-8398",

"description": "dwarf.c in GNU Binutils 2.28 is vulnerable to an invalid read of size 1 during dumping of debug information from a corrupt binary …".

"nvd_score": 5,

"nvd_score_version": "CVSS v2",

"nvd_vectors": "AV:N/AC:L/Au:N/C:N/I:N/A:P",

"nvd_severity": "medium",

"nvd_url": https://web.nvd.nist.gov/view/vuln/detail?vulnId=CVE-2017-8398,

…..

},

Other image scanner products

Clair

Docker Trusted Registry

JFrog Xray

...

Virus scanning

$ docker create --name erlang_scan erlang # Create container from image

$ docker export –output "live_system.tar" erlang_scan # Push the container fs to tar file

$ clamscan live_system.tar

live_system.tar: OK

----------- SCAN SUMMARY -----------Known viruses: 6590083Engine version: 0.100.1Scanned directories: 0Scanned files: 1Infected files: 0Data scanned: 0.00 MBData read: 1029.54 MB (ratio 0.00:1)Time: 9.586 sec (0 m 9 s)

$ docker rm erlang_scan

https://medium.com/@cwgem/thoughts-about-docker-security-8e0df4b43650

Docker bench security

Checking for best practices:

1. Host configuration

2. Docker daemon configuration

3. Docker daemon configuration files

4. Container Images and Build File

5. Container Runtime

6. Docker Security Operations

7. Docker Swarm Configuration

https://github.com/docker/docker-bench-security

[INFO] 2 - Docker daemon configuration

[PASS] 2.1 - Ensure network traffic is restricted between containers on the default bridge

[PASS] 2.2 - Ensure the logging level is set to 'info'

[PASS] 2.3 - Ensure Docker is allowed to make changes to iptables

[PASS] 2.4 - Ensure insecure registries are not used

[PASS] 2.5 - Ensure aufs storage driver is not used

[INFO] 2.6 - Ensure TLS authentication for Docker daemon is configured

[INFO] * Docker daemon not listening on TCP

[INFO] 2.7 - Ensure the default ulimit is configured appropriately

[INFO] * Default ulimit doesn't appear to be set

Securing the daemon

Protect the socket

srw-rw---- 1 root docker 0 Sep 10 21:04 /var/run/docker.sock=

Don't put just anyone to the docker group

Accessing docker over the network

TLS encryption (don't enable port 2375)

Certificate authentication

Firewall the docker host

DOCKER_HOST=tcp://docker.yourdomain.com:2376

DOCKER_TLS_VERIFY=1

https://docs.docker.com/engine/security/https/

https://docs.docker.com/engine/security/certificates/

Securing containers

Resource limits

--memory 2G: The maximum amount of memory the container can use

--memory-swap 2G: The amount of memory the container is allowed to swap to disk

--shm-size 64M: Size of /dev/shm

--cpus=1.5 how much of the available CPU resources a container can use.

--cpuset-cpus=0,1,2: Limit the specific CPUs or cores a container can use

--gpus device=0,2: nvidia gpu access

--pids-limit: Limit number of processes started inside docker container

--ulimit <options>, eg. --ulimit nproc=256:512

https://docs.docker.com/config/containers/resource_constraints/

Stopping a fork bomb

$ docker run --rm --name aaa --pids-limit 30 ubuntu \

bash -c ":() { : | : & }; :; while [[ true ]]; do sleep 1; done"

environment: fork: retry: Resource temporarily unavailable

environment: fork: retry: Resource temporarily unavailable

...

bash: fork: retry: Resource temporarily unavailable

environment: fork: retry: Resource temporarily unavailable

environment: fork: retry: Resource temporarily unavailable

environment: fork: Resource temporarily unavailable

bash: fork: Interrupted system call

Stopping a fork bomb #2

$ docker stats aaa

NAME CPU % MEM USAGE / LIMIT MEM % PIDS

aaa 0.14% 9.172MiB / 23.41GiB 0.04% 30

Make the root fs read-only

$ docker run --rm -ti --read-only ubuntu bash

root@4f8d760aa70b:/# touch /tmp/iii

touch: cannot touch '/tmp/iii': Read-only file system

root@4f8d760aa70b:/#

Use tmpfs to allow write access

$ docker run --rm -ti --read-only --tmpfs /tmp ubuntu bash

root@e28b09f46878:/# touch /tmp/akaka

root@e28b09f46878:/#

Remove all capabilities ...

$ docker run --rm --cap-drop=ALL nginx

2019/09/22 09:37:45 [emerg] 1#1: chown("/var/cache/nginx/client_temp", 101) failed (1: Operation not permitted)

nginx: [emerg] chown("/var/cache/nginx/client_temp", 101) failed (1: Operation not permitted)

man 7 capabilities

… add only what's required

$ docker run --rm --cap-add=chown --cap-drop=ALL nginx

2019/09/22 09:39:28 [emerg] 1#1: bind() to 0.0.0.0:80 failed (13: Permission denied)

nginx: [emerg] bind() to 0.0.0.0:80 failed (13: Permission denied)

… add only what's required #2

$ docker run --rm --cap-add=chown --cap-add=net_bind_service --cap-drop=ALL nginx

2019/09/22 09:43:22 [emerg] 6#6: setgid(101) failed (1: Operation not permitted)

2019/09/22 09:43:22 [alert] 1#1: worker process 6 exited with fatal code 2 and cannot be respawned

… add only what's required #3

$ docker run --rm --cap-add=chown --cap-add=net_bind_service --cap-add=setgid --cap-drop=ALL nginx

2019/09/22 09:43:54 [emerg] 6#6: setuid(101) failed (1: Operation not permitted)

2019/09/22 09:43:54 [alert] 1#1: worker process 6 exited with fatal code 2 and cannot be respawned

… add only what's required #4

$ docker run --rm \

--cap-add=chown \

--cap-add=net_bind_service \

--cap-add=setgid \

--cap-add=setuid \

--cap-drop=ALL \

nginx

User remapping

$ docker run --rm -ti -v /etc:/etc ubuntu bash

root@6ac62e5eb40c:/# touch /etc/hello-world

root@6ac62e5eb40c:/# exit

$ ls -la /etc/hello-world

-rw-r--r-- 1 root root 0 Sep 22 11:53 /etc/hello-world

User remapping #2

$ whoami

john

$ id -u

1000

$id -g

100

User remapping #3

/etc/docker/daemon.json:

{

"userns-remap": "john"

}

/etc/subuid:

john:1000:65536

/etc/subgid:

john:100:65536

User remapping #4

$ docker run --rm -ti -v /etc:/etc ubuntu bash

root@deb50f4847e6:/# touch /etc/hello-world2

touch: cannot touch '/etc/hello-world2': Permission denied

User remapping #5

$ docker run --rm -ti -v /tmp:/tmp ubuntu bash

root@7b66cc086eb4:/# touch /tmp/aaa

root@7b66cc086eb4:/# ls -la /tmp/aaa

-rw-r--r-- 1 root root 0 Sep 22 10:13 /tmp/aaa

root@7b66cc086eb4:/# exit

$ ls -la /tmp/aaa

-rw-r--r-- 1 john users 0 Sep 22 12:13 /tmp/aaa

https://ilya-bystrov.github.io/posts/docker-daemon-remapping/

Don't use privileged mode

"Privileged mode enables access to all devices on the host as well as set some configuration in AppArmor or SELinux to allow the container nearly all the same access to the host as processes running outside containers on the host."

Don't use the host's namespaces

$ docker run --userns=host -ti --rm -v /tmp:/tmpubuntu bash

root@a78119823836:/# touch /tmp/hahaha

root@a78119823836:/# ls -la /tmp/hahaha

-rw-r--r-- 1 root root 0 Oct 3 10:08 /tmp/hahaha

root@a78119823836:/# exit

$ ls -la /tmp/hahaha

-rw-r--r-- 1 root root 0 Oct 3 12:08 /tmp/hahaha

Authorization plugin

dockerd --authorization-plugin=someplugin

Could run locally on a Unix domain socket, or anywhere on http(s)

Authorization plugin #2

https://docs.docker.com/engine/extend/plugins_authorization/

PoC implementation: https://pastebin.com/SFUWdP08

Secrets in container

$ docker run --rm –ti –e SOME_PASSWORD=aaaa ubuntu bash

root@7b66cc086eb4:/#echo $SOME_PASSWORD

aaaa

root@7b66cc086eb4:/#

Environments are often logged!

#less /proc/29487/task/29487/environ

PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin^@HOSTNAME=7b66cc086eb4^@TERM=xterm^@SOME_PASSWORD=aaaa^@HOME=/root

Secrets in config files in container

docker run –v /path/to/1.cfg:/etc/yourapp/1.cfg:ro yourimage

Docker secrets

Container orchestration systems offer some basic secret management

Kubernetes: secrets, configmaps (Namespaces, RBAC)

Docker Swarm: secrets

Not for a standalone docker installation :-(

Setup a single node swarm or k8s deployment :-)

https://www.hashicorp.com/resources/securing-container-secrets-vault

Elevating privileges

FROM ubuntu:latest

RUN apt-get update && \

apt-get install –y sudo && \

echo "sj ALL=(ALL) NOPASSWD: ALL" > /etc/sudoers.d/sj && \

echo "sj:x:1000:100::/home/sj:/bin/bash" >> /etc/passwd

USER 1000

Elevating privileges #2

$ docker run --rm -ti aaa bash

sj@177cd44c70c0:/$ id

uid=1000(sj) gid=100(users) groups=100(users)

sj@177cd44c70c0:/$ sudo bash

root@177cd44c70c0:/# id

uid=0(root) gid=0(root) groups=0(root)

Elevating privileges #3

/etc/docker/daemon.json:

{

"no-new-privileges": true

}

Elevating privileges #4

$ docker run --rm -ti aaa bash

sj@177cd44c70c0:/$ id

uid=1000(sj) gid=100(users) groups=100(users)

sj@177cd44c70c0:/$ sudo bash

sudo: effective uid is not 0, is /usr/bin/sudo on a file system with the 'nosuid' option set or an NFS file system without root privileges?

More daemon.json settings

{

"icc": false, // Disable inter container communication

"userland-proxy": false, // Disable userland proxy for loopback traffic

….

}

Activity monitoring with sysdig/falco

Notify other systems or humans of abnormal behavior.

https://sysdig.com/opensource/falco/

Activity monitoring with sysdig/falco #2

***Action change_thread_namespace

Calling setns() to change namespaces...

***Action create_files_below_dev

Creating /dev/created-by-event-generator-sh...

***Action db_program_spawn_process

Becoming the program "mysql" and then running ls

***Action exec_ls

bin dev etc …

***Action exfiltration

Reading /etc/shadow and sending to 10.5.2.6:8197...

Activity monitoring with sysdig/falco #3

2019-10-03T13:17:21.968443650+0000: Notice Namespace change (setns) by unexpected program (user=root command=event_generatorparent=<NA> <NA> (id=2f5a7b42362a) container_id=2f5a7b42362a image=<NA>)

2019-10-03T13:17:22.968679872+0000: Error File created below /dev by untrusted program (user=root command=event_generatorfile=/dev/created-by-event-generator-sh container_id=2f5a7b42362a image=sysdig/falco-event-generator)

2019-10-03T13:17:23.971571824+0000: Notice Database-related program spawned process other than itself (user=root program=ls parent=mysqldcontainer_id=2f5a7b42362a image=sysdig/falco-event-generator)

2019-10-03T13:17:24.972983032+0000: Warning Sensitive file opened for reading by non-trusted program (user=root program=event_generatorcommand=event_generator file=/etc/shadow parent=<NA> gparent=<NA> ggparent=<NA> gggparent=<NA> container_id=2f5a7b42362a image=sysdig/falco-event-generator)

Host encryption

Don't use fscrypt: no support for namespaces

Use LUKS

Final words

Apply what makes sense in your environment

At the end of the day security must not kill productivity