docker for hpc in a nutshell
TRANSCRIPT
![Page 1: Docker for HPC in a Nutshell](https://reader034.vdocument.in/reader034/viewer/2022042516/55ce2563bb61ebe4068b45a1/html5/thumbnails/1.jpg)
© 2013 Cassini Consulting
Andreas Schmidt | @aschmidt75
Docker: User Friendly Application & Service Containers for HPC Environments Docker In A Nutshell
![Page 2: Docker for HPC in a Nutshell](https://reader034.vdocument.in/reader034/viewer/2022042516/55ce2563bb61ebe4068b45a1/html5/thumbnails/2.jpg)
2 16.07.15 Cassini Consulting
![Page 3: Docker for HPC in a Nutshell](https://reader034.vdocument.in/reader034/viewer/2022042516/55ce2563bb61ebe4068b45a1/html5/thumbnails/3.jpg)
3 16.07.15 Cassini Consulting
![Page 4: Docker for HPC in a Nutshell](https://reader034.vdocument.in/reader034/viewer/2022042516/55ce2563bb61ebe4068b45a1/html5/thumbnails/4.jpg)
4 16.07.15 Cassini Consulting
vs. Pets Cattle
![Page 5: Docker for HPC in a Nutshell](https://reader034.vdocument.in/reader034/viewer/2022042516/55ce2563bb61ebe4068b45a1/html5/thumbnails/5.jpg)
5 16.07.15 Cassini Consulting
![Page 6: Docker for HPC in a Nutshell](https://reader034.vdocument.in/reader034/viewer/2022042516/55ce2563bb61ebe4068b45a1/html5/thumbnails/6.jpg)
16.07.15 Cassini Consulting 6
Technical Perspective
Developer Perspective
Security
Ecosystem & Partners
![Page 7: Docker for HPC in a Nutshell](https://reader034.vdocument.in/reader034/viewer/2022042516/55ce2563bb61ebe4068b45a1/html5/thumbnails/7.jpg)
16.07.15 Cassini Consulting 7
Technical Perspective
Developer Perspective
Security
Ecosystem & Partners
![Page 8: Docker for HPC in a Nutshell](https://reader034.vdocument.in/reader034/viewer/2022042516/55ce2563bb61ebe4068b45a1/html5/thumbnails/8.jpg)
Docker Daemon
Docker Engine core components
8 16.07.15 Cassini Consulting
Containers
Images
Containers
![Page 9: Docker for HPC in a Nutshell](https://reader034.vdocument.in/reader034/viewer/2022042516/55ce2563bb61ebe4068b45a1/html5/thumbnails/9.jpg)
Docker Daemon
Docker Engine core components
9 16.07.15 Cassini Consulting
Containers
Images
Containers
API
Docker Client
Docker Client
![Page 10: Docker for HPC in a Nutshell](https://reader034.vdocument.in/reader034/viewer/2022042516/55ce2563bb61ebe4068b45a1/html5/thumbnails/10.jpg)
The Docker workflow & commands
10 16.07.15 Cassini Consulting
Image
![Page 11: Docker for HPC in a Nutshell](https://reader034.vdocument.in/reader034/viewer/2022042516/55ce2563bb61ebe4068b45a1/html5/thumbnails/11.jpg)
The Docker workflow & commands
11 16.07.15 Cassini Consulting
Registry
Image
pull
![Page 12: Docker for HPC in a Nutshell](https://reader034.vdocument.in/reader034/viewer/2022042516/55ce2563bb61ebe4068b45a1/html5/thumbnails/12.jpg)
The Docker workflow & commands
12 16.07.15 Cassini Consulting
Registry
Image
Docker- file
pull
build
![Page 13: Docker for HPC in a Nutshell](https://reader034.vdocument.in/reader034/viewer/2022042516/55ce2563bb61ebe4068b45a1/html5/thumbnails/13.jpg)
The Docker workflow & commands
13 16.07.15 Cassini Consulting
Registry
Image
Docker- file
push pull
build
![Page 14: Docker for HPC in a Nutshell](https://reader034.vdocument.in/reader034/viewer/2022042516/55ce2563bb61ebe4068b45a1/html5/thumbnails/14.jpg)
The Docker workflow & commands
14 16.07.15 Cassini Consulting
Registry
Image
Container
Docker- file
push pull
run
build
![Page 15: Docker for HPC in a Nutshell](https://reader034.vdocument.in/reader034/viewer/2022042516/55ce2563bb61ebe4068b45a1/html5/thumbnails/15.jpg)
The Docker workflow & commands
15 16.07.15 Cassini Consulting
Registry
Image
Container
Docker- file
push pull
run
commit
build
![Page 16: Docker for HPC in a Nutshell](https://reader034.vdocument.in/reader034/viewer/2022042516/55ce2563bb61ebe4068b45a1/html5/thumbnails/16.jpg)
The Docker workflow & commands
16 16.07.15 Cassini Consulting
Registry
Image
Container
Docker- file
push pull
run
commit
start, stop, ...
build
![Page 17: Docker for HPC in a Nutshell](https://reader034.vdocument.in/reader034/viewer/2022042516/55ce2563bb61ebe4068b45a1/html5/thumbnails/17.jpg)
The Docker workflow & commands
17 16.07.15 Cassini Consulting
Registry
Image
Container
Docker- file
✖
push pull
run
commit
rm
start, stop, ...
build
![Page 18: Docker for HPC in a Nutshell](https://reader034.vdocument.in/reader034/viewer/2022042516/55ce2563bb61ebe4068b45a1/html5/thumbnails/18.jpg)
The Docker workflow & commands
18 16.07.15 Cassini Consulting
Registry
Image
Container
Docker- file
✖ ✖
push pull
run
commit
rm rmi
start, stop, ...
build
![Page 19: Docker for HPC in a Nutshell](https://reader034.vdocument.in/reader034/viewer/2022042516/55ce2563bb61ebe4068b45a1/html5/thumbnails/19.jpg)
The Docker workflow & commands
19 16.07.15 Cassini Consulting
Registry
Image
Container
Docker- file
External Tarball
✖ ✖
push pull
run
commit
rm rmi
start, stop, ...
build
export
![Page 20: Docker for HPC in a Nutshell](https://reader034.vdocument.in/reader034/viewer/2022042516/55ce2563bb61ebe4068b45a1/html5/thumbnails/20.jpg)
Docker Engine technology foundation
Isolation through Kernel Namespaces Linux Capabilities Resource limitation through Linux control groups Filesystem isolation, Copy-On-Write & Union FS
20 16.07.15 Cassini Consulting
![Page 21: Docker for HPC in a Nutshell](https://reader034.vdocument.in/reader034/viewer/2022042516/55ce2563bb61ebe4068b45a1/html5/thumbnails/21.jpg)
Docker Engine technology foundation
21 16.07.15 (*) https://www.docker.com/whatisdocker
![Page 22: Docker for HPC in a Nutshell](https://reader034.vdocument.in/reader034/viewer/2022042516/55ce2563bb61ebe4068b45a1/html5/thumbnails/22.jpg)
Virtual Machines vs. Containers
22 16.07.15 (*) https://www.docker.com/whatisdocker
![Page 23: Docker for HPC in a Nutshell](https://reader034.vdocument.in/reader034/viewer/2022042516/55ce2563bb61ebe4068b45a1/html5/thumbnails/23.jpg)
16.07.15 Cassini Consulting 23
Technical Perspective
Developer Perspective
Security
Ecosystem & Partners
![Page 24: Docker for HPC in a Nutshell](https://reader034.vdocument.in/reader034/viewer/2022042516/55ce2563bb61ebe4068b45a1/html5/thumbnails/24.jpg)
From Development to Production: Challenges ahead!
24 16.07.15 Cassini Consulting
![Page 25: Docker for HPC in a Nutshell](https://reader034.vdocument.in/reader034/viewer/2022042516/55ce2563bb61ebe4068b45a1/html5/thumbnails/25.jpg)
What Docker gives to developers
25 16.07.15 Cassini Consulting
Easy sandbox approach
Easy way to deliver software
Dev / Prod parity
![Page 26: Docker for HPC in a Nutshell](https://reader034.vdocument.in/reader034/viewer/2022042516/55ce2563bb61ebe4068b45a1/html5/thumbnails/26.jpg)
Easy Sandbox approach
§ Create build and test environments
§ Choose Libraries and Framework Dependencies, per application
§ Deal with incompatible version mismatches (Ruby 1.9 vs. Ruby 2.1, including libs, bundler, version switchers, ...)
§ Lightweight alternative to using virtual machines
26 16.07.15 Cassini Consulting
![Page 27: Docker for HPC in a Nutshell](https://reader034.vdocument.in/reader034/viewer/2022042516/55ce2563bb61ebe4068b45a1/html5/thumbnails/27.jpg)
Easy Sandbox approach
A Docker Image contains its own userland libraries and binaries
§ separated from other images
§ take exactly the versions of libraries you need
§ leave out things you do not need
§ reproducible, lightweight, easily testable
§ look at it as a unit of delivery
27 16.07.15 Cassini Consulting
![Page 28: Docker for HPC in a Nutshell](https://reader034.vdocument.in/reader034/viewer/2022042516/55ce2563bb61ebe4068b45a1/html5/thumbnails/28.jpg)
Easy way to deliver software
28 16.07.15 Cassini Consulting
Code Deliverables (i.e. RPM packages) are not runnable.
An installed instance (i.e. in a VM) is hard to transport.
![Page 29: Docker for HPC in a Nutshell](https://reader034.vdocument.in/reader034/viewer/2022042516/55ce2563bb61ebe4068b45a1/html5/thumbnails/29.jpg)
29 16.07.15 Cassini Consulting
https://twitter.com/joyent/status/565243828718678016
![Page 30: Docker for HPC in a Nutshell](https://reader034.vdocument.in/reader034/viewer/2022042516/55ce2563bb61ebe4068b45a1/html5/thumbnails/30.jpg)
Easy way to deliver software
Dockerfile as a Contract between Development and Operations
30 16.07.15 Cassini Consulting
what to base from
set environment params
prepare the image, i.e. install something, configure it
describe the interface
what to run
![Page 31: Docker for HPC in a Nutshell](https://reader034.vdocument.in/reader034/viewer/2022042516/55ce2563bb61ebe4068b45a1/html5/thumbnails/31.jpg)
Easy way to deliver software
Dockerfile as the blueprint for reusable building blocks
31 16.07.15 Cassini Consulting
what to base from
Redis is an official repository at dockerhub
![Page 32: Docker for HPC in a Nutshell](https://reader034.vdocument.in/reader034/viewer/2022042516/55ce2563bb61ebe4068b45a1/html5/thumbnails/32.jpg)
Easy way to deliver software
Dockerfile as the blueprint for reusable building blocks
32 16.07.15 Cassini Consulting
what to base from
Redis is an official repository at dockerhub
![Page 33: Docker for HPC in a Nutshell](https://reader034.vdocument.in/reader034/viewer/2022042516/55ce2563bb61ebe4068b45a1/html5/thumbnails/33.jpg)
Pave the way for Dev/Prod parity
33 16.07.15 Cassini Consulting
§ From Applications to Systems
§ Describe not only compute, but also storage and networking.
§ Example: docker-compose
![Page 34: Docker for HPC in a Nutshell](https://reader034.vdocument.in/reader034/viewer/2022042516/55ce2563bb61ebe4068b45a1/html5/thumbnails/34.jpg)
16.07.15 Cassini Consulting 34
Technical Perspective
Developer Perspective
Security
Ecosystem & Partners
![Page 35: Docker for HPC in a Nutshell](https://reader034.vdocument.in/reader034/viewer/2022042516/55ce2563bb61ebe4068b45a1/html5/thumbnails/35.jpg)
35 16.07.15 Cassini Consulting
Is Docker secure?
![Page 36: Docker for HPC in a Nutshell](https://reader034.vdocument.in/reader034/viewer/2022042516/55ce2563bb61ebe4068b45a1/html5/thumbnails/36.jpg)
Let's ask this question more specific.
Is the docker daemon secure? Are images transported securely? Are images built in a secure fashion? Are containers as secure as virtual machines? Are application processes more or less secure when containerized?
36 16.07.15 Cassini Consulting
![Page 37: Docker for HPC in a Nutshell](https://reader034.vdocument.in/reader034/viewer/2022042516/55ce2563bb61ebe4068b45a1/html5/thumbnails/37.jpg)
Are containers as secure as virtual machines?
Control group-separated, chroot-like, namespaced resources, running on a shared kernel.
37 16.07.15 Cassini Consulting
Application Containers Virtual Machines
Virtualized pieces of hardware, running its own kernel with process/user/network spaces separated on hypervisor level.
![Page 38: Docker for HPC in a Nutshell](https://reader034.vdocument.in/reader034/viewer/2022042516/55ce2563bb61ebe4068b45a1/html5/thumbnails/38.jpg)
Are application processes more or less secure when containerized?
Definitely more secure, if "used properly(*)" Docker Container ~ Application process, ideally a single process, without management daemons Smaller attack surface Namespaced process, network, FS mounts, ...: ~ Application cannot see "the outside OS world" Reduced Linux Capabilities, can be fine tuned Additional isolation mechanisms at hand: SELinux Type Enforcement, AppArmor profiles, Libseccomp System Call Filtering
38 16.07.15 Cassini Consulting (*) http://container-solutions.com/is-docker-safe-for-production
![Page 39: Docker for HPC in a Nutshell](https://reader034.vdocument.in/reader034/viewer/2022042516/55ce2563bb61ebe4068b45a1/html5/thumbnails/39.jpg)
Docker Hardening – Docker Security Benchmark
39 16.07.15 Cassini Consulting
§ Extensive Guide on hardening docker hosts, images and containers, including checks
§ Automated tools are in development
§ benchmarks.cisecurity.org
![Page 40: Docker for HPC in a Nutshell](https://reader034.vdocument.in/reader034/viewer/2022042516/55ce2563bb61ebe4068b45a1/html5/thumbnails/40.jpg)
16.07.15 Cassini Consulting 40
Technical Perspective
Developer Perspective
Security
Ecosystem & Partners
![Page 41: Docker for HPC in a Nutshell](https://reader034.vdocument.in/reader034/viewer/2022042516/55ce2563bb61ebe4068b45a1/html5/thumbnails/41.jpg)
Tooling around Docker
41 16.07.15 Cassini Consulting
Where to run Specialised Operating systems
Where to pull images from Registries (Private, On Premise, ...)
How to operate it Orchestration, Scheduling, Management, Monitoring
From Infrastructure to Applications
Platform-As-a-Service
How to build containers Config Management, Developer Tools
Technical topics Networking, Security, Storage
![Page 42: Docker for HPC in a Nutshell](https://reader034.vdocument.in/reader034/viewer/2022042516/55ce2563bb61ebe4068b45a1/html5/thumbnails/42.jpg)
42 16.07.15 Cassini Consulting
![Page 43: Docker for HPC in a Nutshell](https://reader034.vdocument.in/reader034/viewer/2022042516/55ce2563bb61ebe4068b45a1/html5/thumbnails/43.jpg)
Competitors & Container Runtime Alternatives
LXC + LXD
CoreOS Rocket (rkt), Application Container Spec (appc)
Open Container Project (www.opencontainers.org)
43 16.07.15 Cassini Consulting
![Page 44: Docker for HPC in a Nutshell](https://reader034.vdocument.in/reader034/viewer/2022042516/55ce2563bb61ebe4068b45a1/html5/thumbnails/44.jpg)
16.07.15 Cassini Consulting 44
Technical Perspective
![Page 45: Docker for HPC in a Nutshell](https://reader034.vdocument.in/reader034/viewer/2022042516/55ce2563bb61ebe4068b45a1/html5/thumbnails/45.jpg)
16.07.15 Cassini Consulting 45
Technical Perspective Developer Perspective
![Page 46: Docker for HPC in a Nutshell](https://reader034.vdocument.in/reader034/viewer/2022042516/55ce2563bb61ebe4068b45a1/html5/thumbnails/46.jpg)
16.07.15 Cassini Consulting 46
Technical Perspective Developer Perspective Security
![Page 47: Docker for HPC in a Nutshell](https://reader034.vdocument.in/reader034/viewer/2022042516/55ce2563bb61ebe4068b45a1/html5/thumbnails/47.jpg)
16.07.15 Cassini Consulting 47
Technical Perspective Developer Perspective Security Ecosystem & Partners
![Page 48: Docker for HPC in a Nutshell](https://reader034.vdocument.in/reader034/viewer/2022042516/55ce2563bb61ebe4068b45a1/html5/thumbnails/48.jpg)
16.07.15 Cassini Consulting 48
Technical Perspective Developer Perspective Security Ecosystem & Competitors Thank you!