techday - cambridge 2016 - opennebula corona
TRANSCRIPT
CORONA Dan Kelleher | Scientific Systems Administrator | Harvard FAS Research Computing
Overview • Node Types • Docker Containers
– Base, Libvirt, nfs, ONE server, ONE hypervisor, Sunstone – Docker network, privilege and PID – Systemd vs Supervisord – Compose
• Challenges – Configuration – Oneadmin token and SSH keys
Motivation • Binary roll out and roll back
– Simple deployment – Automated testing – Upgrades
• Elasticity? – Deploy a new node using cluster scheduler – Tear it down when it’s not longer needed
• Kolla – https://github.com/openstack/kolla
Node Types Controller
Hybrid
ONE Server • Host Net • Host PID
Hybrid Hybrid
Hypervisor NFS • Host Net • Host PID • Privileged
Sunstone • Host Net • Host PID • Privileged
Libvirt • Host Net • Host PID • Privileged • Cgroups • /dev • /run • /lib/
modules
NFS • Host Net • Host PID • Privileged
ONE Hyper • Host Net • Host PID • Privileged • Cgroups • /dev • /run
ONE Server • Host Net • Host PID
NFS • Host Net • Host PID • Privileged
Libvirt • Host Net • Host PID • Privileged • Cgroups • /dev • /run • /lib/
modules
ONE Hyper • Host Net • Host PID • Privileged • Cgroups • /dev • /run
Sunstone • Host Net • Host PID • Privileged
Overview • Node Types • Docker Containers
– Base, Libvirt, nfs, ONE server, ONE hypervisor, Sunstone – Docker network, privilege and PID – Systemd vs Supervisord – Compose
• Challenges – Configuration – Oneadmin token and SSH keys
Corona-Base
Libvirt The Libvirt container is straight forward build but:
– In order to setup the virtualization limitations the libvirt container needs access to cgroups
– It also needs access to the device directory and kernel modules directory to control the hardware
Libvirt is also a self contained daemon which makes process management easier
Libvirt
NFS
NFS supervisor
NFS • NFS on the other hand is easy to build and doesn’t
require access to a lot of resources but is complex to configure
• NFS requires 5 intertwined daemons which each need to be configured
• In addition a reload condition needs to be defined • This is one of the cases where systemd would have
excelled. (More on that in a bit)
ONE Server
ONE Hypervisor
ONE • By comparison the OpenNebula software was both easy
to build and simple to configure in supervisord • The challenge in the ONE software lies in configuring the
software it self without jumping into the container.
Sunstone
This container needs a lot more work
CORONA Un-Contained • Host PID is mostly for convenience of management • Host Networking was the best solution to create the
bridges for the trunked vlans • Privilege is required by anything which needs to bind to
the host networking stack in the protected port range – Some of this could likely be reduced using docker port binding – Libvirt needs to be able to inject kernel modules – ONE hypervisor needs to create the bridges
MySQL • There’s a MySQL container straight from docker hub
– https://hub.docker.com/_/mysql/
• Also there is a MariaDB official container on docker hub – https://hub.docker.com/_/mariadb/
• In the future docker should allow the controller container to pick up the database name, password, and username if they are on the same host
How to Supervise
Systemd • Pros:
– Dependencies are handled – Reloads are simple – Configured by the
packages
• Cons: – Running multiple instances
proved challenging – Require a lot of packages
Supervisord • Pros:
– Running multiple instances is simple
– Doesn’t run as root
• Cons: – Has to be hand configured
Docker Compose
Overview • Node Types • Docker Containers
– Base, Libvirt, nfs, ONE server, ONE hypervisor, Sunstone – Docker network, privilege and PID – Systemd vs Supervisord – Compose
• Challenges – Configuration – Oneadmin token and SSH keys
Configuration • The Docker standard is to use environment variable
– At the moment we are using flat files mounted as read only docker volumes.
• A Script needs to be created to parse environment variables and put them in the correct config files – Building this script has been challenging – A lot of configuration variables add to complexity
Token and SSH • In our current setup these are laid down by puppet • Mounting these out in production is not very secure • Hypervisors can’t auto check-in without these
– This is critical for elasticity
Demo?
Questions?