managing container clusters in openstack native way

Managing Container Clusters in OpenStack Native Way

OTSUKA, Motohiro/YuanyingNEC Solution Innovators

OpenStack Magnum Core Reviewer

Haiwei XuNEC Solution InnovatorsOpenStack Senlin Core Reviewer

Qiming TengIBM, Research Scientist

OpenStack Senlin PTL, OpenStack Heat Core Reviewer

Agenda• Why containers if you already have OpenStack

• What are the use cases?• The many roads leading to Roma

• Container as first-class citizens on OpenStack• Deployment and management• Technology Gaps

• Experience Sharing and Outlook• What we can do today• Things to expect in Newton cycle

Why Containers, If You Already Have OpenStack

Photographer: Captain Albert E. Theberge, NOAA Corps (ret.)from http://www.photolib.noaa.gov/coastline/line3174.htm

Containers

http://www.photolib.noaa.gov/coastline/line3174.htm

Nova

X-ray: NASA/CXC/RIKEN/D.Takei et al; Optical: NASA/STScI; Radio: NRAO/VLAfrom http://www.nasa.gov/sites/default/files/thumbnails/image/gkper.jpg

Advantages of container technology

Server

Host OS

Hypervisor

Guest OS

libs / bins

Application

Guest OS

libs / bins

Application

Server

Host OS

libs / bins

Application

libs / bins

Application

Virtual Machine Container

Container Image

libs / bins

Application

Container Image

libs / bins

Application


Server A

Host OS

Container Image

libs / bins

Application

Container Image

Server B

Host OS

libs / bins

Application

DockerfileDocker Registry

Development Production

Version managiment

Major Use Cases• For application/service users

• IF self-serviced THEN deploy/launch; simple configurationENDIF

• go...• For application developers

• Develop, Commit, Test• Build, Deploy, Push• Pull, Patch, Push

• For cloud deployers/operators• Build Infrastructure• Install, Configure, Upgrade• Monitor, Fix, Bill,

All Roads Lead to Roma• How many roads do we have?

• nova lxc• nova docker• heat docker• heat deployment• magnum bay

• docker swarm• kubernetes• mesos

• marathon, ...• openstack ansible• kolla

• kolla-mesos• .....

Nova: Docker / LXC

Ironic

BareMetal

BareMetal

VM VM VM

VM VM VM

Docker/LXC

???

Virtualization Bare metal Container

VM VM VM

VM VM VM

VM VM VM

VM VM VM

libvirt VMware Xen

Nova

driv

er

nova-docker virt driver

LXC (libvirt) driver

Heat: Docker / Deployment

Heat

tcp://$FLOATING_IP:2375

DockerInc::Docker::Container

Heat

OS::Heat::StructuredConfiggroup: docker-compose

heat-config-docker-

compose

unix:///var/run/docker.sock

Magnum: Kubernetes / Swarm• Container Orchestration Engine (COE) as a Service

TenantBay (Kubernetes Type)

KubernetesMaster

KubernetesMinion 1

KubernetesMinion 2

Service

Pod PodReplicationController Pod

Container Container Container Container Container Container

Bay (Swarm Type)

SwarmMaster Docker Node 2

Container ContainerContainer Container




Docker Node 1





Magnum

Kubernetes TypeBaymodel

image: fedora-atomickeypair: my_public_key

external-network: $NIC_IDdns-nameserver: 8.8.8.8

flavor: baremetaltype-of-bay: kubernetes

Swarm TypeBaymodel

image: fedora-atomickeypair: my_public_key

external-network: $NIC_IDdns-nameserver: 8.8.8.8

flavor: baremetaltype-of-bay: swarm

Kolla: Ansible + Container = OpenStack• OpenStack as a Service

nova-api

neutron-server

cinder-api

neutron-agent nova-compute cinder-volume

Then the next question is: How?• Unified API?

• Refer to mailinglist discussion• http://lists.openstack.org/pipermail/openstack-dev/2016-April/091947.html

• Unified Abstraction?• OpenStack Container API ?

• https://blueprints.launchpad.net/magnum/+spec/unified-containers• Kubernetes Driver• Docker Swarm Driver

http://lists.openstack.org/pipermail/openstack-dev/2016-April/091947.html

http://lists.openstack.org/pipermail/openstack-dev/2016-April/091947.html

https://blueprints.launchpad.net/magnum/+spec/unified-containers

https://blueprints.launchpad.net/magnum/+spec/unified-containers

Containers as First-Class Citizens on OpenStack

Balancing across the abstraction layer• Container as another compute API?

• maybe pm, vm, lwVM• so many backends

• An abstraction over all existing container management software?

• it is possible, but many questions to be answered, e.g. why?• do you really need to switch between these software frequently?• are you willing to develop a client software to interact with all of them?

• So ... container clustering• better integration with OpenStack• ease of use

OpenStack Clustering Service

Scalable

Load-Balanced

Highly-Available

Manageable

......

of any (OpenStack) objects

- What is missing from OpenStack?A Clustering Service

- Auto-scaling?Just one of the usage scenario of a cluster.

- Auto-Healing (HA)?Just another usage scenario.

- We can address the concerns by making policies orthogonal

Senlin Architecture

Senlin Engine

Senlin API

Senlin Database

Senlin Client

REST RPC

Profiles

Policies

Senlin Features• Profiles: A specification for the objects to be managed• Policies: Rules to be checked/enforced before/after actions are performed

21

(others)

Senlin

Nova

Docker

Heat

Ironic BareMetal

VMs

Stacks

Containersplacement

deletion

scaling

health

load-balance

affinity

Policies as Plugins Profiles as Plugins Cluster/Nodes Managed

Senlin Server Architecture

openstacksdkidentity

compute

orchestration

network

...

engineengine lock

scheduler

actions

nodecluster

service

registry

receiverparser

drivers

openstack

dummy

(others)

dbapi

sqlalchemy

rpc clientpolicies

placement

deletionscaling

health

load-balance

affinity

receiverwebhoookMsgQueue

extension points for external monitoring services

extension points facilitating a smarter cluster management

extension points to talk to different endpoints for object CRUD operations

extension points for interfacing with different services or clouds

profilesos.heat.stack

(others)

os.nova.server

senlin-apiWSGI

middleware

apiv1

Senlin Server Architecture (for containers)engineengine lock

scheduler

actions

nodecluster

service

registry

receiverparser

drivers

docker-py

dummy

lxc

dbapi

sqlalchemy

rpc clientpolicies

placement

deletionscaling

health

load-balance

affinity

receiverwebhoookMsgQueue

extension points for external monitoring services

extension points facilitating a smarter cluster management

extension points to talk to different endpoints for object CRUD operations

extension points for interfacing with different services or clouds

profilescontainer.docker

(others)

container.lxc

senlin-apiWSGI

middleware

apiv1

Targeted Use Cases: Auto-Scaling

VM1 VM2 VM3

C1-1 C1-2 C1-3C2-1 C2-2 C3-1 C3-2

VM4

C3-3

VM Cluster

Node Node Node Node

Container Cluster

Node Node Node

Container Cluster

Node Node

Container Cluster

Node Node Node

Senlin

Targeted Use Cases: Auto-Healing

VM1 VM2 VM3

C1-1 C1-2 C1-3C2-1 C2-2 C3-1 C3-2

VM4

C3-3

VM Cluster

Node Node Node Node

Container Cluster

Node Node Node

Container Cluster

Node Node

Container Cluster

Node Node Node

C3-2 C1-3

Senlin

Targeted Use Cases: Controller Plane

VM/Baremetal VM2 / Baremetal

key keyn-api n-api g-api

Container Cluster

Node Node

Container Cluster

Node Node

Monitoring(Consul,

Sensu, ... )

Senlin

webhook /message

What can we do today & How to do it?

Container type profile

type: container.dockerversion: 1.0properties: name: container1 image: hello-world command: ‘/bin/sleep 30’ networks: - network: container-network …… …

Nova server type profile spec

type: os.nova.serverversion: 1.0properties: name: cirros_server image: “cirros-0.3.4-x86_64-uec” flavor: m1.small key_name: oskey networks: - network: private-network

Container node and container cluster

node

Heat stack

Nova server

Profile type

Container

Nova server

Heat stack

Container

Nova server

Template for Heat Heat stack

container

cluster

Template for container

Template for Nova Nova serverNova server

Heat stackHeat stack

containercontainer

How to create a container cluster?

container profile

cluster1

vm server

vm server

vm server

cluster1

containervm

vmcontainer

container

vm

cluster2

container

container

container

container

container

The scalability of vm cluster and container cluster

cluster1

containervm

containervm

cluster2

container

container

user

Placement policyDeletion policyScaling policy

Placement policyDeletion policyScaling policy

vm

container container

Outlook

Container Backends• lxc, lxd, docker, rocket, runC, ...• docker

• Docker-py

Container scheduling• Senlin placement policy

• Start container on specified some nodes• Make the policy more intelligent

• Dynamic rescheduling?

Networking support• Kuryr• others?

Storage support• Kuryr• Rexray• flocker

Welcome to join us!• IRC: #senlin• Weekly meeting from UTC 13:00~14:00, Tuesday

Thank you