Transcript
Page 1: Docker vs. Mesos Unified Container

Unified ContainerApr 2016

Guang Ya Liu ([email protected])

Qian Zhang ([email protected])

Yong Feng ([email protected])

IBM Platform Computing

Page 2: Docker vs. Mesos Unified Container

What is a Container

Loosely defined

A lightweight “VM”

To Mesos

A per-task/executor isolated execution

Container History

LXC (2010)

Cgroups (2012)

Linux namespaces (2013)

Docker (2014)

Unified Container (2016)

IBM GCG STG Lab

© 2015 IBM Corporation | IBM Confidential 2

Page 3: Docker vs. Mesos Unified Container

Two Major Containerizers in Mesos

Mesos Containerizer (Default)

Implements unified container.

Provides lightweight containerization and

resource isolation using Linux-specific

functionality such as cgroups and namespaces.

It is composable so operators can selectively

enable different resource isolators.

Docker Containerizer

Docker containerizer delegates container

management to the Docker engine.

Dynamically chosen

Based on ContainerInfo

o ContainerInfo::MESOS

o ContainerInfo::DOCKER

If both are specified, based on the sequence

configured in agent flag of --

containerizers=docke

IBM GCG STG Lab

© 2015 IBM Corporation | IBM Confidential 3

Agent

Mesos Containerizer Docker Containerizer

Docker Daemon

Docker Executor

Isolator

Isolator

Isolator

Isolator

Command/Customer

Executor

Page 4: Docker vs. Mesos Unified Container

Why Mesos Implements Unified Container

Unified container (Mesos Containerizer) will be the long term solution in Mesos for container

management. Docker Containerizer will exist for a long time but sunset eventually.

IBM GCG STG Lab

© 2015 IBM Corporation | IBM Confidential 4

In the middle of 2015, Mesos started to implement its own unified container with Mesos

Containerizer and tried to remove the dependency on Docker daemon, the motivations behind this

initiative are: Docker daemon is not stable and buggy (e.g. crash) in a large scale environment (reported by twitter)

Docker is trying to foster its own ecosystem which is not that open for 3rd part container orchestrators to

integrate with. E.g., Docker relies on CNM/libnetwork for network integration, but the design of libnetwork and its drivers is specific to Docker

which is not easy for integration without depending on Docker daemon. That’s one of the reasons why Mesos and Kubernetes

have chosen CNI for container network integration rather than Docker CNM. CNI follows the UNIX philosophy of doing one

thing well, and it doesn't require daemons, and is cross-platform (i.e., work the same across runtimes).

Docker Containerizer has gaps and hard to resolve.

o Requires Docker installation and maintenance.

o Tasks die with Docker daemon upgrade, etc.

o Cannot compose with Mesos isolators (disk quota, port mapping, volume, CNI).

Mesos is target to support all the major container image specs in a generic way rather than sticking to Docker.o Docker, AppC and OCI.

o Maintain one containerizer for each supported image spec will be hard.

o Introducing a new feature may need to update all containerizers.

Page 5: Docker vs. Mesos Unified Container

Unified Container Architecture

Launcher

Fork the executor process in the

containerized context.

Provisioner

Pull, cache and create rootfs for the

container when launching container

Support AppC and Docker Image format

Universal provisioner for different

container spec

Isolator

Runtime Isolator: Isolate image entry

point, cmd, env, working dir.

CNI Isolator: Manage (add and delete)

network for container with CNI command

line

Docker Volume Isolator: Enable container

to use external storage

IBM GCG STG Lab

© 2015 IBM Corporation | IBM Confidential 5

Page 6: Docker vs. Mesos Unified Container

Docker vs. Mesos Unified Container – Architecture

IBM GCG STG Lab

© 2015 IBM Corporation | IBM Confidential 6

Docker Daemon

Image Store

Create and

access images

Layer Store

Create container

FS via driver

Puller

Pull images from

v1/v2 registry

libcontainerd

Talk to containerd

for container

runtime and

resource isolation

OverlayAufs

. . .VFS

Volume Store

Volume mngt

via driver

FlockerLocal

. . .Convoy

libnetwork

Network mngt

via driver

ipvlanBridge

. . .Overlay

containerdDaemon to control runC

Container Container Container

Mesos Containerizer

Provisioner

LauncherFork executor in the

containerized context

via namespace

DVD Isolator

Container Volume

management via

dvdcli

CNI Isolator

Container network

mngt via CNI

pluginipvlanBridge

. . .Flannel

Container

(Executor)

Container

(Executor)

Container

(Executor)

Image Store

Pull and store Docker/Appc

images

Backend

Create container FS

Copy / bind / overlay

Cgroups IsolatorResource isolation

via Cgroups

Flocker Convoy

Rex-Ray ...

Page 7: Docker vs. Mesos Unified Container

Provisioner Deep Dive

Image Fetch and Store

Vendor specific store which does discover, fetching and processing.

Cache Image for fast provisioning

Force pull image to bypass cache

Provision Backend

Copy

o Small Image less than 1 G

o request IO when copying image

Bind

o Only one layer image, design for large images (multi-GB), request almost

zero IO.

o The rootfs is read-only, need external storage if want to customized some

RW volume.

Overlay

o Support both one layer and multi layer images, request almost zero IO.

o The rootfs is writable.

Roadmap: Smart backend selection based on image size.

IBM GCG STG Lab

© 2015 IBM Corporation | IBM Confidential 7

File System Isolator

Provisioner

Backend Store

Copy

Overlay

Bind

Docker

AppC

OCI

Page 8: Docker vs. Mesos Unified Container

Docker Runtime Isolator

Runtime configuration

Set up environment variables.

Set up working directory.

Set up command to be executed.

If user specifies a command in

CommandInfo, that will override the

default Entrypoint/Cmd in the Docker

image.

Otherwise, the container will use the

default Entrypoint/Cmd and append

arguments specified in CommandInfo

IBM GCG STG Lab

© 2015 IBM Corporation | IBM Confidential 8

Page 9: Docker vs. Mesos Unified Container

CNI (Container Network Interface) is a proposed standard developed by CoreOS for configuring

network interfaces for Linux containers. CNI is adopted by Kubernetes and is bening considered as

an standard by CNCF. Many CNI plugins (e.g., bridge, flannel, calico, etc.) have already been

developed.

CNI isolator invokes CNI plugins to achieve container network management.

Add a container to CNI networks.

Remove a container from CNI networks.

Report the allocated IP addresses of container to framework.

Support both single-host and multi-hosts network.

Containers created with different image specs (e.g., Docker, Appc) in the same CNI network can communicate

smoothly.

Containers created by different container orchestrators (e.g., Mesos, Kubernetes) in the same CNI network can

communicate smoothly.

Cannot work with network/port_mapping isolator as they are in different level layer 3 (IP) Vs layer 4 (TCP/UDP)

IBM GCG STG Lab

© 2015 IBM Corporation | IBM Confidential 9

CNI isolator

Page 10: Docker vs. Mesos Unified Container

IBM GCG STG Lab

© 2015 IBM Corporation | IBM Confidential 10

CNI isolator in Network Management of Mesos Container

• IP Address Management (IPAM) Server

• assigns IPs on demand

• recycles IPs once they have been released

• (optionally) can tag IPs with a given string/id.

• IPAM client

• tightly coupled with a particular IPAM server

• acts as a bridge between the “Network Isolator Module”

and the IPAM server

• communicates with the server to request/release IPs

• Network Isolator Module such as CNI Isolator:

• looks at TaskInfos to detect the IP requirements for the

tasks

• communicates with the IPAM client to request/release

IPs

• communicates with an external network

virtualizer/isolator to enable network isolation

• Cleanup Module:

• responsible for doing a cleanup (releasing IPs, etc.)

during an Agent lost event, dormant otherwise

Page 11: Docker vs. Mesos Unified Container

Docker Volume Isolator

External Storage Integration

Leverage DVD (Docker Volume Driver)

Leverage dvdcli to call DVD API

o Get Volume Path

o Mount Volume

o UnMount Volume

It does not handle the life-cycle of volume

management. User has to call Docker API/CLI

(after Docker 1.9) or DVD API/CLI to create

and manage the life-cycle of volume.

IBM GCG STG Lab

© 2015 IBM Corporation | IBM Confidential 11

Page 12: Docker vs. Mesos Unified Container

Docker vs. Mesos Unified Container - Functionality

IBM GCG STG Lab

© 2015 IBM Corporation | IBM Confidential 12

Provision Image

Mesos: Support both Docker and Appc image specs and will support OCI in future.

Docker: Can only support Docker image spec.

Network Management

Mesos: Depend on CNI which is a generic container network solution and it can support containers created from

any image specs (Docker, Appc, etc.).

Docker: Depend on libnetwork which is specific to Docker container.

Volume Management

Mesos: Depend on dvdcli to leverage Docker Volume Driver Plugin Proto to integrate with 3rd party storage. The

dvdcli will be created as a Mesos C++ library in Mesos roadmap.

Docker daemon: Rely on Docker Volume Driver Plugin to integrate with 3rd party storage.

Page 13: Docker vs. Mesos Unified Container

Docker vs. Mesos Unified Container - Functionality

IBM GCG STG Lab

© 2015 IBM Corporation | IBM Confidential 13

Recovery

Mesos: For the frameworks which enable checkpoint, their container will continue running uninterrupted when

Mesos agent is down, and once Mesos agent is started again, it will take over those containers again.

Docker: All containers will die when Docker daemon is down. Docker may enhance this in the nearly future.

Extensibility

Mesos: Easy to be extended with isolator which can be implemented as a separate module (.so), e.g., if there is

a new type of resource need to be isolated between containers, we can just implement a new isolator for it and no

need to touch any code in Mesos core path.

Docker: Hard to extend and customize isolation for container because there is no way to inject logics during the

lifecycle of container.

Page 14: Docker vs. Mesos Unified Container

Docker vs. Mesos Unified Container - Integration

IBM GCG STG Lab

© 2015 IBM Corporation | IBM Confidential 14

Interface to End User / Framework

Mesos: Provides HTTP API to framework for offer/task management and also provides HTTP endpoints to

operator for operating the Mesos cluster (e.g., set weight/quota for role, perform agent host maintenance, etc.)

Docker: Provides CLI and API to end user for container lifecycle management.

Network Integration

Mesos: Integrate with different network technologies (e.g., bridge, IPVLAN, MACVLAN, etc.) with different CNI

plugins.

Docker: Integrate with different network technologies (e.g., bridge, IPVLAN, MACVLAN, etc.) with different

Docker network drivers.

Storage Integration

Both Docker and Mesos integrate with different storage technologies (e.g., Amazon EBS) with different Docker

volume driver plugins.

Page 15: Docker vs. Mesos Unified Container

Docker vs. Mesos Unified Container - interoperability

IBM GCG STG Lab

© 2015 IBM Corporation | IBM Confidential 15

Network

Mesos: Containers created by Mesos can smoothly communicate with containers created by other container

runtimes as long as they are in the same CNI network.

Docker: It is not easy for Docker container to communicate with containers created by other container runtimes,

there may be some potential issues, e.g., libnetwork has its own way to allocate IP which is not exposed outside,

so it may cause IP conflict if a container created by another runtime wants to join a Docker network.

There is a POC work to make Docker container communicate with container created by Kubernetes, but it

is kind of hacky since it relies on a special label “cni.network” when executing “docker run” so that the

Docker container will be attached to the specified CNI network.

Storage

Since both Mesos and Docker rely on Docker volume driver plugin to do storage integration, it is possible that a

container created by Mesos mounts and uses a volume created by Docker.

Page 16: Docker vs. Mesos Unified Container

Agent Configuration

--isolation

docker/runtime: enable docker runtime isolator

network/cni: enable network cni isolator

docker/dvd: enable docker volume isolator

--image_providers

Enable container image support in Mesos containerizer.

Support docker and appc currently

--appc_simple_discovery_uri_prefix or --

docker_registry

URI prefix to be used for simple discovery of appc images or

Docker images

--appc_store_dir or --docker_store_dir

Directory the appc or Docker provisioner will store images in.

IBM GCG STG Lab

© 2015 IBM Corporation | IBM Confidential 16

Example

$ sudo mesos-slave \

--containerizers=mesos \

--image_providers=appc,docker \

--isolation=filesystem/linux,docker/runtime

Page 17: Docker vs. Mesos Unified Container

Future Work

AppC Enhancement

Need support runtime configuration like docker runtime isolator

Support meta image discovery

o Only simple image discovery

o appc_simple_discovery_uri_prefix specify image prefix

o Search Image via appc_simple_discovery_uri_prefix/{name}-{version}-{os}-{arch}.{ext}

OCI Support

Need support OCI Spec, however OCI Spec in not finalized yet.

Security

Currently launches all tasks without any control on the privilege levels under which they will be run. This makes

the execution environment insecure.

Need enforcing control on the available privileges on the launched task

Mesos CLI

Need some functions like “docker ps”, “docker inspect” to simplify some query operations

IBM GCG STG Lab

© 2015 IBM Corporation | IBM Confidential 17

Page 18: Docker vs. Mesos Unified Container

Thank You


Top Related