docker vs. mesos unified container

Click here to load reader

Post on 13-Feb-2017

224 views

Category:

Documents

1 download

Embed Size (px)

TRANSCRIPT

  • Unified ContainerApr 2016

    Guang Ya Liu ([email protected])

    Qian Zhang ([email protected])

    Yong Feng ([email protected])

    IBM Platform Computing

    mailto:[email protected])mailto:[email protected]:[email protected]

  • What is a Container

    Loosely defined

    A lightweight VM

    To Mesos

    A per-task/executor isolated execution

    Container History

    LXC (2010)

    Cgroups (2012)

    Linux namespaces (2013)

    Docker (2014)

    Unified Container (2016)

    IBM GCG STG Lab

    2015 IBM Corporation | IBM Confidential 2

  • Two Major Containerizers in Mesos

    Mesos Containerizer (Default)

    Implements unified container.

    Provides lightweight containerization and

    resource isolation using Linux-specific

    functionality such as cgroups and namespaces.

    It is composable so operators can selectively

    enable different resource isolators.

    Docker Containerizer

    Docker containerizer delegates container

    management to the Docker engine.

    Dynamically chosen

    Based on ContainerInfo

    o ContainerInfo::MESOS

    o ContainerInfo::DOCKER

    If both are specified, based on the sequence

    configured in agent flag of --

    containerizers=docke

    IBM GCG STG Lab

    2015 IBM Corporation | IBM Confidential 3

    Agent

    Mesos Containerizer Docker Containerizer

    Docker Daemon

    Docker Executor

    Isolator

    Isolator

    Isolator

    Isolator

    Command/Customer

    Executor

  • Why Mesos Implements Unified Container

    Unified container (Mesos Containerizer) will be the long term solution in Mesos for container

    management. Docker Containerizer will exist for a long time but sunset eventually.

    IBM GCG STG Lab

    2015 IBM Corporation | IBM Confidential 4

    In the middle of 2015, Mesos started to implement its own unified container with Mesos

    Containerizer and tried to remove the dependency on Docker daemon, the motivations behind this

    initiative are: Docker daemon is not stable and buggy (e.g. crash) in a large scale environment (reported by twitter)

    Docker is trying to foster its own ecosystem which is not that open for 3rd part container orchestrators to

    integrate with. E.g., Docker relies on CNM/libnetwork for network integration, but the design of libnetwork and its drivers is specific to Docker

    which is not easy for integration without depending on Docker daemon. Thats one of the reasons why Mesos and Kubernetes

    have chosen CNI for container network integration rather than Docker CNM. CNI follows the UNIX philosophy of doing one

    thing well, and it doesn't require daemons, and is cross-platform (i.e., work the same across runtimes).

    Docker Containerizer has gaps and hard to resolve.

    o Requires Docker installation and maintenance.

    o Tasks die with Docker daemon upgrade, etc.

    o Cannot compose with Mesos isolators (disk quota, port mapping, volume, CNI).

    Mesos is target to support all the major container image specs in a generic way rather than sticking to Docker.o Docker, AppC and OCI.

    o Maintain one containerizer for each supported image spec will be hard.

    o Introducing a new feature may need to update all containerizers.

  • Unified Container Architecture

    Launcher

    Fork the executor process in the

    containerized context.

    Provisioner

    Pull, cache and create rootfs for the

    container when launching container

    Support AppC and Docker Image format

    Universal provisioner for different

    container spec

    Isolator

    Runtime Isolator: Isolate image entry

    point, cmd, env, working dir.

    CNI Isolator: Manage (add and delete)

    network for container with CNI command

    line

    Docker Volume Isolator: Enable container

    to use external storage

    IBM GCG STG Lab

    2015 IBM Corporation | IBM Confidential 5

  • Docker vs. Mesos Unified Container Architecture

    IBM GCG STG Lab

    2015 IBM Corporation | IBM Confidential 6

    Docker Daemon

    Image Store

    Create and

    access images

    Layer Store

    Create container

    FS via driver

    Puller

    Pull images from

    v1/v2 registry

    libcontainerd

    Talk to containerd

    for container

    runtime and

    resource isolation

    OverlayAufs

    . . .VFS

    Volume Store

    Volume mngt

    via driver

    FlockerLocal

    . . .Convoy

    libnetwork

    Network mngt

    via driver

    ipvlanBridge

    . . .Overlay

    containerdDaemon to control runC

    Container Container Container

    Mesos Containerizer

    Provisioner

    LauncherFork executor in the

    containerized context

    via namespace

    DVD Isolator

    Container Volume

    management via

    dvdcli

    CNI Isolator

    Container network

    mngt via CNI

    pluginipvlanBridge

    . . .Flannel

    Container

    (Executor)

    Container

    (Executor)

    Container

    (Executor)

    Image Store

    Pull and store Docker/Appc

    images

    Backend

    Create container FS

    Copy / bind / overlay

    Cgroups IsolatorResource isolation

    via Cgroups

    Flocker Convoy

    Rex-Ray ...

  • Provisioner Deep Dive

    Image Fetch and Store

    Vendor specific store which does discover, fetching and processing.

    Cache Image for fast provisioning

    Force pull image to bypass cache

    Provision Backend

    Copy

    o Small Image less than 1 G

    o request IO when copying image

    Bind

    o Only one layer image, design for large images (multi-GB), request almost

    zero IO.

    o The rootfs is read-only, need external storage if want to customized some

    RW volume.

    Overlay

    o Support both one layer and multi layer images, request almost zero IO.

    o The rootfs is writable.

    Roadmap: Smart backend selection based on image size.

    IBM GCG STG Lab

    2015 IBM Corporation | IBM Confidential 7

    File System Isolator

    Provisioner

    Backend Store

    Copy

    Overlay

    Bind

    Docker

    AppC

    OCI

  • Docker Runtime Isolator

    Runtime configuration

    Set up environment variables.

    Set up working directory.

    Set up command to be executed.

    If user specifies a command in

    CommandInfo, that will override the

    default Entrypoint/Cmd in the Docker

    image.

    Otherwise, the container will use the

    default Entrypoint/Cmd and append

    arguments specified in CommandInfo

    IBM GCG STG Lab

    2015 IBM Corporation | IBM Confidential 8

  • CNI (Container Network Interface) is a proposed standard developed by CoreOS for configuring

    network interfaces for Linux containers. CNI is adopted by Kubernetes and is bening considered as

    an standard by CNCF. Many CNI plugins (e.g., bridge, flannel, calico, etc.) have already been

    developed.

    CNI isolator invokes CNI plugins to achieve container network management.

    Add a container to CNI networks.

    Remove a container from CNI networks.

    Report the allocated IP addresses of container to framework.

    Support both single-host and multi-hosts network.

    Containers created with different image specs (e.g., Docker, Appc) in the same CNI network can communicate

    smoothly.

    Containers created by different container orchestrators (e.g., Mesos, Kubernetes) in the same CNI network can

    communicate smoothly.

    Cannot work with network/port_mapping isolator as they are in different level layer 3 (IP) Vs layer 4 (TCP/UDP)

    IBM GCG STG Lab

    2015 IBM Corporation | IBM Confidential 9

    CNI isolator

  • IBM GCG STG Lab

    2015 IBM Corporation | IBM Confidential 10

    CNI isolator in Network Management of Mesos Container

    IP Address Management (IPAM) Server

    assigns IPs on demand

    recycles IPs once they have been released

    (optionally) can tag IPs with a given string/id.

    IPAM client

    tightly coupled with a particular IPAM server

    acts as a bridge between the Network Isolator Module

    and the IPAM server

    communicates with the server to request/release IPs

    Network Isolator Module such as CNI Isolator:

    looks at TaskInfos to detect the IP requirements for the

    tasks

    communicates with the IPAM client to request/release

    IPs

    communicates with an external network

    virtualizer/isolator to enable network isolation

    Cleanup Module:

    responsible for doing a cleanup (releasing IPs, etc.)

    during an Agent lost event, dormant otherwise

  • Docker Volume Isolator

    External Storage Integration

    Leverage DVD (Docker Volume Driver)

    Leverage dvdcli to call DVD API

    o Get Volume Path

    o Mount Volume

    o UnMount Volume

    It does not handle the life-cycle of volume

    management. User has to call Docker API/CLI

    (after Docker 1.9) or DVD API/CLI to create

    and manage the life-cycle of volume.

    IBM GCG STG Lab

    2015 IBM Corporation | IBM Confidential 11

  • Docker vs. Mesos Unified Container - Functionality

    IBM GCG STG Lab

    2015 IBM Corporation | IBM Confidential 12

    Provision Image

    Mesos: Support both Docker and Appc image specs and will support OCI in future.

    Docker: Can only support Docker image spec.

    Network Management

    Mesos: Depend on CNI which is a generic container network solution and it can support containers created from

    any image specs (Do