Modern Storage Platform for Container Environments

Julien Quintard Member of Technical Staff

Docker Inc. (http://infinit.sh)

1. Platform

2. Internals

3. Demo

the

CONTEXT

The uniqueness of containers compared to traditional virtual machines

comes mainly from the fact that they are lightweight, fast and scalable, at

least when it comes to stateless applications.

Containers also bring new concepts such as portability while opening ways

to new techniques such as micro-services in which an application is split

into several containers.

the

PERSISTENCE

Docker has gained in popularity mainly through the use of stateless applications. Developers and enterprises now require statefulness.

However, given their uniqueness, containers need more than just traditional storage but a platform allowing stateful applications to be

deployed and scaled as easily in spite of failures, be it the host or

container itself.

More precisely, a modern storage platform should be both elastic (scalable

and fault tolerant) and customizable.

1. Platform

the

OVERVIEW

the

KEY-VALUE STOREInfinit’s key layer is its key-value store whose role is to aggregate storage capacity from a set of nodes. Unlike most key-value store available on the

market, Infinit’s benefits from the following:

Consensus

Infinit’s consensus algorithm is block-based and allows for a strong

consistency model on top of which others (eventual) can be emulated.

Policies

The key-value store can be configured through a set of policies from

encryption (e.g key size), block size, compression, deduplication, data

placement, redundancy, versioning and more.

On top of the key-value store, a number of interfaces is provided, each with

its tradeoffs from usage, performance etc.

Decentralization

While most distributed systems rely on a leader/follower model, Infinit is completely decentralized allowing for better scalability and fault tolerance.

the

INTERFACES

On top of the key-value store, a number of interfaces is provided, each with

its tradeoffs from usage, performance etc.

File System

The file system interface was the first one introduced in the Infinit storage

platform. It allows for the creation of hierarchical volumes of files and folders. For now only FUSE (Linux/macOS) and Dokan (Windows) are

publicly available.

POSIX-compliant file system interfaces allows for concurrent reader/writer accesses from multiple client nodes/containers, the drawback being that

such interfaces are often quite complex, hence impacting performance.

Examples range from storing logs, configuration files, credentials or just

running a legacy application.

Object Storage

An object storage HTTP-based API allows for storing unstructured data

through binary large objects (blob). The most well known of such APIs is

Amazon S3 which has become the de-factor standard.

Such an interface is more adapted than file system when one is looking for

a simple way to store objects which are often independent from one

another.

Examples range from videos, images etc. in the context of a website for

instance.

Block Device

The most common network-based block storage interfaces are iSCSI and

FCoE. Those provide raw access to a networked partition which can be

formatted in the file system of your choice: ext4, NTFS etc.

Block devices are interesting because of their inherent limitation. Such

devices can only be attached to a single node. As such, concurrent accesses are limited, only possible on the same node, going through the

same kernel handling the block storage.

This limitation implies that the underlying logic can be optimized to remove

the need for concurrent access mechanisms, to store a database for

instance.

the

EXAMPLE

database

block (iSCSI) 512-byte block size

replication

file (NFS) replication versioning

deduplication

object (AWS S3) 1-megabyte block size

datacenter aware erasure coding

compression

ext4

images/videos

confi

gura

tion

files

content

Below is an example of a micro-service application relying on a policy-

based storage platform providing multiple interfaces: Wordpress.

the

DEPLOYMENTSHyperscale

Deploy your storage infrastructure for it

to evolve and scale independently of your cluster of applications (Swarm,

Kubernetes etc.).

#powerful #operators

Hyperconverge

Your storage infrastructure scales along with your cluster of applications.

#simple #developers

2. Internals

the

DISTRIBUTIONInfinit is completely decentralized (peer-to-peer) meaning that any

operation could be carried out by any other node. In other words, every

node plays part of the role of the metadata server. This symmetry property

is critical to ensure the validation of operations a posteriori.

Infinit’s key-value store relies on two basic constructs to benefit from high

scalability and fault tolerance.

Overlay Network (not related to Docker’s overlay)

The overlay network provides a routing mechanism that allows a client to locate the nodes

responsible for a piece of data given an identifier. Noteworthy is that there exist different types of overlay network, each with its own advantages.

Distributed Hash Table

The distributed hash table relies on the overlay network and ensures the redundancy (replication/erasure coding), consistency (Paxos), self-healing

and mutability of the stored items in order to guarantee their availability and durability.

Example: The Chord overlay network can scale to millions of nodes.

Example: In the Kouncil overlay network, every node has complete knowledge of the nodes and blocks.

the

FLOW

We’ve introduced the key-value store which is implemented through an

overlay network and distributed hash table on top of which several

interfaces are provided.

Let’s take an example to understand how all those layers interconnect

through a system call, create(“/foo”), in this case.

the

CONSENSUS

Most distributed system rely on a set of managers. Those special nodes

organize the network and authorize some critical operations.

When such an operation must be performed, the quorum of manager nodes

is contacted to reach an agreement and maintain consistency.

The problem with this design is that the manager nodes can become

bottlenecks, potentially leading to a cascading effect, not to mention an

ideal target for privilege escalation.

Infinit on the contrary, operates in a decentralized manner, meaning that

no central authority is involved.

Unlike traditional distributed systems that rely on a global lock, Infinit

relies on per-block quorums. As such, the complexity of the consensus

algorithm is related to the redundancy factor only, allowing for better

scalability, security and fault tolerance.

Distributed System Decentralized System

the

BLOCKSThe key-value store allows for storing blocks of variable size. There are

many types of blocks the construct allows one to store, from immutable and mutable blocks to more specific ones such as named blocks that link a

string to an address.

Mutable Block

Such blocks evolve over time based on a version

number. As such they are subject to conflicts. Also,

clients need to invalidate such blocks in their cache to make sure the latest version is refetched.

Finally, mutable blocks rely on heavier cryptographic

mechanisms to ensure integrity and security.

Immutable Block

Immutable blocks, on the other hand, can only exist in one version. The address of such a block is

computed by applying a one-way function on the

block’s content (content addressed). Should the content be modified, the address would change as

well, leading to a new block.

Immutable blocks therefore cannot conflict, can be cached forever and are easy to validate by

reapplying the one-way function on the retrieved

content.

For instance, in the context of a file system, metadata a.k.a inode are

stored in a mutable block while actual data blocks are immutable, all

stored through the key-value store.

As such, a vast majority of the blocks are immutable, leading to better performance because those are cheap to write, fetchable from any source

and cachable ad vitam æternam.

3. Demo

the

CONCLUSION

This demo is really all about the key basics. From that point you can

generate passports for other users to connect to your storage

infrastructure, you can control accesses on files and folder, scale out with

more storage capacity from the cloud and also, obviously, access an Infinit

volume in your Docker containers through Infinit’s Docker Volume Plugin.

Infinit is still at a very early stage of its development. In the future, we will

introduce “infinit cluster” to make it easier to deploy a storage

infrastructure (like Docker Swarm through —init/—join), provide more interfaces (block/object) and open source the several layers I talked about

today.

Website infinit.sh

Twitter @infinit

Github github.com/infinit

Reddit /r/infinit