Location-Aware Distributed Virtual Disk Storage for OpenStack

Keiichi SHIMA <keiichi@iijlab.net> IIJ Innovation Institute Inc.

CloudOpen Europe 2014 2014-10-15

1

Note Well

• This proposal is *NOT* an autonomously distributed storage solution

2

Background

• Widely spread virtualization deployment

• Integrated distributed datacenter

• Flexible resource management

3

Virtualization

• There is no service provider who doesn’t use virtualization

• Many Internet services are now built on top of virtualized computers

• Easy to deploy, scale out when necessary, and scale down when shutting down services

4

Integrated Datacenter

• Datacenters become bigger and bigger

• Aiming to reduce operation cost

• Single point vulnerability

• Approaches to distributed datacenters

• Geographically distributed and widely integrated

5

Resource Management• Resources for virtual infrastructures

• CPU and memory

• Network

• Storage

• We need to locate or relocate these resources flexibly

6

So far

• CPU and memory relocation

• Most of the virtualization platforms support this

• Network relocation

• Software defined networking

• Intensively being researched and developed

7

Storage Resource

• Storage resource relocation

• still not in operation status

8

Current Status• Integrated

• NFS or iSCSI storage devices

• Distributed

• Ceph (RBD)

• GlusterFS

• Sheepdog

9

Current Status

• Integrated solution

• Single point of failure (typically)

• Difficult to relocate resources

10

Current Status

• Distributed approaches

• Difficult to adapt non-uniform environment

• Administration issues (in the sense of a resource location control)

11

Idea is

• To use only good parts of the integrated and distributed solutions

12

Objectives

• Flexibility

• Redundancy

• Locality

13

Flexibility• Flexible management of location

• Place the data at the specified location

• not at somewhere in a storage cluster

• Operation unit

• Per virtual disk

• Per block of a virtual disk

• Flexibility in relocation timing

• Try to prevent to disturb other traffic

14

Redundancy

• Replication of a virtual disk

• Even more than two replicas

• Dynamic operation of replication

15

Locality

• Locate the actual data of a virtual disk as near as possible to the virtual machine using it

16

Operation Scenarios• Consolidation

• Consolidate virtual machines to minimize the # of running hypervisors

• Move

• Push virtual machines out from a hypervisor to upgrade the hypervisor

• Move virtual machines from one DC to another DC, e.g. due to new DC launch, or discontinue

• Efficient resource usage

• Use best performing pair of hypervisors and storage nodes

17

UKAI

• A virtual storage device for virtual machines

• Location control per block

• Replication operation per block

• No global lock required

• Since a virtual storage device is owned by only one virtual machine at the same time

18

UKAI Virtual Disk Concept

VMvDisk

Storage node 1

Storage node 2

Storage node 3

19

Metadata Structure{ "name": "disk01", "size": 200000000, "block_size": 50000000, "blocks": [{ "192.0.2.100": {"synced": true}, "192.0.2.101": {"synced": true} },{ "192.0.2.100": {"synced": true}, "192.0.2.101": {"synced": false} },{ "192.0.2.100": {"synced": true}, "192.0.2.101": {"synced": false} },{ "192.0.2.100": {"synced": false}, "192.0.2.101": {"synced": true} }] }

Virtual disk name

Virtual disk size

Disk block size

Block Locations

Location 1

Location 2

Sync status

20

Metadata Server

• The server nodes keeping virtual data disk structure information

• Apache ZooKeeper is used

21

Typical Node Layout

HV +UKAI Storage

HVHV

UKAI Storage

HV

UKAI Storage

HV + UKAI Storage

HVHV

UKAI Storage

HV

UKAI Storage

R WAN

Other DC

Other DC

Other DC

Metadata Server

22

UKAI Software LayersControl

UKAICore

Local FS

FUSE

KVM

Data

VM

Network

Local FS

Data

UKAI RPC

UKAI RPC

Local call Local call

UKAIFuse

FUSE

UKAIFuse

OpenStack

UKAICore

UKAI RPC

ZooKeeper

Local FS

ZK RPC

ZK RPC

Local call

Meta

UKAI RPC

UKAI Storage

HV + UKAI Storage

Metadata Serv.

23

Supported Disk Admin Ops• Create a disk image / Destroy a disk image

• Size and block size is configurable

• Add location / Delete location

• Per block granularity

• Synchronize

• Per block granularity

• Misc (e.g. getting image metadata info)

24

OpenStack Integration• Compute (Nova)

• A new virtual disk mount adapter module for UKAI is required

• Block storage (Cinder)

• A new storage resource management module for UKAI is required

• Object storage (Swift)

• Out of scope of this proposal

25

Demo

26

Hurdles• Performance

• At this moment, worse than NFS

• Possible reasons: FUSE, RPC, no-caching

• Management facility

• 100% manual operation is not always required

• Need some support tools for node selection based on the predefined hypervisor/storage locations

27

Summary• We need a new storage backend system to support

distributed and integrated virtual infrastructure operation

• UKAI: a location-aware distributed storage system for virtual machines

• Flexible location management, redundancy, and locality are provided

• OpenStack integration is possible

• Need more development efforts to get better performance

28

Availability• UKAI

• https://github.com/keiichishima/ukai

• OpenStack support

• The UKAI branch in https://github.com/keiichishima/nova

• The UKAI branch in https://github.com/keiichishima/cinder

29

Questions?• UKAI

• https://github.com/keiichishima/ukai

• OpenStack support

• The UKAI branch in https://github.com/keiichishima/nova

• The UKAI branch in https://github.com/keiichishima/cinder

30