efficient image management using cinder volumes for virtual and

© Hitachi Data Systems Corporation 2015. All rights reserved.

Efficient Image Management

using Cinder Volumes

for Virtual and Baremetal Machines

Tomoki Sekiyama

Mitsuhiro Tanino

1


Background

New features for Efficient Image Handling

Use case of Volume-backed Images

Usage of Volume-backed Images

Current Limitations

Future Work

2

Contents


Background

3


(3) Storage

virtualization

and automatic data

optimization using

thin provision and

tiering

Volume

Requirement for block storage solution

4

VM

(4) Taking backup and

snapshot instantly via

storage feature

(2) Provide stable

IOPS and low

latency using boot

from volume

Volume

(1) Create a boot volume

using COW snapshot

SATA

SAS

Flash

Stable Performance

Virtualization/ Optimization

Business Continuity

Agility of booting an instance


HDD

Problems (1/3)

Agility is an important factor for OpenStack clouds

• Required to rapidly boot instances

When the image size is large, “nova boot” takes a long time

to download the image from Glance

• Nova caches images (per host), but not effective to first boot

• Cinder-volume-boot always downloads the image by default

5

Download Image Cached

Image

Guest

Image

VM

Volume VM

Image

Compute

Node

Storage

Array


Booting baremetal nodes requires to copy the image to local disk

• Download & copy takes a long time

• Causes high network traffic

• Ironic caches the downloaded images

same as Nova but image copy to

attached disk is always required.

Problems (2/3)

6

HDD

1. Boot with

deploy image

2. Export the disk

as iSCSI target

Image

3. Download & Copy Image

4. Reboot

Attach

Baremetal

Node


Image copy may affect disk I/O performance of instances

• Example: sysbench OLTP benchmark during volume creation

– Measured on a KVM instance with a volume (LVM-iSCSI backend)

– During the image copy to a volume, I/O performance degrades

– The interference can be mitigated by image copy I/O bandwidth limit, but

image copy takes longer time

Volume creation from a snapshot is rapid, but snapshots cannot be

shared among tenants

• Not suitable for public images (e.g. operating systems)

Problems (3/3)

7


Target Environment

Nova: VM, Baremetal (Ironic)

Cinder + Storage array

Cinder

Glance

Nova

Ironic

VM

* Currently baremetal does not

support Cinder volumes

8

Baremetal nodes

VM

VM VM

VM VM

VM VM

Storage Array

Compute Nodes

Control Node

SAN (FC/iSCSI)

etc.


New features for Efficient Image Handling

9


Cinder New features for Image Handling In Liberty release, some features are added to Cinder for efficient

image management

• Image-Volume Cache

– Cache recently used images as “image volumes”

• Volume-backed Image

– Store an Glance image data in a Cinder volume

In both features, a new bootable volume can be created rapidly by

cloning the image volume

Both features are disabled by default

10


Internal tenant

Overview of Image-Volume Cache

Images recently used by Cinder are cached as Cinder volumes

• Each image is stored in a volume (Image-Volume)

• Image volumes are placed in the internal tenant

Automatic management

• If cache volumes exceed the specific amount size, recently

unused cache volumes are deleted

Support various disk formats (converted to raw before cached)

Can coexist with the volume-backed Image feature

11

Download

(First time only)

Image Vol.

(Cache)

Image

Volume VM

Clone Boot Storage

Array


Overview of Volume-backed Image

Register an image volume as a Glance image

• Utilize Glance’s Cinder store

– The image data is stored in a Cinder volume (Image-Volume)

• No image data transfer between Cinder and Glance

New volume can be created rapidly by cloning the image volume

12

Image

Volume

Volume-

backed

Image

Volume VM

Clone Boot

location = cinder://1234-abc..

Storage

Array


Use case of Volume-backed Images

13


Rapid Boot for Virtual Machine Instances

Rapidly boot an instance with a new volume from an image

• Reduce time to launch instances

• Reduce I/O workload

Significant on booting multiple instances

Example:

• Booting an instance from a new Cinder volume

created from 20 GB operating system image

– Measured with thin-provisioning LVM backend

14

0 50 100 150 200 250

default

Volume-backed

Setup instance Volume creation Volume attach Spawning

[s]

246s

11s


Boot Volume Boot Volume

Rapid Boot for Baremetal Instances (TBD) Currently baremetal boot requires image copy to local HDD drive

Ongoing work to support volume boot for Ironic

Combined with volume-backed images, deploy baremetal nodes

without copying image data to local HDD

15

Image Volume

Boot Volume

iSCSI / FC

1. Clone

2. Attach & Boot

Baremetal Nodes


Copy-offload of Image Data

Leverage the storage array’s copy-offloading features

• Some storages support copy-on-write based cloning

– No data transfer

– No interference to instances’ performance

Example: sysbench OLTP benchmark during volume creation

• When thin-provisioning is supported, this feature also improve

storage capacity

16


Admin

tenant

Sharing volume data among tenants

Sharing volume data among tenants

• The visibility of the Volume-backed image is managed by

Glance’s ACL feature

– Public image volume

– Sharing among specific members

• Useful to share base images such as operating system images,

master datasets, etc.

17

Base OS

image

Tenant A Tenant B

Guest 2 Guest 4

Guest 1 Guest 3

Public

Volume-

backed

Image

*The image must be

registered by volume owner


Usage of Volume-backed Images

18


Registration of Volume-backed Image (1/2) Using Glance CLI (with Image API v2)

• glance image-create --disk-format raw \ --container-format bare --name <image-name>

• glance location-add <image-uuid> \ --url cinder://<volume-uuid>

• NOTE: the registered volume data shouldn’t be modified

+------------------+-------------------------------------------------------------+ | Property | Value | +------------------+-------------------------------------------------------------+ | checksum | None | | container_format | bare | | created_at | 2015-09-22T21:31:34Z | | disk_format | raw | | id | f698173d-b96f-43be-aae9-fa0d13751c09 | | locations | [{"url": "cinder://95e571f9-5ccd-45eb-b282-441c3ce9a5db", | | | "metadata": {}}] | … | size | 1073741824 | | visibility | private | +------------------+-------------------------------------------------------------+

19


Registration of Volume-backed Image (2/2) “cinder upload-to-image” command can be optionally configured

to create a volume-backed image

• cinder upload-to-image <volume> <new-image-name>

• The specified volume is cloned to create a new image volume

• The cloned volume is set to read-only

• The cloned volume’s URL is registered to a new Glance image

• To enable this behavior, the following options is required:

– image_upload_use_cinder_backend = True

• Only raw format is supported

20

Image

Volume

New

Image

Volume

Clone

Register

location

Storage

Array


New Volume from Volume-backed Image

To create a new volume from a volume-backed Image:

• cinder create --image <image-uuid> <size>

• The new volume is cloned from the image volume

• The volume is extended when the volume size is larger than the

image volume

In Horizon, new instance

can be launched from a new

volume cloned from the

volume-backed image

21


Enable Volume-backed Image Features

Glance settings (/etc/glance/glance-api.conf)

• Enable Cinder store

– [glance_store] :: stores = file,http,swift,cinder

• Expose image locations (URL)

– [DEFAULT] :: show_multiple_locations = True

Cinder settings (/etc/cinder/cinder.conf)

• Enable Glance API version 2

– [DEFAULT] :: glance_api_version = 2

• Enable volume creation by cloning image volumes

– [DEFAULT] :: allowed_direct_url_schemes = cinder

• (optional) To use “cinder upload-to-image” to create volume-backed image

– Backend section :: image_upload_use_cinder_backend = True

22


Enable Image-Volume Cache

Image-Volume Cache and Volume-backed image feature can

coexist

Cinder settings (/etc/cinder/cinder.conf)

• Enable internal tenant

– [DEFAULT] :: cinder_internal_tenant_project_id = ...

– [DEFAULT] :: cinder_internal_tenant_user_id = ...

• Enable Image-Volume Cache

– Backend section :: image_volume_cache_enabled = True

• (optional) Limit max capacity / number of cache volumes

– Backend section :: image_volume_cache_max_size_gb = ...

and/or

– Backend section :: image_volume_cache_max_count = ...

23


Current Limitations

24


Current Limitation (1/2)

Cannot clone volumes between multiple hosts / backends

• Need to create an image volume on each host / backend

• and register their locations to the volume-backed image

Volume-backed images must be in raw format

• Image-Volume Cache supports various disk formats

– Automatically converts to raw type

25


Current Limitation (2/2) Volume-backed images can only be used to create new volumes

• Nova and other components cannot access volume-backed image

contents

– Currently Glance’s Cinder store functionality is quite limited

– We are proposing a patch to enable Glance download from /

upload to volume-backed images

Image volumes created by “upload-to-image” are visible

• The volume owner may destroy the image volumes

• The image volumes can be hidden from users by storing them in

the internal tenant

– “upload_image_use_internal_tenant = True” in cinder.conf

– Require the Glance patch

26


Current Situation of Glance Cinder-store Lacking important features (considered almost “broken”)

• Only provides a pointer to an existing volume

• Only can be used to create a new volume

• Glance cannot access to Image Volume contents

→ Clients cannot upload & download images

• Doesn’t delete the image volume when the image is deleted

• Cannot specify user/tenant to access Cinder

– Always use current context

– The image volume is visible to users

27

Image

Volume

Volume

backed

Image

Volume


Proposed Cinder-store feature for Glance We are proposing patches for cinder store:

Glance patch: https://review.openstack.org/#/c/186201/

Glance_store patch: https://review.openstack.org/#/c/166414/

• Implement download/upload

– Attach Cinder volumes to Glance node using os-brick library

• Support image volume deletion

• Add settings for user/tenant to store images

– Useful to store images in the internal tenant

This enables cinder store to be used as default store

Nova will also be able to download the image for image-boot

28

Image

Volume

Volume

backed

Image Attach

Volume

https://review.openstack.org/#/c/186201/



Future Work

29


Boot Volume Boot Volume

Ironic: Baremetal Volume-Boot

Specs for supporting volume-boot of baremetal nodes is proposed:

• Ironic-spec: https://review.openstack.org/#/c/200496/

• Nova-spec: https://review.openstack.org/#/c/211101/

Combining with volume-boot and cinder-backed images, the image

can be rapidly deployed to the baremetal node.

30

Image Volume

Boot Volume

iSCSI / FC

1. Clone

2. Attach & Boot

Baremetal Nodes








Nova: Copyless Image Boot

Attaching volume to Nova nodes instead of download the base

image

• Bypass the first image download

• Improve boot time of image-boot instances

31

Guest

Image

VM

Volume-

backed

Image

Compute

Node

Storage

Array

Image

Volume

Attach

Image

Volume


Wrap-up

Cinder volume-backed images are useful to:

• Rapid boot of volume-boot VM instances

• Rapid boot of baremetal instances in the future

– Ironic work for volume-boot is ongoing

• Share volume data (e.g. base OS image) among tenants

• Leverage the storage features for image management

32


Before Liberty & after: Image and Volume boot

Wrap-up

33

# Boot method Boot image type Kilo Liberty Mitaka

1 Boot from volume Image file

2 Boot from snapshot

(creates a new volume) Cinder snapshot

3 Boot from image

(creates a new volume)

Cinder volume *1

*2

4 Boot from image Image file

*1 Limitation: Clients cannot upload & download images at this version.

*2 We are proposing Glance to improve cinder store to enable Cinder volumes to

be used as default images store

• Reviews are welcome!


Disclaimer

The OpenStack® Word Mark and OpenStack Logo are either registered

trademarks/service marks or trademarks/service marks of the OpenStack

Foundation in the United States and other countries and are used with the

OpenStack Foundation's permission. We are not affiliated with, endorsed or

sponsored by the OpenStack Foundation, or the OpenStack community.

Other company, product or service names may be trademarks or service mark

of others.

34

efficient image management using cinder volumes for virtual and

Documents