efficient image management for openstack using block ...for openstack using block storage features...
TRANSCRIPT
© Hitachi Data Systems Corporation 2015. All rights reserved.
Efficient Image Management
for OpenStack using
Block Storage Features
Tomoki Sekiyama
1
© Hitachi Data Systems Corporation 2015. All rights reserved.
Background
New Features for Efficient Image Handling
Use-case of Volume-backed Images
Usage of Volume-backed Images
Future Work
Wrap-up
2
Contents
© Hitachi Data Systems Corporation 2015. All rights reserved.
Background
3
© Hitachi Data Systems Corporation 2015. All rights reserved.
OpenStack Components and Features
OpenStack cloud consists of many components:
Cinder Glance
Nova Ironic
4
Storage Array
VM VM
VM VM
VM VM
VM VM
Compute Nodes
•Compute: Manage VMs
•Volume: Provides block storages
service
•Manage images of OS & data for
VMs and baremetal nodes
•Support various image stores:
Local filesystem, Ceph, Swift, volumes, …
•Manage baremetal nodes
Baremetal nodes
Volume
Volume
Volume
Image Image Image
© Hitachi Data Systems Corporation 2015. All rights reserved.
Example Target Environment
Each services are connected via network (LAN, SAN)
• Cinder + Storage array
Cinder
Glance
Nova-api
Ironic
VM
* Currently baremetal does not
support Cinder volumes
5
Baremetal nodes
VM
VM VM
VM VM
VM VM
Storage Array
Compute Nodes
Control Node
SAN (FC/iSCSI)
etc.
Nova-compute
Nova-compute
Nova-compute
LAN
© Hitachi Data Systems Corporation 2015. All rights reserved.
HDD
Problems (1/2)
Agility is an important factor for OpenStack clouds
• Required to rapidly boot instances
When the image size is large, “nova boot” takes a long time
to download the image from Glance
• Nova caches images (per host), but not effective to first boot
• Cinder-volume-boot always downloads the image by default
6
Download Image Cached
Image
Guest
Image
VM
Volume VM
Image
Compute
Node
Storage
Array
(Image cache is optional) Control Node
© Hitachi Data Systems Corporation 2015. All rights reserved.
Image copy may affect disk I/O performance of instances
• Example: sysbench OLTP benchmark during volume creation
– Measured on a KVM instance with a volume on LVM-iSCSI backend
– During the image copy to a volume, I/O performance degrades
– The interference can be mitigated by image copy I/O bandwidth limit, but
image copy takes longer time
Volume creation from a snapshot is rapid, but snapshots cannot be
shared among projects (tenants)
• Not suitable for public images (e.g. operating systems)
Problems (2/2)
7
© Hitachi Data Systems Corporation 2015. All rights reserved.
(3) Storage
virtualization
and automatic data
optimization using
thin provision and
tiering
Volume
Features of Block Storage Arrays
8
VM
(4) Taking backup and
snapshot instantly via
storage feature
(2) Provide stable
IOPS and low
latency using boot
from volume
Volume
(1) Create a boot volume
using COW snapshot
SATA
SAS
Flash
Stable Performance
Virtualization/ Optimization
Business Continuity
Agility of booting an instance
© Hitachi Data Systems Corporation 2015. All rights reserved.
New Features for Efficient Image Handling
9
© Hitachi Data Systems Corporation 2015. All rights reserved.
Cinder New features for Image Handling In Liberty release, some features are added to Cinder for efficient
image management
• Image-Volume Cache
– Cache recently used images as “image volumes”
• Volume-backed Image
– Store an Glance image data in a Cinder volume
In both features, a new bootable volume can be created rapidly by
cloning the image volume
Both features are disabled by default
10
© Hitachi Data Systems Corporation 2015. All rights reserved.
Internal project
Overview of Image-Volume Cache
Images recently used by Cinder are cached as Cinder volumes
• Each image is stored in a volume (Image-Volume)
• Image volumes are placed in the internal project
Automatic management
• If cache volumes exceed the specific amount size, recently
unused cache volumes are deleted
Support various disk formats (converted to raw before cached)
Can coexist with the volume-backed Image feature
11
Download
(First time only)
Image Vol.
(Cache)
Image
Volume VM
Clone Boot Storage
Array
© Hitachi Data Systems Corporation 2015. All rights reserved.
Overview of Volume-backed Image
Register an image volume as a Glance image
• Utilize Glance’s Cinder store
– The image data is stored in a Cinder volume (Image-Volume)
• No image data transfer between Cinder and Glance
New volume can be created rapidly by cloning the image volume
12
Image
Volume
Volume-
backed
Image
Volume VM
Clone Boot
location = cinder://1234-abc..
Storage
Array
© Hitachi Data Systems Corporation 2015. All rights reserved.
Use-case of Volume-backed Images
13
© Hitachi Data Systems Corporation 2015. All rights reserved.
Rapid Boot for Virtual Machine Instances
Rapidly boot an instance with a new volume from an image
• Reduce time to launch instances
• Reduce I/O workload
Significant on booting multiple instances
Example:
• Booting an instance from a new Cinder volume
created from 20 GB operating system image
– Measured with thin-provisioning LVM backend
14
0 50 100 150 200 250
default
Volume- backed
Setup instance Volume creation Volume attach Spawning
[s]
246s
11s
© Hitachi Data Systems Corporation 2015. All rights reserved.
Copy-offload of Image Data
Leverage the storage array’s copy-offloading features
• Some storages support copy-on-write based cloning
– No data transfer
– No interference to instances’ performance
Example: sysbench OLTP benchmark during volume creation
• When thin-provisioning is supported, this feature also improve
storage capacity
15
© Hitachi Data Systems Corporation 2015. All rights reserved.
Project C
(app dev)
Sharing volume data among projects
Sharing volume data among projects
• The visibility of the Volume-backed image is managed by
Glance’s ACL feature
– Public image volume
– Sharing among specific members
• Useful to share base images such as operating system, common
application images, master datasets, etc.
16
Master
image
Project A Project B
Guest 2 Guest 4
Guest 1 Guest 3
Public
Volume-
backed
Image
*The image must be
registered by volume owner
© Hitachi Data Systems Corporation 2015. All rights reserved.
Usage of Volume-backed Images
17
© Hitachi Data Systems Corporation 2015. All rights reserved.
Cinder-store for Glance Enables Glance to store the image as a Cinder volume
• Until Kilo release, it can only “point” the volume, but read/write
access to the image data was not provided.
– Image volumes were only to create new volumes inside Cinder
• In Mitaka release, it supports read/write access experimentally.
– The image volume is attached to the Glance node using “os-brick” library
– Glance transfers the data to its clients (e.g. Nova, OpenStack CLI)
18
Image
Volume
Volume
backed
Image Attach
Volume
© Hitachi Data Systems Corporation 2015. All rights reserved.
Enable Volume-backed Image Feature (1/2)
Glance settings (/etc/glance/glance-api.conf)
• Enable Cinder store
– [glance_store] :: stores = file,http,swift,cinder
• Expose image locations (URL)
– [DEFAULT] :: show_multiple_locations = True
Cinder settings (/etc/cinder/cinder.conf)
• Enable Glance API version 2
– [DEFAULT] :: glance_api_version = 2
• Enable volume creation by cloning image volumes
– [DEFAULT] :: allowed_direct_url_schemes = cinder
• (optional) To use “cinder upload-to-image” to create volume-backed image
– Backend section :: image_upload_use_cinder_backend = True
19
© Hitachi Data Systems Corporation 2015. All rights reserved.
Enable Volume-backed Image Feature (2/2)
By default, image volumes are stored in current user’s project
• Only accessible from the user → cannot be shared among projects
• Visible to users → users may delete them by mistake
– All image volumes should be stored in the internal service project to avoid
these issues
Glance settings (/etc/glance/glance-api.conf)
• Specify the internal project to store image volumes
– [glance_store] :: cinder_store_project_name = service
– [glance_store] :: cinder_store_user_name = glance URL of “keystone” (auth service)
– [glance_store] :: cinder_store_password = ********
– [glance_store] :: cinder_store_auth_address = http://1.2.3.4/identity/v2.0
Cinder settings (/etc/cinder/cinder.conf)
• To use “cinder upload-to-image” to store the image volume to internal project
– Backend section :: image_upload_use_internal_tenant = True
– [DEFAULT] :: cinder_internal_tenant_project_id = 0123456789abcdedf
– [DEFAULT] :: cinder_internal_tenant_user_id = abcdedf01234567890
20
© Hitachi Data Systems Corporation 2015. All rights reserved.
Registration of Volume-backed Image (1/3) When “image_upload_use_cinder_backend = True”:
• openstack image create --volume <volume> <image-name> OR
• cinder upload-to-image <volume> <new-image-name>
• The specified volume is cloned to create a new image volume
• A new Glance image is created
• The cloned volume’s URL is registered to the image
• Only “raw” format is supported
– If other format (qcow2 etc.) is specified, image is converted and
transferred to Glance.
21
Image
Volume
New
Image
Volume
Clone
Register
location
Storage
Array
© Hitachi Data Systems Corporation 2015. All rights reserved.
Registration of Volume-backed Image (2/3) Using Glance CLI (with Image API v1)
• glance image-create --disk-format raw \ --container-format bare --name <image-name> \ --file <path-to-image-file> --store cinder
• --store option is only available in Image API v1, which might be deprecated in
future releases
• In Image API v2, need to set cinder to default store …
• NOTE: This is only available in Mitaka release
22
© Hitachi Data Systems Corporation 2015. All rights reserved.
Registration of Volume-backed Image (3/3) Manual registration of volumes (using Image API v2)
• Create the image volume(s) manually first – NOTE: the registered volume data must not be modified.
• glance image-create --disk-format raw \ --container-format bare --name <image-name>
• glance location-add <image-uuid> \ --url cinder://<volume-uuid>
• If Cinder has multiple backends, location-add should be repeated for each.
+------------------+-------------------------------------------------------------+ | Property | Value | +------------------+-------------------------------------------------------------+ | checksum | None | | container_format | bare | | created_at | 2015-09-22T21:31:34Z | | disk_format | raw | | id | f698173d-b96f-43be-aae9-fa0d13751c09 | | locations | [{"url": "cinder://95e571f9-5ccd-45eb-b282-441c3ce9a5db", | | | "metadata": {}}] | … | size | 1073741824 | | visibility | private | +------------------+-------------------------------------------------------------+ 23
© Hitachi Data Systems Corporation 2015. All rights reserved.
New Volume from Volume-backed Image
To create a new volume from a volume-backed Image:
• cinder create --image <image-uuid> <size>
• The new volume is cloned from the image volume
• The volume is extended when the volume size is larger than the
image volume
In Horizon, a new instance
can be launched from a new
volume cloned from the
volume-backed image
24
© Hitachi Data Systems Corporation 2015. All rights reserved.
Current Limitation
Cannot clone volumes between multiple hosts / backends
• Need to create an image volume on each host / backend
• and register their locations to the volume-backed image
Volume-backed images must be in raw format
• Image-Volume Cache supports various disk formats
– Automatically converts to raw type
25
© Hitachi Data Systems Corporation 2015. All rights reserved.
Future Work
26
© Hitachi Data Systems Corporation 2015. All rights reserved.
Booting baremetal nodes requires to copy the image to local disk
• Download & copy takes a long time
• Causes high network traffic
• Ironic caches the downloaded images
same as Nova but image copy to
attached disk is always required.
Problems in Ironic boot time
27
HDD
1. Boot with
deploy image
2. Export the disk
as iSCSI target
Image
3. Download & Copy Image
4. Reboot
Attach
Baremetal
Node
© Hitachi Data Systems Corporation 2015. All rights reserved.
Boot Volume Boot Volume
Ironic: Baremetal Volume-Boot
Specs for supporting volume-boot of baremetal nodes is proposed:
• Ironic-spec: https://review.openstack.org/#/c/200496/
• Nova-spec: https://review.openstack.org/#/c/211101/
Combining with volume-boot and cinder-backed images, the image
can be rapidly deployed to the baremetal node.
28
Image Volume
Boot Volume
iSCSI / FC
1. Clone
2. Attach & Boot
Baremetal Nodes
© Hitachi Data Systems Corporation 2015. All rights reserved.
Wrap-up
29
© Hitachi Data Systems Corporation 2015. All rights reserved.
Wrap-up
Cinder volume-backed images are useful to:
• Rapid boot of volume-boot VM instances
• Rapid boot of baremetal instances in the future
– Ironic work for volume-boot is ongoing
• Share volume data (e.g. base OS image) among tenants
• Leverage the storage features for image management
30
© Hitachi Data Systems Corporation 2015. All rights reserved.
Wrap-up
31
# Action Source Kilo Liberty Mitaka
1 Boot from volume Glance Image
2 Cinder snapshot
3 Image volume
4 Boot from image Glance Image
5 Image volume
6 Create image volume Cinder volume
7 Local file (upload) *1
8 Nova instance
image
*2
*1: It requires cinder-store to be set to default store, or usage of Image API v1 (may be deprecated in the future).
*2: It requires cinder-store to be set to default store.
© Hitachi Data Systems Corporation 2015. All rights reserved.
Disclaimer
The OpenStack® Word Mark and OpenStack Logo are either registered
trademarks/service marks or trademarks/service marks of the OpenStack
Foundation in the United States and other countries and are used with the
OpenStack Foundation's permission. We are not affiliated with, endorsed or
sponsored by the OpenStack Foundation, or the OpenStack community.
Other company, product or service names may be trademarks or service mark
of others.
32
33
© Hitachi Data Systems Corporation 2015. All rights reserved.
Backup slides
34
© Hitachi Data Systems Corporation 2015. All rights reserved.
Enable Image-Volume Cache
Image-Volume Cache and Volume-backed image feature can
coexist
Cinder settings (/etc/cinder/cinder.conf) • Enable internal tenant
– [DEFAULT] :: cinder_internal_tenant_project_id = ...
– [DEFAULT] :: cinder_internal_tenant_user_id = ...
• Enable Image-Volume Cache
– Backend section :: image_volume_cache_enabled = True
• Limit max capacity / number of cache volumes
– Backend section :: image_volume_cache_max_size_gb = ...
and/or
– Backend section :: image_volume_cache_max_count = ...
35