openstack storage new and upcoming from ibm research

© 2012 IBM Corporation

OpenStack Storage

New and Upcoming from IBM Research

Ronen Kat, IBM Research – Haifa

[email protected]

© 2012 IBM Corporation2IBM Research - Haifa

IBM and OpenStack

OpenStack Summit(San Diego 2012)

by Jeff Borek

Link

http://www.openstack.org/summit/san-diego-2012/open stack-summit-sessions/presentation/enabling-choice-for-the-opens tack-community


IBM and OpenStack

� IBM has a strong history of working with open standards and opensource�E.g., Linux, Apache, Eclipse

� Community approach to developing software can help meet clients’needs faster�Drive agreement on needed cloud standards and help remove threat of

proprietary lock in

� IBM’s value�Maintaining and supported by IBM

�IBM embraces OpenStack, contributes code and resources and consumes code

�Providing support for IBM platforms�Leveraging IBM software and management for OpenStack


Contributions to OpenStack Success Deliver Value

IBM Storwize V7000 IBM XIV

� Membership Services from HSLT

IBM Power Systems

� Drivers for IBM SVC & XIV� PowerVM driver� Dynamic hypervisor

support

� Implementations of 2 important open cloud standards

� Globalization & localization enablement� Localization for Simplified Chines� Crowd-sourced translation capability

� API, quotas, Nova integration

� Legal support for drafting bylaws

� Improvements to stability and quality

� Community sprint days� Permission building in the

China market� Three IBMers named “core

contributors”� 124 IBMers collaborating

across divisional lines

� Legal support for drafting bylaws

� Improvements to stability and quality

� Community sprint days� Permission building in the

China market� Three IBMers named “core

contributors”� 124 IBMers collaborating

across divisional lines


Different cloud workloads need different classes of storage

High-performance, co-located storage for XaaS• Blocks/file to support compute

General purpose data center NAS extension • Files

Fixed content depot• Objects


IBM Block Storage Enablement for OpenStack

OpenStack volume management drivers• SAN Volume Controller and Storwize family• IBM XIV

Storwize V7000

Storwize V7000 Unified

Storwize V3700

Flex System V7000

SAN Volume Controller

IBM XIV

New in FOLSOM

New in FOLSOM

IBM Research is exploring additional drivers enhancements…


Adding Functionality Beyond “just drivers”

� Not all storage was made equal�Allocate data by business needs and requirements

�Enable QoS and prioritization for storage

�Enable IaaS provider to “hand out” different storage types

� Mechanism�Cinder Volume types – new in Folsom�Cinder Scheduler – support for “filters” and back-end capabilities

�Supporting volume differentiation in drivers


Adding Functionality Beyond “just drivers”

� Fibre Channel support (FC and FCoE)�Enable use of OpenStack in enterprises

� IBM Research is investigating potential featuressuch as

�Federation of storage systems

�Storage system support for fast VM provisioning�High availability and QoS options for volumes

�Backup and DR

�Data reduction for Openstack storage�Etc…


VISION CloudVirtualized Storage Services Foundation for the Future Internet

Architect and build the next generation, standard-based, scalable, low-cost and secure cloud storage system

Key Innovations:• Raise Abstraction Level of Storage • Computational Storage• Content-Centric Storage• Advanced Capabilities• Data Mobility and Federation

Four use cases to demonstrate data-intensive services• Telco, Media, Healthcare and Enterprise

A 3-year project, European project led by IBM • Started Oct 2010

Now considering features to port to OpenStack Swift…

www.visioncloud.eu


� RESTful HTTP(s) Interface: Create, Retrieve, Update and Delete objects and containers (along with other abstractions)

►Capabilities: Allow implementation to define which subset it is supports►CDMI-aware and non-CDMI-aware clients

� Defined by SNIA (Storage Networking Industry Associ ation) and v1.01 in process of ISO standardization

CDMI CloudClient issues:HTTP(s) GET, PUT, POST, DELETE

Requests/Responses can include:Mime-type, data, metadata

Implementation responds:HTTP(s) Status

April 2009 Cloud TWG launched

April 2010CDMI V1.0 published

September 2011 CDMI v1.0.1 errata published

April 2011 Submitted for ISO standard

March 2011 CDMI reference implemen-tation

Current: Work on CDMI 1.1

Cloud Data Management Interface (CDMI):An emerging standard interface for storage cloud


Rich Meta Data Support for Objects

Description• Metadata integral part of objects

• Can describe content and how handled• Provide queries over metadata

Benefits• Increases the value of object stores as an infrastructure for building value-add

applications over the stored data, e.g. for healthcare, telco and media.


Rich Meta Data Support for ObjectsIndex and queries for user metadata

A catalog maintains for each object in a container a list of the attributes and attribute-value pairs

• A content-centric query requires a look-up in the catalog

Example (schematic) – list all red objectsGET /MyContainer/ HTTP/1.1. . . x-Match-md: x-Attribute=‘color’ x-

Value=‘red’

Response (schematic)HTTP/1.1 200 OKContent-Type: application/json

{"children" : [

“Obj 2",“Obj 3" ]

} Obj 2redcolor

Obj 3squareshape

Obj 1bluecolor

Obj 1triangleshape

Obj 2squareshape

Obj 3redcolor

ObjectValueAttribute

Obj 1

Obj 2

Obj 3

MyContainer


Computational Support via Storlet Engine

Description• “Stored procedures” for a storage cloud

• Provide ability to run computations (storlets) safely and securely, close to the data

Benefits• Reduce bandwidth, prevent exposure of sensitive data

• Enables extending Swift without changing its code• Create customized solutions


PUT Pudong Feb 2012mimetype = jpegcategory = vacation picturelocation = Shanghai

Storlets are the “stored procedure” of object clouds

Storlets provide a safe and secure way to execute computations in a storage cloud

• Typically run in a sandboxStorlets are uploaded as objects

• Distinguished from other objects by metadataStorlets are triggered by events on objects (e.g., put/get) and associated metadata attributes

• Synchronous or asynchronousBenefits

• Locality – avoid network overhead• Security – avoid transferring data outside of cloud• Timeliness• Automated execution• Stronger provenance

Use cases• Transformations on data, e.g., transcoding,

computing thumbnails• Extraction/derivation of metadata• Simple computations

Thumbnail CreatorObject-type = storletPut object trigger:

mimetype = jpegcategory = vacation picture

Code:. . . .

Pudong Feb 2012 thumbnailmimetype = jpegcategory = vacation pictureLocation = Shanghai


Transcode intoadditional formats.

Metadata indicates formats

Use metadata to select optimum format

for device/browser

Telco use case

Media use case

UploadMPEG-4

Meta data and Storlets in actionManaging and Serving Content

Low-res copy


Supporting Secure Multi-Tenancy

Description• Provide secure logical isolation between tenants to enable hosting of many

tenants over the same shared infrastructure

• User of one tenant cannot access storage of another tenant

• Security breach in one tenant cannot be leveraged to breach another tenant

Benefits• Feature required in order to provide secure public object cloud


We want to allow secure lightweight isolation between tenants while allowing complete sharing of physical resources

ApproachRun time model and security

• Principle of least privilege: Every sub component should operate using the least set of privileges required for the job completion.

Multi-tenancy and isolation• Build a system with separate tenant privileges. If the system

is compromised the damage should be confined to a single tenant.

• There should be a complete isolation of all tenant related information to prevent any cross-tenant leakage.

• All data-at-rest should be encrypted with a per tenant keyScalability and performance

• Security that can scale• Limit the performance affecting overheads.

Container

Object

User

Tenant

Model


Secure WAN De-duplication

Description• Phase 1: Support full object deduplication in the storage and over the network.

• Phase 2: Add Proof of Ownership (PoW) mechanisms to enable secure WAN deduplication

Benefits• Capacity and bandwidth efficiency for applications like Mail and content depots

• Security for client use is unique


Client-side deduplication in a cloud has a potential for significant savings, capacity and bandwidth but entails security challenges

Allows savings both bandwidth and capacity

Basic protocol:� Client computes a deterministic short hash of the

data� Client sends hash value to the cloud server� Cloud asks for the actual data only if the hash

(and data) are unfamiliar

One problem to address is spoofing uploads� Attacker obtains hash of victim’s file� Attempts to upload a file, but swaps the hash value

with that of the victim’s file.� File is now registered to attacker� Download file…

� Only need to get hold of a very small(not necessarily secret) piece of informationExample: 160 bits to get hold of a 1.5GB objectfrom a popular backup server

We have developed a solution called Proofs of Ownership (PoW)� A challenge response phase during uploads

Client Swift

Content already exists

Data Content

SHA1

2fd4e1c6

Create: object2Content-id: 2fd4e1c6

Success response

Client Swift

Content is new

Data Content

SHA1

2fd4e1c6


Object w/ content ID 2fd4e1c6 doesn’t exist

Special error response

Data Content

Create ‘object1’

Success response

Object w/ content ID 2fd4e1c6 exists



Extending to a Global Storage Pool

Description• Tie together multiple Swift clusters into a single namespace• Enable a geographically distributed Swift installation• Support active/active replication• Support geographic placement constraints

Benefits• Reduce TCO• Increase availability• Enable large cloud-based implementations


Extending Swift to a Global Storage Pool enables large cloud-based implementations

Global Distribution• Replicate objects in different

data centers• Simplify failure recovery/DR

• Use for availability, recovery and performance


FI-WARE – Foundation for Future Internet

FI-WARE

Mission: provide core platform for FI applications in multiple industries* (‘Usage Area’ projects*)

**

*

*

* *

FI-WARE Budget: €40M (FI-PPP in total: €300M over 5 years)

*

*


More OpenStack focus at IBM Research - Haifa

NetworkVirtualization

Object Storage

Block Storage

ComputeHA, powerVM, VM placement


Questions…

Thank you…

openstack storage new and upcoming from ibm research

Technology