vmware vsan technical deep dive - march 2014

© 2014 VMware Inc. All rights reserved.

VMware Virtual SAN 5.5Technical Deep Dive – March 2014

Alberto Farronato, VMwareWade Holmes, VMwareMarch, 2014

Software-Defined Storage

2

Bringing the efficient operational model of virtualization to storage

Virtual Data Services

Data Protection Mobility Performance

Policy-driven Control Plane

SAN / NAS

SAN/NAS Pool

Virtual Data Plane

x86 Servers

Hypervisor-convergedStorage pool

Object Storage Pool

Cloud Object Storage

Virtual SAN

3

Virtual SAN: Radically Simple Hypervisor-Converged Storage

vSphere + VSAN

…

• Runs on any standard x86 server

• Policy-based management framework

• Embedded in vSphere kernel

• High performance flash architecture

• Built-in resiliency

• Deep integration with VMware stack

The Basics

Hard disksSSD

Hard disksSSD

Hard disksSSD

VSAN Shared Datastore

4

12,000+Virtual SAN Beta

Participants

95% Beta customersRecommend

VSAN

90%Believe VSAN will

Impact Storage like vSphere did to

Compute

Unprecedented Customer Interest And Validation

5

Why Virtual SAN?

• Two click Install

• Single pane of glass

• Policy-driven

• Self-tuning

• Integrated with VMware stack

Radically Simple

• Embedded in vSphere kernel

• Flash-accelerated

• Up to 2M IOPs from 32 nodes cluster

• Granular and linear scaling

High Performance Lower TCO

• Server-side economics

• No large upfront investments

• Grow-as-you-go

• Easy to operate with powerful automation

• No specialized skillset

6

Two Ways to Build a Virtual SAN NodeCompletely Hardware Independent

1. Virtual SAN Ready Node

…with multiple options available at GA + 30

Preconfigured server ready to use Virtual SAN…

2. Build Your Own

…using the Virtual SAN Compatibility Guide*

Choose individual components …

SSD or PCIe

SAS/NL-SAS/ SATA HDDs

Any Server on vSphere Hardware Compatibility List

HBA/RAID Controller

?⃰ Note: For additional details, please refer to Virtual SAN VMware Compatibility Guide Page?⃰ Components for Virtual SAN must be chosen from Virtual SAN HCL, using any other components is unsupported

http://www.vmware.com/resources/compatibility/search.php?deviceCategory=vsan

7

Broad Partner Ecosystem Support for Virtual SAN

StorageServer / Systems

SolutionData Protection

Solution

8

Virtual SAN Simplifies And Automates Storage Management Per VM Storage Service Levels From a Single Self-tuning Datastore

Storage Policy-Based Management

Virtual SAN Shared Datastore

vSphere + Virtual SAN

SLAs

Software Automates Control of Service Levels

No more LUNs/Volumes!

Policies Set Basedon Application Needs

Capacity

Performance

Availability

Per VM Storage Policies

“Virtual SAN is easy to deploy, just a few check boxes. No need to configure RAID.”— Jim Streit IT Architect, Thomson Reuters

9

Virtual SAN Delivers Enterprise-Grade Scale

2MIOPS

3,200VMs

4.4 Petabytes

Maximum Scalability per Virtual SAN Cluster

32Hosts

“Virtual SAN’s allows us to build out scalable heterogeneous storage infrastructure like the Facebooks and Googles of the world. Virtual SAN allows us to add scale, add resources, while being able to service high performance workloads.”— Dave Burns VP of Tech Ops, Cincinnati Bell

High Performance with Elastic and Linear Scalability

10

4 8 16 24 32

80K 160K320K

480K640K

253K

505K

1010K

1515K

2020K

Mixed 100% ReadLinear (100% Read)

Number of Hosts In Virtual SAN Cluster

IOP

S

3 5 7 8

286

473

677

767805

Number of VDI VMs

VSAN Linear (VSAN) All SSD Array

Number of Hosts In Virtual SAN Cluster

Notes: based on IOmeter benchmarkMixed = 70% Read, 4K 80% random Notes: Based on View Planner benchmark

Up to 2M IOPs in 32 Node Cluster Comparable VDI density to an All Flash Array

Virtual SAN is Deeply Integrated with VMware Stack

11

Ideal for VMware Environments

CONFIDENTIAL – NDA ONLY

vMotionvSphere HA

DRSStorage vMotion

vSphere

SnapshotsLinked Clones

VDP AdvancedvSphere Replication

Data Protection

VMware View

Virtual Desktop

vCenter Operations ManagervCloud Automation Center

IaaS

Cloud Ops and Automation

Site Recovery Manager

Disaster Recovery

Site A Site B

Storage Policy-Based Management

12

Virtual SAN 5.5 – Pricing And Packing

VSAN Editions and Bundles

Virtual SANVirtual SAN with Data

ProtectionVirtual SAN for Desktop

Overview• Standalone edition• No capacity, scale or

workload restriction

• Bundle of Virtual SAN and vSphere Data Protection Adv.

• Standalone edition• VDI only (VMware or Citrix)• Concurrent or named users

Licensing Per CPU Per CPU Per User

Price (USD) $2,495 $2,875(Promo ends Sept 15th 2014)

$50

Features

Persistent data store

Read / Write caching

Policy-based Management

Virtual Distributed Switch

Replication(vSphere Replication)

Snapshots and clones(vSphere Snapshots & Clones)

Backup(vSphere Data Protection Advanced)

Not for Public DisclosureNDA Material only

Do not share with Public until GA

Note: Regional pricing in standard VMware currencies applies. Please check local pricelists for more detail.

Virtual SAN – Launch Promotions

13

Virtual SANwith Data Protection

Virtual SAN(1 CPU)

vSphere Data Protection Advanced

(1 CPU)

VSA to VSAN upgrade

Virtual SAN(6 CPUs per

bundle)

Register and download promo

Virtual SAN

(1 CPU)

Beta PromoBundle Promos

20% 20% 20%



$9,180 / bundle$2,875 / CPU $1,996 / CPU

Promo Discount

Promo Price

End Date

Terms

9/15/2014 9/15/2014 6/15/2014

• Min purchase of 10 CPUs• First purchase only

Note: Regional pricing for promotions exist in standard VMware currencies. Please check local pricelists for more detail.

14

Virtual SAN Reduces CAPEX and OPEX for Better TCO

CAPEX• Server-side economics• No Fibre Channel network• Pay-as-you-grow

OPEX• Simplified storage configuration• No LUNs• Managed directly through

vSphere Web Client• Automated VM provisioning• Simplified capacity planning

As Low as $0.50/GB2

As Low as $0.25/IOPS

5X Lower OPEX4

Up to 50% TCO

Reduction

As Low as $50/Desktop

1

1. Full clones2. Usable capacity3. Estimated based on 2013 street pricing, Capex (includes storage hardware + Software License costs)

4. Source: Taneja Group



15

Flexibly Configure For Performance And Capacity

Performance

2xCPU – 8-core128GB Memory



1x400GB MLC SSD

(~15% of usable capacity)

1x400GB MLC SSD


2x400GB MLC SSD


5x1.2TB 10K SAS

7x2TB 7.2K NL-SAS

10x4TB 7.2K NL-SAS

IOPS1

Raw Capacity

~20-15K

6TB

~15-10K

14TB

~10-5K

40TB

Capacity

1. Mix workload 70% Read, 80% RandomEstimated based on 2013 street pricing, Capex (includes storage hardware + Software License costs)

$0.32/IOPS

$2.12/GB

$0.57/IOPS

$1.02/GB

$1.38/IOPS

$0.52/GB



• Compared to external storage at scale

• Estimated based on 2013 street pricing, Capex (includes storage hardware + Software License costs)

• Additional savings come from reduced Opex through automation

• Virtual SAN configuration: 9 VMs per core, with 40GB per VM, 2 copies for availability and 10% SSD for performance

Granular Scaling Eliminates OverprovisioningDelivers Predictable Scaling and ability to Control Costs

VSAN enables predictable linear

scaling

Spikes correspond to scaling out due to IOPs requirements

16

500 1000 1500 2000 2500 3000$40

$90

$140

$190

$240

$/VDI Storage CostVirtual SAN Midrange Hybrid Array

Number of Desktops

Sto

rag

e C

os

t P

er

De

sk

top



17

Running a Google-like DatacenterModular infrastructure. Break-Replace Operations

"From a break fix perspective, I think there's a huge difference in what needs to be done when a piece of hardware fails. I can have anyone on my team go back and replace a 1U or 2U servers. … essentially modularizing my datacenter and delivering a true Software-Defined Storage architecture."

— Ryan HoenleDirector of IT, DOE Fund

18

Hardware Requirements

Any Server on the VMware Compatibility Guide

•SSD, HDD, and Storage Controllers must be listed on the VMware Compatibility Guide for VSAN http://www.vmware.com/resources/compatibility/search.php?deviceCategory=vsan

•Minimum 3 ESXi 5.5 Hosts, Maximum Hosts “I’ll tell you later……”

1Gb/10Gb NIC

SAS/SATA Controllers (RAID Controllers must work in “pass-through” or RAID0” mode

SAS/SATA/PCIe SSD

SAS/NL-SAS/SATA HDD

At least 1 of each

4GB to 8GB USB, SD Cards




19

Flash Based Devices

VMware SSD Performance Classes

– Class A: 2,500-5,000 writes per second

– Class B: 5,000-10,000 writes per second

– Class C: 10,000-20,000 writes per second

– Class D: 20,000-30,000 writes per second

– Class E: 30,000+ writes per second

Examples

– Intel DC S3700 SSD ~36000 writes per second -> Class E

– Toshiba SAS SSD MK2001GRZB ~16000 writes per second -> Class C

Workload Definition

– Queue Depth: 16 or less

– Transfer Length: 4KB

– Operations: write

– Pattern: 100% random

– Latency: less than 5 ms

Endurance

– 10 Drive Writes per Day (DWPD), and

– Random write endurance up to 3.5 PB on 8KB transfer size per NAND module, or 2.5 PB on 4KB transfer size per NAND module

Flash Capacity Sizing

The general recommendation for sizing Virtual SAN's flash capacity is to have 10% of the anticipated consumed storage capacity before the Number of Failures To Tolerate is considered.

Total flash capacity percentage should be based on use case, capacity and performance requirements.

– 10% is a general recommendation, could be too much or it may not be enough.

Measurement Requirements Values

Projected VM space usage 20GB

Projected number of VMs 1000

Total projected space consumption per VM 20GB x 1000 = 20,000 GB = 20 TB

Target flash capacity percentage 10%

Total flash capacity required 20TB x .10 = 2 TB

Multi-level cell SSD (or better) or PCIe SSD

SAS/NL-SAS HDDSelect SATA HDDs

Any Server on vSphere Hardware Compatibility List

* Note: For additional details, please refer to Virtual SAN VMware Compatibility Guide

6Gb enterprise-grade HBA/RAID Controller

1 2 Build your ownVSAN Ready Node

…with 10 different options between multiple 3rd party vendors available at GA

Preconfigured server ready to use VSAN…

…using the VSAN Compatibility Guide*

Choose individual components …

Two Ways to Build a Virtual SAN Node

Radically Simple Hypervisor-Converged Storage

22

Virtual SAN Implementation Requirements

• Virtual SAN requires:– Minimum of 3 hosts in a cluster configuration

– All 3 host MUST!!! contribute storage• vSphere 5.5 U1 or later

– Locally attached disks• Magnetic disks (HDD)• Flash-based devices (SSD)

– Network connectivity• 1GB Ethernet• 10GB Ethernet (preferred)

esxi-01

local storage local storage local storage

vSphere 5.5 U1 Cluster

esxi-02 esxi-03

cluster

HDDHDD HDD

23

Virtual SAN Scalable Architecture

• Scale up and Scale out architecture – granular and linearly storage, performance and compute scaling capabilities– Per magnetic disks – for capacity

– Per flash based device – for performance

– Per disk group – for performance and capacity

– Per node – for compute capacity

disk group disk group disk group

VSAN network VSAN networkVSAN network

vsanDatastore

HDD

disk group

HDD HDD HDD

disk group

VSAN network

HDDscal

e up

scale out

24

Oh yeah! Scalability…..

vsanDatastore

4.4 Petabytes

2 Million IOPS

32 Hosts

25

Storage Policy-based Management

• SPBM is a storage policy framework built into vSphere that enables virtual machine policy driven provisioning.

• Virtual SAN leverages this new framework in conjunction with VASA API’s to expose storage characteristics to vCenter:

– Storage capabilities• Underlying storage surfaces up to vCenter and what it is capable of offering.

– Virtual machine storage requirements• Requirements can only be used against available capabilities.

– VM Storage Policies• Construct that stores virtual machine’s storage provisioning requirements based on storage capabilities.

Storage Policy Wizard

SPBM

VSAN object

VSAN object manager

virtual disk

VSAN objects may be (1) mirrored across hosts & (2) striped across disks/hosts to meet VM storage profile policies

Datastore Profile

Virtual SAN SPBM Object Provisioning Mechanism

27

Virtual SAN Disk Groups

• Virtual SAN uses the concept of disk groups to pool together flash devices and magnetic disks as single management constructs.

• Disk groups are composed of at least 1 flash device and 1 magnetic disk.– Flash devices are use for performance (Read cache + Write buffer).

– Magnetic disks are used for storage capacity.

– Disk groups cannot be created without a flash device.

disk group disk group disk group disk group

Each host: 5 disk groups max. Each disk group: 1 SSD + 1 to 7 HDDs

disk group

HDD HDDHDDHDDHDD

28

Virtual SAN Datastore

• Virtual SAN is an object store solution that is presented to vSphere as a file system.

• The object store mounts the VMFS volumes from all hosts in a cluster and presents them as a single shared datastore.– Only members of the cluster can access the Virtual SAN datastore

– Not all hosts need to contribute storage, but its recommended.

disk group disk group disk group disk group

Each host: 5 disk groups max. Each disk group: 1 SSD + 1 to 7 HDDs

disk group

VSAN network VSAN network VSAN network VSAN networkVSAN network

vsanDatastore

HDD HDDHDDHDDHDD

29

Virtual SAN Network

• New Virtual SAN traffic VMkernel interface.– Dedicated for Virtual SAN intra-cluster communication and data replication.

• Supports both Standard and Distributes vSwitches– Leverage NIOC for QoS in shared scenarios

• NIC teaming – used for availability and not for bandwidth aggregation.

• Layer 2 Multicast must be enabled on physical switches.– Much easier to manage and implement than Layer 3 Multicast

Management Virtual Machines vMotion Virtual SAN

Distributed Switch

20 shares 30 shares 50 shares 100 shares

uplink1 uplink2

vmk1 vmk2vmk0

Virtual SAN Network

• NIC teamed and load balancing algorithms:

– Route based on Port ID• active / passive with explicit failover

– Route based on IP Hash• active / active with LACP port channel

– Route based on Physical NIC load• active / active with LACP port channel

Management Virtual Machines vMotion Virtual SAN

Distributed Switch

100 shares 150 shares 250 shares 500 shares

uplink1 uplink2

vmk1 vmk2vmk0

Multi chassis link aggregation capable switches

VMware Virtual SANInteroperability Technologies and Products

VMware Virtual SANConfiguration Walkthrough

33

Configuring VMware Virtual SAN

• Radically Simple configuration procedure

Setup Virtual SAN Network

Enable Virtual SAN on the Cluster

Select Manual or Automatic

If Manual, create disk groups

34

Configure Network

• Configure the new dedicated Virtual SAN network– vSphere Web Client network template configuration feature.

35

Enable Virtual SAN

• One click away!!!

– Virtual SAN configured in Automatic mode, all empty local disks are claimed by Virtual SAN for the creation of the distributed vsanDatastore.

– Virtual SAN configured in Manual mode, the administrator must manually select disks to add the the distributed vsanDatastore by creating Disk Groups.

36

Virtual SAN Datastore

• A single Virtual SAN Datastore is created and mounted, using storage from all multiple hosts and disk groups in the cluster.

• Virtual SAN Datastore is automatically presented to all hosts in the cluster.

• Virtual SAN Datastore enforces thin-provisioning storage allocation by default.

37

Virtual SAN Capabilities

• Virtual SAN currently surfaces five unique storage capabilities to vCenter.

38

Number of Failures to Tolerate

• Number of failures to tolerate– Defines the number of hosts, disk or network failures a storage object can tolerate. For “n” failures

tolerated, “n+1” copies of the object are created and “2n+1” host contributing storage are required.

vsan network

vmdkvmdk witness

esxi-01 esxi-02 esxi-03 esxi-04

~50% of I/O ~50% of I/O

Virtual SAN Policy: “Number of failures to tolerate = 1”

raid-1

39

Number of Disk Stripes Per Object

• Number of disk stripes per object– The number of HDDs across which each replica of a storage object is distributed. Higher values may

result in better performance.

vsan network

stripe-2b witness

esxi-01 esxi-02 esxi-03 esxi-04

stripe-1b

stripe-1a stripe-2a

raid-0raid-0

VSAN Policy: “Number of failures to tolerate = 1” + “Stripe Width =2”

raid-1

40

Managing Failure Scenarios

Through policies, VM’s on Virtual SAN can tolerate multiple failures– Disk Failure – degraded event

– SSD Failure – degraded event

– Controller Failure – degraded event

– Network Failure – absent event

– Server Failure – absent event

VM’s continue to run

Parallel rebuilds minimize performance pain– SSD Fail – immediately

– HDD Fail – immediately

– Controller Fail – immediately

– Network Fail – 60 minutes

– Host Fail – 60 minutes

41

Virtual SAN Storage Capabilities

• Force provisioning– if yes, the object will be provisioned even is the policy specified in the storage policy is not satisfiable

with the resources currently available.

• Flash read cache reservation (%)– Flash capacity reserved as read cache for the storage object. Specified as a percentage of logical size

of the object.

• Object space reservation (%)– Percentage of the logical size of the storage object that will be reserved (thick provisioned) upon VM

provisioning. The rest of the storage object is thin provisioned.

42

VM Storage Policies Recommendations

• Number of Disk Stripes per object– Should be left at 1, unless the IOPS requirements of the VM is not being met by the flash layer.

• Flash Read Cache Reservation– Should be left at 0, unless there is a specific performance requirement to be met by a VM.

• Proportional Capacity– Should be left at 0, unless thick provisioning of virtual machines is required.

• Force Provisioning– Should be left disabled, unless the VM needs to be provisioned, even if not in compliance.

Failure Handling Philosophy

Traditional SANs– Physical drive needs to be replaced to get back to full redundancy

– Hot-spare disks are set aside to take role of failed disks immediately

– In both cases: 1:1 replacement of disk

Virtual SAN– Entire cluster is a “hot-spare”, we always want to get back to full redundancy

– When a disk fails, many small components (stripes or mirrors of objects) fail– New copies of these components can be spread around the cluster for balancing

– Replacement of the physical disk just adds back resources

44

Understanding Failure Events

Degraded events are responsible to trigger the immediate recovery operations.– Triggers the immediate recovery operation of objects and components

– Not configurable

Any of the following detected I/O errors are always deemed degraded: – Magnetic disk failures

– Flash based devices failures

– Storage controller failures

Any of the following detected I/O errors are always deemed absent:– Network failures

– Network Interface Cards (NICs)

– Host failures

Maintenance Mode – planned downtime

3 Maintenance mode options:

Ensure accessibility

Full data migration

No data migration

For more information, visit:http://www.vmware.com/products/virtual-san

vmware vsan technical deep dive - march 2014

Technology