bco2807-vsphere ha and datastore access outages – current-capabilities deep-dive and tech...

7/27/2019 BCO2807-vSphere HA and Datastore Access Outages Current-Capabilities Deep-Dive and Tech Preview_Final_US.pdf

1/57

vSphere HA and Datastore

Access Outages Current-Capabilities Deep-Dive andTech Preview

Smriti Desai, VMware, Inc.

Keith Farkas, VMware, Inc.

INF-BCO2807

#vmworldinf


2/57

2

Disclaimer

This session may contain product features that are

currently under development.

This session/overview of the new technology represents

no commitment from VMware to deliver these features in

any generally available product.

Features are subject to change, and must not be included in

contracts, purchase orders, or sales agreements of any kind.

Technical feasibi lity and market demand will affect final delivery.

Pricing and packaging for any new technologies or features

discussed or presented have not been determined.


3/57

3

VMware Business Continuity Solutions

Local Availability

vSphere High Availability

vSphere Fault Tolerance

vMotion and Storage vMotion

Data Protection

vSphere Data Protection

Storage APIs for Data Protection

Local Site Failover Site

vSphere vSpherevSphere vSphere vSphere

This talk

Disaster Recovery

vCenter Site Recovery Manager and

vSphere Replication


4/57

4

vSphere HA Recap

vSphere HA minimizes unplanned downtime

Provides automatic VM recovery in minutes

Protects against 3 types of failures

Does not require complex configuration

Is OS and application-independent

Infrastructure Connectivity Application

Host failures Host network isolated GuestOS hangs/crashes

VM crashes Datastore incurs PDL Application hangs/crashes


5/57

5

Talk Focus

Datastore accessibility outages occur infrequently but have a

large cost

Loss of accessibility is due to

Network or switch failure

Array, NFS sever, etc. misconfiguration

VM manageabil ity and availabil ity is affected

Applications with vdisks on inaccessible datastores hang, crash, or fail

May not be able to manage VMs on the affected hosts

vSphere HAs protection impacted

VMwareESX VMwareESX


6/57

6

Agenda and Objectives

Session has two major parts

1. Impact of datastore inaccessibi lity on HA failover workflows

2. Expanding HA protection against datastore inaccessibi lity

Objectives

Learn how HA workflows are impacted by datastore accessibility

Understand how vSphere 5.0/1 reduces the impact of inaccessibility

Preview the future - protecting VMs against datastore inaccessibility


7/57

7

Agenda: Part 1

1. Impact of datastore inaccessibil ity on HA failover workflows

Architecture overview

Datastore usage HA workflows and responses

2. Expanding HA protection against datastore inaccessibil ity


8/57

8

vSphere 5.0+ Architecture

HA Agent

Called the Fault Domain Manager (FDM)

Provides all the HA on-host functionality

Operation

vCenter Server (VC) manages the cluster Failover operations are independent of VC

Communicate over

Management network

Datastores

vCenter Server (VC)


9/57

9

FDM Master and Slave Roles

Any FDM can be master, selected by election

All others assume the role of FDM slaves

The FDM master

Monitors hosts and VMs

Manages VM restarts after failures

Reports cluster state to VC

The FDM slave

Forwards critical state changes to the master

Restart VMs when directed by the master

Elect new master

vCenter Server (VC)


10/57


11/57

11

Datastores Used for Communication

Datastores are used when management network is not available

Heartbeat datastores

Used by a master to monitor a partitioned/isolated slave

Enables a master to detect VM power state changes

VC chooses two (by default) for each host

Reselected after datastore accessibility changes

Home datastore of each VM

Used by isolated slaves to determine if a master owns the VM


12/57

12

Datastorecommunication HA host states Response

Persisted

Configuration

used in

triggers

used in

influence


13/57

13

Information Sources for HA Host States

The HA state reported for each host is derived using information f rom

VC

The master VC is communicating with

The FDM on the host

State Source

Election FDM on the host

Running (Master) VC

Connected (Slave) Master

Unreachable Master or VC

Isolated FDM on the host, reported by master

Partitioned Master

Dead Master


14/57

14

Information Sources for HA Host States

The HA state reported for each host is derived using information f rom

VC

The master VC is communicating with

The FDM on the host

State Source

Election FDM on the host

Running (Master) VC

Connected (Slave) Master

Unreachable Master or VC

Isolated FDM on the host, reported by master

Partitioned Master

Dead Master

Determined using

datastore

communication


15/57

15

How a Master Determines a Slaves State

Connected Slave

Unreachable

Partitioned / Isolated*

Dead

Not connected

No datastore heartbeats

No response to pings

Datastore heartbeats

No datastore heartbeats

Network available

* See slide notes


16/57

16

Impact of datastore accessibili ty on responses

Datastorecommunication HA host states Response

Persisted

Configuration

used in

triggers

used in

influence


17/57

17

Network Isolated FDM - VM Isolation Workflow

Apply isolationresponse

Homedatastore

accessible?

Master

owns VM?

Determine

VMs to

power off /

shutdown

No. Wait

Yes

Yes

No

Report VM poweroff

1 2 3

4 5


18/57

18

Network Isolated FDM - Home Datastore Inaccessibili ty

If all FDMs are isolated, all wil l apply isolation responses

VMs not restarted until master has access to VM datastores

Best practices

Redundant management networks

Reconfigure storage to reduce likelihood of inaccessible datastores

Use leave powered on isolation option


Homedatastore

accessible?

Master

owns VM?

No. Wait

Yes

Yes

No

Report VM poweroff

2 3

4 5

Determine

VMs to

power off /

shutdown

1


19/57

19

Network Isolated FDM - Heartbeat Datastore Accessibil ity

Isolated host and master have access to heartbeat datastores

Master will attempt failover on power off notification

Otherwise

Master will declare host dead and start failover immediately*

Same situation applies to partitioned hosts


Homedatastore

accessible?

Master

owns VM?

No. Wait

Yes

Yes

No

Report VM poweroff

* More info in backup slides

2 3

4 5

Determine

VMs to

power off /

shutdown

1


20/57

20

Host Declared DeadThe VM Failover Response


21/57

21

Retry

after adelay

Wait for

capacitychange

Host Dead: FDM Masters Workflow

Host declared dead

Determine VMs to be failed

over

Found

place?

No

Error encountered

Try to place each VMRestarted

?

Yes

No

End

Yes

1

2

3 4

5

6

7

Note: steps 2 to 7 apply anytimea VM is to be restarted


22/57

22

Impact of Datastore Accessibili ty: VMs to Failover

Case 1: home datastore of VM is not accessible on masters host

Master will proxy all accesses via a slave with access

Case 2: master may not know the VM is protected

Reason #1: VMs home datastore is inaccessible

VM cant be powered on in any case

Master will retry once datastore is accessible

Reason #2: partition, multiple masters, other master owns VM

But other master knows and will restart it if needed


23/57

23

Retry

after adelay

Wait for

capacitychange

Host Dead: FDM Masters Workflow

Host declared dead

Determine VMs to be failed

over

Found

place?

No

Error encountered

Try to place each VMRestarted

?

Yes

No

End

Yes

1

2

3 4

5

6

7

Note: steps 2 to 7 apply anytimea VM is to be restarted


24/57

24

Impact of Datastore Accessibil ity: VM Restart

Case 1: host manageabili ty impacted by datastore inaccessibi lity

Master will retry failovers on another host after timeout

Could take a long time to restart failed VMs

vSphere 5.0 and 5.1 enhancements significantly reduces impact

Case 2: one of VMs datastores is inaccessible on some/all hosts

Master will retry but could exhaust 5 retries before success

Future opportunity to enhance HA

Both are discussed next in part 2 of this session


25/57

25

Agenda: Part 2

1. Impact of datastore inaccessibility on HA failover workflows

2. Expanding HA protection against datastore inaccessibil ity

Technical direction

VM manageability and availability

Tech preview


26/57

26

Solution Approach for Inaccessible Datastores

Improve VM availabili ty by ensuring

1. VMs are manageable

2. VMs that use the datastore are moved to healthy hosts

Address #1 by enhancing ESX, #2 by enhancing HA

VMwareESX VMwareESXvCenter Server Manage


27/57

27

Types of Inaccessibil ity: PDL and APD

Are ESX storage-device states that indicate inaccessibility

PDL (Permanent Device Loss): device is permanently inaccessible

E.g., caused by removing a LUN using array management software

ESX infers state from

SCSI sense codes returned by an array

iSCSI login reject (target is gone or access not authorized)

Device must be recreated to restore normal operation

APD (All Paths Down): device is possibly temporarily inaccessible E.g., caused by unplugging a network cable

Device could become accessible at any time


28/57

28

T=140s (default)

APD timeout declared

I/O fast failing starts

Datastore reachable

APD cleared

Normal I/O behavior

ESX Enhancements for VM Manageabili ty

Idea: if a datastore is under APD/PDL, fail I/Os quickly

Impacted operations notified faster and allows others to proceed

T=0

APD detection

ESX PDL Support (vSphere 5.0) ESX APD Support (vSphere 5.1)

When under PDL, I/Os are failed

immediately

When under APD, non guest I/Os

are failed immediately after a delay


29/57


30/57

30

HA Enhancements for VM Availabil ity: The Future

We are exploring a signif icant extension to this mechanism

Design goals

Add support for APD

Triggered by PDL/APD declaration rather than guest I/Os

Full customization of responses (e.g., event only option)

Full user interface and detailed reporting

VM placement sensitive to accessibility


31/57

31

VM Component ProtectionCaveat: what follows is a prototype and a feature based on it

may look quite different, if and when we offer it

Protection Workflow:


32/57


PDL

Datastore

inaccessible

Terminate andrestart VM

PDL

Determine

per VM

response

End

FailoverNo action

1

2

3

4

Protection Workflow: 2


33/57


APD

Datastore

inaccessible


End

No actionFailover

APD

No

Wait for APD

declaration

Wait for

optional delay

Determine

per VM

response

Could

reservecapacity?

Yes

2

3

4

56

7

1

Protection Workflow: 2


34/57


APD

Datastore

inaccessible


End

No actionFailover

APD

No

Wait for APD

declaration

Wait for

optional delay

Determine

per VM

response

Could

reservecapacity?

Yes

2

3

4

56

7

1

Reset guest if

requested

APDc

le

ared

8

Combined Workflow:


35/57

APD and PDL

Restart guest if

requested

Datastore

inaccessible


APDc

le

ared

PDL

Determine

per VM

response

End

FailoverNo action No actionFailover

APD

No

Wait for APD

declaration

Wait for

optional delay

Determine

per VM

response

Could

reservecapacity?

Yes

D O i


36/57

36

Demo Overview

2 VMs on NFS

OracleDBServer

ExchangeServer

3 VMs on SAN

Webserver 1

Webserver 2

Ubuntu

APD impacts host A SAN

Converged NFS/iSCSI storage array

FC Switch Ethernet Switch

SAN NFS

WS1 WS2

OR EX

UB


37/57


38/57

Summary: Protection Against Datastore Inaccessibility


39/57

39

Summary: Protection Against Datastore Inaccessibility

Several platform enhancements in recent years

vSphere 5.0: PDL support

vSphere 5.1: APD support

vSphere 5.0U1: HA restarts VMs if they fail during a PDL/APD

The future: HA recovering VMs impacted by PDL/APD

Comprehensive: APD and PDL, and covers all VM I/Os

Configurable: Various levels of VM remediation

Usable: Enable with 1 click, detailed error reporting

Please send us your feedback on the proposed feature


40/57


41/57


42/57

42

Questions?


43/57

FILL OUT

A SURVEY

EVERY COMPLETE SURVEY

IS ENTERED INTO

DRAWING FOR A

$25 VMWARE COMPANY

STORE GIFT CERTIFICATE


44/57


45/57


46/57

46

VM Protection Workflow Example

2 VM Protection Workflow Power On


47/57

47

2. VM Protection Workflow Power On

6. Master updates protection list on disk

5. VC tells master to protect VM

4. VC reports VM is unprotected

2. User powers on VM

7. Master informs VC that it did

8. VC reports VM is protected

3. Host reports to VC VM powered on

1. VM is off. VM protection is N/A


48/57


49/57

Applying Concepts: Host Failures Host Dead


50/57

50

Applying Concepts: Host Failures Host Dead

Master declares a host dead when:

Master cant communicate with it over the network

Host is not connected to master Host does not respond to ICMP pings

Master observes no datastore heartbeats

Results in: Master attempts to restart all VMs from host

Restarts on network-reachable hosts andits own host

ESX 1 ESX 3

ESX 4ESX 2


51/57

Troubleshooting vSphere HA 5.0


52/57

52

Troubleshooting vSphere HA 5.0

HA issues proactive warning about possible future conditions

VMs not protected after powering on

Management network discontinuities Isolation addresses stop working

HA host states provide granularity in to error conditions

Al l HA conditions reported via events; config issues/alarms for some

Event descriptions describe problem and actions to take

All event messages contain vSphere HA so searching for HA issues easier

HA alarms are more fine grain and auto clearing (where appropriate)

5.0+ Troubleshooting guide which discusses likely top issues. E.g.,

Implications of each of the HA host states Topics on HB datastores, failovers, admission control

Will be updated periodically


53/57

Log File Format


54/57

54

og e o at

Log fi le contains time stamped rows

Many rows report the HA agent (FDM) module that logged the info

E.g.,

2011-06-01T05:48:00.945Z [FFFE2B90 info 'Invt' opID=SWI-a111addb] [InventoryManagerImpl::ProcessClusterChange] Cluster state

changed to Startup

Noteworthy modules are

Cluster module responsible for cluster functions Invt module responsible for caching key inventory details

Policy module responsible for deciding what to do on a failure

Placement module responsible for placing failed VMs

Execution module responsible for restarting VMs Monitor modules responsible for periodic health checks

FDM module responsible for communication with vCenter Server


55/57

Heartbeat Datastores(HB): Mechanisms


56/57

56

( )

Used by master for slaves not connected to it over network

Determine if a slave is alive

Rely on heartbeats issued to slaves HB datastores

Each FDM opens a file on each of its HB datastores for heartbeating purposes

Files contain no information. On VMFS datastores, file will have the minimum-allowedfile size

Files are named X-hb, where X is the (SDK API) moID of the host

Master periodically reads heartbeats of all partitioned / isolated slaves

Determine the set of VMs running on a slave

A FDM writes a list of powered on VMs into a file on each of its HB datastores

Master periodically reads the files of all partitioned/isolated slaves

Each poweron file contains at most 140 KB of info. On VMFS datastores, actual diskusage is determined by the file-sizes supported by the VMFS version

They are named X-powereon, where X is the (SDK API) moID of the host


57/57

bco2807-vsphere ha and datastore access outages – current-capabilities deep-dive and tech...

Documents