virtualizing exchange

APP-BCA1684

#vmworldapps

Virtualizing

Exchange Best

Practices

Scott Salyer

VMware, Inc.

2

Disclaimer

This session may contain product features that are

currently under development.

This session/overview of the new technology represents

no commitment from VMware to deliver these features in

any generally available product.

Features are subject to change, and must not be included in

contracts, purchase orders, or sales agreements of any kind.

Technical feasibility and market demand will affect final delivery.

Pricing and packaging for any new technologies or features

discussed or presented have not been determined.

3

Agenda

Exchange Design on vSphere

Support Considerations

vSphere Best Practices for Exchange

vSphere HA, DRS and vMotion with Exchange

Site Resiliency for Exchange

4

Exchange Design Process

Set aside virtualization for a moment…remember, we’re designing

an Exchange environment first

Complete pre-requisite work before diving into the design:

• Understand the business and technical requirements

• Evaluate the current workload, or estimated workload for a new environment

• Know the health of the current and supporting infrastructure

• Understand support and licensing considerations

Follow a design methodology:

1. Establish the design requirements

2. Gather the compute requirements based on the Exchange design

requirements (Exchange Role Requirements Calculator)

3. Design the virtualization platform based on the Exchange design and

compute requirements

4. Determine virtual machine sizing and distribution among virtualization hosts

5

Design Requirements

High availability requirements

• Database Availability Groups (DAG)

• vSphere High Availability

Site resiliency requirements

• DAG

• Site Recovery Manager

Dedicated or Multi-role servers

Database Sizing

• Few, large databases or many, small databases

• Consider backup and restore times

Number of mailboxes

• Growth over next X years

Mailbox tiers

• Quota, activity, deleted item retention, etc

6

Compute Requirements based on Exchange Design

Requirements

• 1 site, high availability using DAG and vSphere HA

• 10,000 heavy users (2GB quota, 150 messages/mbx/day)

• Dedicated mailbox role, combined client access and hub transport servers

*CPU recommendations based on Intel x5660 processor

Physical Compute Requirements

• (7) mailbox cores to support activated databases

• (7) client access/hub transport cores

7

vSphere Compute Requirements

Exchange compute requirements

• 14 total mailbox cores to support failover

• 14 total client access/hub transport cores to support failover

• 256 GB memory for mailbox role to support failover (assuming two node DAG)

• 48 GB memory for CAS/HT role to support failover (assuming four CAS/HT VMs)

• No. of CAS/HT VMs * (4 GB + (2 GB * No. of vCPUs))

Sample vSphere host configuration

• CPU: (2) six-core Intel x5660

• Memory: 144 GB

vSphere minimum requirements

• Three vSphere hosts provide 36 cores, 432 GB

8

Virtual Machine Sizing and Placement

Option 1:

• (2) DAG nodes (77% utilized during failover)

• 8 vCPU, 128GB

• (4) client access/hub transport nodes

• 4 vCPU, 12GB (2GB/vCPU + 4GB)

• CPU and Memory overcommitted during host failure

Option 2

• (3) DAG nodes (54% utilized during failover)

• 6 vCPU, 64GB

• (4) client access/hub transport nodes

• 4 vCPU, 12GB (2GB/vCPU + 4GB)

• Sufficient resources for peripheral services

9

Support Considerations (What is and what isn’t?)

Support for Exchange has evolved drastically over the last two

years leading to confusion and misconceptions

What is Supported?

• Virtualization of all server roles, including Unified Messaging with Exchange

2010 SP1

• Combining Exchange 2010 SP1 DAG and vSphere HA and vMotion

• Thick virtual disks and raw-device mappings (pass-thru disk)

• Fibre channel, FCoE, iSCSI (native and in-guest)

Not Supported?

• NAS Storage for Exchange files (mailbox database, HT queue, logs)

• Thin virtual disks

• Virtual machine snapshots (what about backups?)

MS TechNet – Understanding Exchange 2010 Virtualization:

(http://technet.microsoft.com/en-us/library/jj126252)

10

Virtual CPUs

Best Practices for vCPUs

• vCPUs assigned to all Exchange virtual machines should be equal to or less than the total number of physical cores on the ESX host

• Enable hyper-threading, but understand that logical processor cores are not equal to physical processor cores for sizing

• Enable NUMA -- Exchange is not NUMA-aware, but ESX is

• Determine the CPU requirements based on physical CPU throughput and mailbox profile

• Use the SPECInt2006 rating for proposed processor

• Exchange Processor Query Tool

• Calculate adjusted megacycles required and match up to proposed hardware

• Exchange Mailbox Role Calculator

11

Hyper-threading Confusion

VMware’s Performance Best Practices for vSphere whitepaper

recommends enabling hyper-threading in the BIOS

• http://www.VMware.com/pdf/Perf_Best_Practices_vSphere4.0.pdf (page 15)

• www.VMware.com/pdf/Perf_Best_Practices_vSphere5.0.pdf (page 20)

Microsoft recommends disabling hyper-threading for production

Exchange implementations

• http://technet.microsoft.com/en-us/library/dd346699.aspx#Hyper

• “Hyper-threading causes capacity planning and monitoring challenges, and as

a result, the expected gain in CPU overhead is likely not justified…”

“So, do I or don’t I?”

YES! Enable hyper-threading to take advantage of ESXi’s

intelligent scheduler, but size to the capability of physical cores

12

Sizing Exchange VMs to the NUMA Node

NUMA Node 1 NUMA Node 2

NUMA Interconnect 4 vCPU/32 GB

Within NUMA Node

6 vCPU/48 GB

Beyond NUMA Node

Memory Bank 1 Memory Bank 2

4 Core / 32 GB 4 Core / 32 GB

13

NUMA Considerations

The following recommendations should be followed whenever possible to

ensure the best performance on systems that support NUMA:

• Make sure the NUMA architecture is enabled in the BIOS. On many systems

enabling NUMA is achieved by disabling Node Interleaving (typically default)

• When possible size Exchange virtual machines so that the amount of vCPUs

and memory does not exceed the number of cores and memory in a single

NUMA node

• The size of a NUMA node is not always the number of cores on a chip, for

example; AMD Magny Cours place two six-core chips in a single die/socket

• Avoid using CPU affinity features in vSphere, as this can circumvent the NUMA

architecture and reduce performance.

14

Virtual Memory

Best Practices

• Avoid memory over-commitment

• Only use memory reservations to avoid over-commitment, guarantee available physical memory, or to reclaim VM swap file space.

• “Right-size” the configured memory of a VM.

• More memory means less IOs, but only slightly. Test in your environment and fine-tune.

• Check “Unlimited” in resource allocation or set the limit to higher than configured memory

15

Storage: Best Practices

Follow storage vendor recommendations for path policy;

no restrictions like with MSCS

Format NTFS partitions from within the guest with a 64KB

allocation unit size

Verify partition is aligned

• StartingoffSet StripeUnitSize

Eagerzeroedthick virtual disks eliminate first write penalty

• Do not use if using array thin provisioning

Set power policy to High Performance

• Or disable power management in BIOS

16

Storage: Common Questions

PVSCSI vs LSI Logic

• PVSCSI can provide better throughput with less CPU utilization, test using

JetStress and your proposed workload

More virtual SCSI adapters • Evenly distribute virtual disks and

RDMs across four vSCSI adapters

VMFS

VMFS

VMDK or RDM? • Next Slide…

One virtual disk per VMFS or

consolidated virtual disks? • Performance – VMFS datastore must be sized

to accommodate the combined workload

• Management – Fewer datastores eases

management

VMFS VMFS

17

When should I use raw-device mappings (RDMs)?

Performance

Performance is no longer a deciding factor for using RDMs

VMDK disks perform comparably to RDMs

Capacity

VMDK files are limited to 2TB

Physical mode RDMs support up to 64TB

Storage Interaction

Backup solutions may require RDMs due to storage interaction needed

for hardware based VSS

Considerations

Easier to exhaust 255 LUN limitation in ESXi

VMFS volumes can support multiple virtual disks

vSphere storage features leverage virtual disks

18

What about NFS and In-guest iSCSI?

NFS

• Not supported for Exchange data (databases or logs) or shared-disk MSCS

configs (it works great, but consider support implications)

• Consider using for guest OS and app data

In-guest iSCSI

• Supported for DAG database storage

• Facilitates easy storage zoning and access masking

• Useful for minimizing number of LUNs zoned to an ESXi host

• Offloads storage processing resources away from ESXi hosts

19

Networking

Best Practices

• vSphere Distributed Switch or Standard vSwitch?

• Choice is yours, but distributed switches require less management overhead

• Separate traffic types:

• Management (vmkernel, vMotion, FT)

• Storage (iSCSI, FCoE)

• Virtual Machine (MAPI, replication, DMZ, etc)

• Configure vMotion to use multiple NICs to increase throughput

• Allocate at least 2 NICs per virtual switch to leverage NIC teaming capabilities

• Use the VMXNET3 para-virtualized network interface within the guest

• Following Microsoft best practices, allocate multiple NICs to Exchange VMs participating in a DAG

20

Exchange DAG Networking

DAG VMs *should* have two virtual network adapters for

replication and client access (MAPI)

• If separate networks are not possible use a single virtual NIC

Single vSwitch using VLAN trunking to

separate traffic

Physical Separation using

multiple vSwitches and physical NICs

21

vSphere High Availability

Best Practices

Enable HA to protect all VMs in the case of an unexpected host failure

Configure Admission Control policy to ensure failover capacity is available

Enable VM Monitoring to restart non-responsive VMs

Protect database availability groups with vSphere HA

• Deploy vSphere clusters as N+1, where N

is the number of nodes in a DAG

22

VMware Distributed Resource Scheduling

Best Practices

Enable DRS in fully automated mode for entire cluster

• Set individual automation levels for one-off needs

When possible keep VMs small

• VMs with less configured memory and fewer vCPUs can be placed by DRS

more efficiently

vSphere cluster members should have compatible CPU models for vMotion compatibility

• EVC Mode can guarantee host compatibility

Use host and VM groups and affinity rules

23

DRS VM and Host Groups and Rules

Host DRS Groups – Use to group like hosts; hosts in a rack or

chassis, or for licensing purposes

VM DRS Groups – Use to group like or dependent VMs together;

all CAS/HT VMs, all DAG nodes, etc.

Rules:

• Keep VMs Together

• Separate VMs

• VMs to Hosts

• Must run on hosts in group

• Should run on hosts in group

• Must not run on hosts in group

• Should not run on hosts in group

Use for DAG VMs

Use for non-clustered

VMs

24

Should Run On Versus Must Run On Rules

“Should run on” rules preferred to

keep like VMs separated across

chassis, racks, datacenters, etc.

In the case of a failure, VMs can be

powered on surviving hosts to

maintain performance

“Must run on” rules preferred when a

VM must never run on a particular

host or set of hosts

In the case of a failure, VMs will NOT

be allowed to run on the specified

hosts unless the rule is removed,

even if vCenter Server is offline

25

Avoid Database Failover during vMotion

When using vMotion with DAG nodes…

If supported at the physical networking layer enable jumbo frames on all

vmkernel ports to reduce the frames that must be generated and

processed

If jumbo frames is not supported modify cluster heartbeat setting

samesubnetdelay parameter to a maximum of 2000ms (default = 1000ms)

Always dedicate vMotion interfaces for the best performance

Always use multiple vMotion interfaces for increased throughput

C:\> C:\cluster.exe /cluster:dag-name /prop

samesubnetdelay=2000

PS C:\> $cluster = get-cluster dag-name;

$cluster.SameSubnetDelay = 2000

26

Multi-NIC vMotion Support

vDS management port group configured with both vmnics active

27

Multi-NIC vMotion Support

vDS vMotion port group 1 configured with vmnic0 active, vmnic1 standby

28

Multi-NIC vMotion Support

vDS vMotion port group 2 configured with vmnic0 standby, vmnic1 active

29

Multi-NIC vMotion Support

Configure in teaming properties of port group

30

Backups

Virtual Machine Backups

• Requires Exchange-aware agent to ensure a supported Exchange backup

• Large virtual machines (many virtual disks) take longer to snapshot and

commit – may affect DAG heartbeats

Software VSS

• Wide third party adoption

• Flexible configuration support

Hardware VSS

• Storage array level protection, either full clones or snapshots

• Storage vendor provides VSS integration

• Most solutions require physical mode raw-device mappings (unless using in-

guest attached iSCSI)

31

Site Resiliency – Exchange Native

Deployment may involve two or more sites serving as failover targets for

each other

Passive databases located in primary and secondary sites provide local

high availability and site resilience

“Datacenter switchover” process:

1. Terminate services in primary site

(stop-databaseavailabilitygroup)

2. Validate dependencies in second

datacenter

3. Activate mailbox servers

(restore-databaseavailabilitygroup)

4. Update DNS records for client access

endpoints

32

Site Resiliency – Site Recovery Manager

Deployment may involve two or more sites serving as failover targets for

each other

Passive databases at the primary site provide high availability, storage

replication provides site resiliency

SRM failover process:

1. Initiate SRM recovery plan (also

updates DAG node IP addresses)

2. Configure DAG IP address and

witness server (may be part of

SRM recovery plan)

3. Update DAG and DAG node A

records (may be part of SRM

recovery plan)

4. Update DNS records for client

access endpoints (may be part of

SRM recovery plan)

33

Choosing a Site Resilience Plan

Exchange Native Site Resilience

• Active-active and Active-passive deployments supported

• Manual activation during site failure

• Requires application specific site recovery procedure

• Testing recovery procedure requires failing over production services

Site Recovery Manager

• Active-active and Active-passive deployments supported

• Manual activation during site failure

• Application independent recovery

• Testing of recovery plans can be done without impacting production

34

Take Aways…

100% support from VMware and Microsoft

Virtualization doesn’t change the compute requirements for Exchange

Design the Exchange environment based on requirements, base vSphere design on Exchange design

Visit our Exchange Page on VMware.com

http://www.vmware.com/go/exchange

VMware Communities: Exchange, Domino and RIM

FILL OUT A SURVEY

AT WWW.VMWORLD.COM/MOBILE

COMPLETE THE SURVEY

WITHIN ONE HOUR AFTER

EACH SESSION AND YOU WILL

BE ENTERED INTO A DRAW

FOR A GIFT FROM THE

VMWARE COMPANY STORE

APP-BCA1684

#vmworldapps

Virtualizing

Exchange Best

Practices

Scott Salyer

VMware, Inc.