1801 - netbackup 7.6 best practices: protecting virtual...

1801 - NetBackup 7.6 Best Practices: Protecting Virtual Environments George Winter Technical Product Manager, NetBackup

Jim Olson Technical Director, NetBackup

SYMANTEC VISION 2014

Any information regarding pre-release

Symantec offerings, future updates or other

planned modifications is subject to ongoing

evaluation by Symantec and therefore subject

to change. This information is provided without

warranty of any kind, express or implied.

Customers who purchase Symantec offerings

should make their purchase decision based

upon features that are currently available.

Disclaimer


Agenda

• General Concepts for Better Virtualization Backups

• VADP Transport Mode & Heat Map – Which Is Best?

• vStorage API for Data Protection (VADP) Snapshots

• Maximizing Backup Throughput

• Resource Limits and Managing Redo Logs

• NetBackup Technologies For Improving Efficiency

• Hyper-V Backup Technologies

• What About VMware Restores?

• Summary


General Concepts for Better Virtualization Backups


General Concepts for Better Virtualization Backups

• Understand virtualization backup API’s

– New API’s significantly change backup dynamics – we will discuss

• You have lots of ESXi hosts – use them!

– Spread backup streams across ESXi hosts – NetBackup can help!

• First NBD stream per ESXi is fastest (~ 100 MB/sec – 10GbE)

– Subsequent streams from same ESXi will be slower

• Large redo logs are bad

– We’ll tell you how to design backups to avoid large redo

• Never assume your servers, storage, network, SAN is performing at optimal levels

– Test your environment to ensure that hardware is performing correctly


VADP Transport Mode & Heat Map – Which Is Best?


These boxes describe relative backup load on systems

Red lines and arrows describe data path of specific transport mode:

Transport Mode Diagram Key: RAM, CPU, I/O Load

Small Load

Medium Load

Heavy Load


NBD Transport Mode – Data Path

VMware ESXi

Media Server

ESXi Datastore

LAN

SAN

Simplest, most popular transport mode

No SW req’d on VMware

All data through VMkernel port

VMkernel port limit (~100MB/sec)

VMkernel Port


NBD Transport Mode – Load/Heat Path

VMware ESXi

Media Server

ESXi Datastore

LAN

SAN

VMkernel Port


SAN Transport Mode – Data Path

VMware ESXi

Media Server

ESXi Datastore

LAN

SAN

True off-host backups Lowest impact on ESXi host No VMkernel perf limits Fastest transport method


SAN Transport Mode – Load/Heat Map

VMware ESXi

Media Server

ESXi Datastore

LAN

SAN


HotAdd Transport Mode – Data Path

VMware ESXi

Media Server

ESXi Datastore

LAN

SAN

Bypasses VMkernel port No VMkernel perf limit Requires dedicated VM(s) Most impactful Transport Mode


HotAdd Transport Mode – Load/Heat Map

VMware ESXi

Media Server

ESXi Datastore

LAN

SAN


Which Transport Mode To Use?

• NBD Transport

– Easiest and most commonly used

– Need to understand performance implications (VMkernel QoS)

– VMware’s VVOL may mandate this

• SAN Transport (Fibre and iSCSI)

– Lowest impact on ESXi

– Typically used on more important VM’s

– Explicit access to SAN & Datastore required

• HotAdd

– All backup processing load placed on ESXi hosts

– Additional HotAdd VM’s to install, maintain

– ESXi hosting HotAdd VM must have direct access to Datastores

– Makes sense in certain environments – e.g. remote office


vStorage API for Data Protection (VADP) Snapshots


Problems With VADP Snapshots?

• VM Admin complaining about snapshots?

– Stuck snapshots?

– Failed snapshots & backups?

– Datastores filling up?

– Too much I/O impact?

• Let’s look at snapshots in more detail

– You can’t fix what you don’t understand

REDO LOG

VMDK


VADP Snapshot Process & Redo Log

1. Backup initiated

2. VADP creates snapshot of VM

3. VMware redo log created

– All writes sent to redo during backup

4. Backup processed

5. Backup finished

– Lock Released

6. Redo consolidated

Important to understand how redo impacts backup….

REDO LOG

VMDK


Understanding Redo Logs

• Redo logs are necessary

– Capture changes to active VM’s during backups

• Large redo logs are bad

• Larger redo logs are even badder

• Slower backups = redo logs become larger

– More data change captured during longer backup

• Larger redo = requires more temp storage

– VM admin is mad

– Storage admin is mad

• Larger Redo = more I/O load

– Consolidation process

• More I/O = Less reliable backups

REDO LOG


Maximizing Backup Throughput – Understanding Resource Limits


0 20 40 60 80 100 120

4 Stream

3 Streams

2 Streams

1 Stream

MB/Sec

MB/Sec

NBD Transport – Single ESXi Performance Per Stream

Stream Rate = 100 MB/sec




This concept generally similar with all transport types NBD VMkernel limit makes this more pronounced with NBD transport

Fastest Per Stream Rate


Resource Limit Exercise

ESXi 1

ESXi 2

ESXi 3

ESXi 4

ESXi 5

ESXi 6

Network = 10 GbE Measured Perf at Media Server = 600 MB/sec

600 MB/sec


VADP Backups Without Resource Limits

ESXi 1

ESXi 2

ESXi 3

ESXi 4

ESXi 5

ESXi 6

Backup Aggregate Performance: Number of streams = 15 Aggregate throughput = 565 MB/s What’s wrong with this?

Streams = 5 Perf = 165 MB/sec






600 MB/sec

5

3

0

4

3

0



ESXi 1

ESXi 2

ESXi 3

ESXi 4

ESXi 5

ESXi 6

Backup Aggregate Performance: Number of streams = 15 Aggregate throughput = 565 MB/s What’s wrong with this?







600 MB/sec

165

125

0

145

130

0



ESXi 1

ESXi 2

ESXi 3

ESXi 4

ESXi 5

ESXi 6







Why is this bad? • ESXi hosts are busy without B/U’s • ESXi resources required for

backups not available • Some ESXi hosts doing zero

backups • VADP snapshot active longer &

will be larger • More redo space req’d


NBD Transport NetBackup VIP Resource Limits

• Set ESXi & Datastore Resource limits to “1”

• Balance backups across ESXi hosts

• Minimize load on each ESXi host

• Maximize throughput to NetBackup media server

• Redo smaller at any given time

• I/O spread out across Datastores

• Less is more – backups are actually faster and more reliable


Unpaid customer testimonial

The Resource Limit setting is the single

best feature in NetBackup for VMware


VADP Backups With Resource Limits

ESXi 1

ESXi 2

ESXi 3

ESXi 4

ESXi 5

ESXi 6

Backup Aggregate Performance: Streams per ESXi host = 1 Total number of streams = 6 Aggregate throughput = 600







600 MB/sec

1

1

1

1

1

1


VADP Backups With Resource Limits

ESXi 1

ESXi 2

ESXi 3

ESXi 4

ESXi 5

ESXi 6

Backup Aggregate Performance: Streams per ESXi host = 1 Total number of streams = 6 Aggregate throughput = 600







600 MB/sec

100

100

100

100

100

100


How does this improve the redo log process?


Real World Example: Resource Limits and Redo Logs


Scenario 2: ESXserver Resource Limit Set to 4

Configuration:

• Transport = NBD • 10 GbE – 600 MB/sec • Data Change = 5% • VM Size = 100 GB

Scenario 1: ESXserver Resource Limit Set to 1

Exercise – How Resource Limits Impact Redo Size

6 ESXi Hosts

Q: Which setting will provide fastest backups?

Q: Which setting will require most redo log space?



Scenario 2: ESXserver Resource Limit = 4 Snapshot per Datastore = 4 B/U Rate Each VM = 25 MB/s Max Redo Space Req’d = 80 GB VM’s B/U per hour = 21.6


6 ESXi Hosts

Let’s take a look at these numbers…




6 ESXi Hosts Bandwidth is limited by two things:

1) VMkernel port used by other VMware services

2) Bandwidth to Media Server limited

600 MB/s input to backup server must be divided by 24 streams (6 ESXi x 4 Streams)




6 ESXi Hosts

Redo is significantly higher because:

• VM’s backup perf is 25% rate of scenario 1 (25MB/s)

• VADP snapshot open 400% longer

• 4 redo logs active at any given time (per Datastore)




6 ESXi Hosts

Bandwidth to media server limited by network technology

Number of VMs backed up per hour is unchanged by increasing streams per ESXi


Summary – How Resource Limits Impact Redo Size



6 ESXi Hosts

Scenario 1 provides:

• Fastest possible backup

• Smallest redo creation

• Less redo consolidation

• Much less I/O

• More reliable backups

• Same VM/hr backup rate


NetBackup Technologies For Improving Efficiency


NetBackup 7.6 VMware Accelerator

• Eliminate full backups

– Media server synthesizes a full image using data from previous backups

– Only the blocks that have changed are transferred from client to media server

• Key Benefits:

– Significantly reduce backup impact on VMware environment

– Shorten backup windows by over 90%

– Zero loss of any restore functionality

– Zero modifications to VMware environment

– Less backup hardware required – increases scalability of existing env.

– Supported with any NetBackup deduplication technology

– Integrated with V-RAY, AIR, SLP, applications


NBU for Accelerator: Non-Accelerator Backup Time

• Backup of 6 TB VM – 104 MB/sec

• Backup is to NBU Appliance dedupe target

• Backup perf is limited by VMkernel “throttling”

• Note backup performance of 105 MB/sec

• Significantly slower than 10 GbE nominal throughput


NBU for Accelerator: Accelerator Enabled Backup

• Enabled via simple checkbox selection

• Zero modification to VMware environment

• Zero loss of any restore functionality

• Enables incremental database backups


NBU for Accelerator: Accelerator Enabled Backup Time

• 1,749 MB/sec effective backup rate

• Effectively bypass the VMkernel performance limitation

• After initial full, all subsequent backups are effectively incrementals

• Zero loss of any restore functionality

• Search and restore files, folders, DB objects (e.g. individual mail message)


35x faster Backup with NetBackup 7.6 Accelerator for VMware


NetBackup Accelerator For VMware Additional Benefits


35x faster Backup – VM Backup Capacity

Without Accelerator Full/Incr Backup

NetBackup for VMware Accelerator Backup

864 VMs

VM Size = 100 GB - 10 GbE - NBD transport - 48/10 Backup Window

3,600 VMs


Total Amount Data Transferred in 4 weeks / 500VMs

Standard Full/Incr Backup

NetBackup for VMware Accelerator Backup

240 TB

50 TB

VM Size = 100 GB - 10 GbE - NBD transport - 48/10 Backup Window


Two advantages of the NBU Accelerator Backup (CPU)

– Significantly less CPU utilization saves HW and VMware license costs

– Backup is significantly faster – backup more VMs with same/less HW

Backup Processing Impact ESXi Host - CPU Utilization

Standard VADP backup (NBD transport)

NetBackup Accelerator For VMware (NBD transport)


Advantages of the NBU Accelerator Backup (network)

– Significantly less network traffic on already busy VMkernel port (NIC)

– Faster backup

Backup Processing Impact ESXi Host - Network Utilization

Standard VADP backup (NBD transport)

NetBackup Accelerator For VMware (NBD transport)


Hyper-V Backup Technologies


Configuring Hyper-V Backups

• Currently no resource limits

• Hyper-V backups configured at host/cluster level

– Limit backups within the policy

• Significant improvements to Hyper-V are in the works

– Resource limits

– Hyper-V Intelligent Policy

– Etc.


Deduplication On Hyper-V Host

• Deduplicate data on the Hyper-V host

• Transfer less data over LAN

• More load on Hyper-V host

NetBackup Hyper-V

LAN Traffic


NetBackup SAN Client On Hyper-V Host

• Direct Fibre connection for Hyper-V to NetBackup

• Zero LAN traffic

• Average load on Hyper-V host

• Dedicated, FAST Fibre connection

NetBackup Hyper-V

Fibre Traffic


VSS Transportable Snapshots

• Snapshot created quickly

• Backup snapshot mounted on alternate host

• Zero LAN traffic from Hyper-V host

• Array must support

NetBackup Hyper-V

VHDX


What About VMware Restores?


NetBackup 7.6 Best Practices: Improving Recovery Times

Wednesday 2:00 PM - PALACE 3


Summary


Summary

• Understand virtualization backup API’s

– New API’s significantly change backup dynamics

• Understand dynamics of different transport modes

– Each transport mode has different performance characteristics

• Resource limits are “the best thing ever”

• Upgrade NetBackup for latest performance technologies

– NetBackup Accelerator for VMware

– Instant Recovery for VMware


NetBackup for VMware Setting Suggestions

• For 10 or more ESXi host environments:

– Set resource limits for both ESXi server and Datastore

– ESXserver = 1

– Datastore = 1

– Rarely ever have limits higher than 2 for both

• Suggested max backup streams per media server = 10

– Assume 10 GbE or 8 Gb Fibre

– More typically is not faster

• Backup during periods of low VM activity

– Consider possibility of backups during day

Thank you!

Please take a few minutes to fill out the short session survey available on the mobile app—the survey will be available in the mobile app shortly after the session ends. And then watch for and complete the more extensive post-event survey that will arrive via email a few days after the conference.

To download the app, go to https://vision2014.quickmobile.com or search for Vision 2014 in the iTunes or Android stores.

https://vision2014.quickmobile.com/https://vision2014.quickmobile.com/

Thank you!

Copyright © 2014 Symantec Corporation. All rights reserved. Symantec and the Symantec Logo are trademarks or registered trademarks of Symantec Corporation or its affiliates in the U.S. and other countries. Other names may be trademarks of their respective owners. This document is provided for informational purposes only and is not intended as advertising. All warranties relating to the information in this document, either express or implied, are disclaimed to the maximum extent allowed by law. The information in this document is subject to change without notice.

George Winter

[email protected]

Jim Olson

[email protected]


Suggestions

• Add summary / conclusion slides

• Use “ESXi Host”

• Highlight V-Ray added value

• Make sure math is understood

• Add summary slide at end

• Add slide to advertise other VMware / NBU slides


Instant Recovery for VMware Process

VMware ESXi

NetBackup Appliance

ESXi Datastores

LAN

REDO At this point VM1 is 100% accessible to all users All changes that occur are safely captured in REDO This entire process can take less than 60 seconds

VM1

Temp NetBackup Read Only Datastore


SAN Transport Mode – Data Path

VMware ESXi

Media Server

ESXi Datastore

LAN

SAN


Thermometers

1801 - netbackup 7.6 best practices: protecting virtual...

Documents