3/6/13 Backup Exec, NetBackup, VADP, VDDK, vStorage : Wait Wait Back it Up! NetBackup for 101 VMware Professionals

www.mrvray.com 1/11

Wait. Wait. Back it up!

Search

About MeOn ConnectNetBackup 101 for VMware ProfessionalsComments?

What’s up with VADP backups and VDDK on vSphere 5.1?

Published by Abdul Rasheed in Avamar, Backup Exec, CommVault, Data Protection, NetBackup, Veeam, VMware, vRanger, vSphere, vStorage on November29th, 2012

VMware vSphere 5.1 has been in the market for more than a few months now and the interest in the new capabilities is high. Because of this the market sawmany backup vendors rush to announce support for vSphere 5.1 in their VADP (vStorage APIs for Data Protection) integration. Everything looked clean andshiny and new.

On November 21, Symantec made an interesting announcement1. In a nutshell, the statement was that support for vSphere 5.1 would be delayed in itsNetBackup and Backup Exec products. It was because they discovered issues while testing the VADP 5.1 API for integration. The API in the current form mayintroduce risk in performing consistent backups and ensuring reliable restores. All vendors receive the same API, not all vendors perform the same level oftesting.

In order to explain the intricacies, first we need to take a quick look at how a backup product is integrated with VMware vSphere. With each release of vSphere,VMware publishes a set of APIs known as VMware APIs for Data Protection or VADP. One of the key components of VADP is Virtual Disk Development kitaka VDDK. This is the component through which third party code receives authenticated access to vSphere Datastores and virtual machine disk files. VMwaremakes this component available to its technology partners. Partners (backup product vendors in this case) ship this along with their product that has calls tovStorage APIs.

With each version of vSphere, an equivalent version of VDDK is released. The VDDK is generally backward compatible to one or more earlier versions of

vSphere. For example, VDDK 5.1 supports2 vSphere 5.1, 5.0 and 4.1. VDDK 5.0 supports3 vSphere 5.0, 4.1, 4.0 and VI 3.5. Since the updated VDDK isrequired to understand the modified data structures in a new version of vSphere, lower versions of VDDK are in general not supported for

accessing a higher version of vSphere. For example, VMware historically and currently (as of today) does not support the use of VDDK 5.0 to access

datastores in vSphere 5.1. VMware documents supported versions of vSphere for each of its VDDK versions in release notes.

The key to remember is the statement in bold face above. VMware does not support any violated combinations because of the risks and uncertainties. The

partners are expected to ship the correct version of VDDK when they announce the availability of support for a given vSphere release.

What Symantec announced and VMware confirmed4 is that VDDK 5.1 has issues and hence the support for vSphere 5.1 in its products will be delayed. Thismakes sense since VDDK 5.1 is the only version currently allowed to access vSphere 5.1. The face-saving reactions from other vendors to this

announcement revealed some of the dirty games and ugly truths to come out in the area of VADP/VDDK integration.

1. Vendors were claiming support for vSphere 5.1 but still shipping VDDK 5.0 with their products. This is currently not supported by VMware because of theuncertainties. This may change but at the time vendors claiming support, they were taking risks that typically are not acceptable in field of data protectionbusiness.

2. Vendors were mucking with API calls and silently killing hung processes. That may work for an isolated or random hang. But will not work when there arerepeatable hang situations like those observed in VDDK 5.1. Plus, there are performance and reliability concerns in abruptly ending sessions withvSphere.

3. Most vendors weren’t testing all the edge cases and never realized the problems in VDDK 5.1, thus prematurely announcing support for 5.1

If your backup vendor currently supports vSphere 5.1, be sure to ask what their situation is.

Sources and references:

1. Quality wins every time: vSphere 5.1 support update, Symantec official blog.

2. VDDK 5.1 Release Notes, VMware Support resources

3. VDDK 5.0 Release Notes, VMware Support resources

4. Third-party backup software using VDDK 5.1 may encounter backup/restore failures, VMware Support KB

3/6/13 Backup Exec, NetBackup, VADP, VDDK, vStorage : Wait Wait Back it Up! NetBackup for 101 VMware Professionals

www.mrvray.com 2/11

8Like 31 people like this. Be the first of your friends.

Tags:Backup Exec, NetBackup, VADP, VDDK, vStorage

2 Comments »

Dear EMC Avamar, please stop leeching from enterprise vSphere environments

Published by Abdul Rasheed in Avamar, Data Domain, Data Protection, NetBackup, VMware, vSphere, vSphere Data Protection, vStorage on October 23rd,2012

VMware introduced vStorage APIs for Data Protection (VADP) so that backup products can do centralized, efficient, off-host LAN free backup of vSpherevirtual machines.

In the physical world, most systems have plenty of resources, often underutilized. Running backup agent in such a system wasn’t a primary concern for mostworkloads. The era of virtualization changed things drastically. Server consolidation via virtualization allowed organizations to get the most out of their hardwareinvestment. That means backup agents do not have the luxury to simply take up resources from production workloads anymore as the underlying ESXiinfrastructure is optimized and right-sized to get line of business applications running smoothly.

VMware solved the backup agent problem from the early days of ESX/ESXi hosts. The SAN transport method for virtual machine backup was born during theold VCB (VMware Consolidated Backup) days and further enhanced in VADP (vStorage APIs for Data Protection). The idea is simple. Let the snapshots ofvirtual machine be presented to a workhorse backup host and allow that system do the heavy lifting of processing and moving data to backup storage. TheCPU, memory and I/O resources on ESX/ESXi hosts are not used during backups. Thus the production virtual machines are not starved for hypervisorresources during backups.

For non-SAN environments like NFS based datastores, the same dedicated host can use Network Block Device (NBD) transport to stream data throughmanagement network. Although it is not as efficient as SAN transport, it still offloaded most of the backup processing to the dedicated physical host.

Dedicating one or more workhorse backup systems to do backups was not practical for small business environments and remote offices. To accommodatethat business need, VMware allowed virtual machines to act as backup proxy hosts for smaller deployments. This is how hotadd transport was introduced.

Thus your backup strategy is to use a dedicated physical workhorse backup system to offload all or part of backup processing using SAN or NBD transports.For really small environments, a virtual machine with NBD or hotadd transport would suffice.

Somehow EMC missed this memo. Ironically, EMC had been the proponent of running Avamar agent inside the guest instead of adopting VMware’s VADP.The argument was that the source side deduplication at Avamar agent minimizes the amount of data to be moved across the wire. While that is indeed true,EMC conveniently forgot to mention that CPU intensive deduplication within the backup agent would indeed leech ESXi resources away from productionworkloads!

Then EMC conceded and announced VADP support. But the saga continues. What EMC had provided is hotadd support for VADP. That means you allocatemultiple proxy virtual machines even in the case of enterprise vSphere environments. Some of the best practice documents for Avamar suggest deploying abackup proxy host for every 20 virtual machines. Typical vSphere environment in an enterprise would have 1000 to 3000 virtual machines. That translates to 50to 150 proxy hosts! These systems are literally the leach worms in vSphere environment draining resources that belong to production applications.

3/6/13 Backup Exec, NetBackup, VADP, VDDK, vStorage : Wait Wait Back it Up! NetBackup for 101 VMware Professionals

www.mrvray.com 3/11

The giant tower of energy consuming nodes in Avamar grid is not even lifting a finger in processing backups! It is merely a storage system. The real workhorsesare ESXi hosts giving in CPU, memory and I/O resources to Avamar proxy hosts to generate and deduplicate backup stream.

The story does not change even if you replace Avamar Datastore with a Data Domain device. In that case, the DD Boost agent running on Avamar proxy hostsare draining resources from ESXi to reduce data at source and send deduplicated data to Data Domain system.

EMC BRS should seriously look at the way Avamar proxy hosts with or without DD Boost are leaching resources from precious production workloads. Themethod used by Avamar is recommended only for SMB and remote office environments. Take the hint from VMware engineering as to why Avamar technologywas borrowed to provide a solution for SMB customers in VMware Data Protection (VDP) product. You can’t chop a tree with a penknife!

The best example for effectively using VADP for enterprise vSphere is NetBackup 5220. EMC BRS could learn a lesson or two from how Symantec integrateswith VMware in a much better way. This appliance is a complete backup system with intelligent deduplication and VADP support built right in for VMwarebackups. This appliance does the heavy lifting so that production workloads are unaffected by backups.

How about recovery? For thick provisioned disks SAN transport is indeed the fastest. For thin provisioned disks, NBD performs much better. The good newson Symantec NetBackup 5220 is that the user could control the transport method for restores as well. You might have done the backup using SAN transport,

3/6/13 Backup Exec, NetBackup, VADP, VDDK, vStorage : Wait Wait Back it Up! NetBackup for 101 VMware Professionals

www.mrvray.com 4/11

8

8

however you can do the restore using NBD if you are restoring thin provisioned virtual machines. For Avamar, hot-add is the end-all for all approaches. NBD ona virtual proxy isn’t useful, hence using that is a moot point when the product offers just virtual machine proxy for VADP.

The question is…

Dear EMC Avamar, when will you offer an enterprise grade VADP based backup for your customers? They deserve enterprise grade protection for theinvestment they had done for large Avamar Datastores and Data Domain devices.

Like 5 people like this. Be the first of your friends.

Tags:Direct vSphere Backup, Hotadd, NBD, NetBackup 5220, SAN Transport

1 Comment »

VMware announces vSphere Data Protection (VDP), what is in it for you?

Published by Abdul Rasheed in Avamar, Data Protection, VMware, vSphere, vSphere Data Protection, vStorage on August 28th, 2012

vSphere Data Protection (VDP) is VMware’s new virtual backup appliance for SMB available in VMware vSphere 5.1. It replaces the older VMware DataRecovery (vDR) product. There had been a number of confusions around this announcement; partly due the way EMC, VMware’s parent company, made somepress releases.

Is VDP the same as EMC’s Avamar Virtual Edition (Avamar VE)?

No, it is not. VDP is a product from VMware. The only technology VMware had used from Avamar is its deduplication engine. The older vDR had limiteddedupe capabilities as it was mainly coming from change block tracking (CBT) in vStorage APIs for Data Protection (VADP). With Avamar’s technology, VDPnow provides variable block based deduplication.

I heard that I can upgrade from VDP to EMC Avamar if I need to grow beyond 2TB, is that true?

No, VDP is not a ‘lite’ version of Avamar. It is a different product altogether.

What are my options if I need to grow beyond 2TB?

You could add additional VDP appliances. Up to 10 VDP appliances are supported under one vCenter server. However, these are separate islands of storage.These appliances do not provide global deduplication among these storage pools.

Having said that it is more likely for you to hit other limitations in VDP before hitting the 2TB limit. Note that Avamar based deduplication engine is suitable onlyfor SMBs who could afford to have black out windows and maintenance windows in their backup solution. These are the periods of time where the housekeeping work is being done by dedupe engine. The system is not available for running backup jobs.

Only 8 virtual machines can be backed up concurrently that might increase backup windows. There is no SAN transport capability to offload production ESXihosts from backup tasks. There is no good way to make additional copies for redundancy or extended retention like replication to remote location or cloud.VMware has made it clear that VDP is truly for SMBs and encourages customers to look at enterprise class backup solutions from partners for largerenvironments.

Why would EMC let VMware use its Avamar technology at no additional cost to customers? Is EMC trying to promote its products?

Just like how Windows/UNIX/Linux operating environments provide basic utilities for backups, VMware had always provided basic backup solution with itsofferings. In the days of ESX service console, the Linux based console provided tools like tar and cpio. With ESXi where service console is no more, vDR wasbrought to the table. vDR had its limitations. Now the choice is to innovate vDR or license a relatively mature technology. As parent company has a solution,VMware went the route of taking Avamar dedupe engine for storage and build its own capabilities for scheduling backups and managing recovery points.

EMC’s Avamar is a popular product in small environments. Although EMC had been trying hard to make Avamar enterprise ready, its deduplication engine hassignificant limitations. It requires blackout and maintenance windows. With larger capacities, the duration of these windows also increases. With the acquisitionof Data Domain, EMC is now focusing more on using its DD Boost technology for distributing the deduplication workload. In fact, EMC recommends the use ofData Domain Boost with Avamar (instead of using Avamar’s dedupe engine) for larger workloads. I believe it was a good decision to support VMware’s SMBmarket with a technology that was meant for SMB in the first place. I think Avamar dedupe engine is counting its days as a technology that can make money.See my earlier blog on EMC’s backup portfolio.

Stay tuned. More on VDP coming soon!

Like 2 people like this. Be the first of your friends.

No Comments »

Client Direct: NetBackup vs. NetWorker

Published by Abdul Rasheed in Data Protection, NetBackup, NetWorker on July 17th, 2012

NetBackup introduced Client Direct capability a few years back with NetBackup 7.0 release. This is a break-through innovation in backup infrastructurearchitecture. Traditionally backup is a process where data is read from production client, transmitted over wire in its entirety to a backup server and then writtento storage. The emergence of target dedupe appliances behind a backup server meant that backup can now take three hops through network. It hops fromclient to backup server first, then it hops from backup server to deduplication appliance. NetBackup changed this game. NetBackup client can dedupe backup

3/6/13 Backup Exec, NetBackup, VADP, VDDK, vStorage : Wait Wait Back it Up! NetBackup for 101 VMware Professionals

www.mrvray.com 5/11

8

stream at source and send deduplicated data directly to NetBackup’s deduplication pool, for example a NetBackup 5020 deduplication appliance, asillustrated below.

NetBackup Client Direct

This architecture is possible in NetBackup because it has several innovations that reduce the impact of running deduplication at the production client.

1. NetBackup Accelerator: This technology features a platform independent track log that intelligently detects changed files without the need for enumeratingthe entire file system. Then it optimally synthesizes a full backup image at the storage. The result: Full backups can be run using the resources needed torun an incremental backup.

2. NetBackup Client Side Deduplication Cache: This enables the production client to run deduplication by comparing the generated fingerprints for thechucks in the changed file (detected intelligently as explained in 1) against the previous backup set without shipping the fingerprint to storage forcomparison. The result: Superior federated deduplication without the slow chatter across network.

3. Intelligent Hybrid Chunking that is not CPU bound: Deduplication chunking is done typically using variable block method or fixed block method. The firstone is CPU intensive and the second one is less efficient in data reduction rate. NetBackup uses the best of both worlds by using intelligent hybridchunking. As deduplication-fingerprinting logic is built into the client, it can start the chunking exactly after identifying the object boundaries. Thus you getthe advantage of not being CPU bound while also not suffering from low deduplication rate.

Reducing of impact on production client’s resources, reducing the impact on production network, reducing the number of hops and reducing the impact onbackup server (translation, increased scalability) make NetBackup Client Direct a unique feature. The popularity of this feature had made ‘Client Direct’ acommon innovation name that appears in RFPs for backup solutions.

The pressure is causing other backup vendors to come up with ‘Client Direct’. EMC announced last week that NetWorker 8.0 will have this capability and evennamed it ‘Client Direct’ so that the checkboxes in RFP can be ticked. A closer look reveals that NetWorker Client Direct is suitable for checkbox in RFP, butreally not ready for primetime as is.

1. NetWorker Client has no intelligent detection of changed files. NetWorker also does not have any sort of optimized synthetics. The result: Running fullbackups with NetWorker Client Direct will use significant amount of processing power from production clients.

2. The NetWorker client and does not federate deduplication; it is done by DD Boost. As these two are essentially unaware of each other’s format, there isno way to cache fingerprints of the chunks from previous backups. That means excessive chitchat with the target Data Domain device during backups.

3. DD Boost is the process of offloading some of the Data Domain deduplication processing to other systems. In this case, the production clients are takingthat load. As clearly documented in Data Domain SISL architecture “SISL takes the pressure off of disk accesses as a bottleneck so that the systemrelies on the speed of the CPU to deliver inline deduplication performance”. Translation: CPU bound chunking. When this is offloaded to productionclients, it can severely affect the performance of production systems with large backup workloads.

Even though EMC can mark the checkboxes in RFPs; their specialists are less likely to encourage POCs with NetWorker Client Direct. In a neck-to-neck battle,it appears that NetWorker has a long road ahead to match NetBackup Client Direct.

Like Be the first of your friends to like this.

Tags:Client Direct, Deduplication, NetBackup Accelerator

No Comments »

What do NetApp ONTAP and Symantec NetBackup have in common?

Published by Abdul Rasheed in Avamar, Data Domain, Data Protection, NetBackup, NetWorker on July 6th, 2012

A friend of mine forwarded this link to the interview SearchStorage.com recently did with Dave Hitz, one of the founders of NetApp. It is an interesting read andthe major topic is the new clustering capabilities in OnTap 8. When he was asked about EMC’s Isilon, I found his response to hit a home run.

“If you look at features EMC can support, you end up with a complete list. If you break apart their architectures and look at the same feature list byarchitecture, you end up finding the main feature Isilon has is clustering, which is great. Unfortunately, it’s not in combination with the full suite of rich data

management capabilities. That’s the No. 1 difference Ontap has — it’s the same Ontap that has all this cool stuff in it.” , said Dave Hitz.

The context here is the fact that the foundational technology powering all storage systems from NetApp is ONTAP (with E-series being an outlier) andcustomers get the choice of footprint and features to match their workloads. EMC’s storage division, on the other hand, provides different products foroverlapping set of workloads like VNX, VMAX, Isilion etc.

If you think about it, this response is applicable even when you look at other business units from EMC as well. My favorite is EMC’s Backup and Recovery

3/6/13 Backup Exec, NetBackup, VADP, VDDK, vStorage : Wait Wait Back it Up! NetBackup for 101 VMware Professionals

www.mrvray.com 6/11

8

8

Services (BRS) division. They have four different products; Avamar, Data Domain, NetWorker and HomeBase, pretty much serving the same market. If I wereto fit Dave’s quote in the context of Backup and Recovery and use Symantec’s NetBackup as the competitor for EMC Backup, it would go something like this.

If you look at features EMC can support as a vendor for backup and recovery, you end up with a near-complete list. If you break apart their architectures and

look at the same feature list by architecture, you end up finding that the main value Data Domain has is storage reduction at target with federationcapabilities for limited application workloads. Avamar has full management capabilities but only for smaller workloads. NetWorker has decent long-term

retention capabilities and track record but had been on life support. HomeBase provides Bare Metal Recovery. Unfortunately, none of these products are

with a full suite of rich data management capabilities for end-to-end protection that can bring down capital and operational expenses in managing recovery

points. That’s the No. 1 difference NetBackup has — it’s the same NetBackup that has all those cool stuff in one platform and a lot more innovations likemanaging snapshots, replicas, virtualized applications, backup acceleration etc.

As always, the standard disclaimer applies here. This is just my opinion. Although I work for Symantec, the above statement should not be considered as theview of my employer.

Like 2 people like this. Be the first of your friends.

No Comments »

EMC or HP: Who is stretching the truth on deduplication system performance?

Published by Abdul Rasheed in Data Protection on June 6th, 2012

EMC proudly announced the availability of Data Domain 990 during EMC World 2012 on May 21st. The claim in the news release was that the system couldbackup up to 248 TB in 8-hour backup window with 31 TB/hr throughput. Further, it claimed that it is 6x faster than closer competitor.

The pride was shattered within 2 weeks. Even Kardashion’s marriage lasted longer than the claim. HP announced that it could protect up 100 TB/hr using itsStoreOnce family of products. EMC looked at it with tears and finally responded as given here

EMC said HP’s decision was “puzzling”, and argued the comparison was not fair because HP’s claim was for four hardware systems working on four storage

pools compared to EMC’s figures for one system and one pool. Deduplication, which removes copies of data from storage to improve usage, only workswithin pools of data.

Now is time for a reality check.

Number of systems involved in deduplication processing: EMC’s claim is that Data Domain 990 is a single head unit while HP StoreOnce B6200 is a

multi-node system. From the first look, it sounds like a legitimate argument; but the reality is that EMC has no reason to shed crocodile’s tears about this. Hereis why.

The 31 TB/hr rate for Data Domain 990 is coming from Data Domain Boost, the software component that offloads most of the processor-intensivededuplication processing to backup servers and/or application servers. The unit by itself is not doing all the work. The story is not different for HP B6200 either;it is making use of StoreOnce Catalyst software, which does similar to what Data Domain Boost does for Data Domain 990.

The absolute number of processing heads shouldn’t matter in this case as the actual performance numbers are skewed on account of distributed processing. Iwould even give credit to HP, as their solution is highly available with two nodes serving one storage pool. Backups are the last line of defense in an enterprise.High Availability brings additional customer value.

Number of name spaces: Single name space provides deduplication across all the workload ingested into the storage pool. Data Domain 990 is a singlename space device with one processing head. You buy HP B6200 in the form of two nodes and storage known as couplets. It is not crystal clear from HP’sdocumentation whether multiple couplets can share the same name space or they use dedicated name spaces. I am giving the benefit of doubt that EMC didthe research and made the statement on this. Some of the defensive comments HP did after EMC’s reaction tend to indicate the HP stretched the truth a littlehere.

HP marketing veep Craig Nunes says an 8-node B6200 is a single system because it is managed as one and has a single namespace. The single

namespace is segmented into four individual namespaces, one per couplet, and, he says, “next year I could do a firmware update and change that”.

So, I am inclined to support EMC from this point unless someone can confirm from HP’s documentation that a four-couplet unit uses a single name space.

Truth in comparisons:

EMC’s claim: 6x faster than closer competitor. HP’s claim: 3 times faster (backups) than closest competitor

The statements won’t actually tell you how ‘closer/closest’ competitor is decided. EMC is defining closer competition based on IDC’s report on market share onPurpose-Built Backup Appliances (PBBA) and they are referring to IBM. They selected to compare IBM because they have the poorest number. The othervendors in the list with– HP at 25 TB/hr without Catalyst and Symantec at 23.7 TB/hr for its NetBackup 5220– have solutions superior to IBM! EMC cannot evenclaim 2x (let alone 6x) if the closest comparison was based on performance itself.

HP defined closest competitor in terms of the actual performance. They compared against EMC’s 31 TB/hr to make the 3 times faster claim with 100 TB/hr.

Verdict: Always ask questions on metrics! It is easy to make a claim while staying vague on details.

Not seeing your comments on this post? Please read this note.

Like 6 people like this. Be the first of your friends.

Tags:Boost, Catalyst, Data Domain, Deduplication, EMC, HP, StoreOnce

3/6/13 Backup Exec, NetBackup, VADP, VDDK, vStorage : Wait Wait Back it Up! NetBackup for 101 VMware Professionals

www.mrvray.com 7/11

8

8 Comments »

Will EMC BRS kill Avamar or NetWorker?

Published by Abdul Rasheed in Avamar, Data Domain, Data Protection, NetWorker on May 28th, 2012

EMC World 2012 has come and gone. For those watching the Backup and Recovery Services (BRS) division would notice a drastic shift in strategy since lastyear. Is Avamar counting its days?

Surprised? Let me explain. Remember the “Tape sucks! Move on!” Campaign sung by BRS last year? They even mocked Google for recovering from tapes.They wanted the world to look at Avamar and Data Domain, the two products with spinning disks as the houses of backups. The other child NetWorker wasmostly ignored and was on life support just to get by with the era of tapes.

BRS seems to have come to grip with the reality to some extent. The incremental updates to Avamar and revelation of NetWorker 8 features tend to indicatethat BRS is taking a 180-degree turn.

No real updates for Avamar Data Store: All the announced business critical applications support in Avamar are for both Data Domain Boost and Avamar

native client. Hyper-V that is popular among SMB workloads is now available through Boost to a Data Domain target. Last year, BRS’ announcement was thatDD is for specific work loads and Avamar Data Store is for everything else. Now Boost is getting more attention and Avamar engine by itself pretty much staysthe same. The blackout windows in Avamar Data Store already annoy customers. Data Domain deduplication engine is preferred for target dedupe and DDBoost will replace source side deduplication eventually? Inspired by Symantec’s Dedupe Everywhere strategy?

Note: Thank to Ian’s comment on clarifying that newer application support is available for Avamar as well. Not just for Data Domain through DD Boost.

Emergence of Media Access Node: BRS realized that customers with longer retention requirements would not buy in on ‘keep it on disk’ message. Tape

provides economies of scale. Modern tape technologies are superior in performance and reliability. Now, BRS ships a NetWorker node underneath the coveras Media Access Node in Avamar to copy rehydrated data into tape in NetWorker tape format.

NetWorker 8.0 getting some facelift: Although NetWorker was ignored in keynotes, BRS made a deliberate attempt this year to show what is happening to

NetWorker. It was expecting the morgue but now pulled back and is getting revved up. There is a long road ahead to convince customers, but BRS says it isputting equal number of resources on NetWorker as was done on Avamar. Not to mention about the newfound love, Spectralogic, to compete with IBM andOracle.

If you pay closer attention, all that Avamar got is to make things better for Data Domain (Boost expansion, multi-stream support…) and NetWorker (data storedin NetWorker tape format). In a nutshell, BRS wants everyone to keep backup data on either Data Domain dedupe format or NetWorker tape format. OnceNetWorker and Data Domain Boost combination can support backups through WAN, Avamar may not have anything to offer. From operating marginperspective, Avamar as a product may become a dog in BCG Growth-share matrix? The one eventually going to morgue looks to be Avamar Dedupe engine?

Not seeing your comments about this post? Please read this note.

Like 2 people like this. Be the first of your friends.

Tags:Deduplication

2 Comments »

Deduplication Storage Pool Reliability: The devil is in the details

Published by Abdul Rasheed in Data Domain, Data Protection, NetBackup on April 3rd, 2012

As you guys already know, I do travel a lot and attend trade shows where I represent Symantec. While I was briefing a visitor at Symantec booth on NetBackup5020 appliance, he asked a question which was quite interesting. “We have requested RFPs from multiple vendors for deploying deduplication solution forbackups. EMC sales team told us that Data Domain 800 series is better than NetBackup 5020 appliances in terms of reliability. They said that if one node in amulti-node NetBackup 5020 goes down, the entire deduplication pool goes down. What do you think about it?”

I thanked him for his question. I took a good 20 minutes to explain the situation. I thought it will be nice to document this in a blog for a fair comparison.

Let us compare configurations based on Data Domain 860 and NetBackup 5020. Let us say that the customer is looking to create 96TB of deduplication poolright now. He may need more storage in future.

With Data Domain 860, it would require four ES30 shelves (with 2TB drives) to create this capacity. Plus you need the 860 head unit. With NetBackup 5020,you would need three nodes.

3/6/13 Backup Exec, NetBackup, VADP, VDDK, vStorage : Wait Wait Back it Up! NetBackup for 101 VMware Professionals

www.mrvray.com 8/11

Implementing a 96TB deduplication pool

Thus, the EMC solution has a total of 5 components (1 head and 4 shelves). EMC’s 96TB deduplication pool will go down if any of the five components fail.

Symantec solution has a total of three components (3 NetBackup 5020 nodes). Symantec’s 96TB deduplication pool will go down if any of the threecomponents fail.

Observation 1: EMC solution has more single points of failure than Symantec’s solution for a given capacity.

Let us dig deeper. Let us look at the components that actually store data, the storage modules.

Each Data Domain ES30 shelf will have 15 spindles: 12 data drives, 2 parity drives and 1 hot spare. Each shelf can withstand 3 concurrent drive failures.

Each NetBackup 5020 nodes have 22 spindles (not counting the two drives in RAID1 for system disk): 18 data drives, 2 parity drives and 2 hot spares. Thisconfiguration can withstand four concurrent drive failures.

Both systems use SATA drives. The theoretical1 annualized failure rate (AFR) for a SATA drive is approximately 1.46%. Robin Harris’ StorageMojo2 blog hassome great information on a study done by Google. He quotes the idea of calculated AFR to be 2.88%

Since we are actually comparing the overall storage modules (ES30 storage shelf vs. NetBackup 5020 storage shelf), let us not worry about the absolute valueof AFR of a disk drive. For our discussion, let us assume that both Symantec and Data Domain are buying disks from the same manufacturer. Let the AFR be3% to simplify probability calculations.

An AFR of 3% indicates that the probability of a SATA drive to fail within a year is 3/100.

In case of Data Domain 860 with ES30 shelves, you will lose data if more than 3 drives fail in a year and failed drives were not replaced. The probability of four

drives failing in a year can be calculated using conditional probability3. The value is (3/100)4 = 0.000081%

In case of a NetBackup 5020 node, you will lose data if more than 4 drives fail in a year and were not replaced. The probability here is (3/100)5 =0.00000243%

Note the probability of data loss is low in both cases even if you don’t replace the failed drives for a year. This is why RAID6 and hot spare play a significant role

in delivering storage reliability. That is the main point I want to make here. However the probability of losing data on ES30 shelf is 33 times higher than the

probability of losing data in NetBackup 5020! The reason here is the extra hot spare that you have in NetBackup 5020 node that provides additionalprotection.

Observation 2: From storage module perspective, although the absolute probability of losing data is quite low for both EMC and Symantec solutions, the

relative probability of losing data on EMC’s ES30 shelf is 33 times higher than that in NetBackup 5020 if drives have identical AFR.

So don’t you disagree with what EMC sales rep has reportedly told about NetBackup 5020 appliances? The devil is always in the details, isn’t it?

Disclaimer: As I had already stated in About Me page in MrVRay.com, the thoughts expressed here are my own. My employer or school has not

endorsed/supported any of the content in this blog. If there are errors in this post, contact me at @AbdulRasheed127 on Twitter and I will be happy to correct it. Iam not entertaining comments until I invest in a good spam blocker, sorry for the inconvenience

References:

1. Annualized Failure Rate (AFR) and Mean Time between Failures (MTBF) in: Seagate Barracuda ES SATA Product Manual, Page 29, Chapter 2.12:Reliability

2. Robin Harris. Google’s Disk Failure Experience3. Conditional Probability: P(AB) = P(A)*P(B|A)

If A and B are independent outcomes, P(B|A) = P(B)

In which case, P(AB) = P(A) * P(B)

3/6/13 Backup Exec, NetBackup, VADP, VDDK, vStorage : Wait Wait Back it Up! NetBackup for 101 VMware Professionals

www.mrvray.com 9/11

8

8

Like 2 people like this. Be the first of your friends.

Tags:Backup Exec, Data Domain, Deduplication, NetBackup, OpenStorage

No Comments »

vSphere changed block tracking: A powerful weapon for backup applications to shrink backup

window

Published by Abdul Rasheed in Avamar, Backup Exec, Data Protection, NetBackup, VMware, vSphere, vStorage on March 12th, 2012

Changed block tracking is not a new technology. Those who have used Storage Foundation for Oracle would know that VERITAS file system (VxFS) provides

no-data check points which can be used by backup applications to identify and backup just the changed blocks from the file systems where database files arehoused. This integration was in NetBackup since version 4.5 that was released 10 years ago! It is still used by Fortune 500 companies to protect missioncritical Oracle databases that would otherwise require a large backup window with traditional RMAN streaming backups.

VMware introduced change block tracking (CBT) since vSphere 4.0 and is available for virtual machines version 7 or higher. NetBackup 7.0 added support forCBT right away. Backing up VMware vSphere environments got faster. When a VM has CBT turned on, it can track changes to virtual machine disk (VMDKs)sectors. Its impact on VM performance is marginal. Backup applications with VADP (vStorage APIs for Data Protection) support can use an API (named

QueryChangedDiskAreas) to identify and copy changed blocks from a particular point in time. This time point is identified using an argument named ChangeId

in the API call.

VMware has made this quite easy for backup vendors to implement. Powerful weapons can be dangerous when not used with utmost care. An unfortunateproblem in Avamar’s implementation of CBT came to light recently. I am not picking on Avamar developers here, it is not possible to predict all the edge casesduring development and they are working hard to fix this data loss situation. As an engineer myself, I truly empathize with Avamar developers for gettingthemselves into this unfortunate situation. This blog is a humble attempt to explain what had happened as I got a few questions from the field seeking input onthe use of CBT after the EMC reported issues in Avamar.

As we know, VADP lets you query the changed disk areas to get all the changes in a VMDK since a point in time corresponding to a previous snapshot. Oncethe changed blocks are identified, those blocks are transferred to the backup storage. The way the changed blocks are used by the backup application tocreate the recovery point (i.e. backup image) varies from vendor to vendor.

No matter how the recovery point is synthesized, the backup application must make sure that the changed blocks are accurately associated with the correctVMDK because a VM can have many disks. As you can imagine if the blocks were associated with the wrong disk in backup image; the image is not anaccurate representation of source. The recovery from this backup image will fail or will result in corrupt data on source.

The correct way to identify VMDK is using their UUIDs which are always unique. Using positional identifies like controller-target-LUN at the VM level are notreliable as those numbers could change when some of the VMDK are removed or new ones are added to a VM. This is an example of disk re-order problem.This re-order can also happen for non-user initiated operations. In Avamar’s case, the problem was that the changed blocks belonging one VMDK was gettingassociated with a different VMDK in backup storage on account of VMDK re-ordering. Thus the resulting backup image (recovery point) generated did notrepresent the actual state of VMDK being protected.

To make the unfortunate matter worse, there was a cascading effect. It appears that Avamar’s implementation of generating a recovery point is to use the

previous backup as the base. If disk re-order happened after nth backup, all backups after nth backup are affected on account of the cascading effect becausenew backups are inheriting the base from corrupted image.

This sounds scary. That is how I started getting questions on reliability of CBT for backups from the field. Symantec supports CBT in both Backup Exec andNetBackup. Are Symantec customers safe?

Yes, Symantec customers using NetBackup and Backup Exec are safe.

How do Symantec NetBackup and Backup Exec handle re-ordering? Block level tracking and associated risks were well thought out during the implementation.Implementation for block level tracking is not something new for Symantec engineering because such situations were accounted for in the design forimplementing VxFS’s no-data check point block level tracking several years ago.

There are multiple layers of resiliency built-in Symantec’s implementation of CBT support. I shall share oversimplified explanations for two of those relevant inensuring data integrity that are relevant here.

Using UUID to accurately associate ChangeId to correct VMDK: We already touched on this. UUID is always unique and using it to associate the previous

point in time for VMDK is safe. Even when VMDKs get re-ordered in a VM, UUID stays the same. Thus both NetBackup and Backup Exec always associatethe changed blocks to the correct VM disk.

Superior architecture that eliminates the ‘cascading-effect’: Generating a corrupted recovery point is bad. What is worse is to use it as the base for

newer recovery points. The corruption goes on and hurt the business if left unnoticed for long time. NetBackup and Backup Exec never directly inject changedblocks to an existing backup to create a new recovery point. The changed blocks are referenced separately in the backup storage. During a restore,NetBackup recreates the point in time during run-time. This is the reason NetBackup and Backup Exec are able to support block level incremental backupseven to tape media! Thus a corrupted backup (should that ever happen) never ‘propagates’ corruption to future backups.

Like 4 people like this. Be the first of your friends.

Tags:Backup Exec, BLIB, CBT, NetBackup, VADP, VMware, vSphere, vStorage

No Comments »

Introduction to VMware vStorage APIs for Data Protection aka VADP

Published by Abdul Rasheed in Data Protection, NetBackup, NetBackup 101 for VMware Professionals, VMware, vSphere, vStorage on February 21st, 2012

3/6/13 Backup Exec, NetBackup, VADP, VDDK, vStorage : Wait Wait Back it Up! NetBackup for 101 VMware Professionals

www.mrvray.com 10/11

8

6. Getting to know NetBackup for VMware vSphere

Note: This is an extended version of my blog in VMware Communities: Where do I download VADP?

Now that we talked about NetBackup master servers and media servers, it is time to get into learning how NetBackup Client on VMware backup host(sometimes known as VMware proxy host) protects the entire vSphere infrastructure. In order to get there, we first need a primer on vStorage APIs for DataProtection (VADP) from VMware. We will use two blogs to cover this topic.

Believe it or not, this question of what VADP really is comes up quite often in VMware Communities, especially in Backup & Recovery Discussions

Backup is like an insurance policy. You don’t want to pay for it, but not having it is the recipe for sleepless nights. You need to protect data on your virtualmachines to guard against hardware failures and user errors. You may also have regulatory and compliance requirements to protect data for longer term.

With modern day hardware and cutting edge hypervisors like that from VMware, you can protect data just by running a backup agent within the guest operatingsystem. In fact, for certain workloads; this is still the recommended way.

VMware had made data protection easy for administrators. That is what vStorage APIs for Data Protection (VADP) is. It available since vSphere 4.0 release. Itis a set of APIs (Application Programming Interfaces) made available by VMware for independent backup software vendors. These APIs make it possible forbackup software vendors to embed the intelligence needed to protect virtual machines without the need to install a backup agent within the guest operatingsystem. Through these APIs, the backup software can create snapshots of virtual machines and copy those to backup storage.

Okay, now let us come to the point. As a VMware administrator what do I need to do to make use of VADP? Where do I download VADP? The answer is…Ensure that you are NOT using hosts with free ESXi licenses.

1. Ensure that you are NOT using hosts with free ESXi licenses2. Choose a backup product that has support for VADP

The first one is easy to determine. If you are not paying anything to VMware, the chances are that you are using free ESXi. In that case, the only way to protectdata in VMs is to run a backup agent within the VM. No VADP benefits.

Choosing a backup product that supports VADP can be tricky. If your organization is migrating to a virtualized environment, see what backup product iscurrently in use for protecting physical infrastructure. Most of the leading backup vendors have added support for VADP. Symantec NetBackup, SymantecBackup Exec, IBM TSM, EMC NetWorker, CommVault Simpana are examples.

If you are not currently invested in a backup product (say, you are working for a start-up), there are a number of things you need to consider. VMware has a freeproduct called VMware Data Recovery (VDR) that supports VADP. It is an easy to use virtual appliance with which you can schedule backups and store it indeduplicated storage. There are also point products (Quest vRanger, Veeam Backup & Replication etc.) which provide additional features. All these productsare good for managing and storing backups of virtual machines on disk for shorter retention periods. However, if your business requirements need long termretention, you would need another backup product to protect the backup repositories of these VM only solutions which can be a challenge. Moreover, it is lessunlikely to see businesses that are 100% virtualized. You are likely to have those NAS devices for file serving, desktops and laptops for end users and so on. Hence a backup product that supports both physical systems and VADP are ideal in most solutions.

Although VADP support is available from many backup vendors, the devil is in the details. Not all solutions use VADP the same way. Furthermore, manyvendors add innovative features on top of VADP to make things better. We will cover this next.

Back to NetBackup 101 for VMware Professionals main page

Next: Coming Soon!

Like Be the first of your friends to like this.

Tags:NetBackup, VADP, VMware, vStorage

1 Comment »« Older Entries

Most Recent Posts

What’s up with VADP backups and VDDK on vSphere 5.1?Dear EMC Avamar, please stop leeching from enterprise vSphere environmentsVMware announces vSphere Data Protection (VDP), what is in it for you?Client Direct: NetBackup vs. NetWorkerWhat do NetApp ONTAP and Symantec NetBackup have in common?

Categories

AvamarBackup ExecCommVaultData DomainData ProtectionNetBackupNetBackup 101 for VMware ProfessionalsNetWorkervCloudVeeamVMwarevRangervSphere

3/6/13 Backup Exec, NetBackup, VADP, VDDK, vStorage : Wait Wait Back it Up! NetBackup for 101 VMware Professionals

www.mrvray.com 11/11

vSphere Data ProtectionvStorage

RSS Feed

What’s up with VADP backups and VDDK on vSphere 5.1?Dear EMC Avamar, please stop leeching from enterprise vSphere environmentsVMware announces vSphere Data Protection (VDP), what is in it for you?Client Direct: NetBackup vs. NetWorkerWhat do NetApp ONTAP and Symantec NetBackup have in common?

NetBackup on Twitter

Archives

November 2012October 2012August 2012July 2012June 2012May 2012April 2012March 2012February 2012December 2011November 2011

© Mr VRay

About MeComments?NetBackup 101 for VMware ProfessionalsOn Connect

Page optimized by WP Minify WordPress Plugin

NetBackup

NetBackup

Join the conversation

NetBackup Which #DR Strategy is More Efficient:#NetBackup AIR or Sealed Store? Check this blogout bit.ly/13PW1oP... fb.me/1yNbJdwdj7 day s ago · reply · retweet · fav orite

NetBackup RT @kmitchel: Come see how the@NetBackup Analyzer tool can help partnersuncover huge $$$ - come to the @symantecbooth @VMwarePEX to s ...7 day s ago · reply · retweet · fav orite

NetBackup GREAT BLOG: #waytooforwardprotection w/ #NetBackup Appliances pushes#VMware VADPs to the limit... fb.me/1lJsHuA0M6 day s ago · reply · retweet · fav orite

NetBackup Symantec's Abdul Rasheed's talks aboutthe joint benchmark with @Cisco, @NetApp and@VMware at #VMwarePEX... fb.me/2br876Xl56 day s ago · reply · retweet · fav orite

NetBackup RT @NetApp: [Video] @Symantec's