wNew software versions are often plagued by bugs.

Symantec has fixed numerous bugs in VeritasNetBackup 6.0 as some users have had major

problems upgrading to the newest version of thepopular backup application. By Jo Maitland

cover story

Photograph by Justin Hession

hen symantec corp. released the latest upgradeto Veritas NetBackup 6.0—Maintenance Pack 3 (MP3)—on June 30, Graeme Hackland took a deep breath. AsIT manager for the Renault Formula One (F1) Team inEnstone, U.K., he uses Veritas NetBackup 6.0 to protect80TB of storage that supports the company’s F1 racingcars on the track and during the manufacturing process.

“It’s working well for me now with Maintenance Pack2 [MP2], so unless there’s a compelling reason for meto go with MP3, I’m not going to rush into it,” saysHackland. He has no plans to install MP3, despite therecent e-mail alert from Symantec urging customers toupgrade right away. His cautious approach is born ofexperience.

anatomy of an upgrade:Veritas NetBackup6.0

September 2006 Storage 21

Although he thinks NetBackup OperationsManager (the new reporting tool in VeritasNetBackup Version 6.0) is an improvement,Flavio Hürlimann, IT specialist at Sunrise/TDC Switzerland AG in Zurich, is stickingwith a third-party reporting tool.

22 Storage September 2006 Photograph by Steve Double

By the time the Renault F1 Team upgraded fromVeritas NetBackup 5.1 to 6.0 MP2 in May, the producthad been shipping for seven months. “We tend not tojump on new revisions, and I was a little nervous aboutNetBackup 6 because there had been quite a lot of badpress,” says Hackland. “I wanted to be sure it was theright thing to do.”

Hackland is referring specifically to issues with thenew online (hot) catalog backup feature in VeritasNetBackup 6.0, which many early adopters reportedwas shaky at best and caused data corruption issues inthe worst cases. The catalog is the heart of the back-up server; it tracks all the backup jobs, file changes, allof the tapes and who has access to what information.

In previous versions of Veritas NetBackup, clientbackups couldn’t run when backing up the catalog,which meant users had to schedule a time during theday to specifically back up their catalog. With VeritasNetBackup 6.0’s hot catalog backup feature, a catalogcan be backed up at any time, even when multipleclient backups are running. This makes schedulingeasier and it’s much less tedious to back up the cata-log more frequently.

To enable this new feature, Symantec rearchitectedVeritas NetBackup, moving from a flat-file catalog to aSybase database catalog to improve scalability. In the oldflat-file environment, users had to read through the filesto find the information they needed; the Sybase data-base architecture lets them perform queries against thecatalog. “It allows them to scale up to much larger en-vironments,” says Matt Kixmoeller, Symantec’s seniordirector of product management, Veritas NetBackup.

Acxiom Corp., an IT services company in Little Rock,AR, upgraded from Version 5.1 to Version 6.0 Mainte-nance Pack 1 (MP1) on March 21. There were addi-tional scripts to convert everything from flat files to thedatabase format, but that was the least of their worries,says Mark Lutgen, team leader, Unix engineering atAcxiom. It took the company 20 hours to back up a50GB catalog, a process that would normally take 90minutes. A bug in the integration between the vault-ing feature in Veritas NetBackup—which automatesthe process of shipping tapes offsite to a disaster re-covery (DR) facility—and the hot catalog backup fea-ture caused the vault to keep running, leaving Acxiomno time to run its backups. It also meant the companyhad to hand-pick the tapes to be sent offsite.

Acxiom had “major issues with MP1,” says Lutgen.“Symantec allows you to run powerful commandsagainst their catalog, but you better know whatyou’re doing or it can cause corruptions.”

Acxiom also ran into configuration issues during aDR test at its recovery center that delayed client restoreby several hours. These problems were related to anothernew feature in Veritas NetBackup 6.0, the EnterpriseMedia Manager (EMM), which consolidates VeritasNetBackup media and device databases onto oneserver to simplify recovery in a DR scenario. One is-sue had to do with some aliases Acxiom had configuredat its home site that weren’t configured at the recov-ery site. The other issue was a switch set in the EMMdatabase that indicated Veritas NetBackup was in anupgrade process and prevented further configuration(adding devices such as tape robots or tape drives). Totheir credit, says Lutgen, Symantec technicians re-solved the problems during Acxiom’s DR test. “Theproblems were 6.0 related … the fix involved us issu-ing commands against the new database,” he adds.

Fortunately, MP2 was released a week after Acxiominstalled MP1, which fixed the catalog backup andvault bug the company discovered. “We are operationaland in no rush to upgrade to MP3,” notes Lutgen.

Version 6.0 changes the catalog recovery processof Veritas NetBackup so significantly that many cus-tomers are still performing an offline (cold) catalogbackup in case the online catalog recovery doesn’twork or, worse, causes data corruption issues.

Warning signsAfter hearing similar stories, Hackland decided to per-form his upgrade to MP2 in two steps. “We felt that wassafer,” he says. He upgraded the master server (which in-cludes the catalog) in the first part of the week. “That’sthe one thing that worried me because I’d heard about

GRAEME HACKLAND, IT manager for the RenaultFormula One Team in Enstone, U.K., isn’t in abig rush to upgrade toVeritas NetBackup 6.0Maintenance Pack 3(MP3). “I wanted to besure it was the rightthing to do,” he says.

1 VERIFY CATALOG CONSISTENCY. Run the Symantec Veritas Net-Backup catalog consistency tool and work with either Syman-tec support or consulting to resolve any inconsistencies.

2 DO A PRE-UPGRADE CATALOG BACKUP. Capture a catalog backupprior to the upgrade.

3 UPDATE THE CONFIG FILE. Remove configuration file referencesto media servers that no longer exist.

4 CHECK THE GLOBAL DATABASE. Review global database infor-mation (vmglob) to determine if there are host names thatare no longer media servers.

5 ENSURE HARDWARE COMPATIBILITY. Verify that hardware run-ning Version 4.5/5.x is supported under Version 6.0 by check-ing the support matrix. Also, ensure the server hardware hasenough processor power and memory (a minimum of 0.5GBfor Windows and 1GB for Unix).

Source: Symantec Corp.

Top five best practices for upgrading to Veritas NetBackup 6.0

Ad pagegoes here

25

24 Storage September 2006

issues with the catalog,” he says. However, he notes,the firm’s catalog isn’t particularly big (approximately100MB). “If there were going to be major problems, Ithought it was going to be at that stage.”

Not so, but there were other surprises. During the up-grade process, users must disable all their policies, per-form the upgrade and then re-enable the policies. TheRenault F1 Team discovered a problem when it cameto reactivating its policies. A script that should have re-activated all the policies didn’t work, so an admin hadto reactivate all the policies manually. It ended up tak-ing almost a day to upgrade the master server, consid-erably longer than the company had expected.

Engineers then performed three test backups andthree test restores, which all worked fine. “Off theywent home, happy as Larry,” says Hackland. But whenthey returned to work the following day, they discov-ered that none of the scheduled backups had run. “I amstill not entirely clear what the issue was,” says Hack-

land. Symantec told him it was due to a “storage unitproblem,” which is a part of Veritas NetBackup. He saysSymantec engineers corrected the issue that day bystopping and restarting NetBackup, which they haddone a number of times during the installation.

“That was a little nerve-wracking, coming in the nextmorning and getting the report that none of the back-ups had run,” he says. “The rest of the upgrade that weekwas fine … It took an age to import the media database,but there were no other issues on that day.”

NetBackup Operations ManagerThe final part of Renault F1’s upgrade was to replaceAdvanced Reporter and Global Data Manager with thenew reporting tool in Veritas NetBackup 6.0 calledNetBackup Operations Manager (NOM).

“NOM is much easier to use; it’s easier to see if back-ups have failed,” says Hackland. However, there’s no mi-gration path between Advanced Reporter and NOM, soall of the historical data collected by Advanced Reporteris lost once NOM is installed. “I wish there had beensome way of archiving the old Advanced Reporter stuff… It’s not a major deal because we have the informationanyway, but it seems a strange thing that you couldn’tgo back and look at the historical logs,” he says.

According to Symantec, Advanced Reporter wasn’tserving the needs of its users and had to be over-hauled. “It was a design question. Did we want fea-tures x, y and z, or did we want a very tight link in withthe past?” says Mike Adams, Symantec’s senior groupmanager, product marketing, Veritas NetBackup. “Wethought it best to add feature sets to NOM [rather]than to create a vast link from Advanced Reporter toOperations Manager.”

Symantec’s Kixmoeller adds that historical data isreally only used to see how well the current backupenvironment is performing. “When you move to [Net-Backup] 6, you have changed your environment andyou’ll want to look at a new baseline of data,” he says.

Flavio Hürlimann, IT storage specialist at Sunrise/TDC Switzerland AG, a major telecom provider inZurich, says NOM is an improvement over the previ-ous reporting tools in Veritas NetBackup, but he feelsit still doesn’t meet requirements. Because it’s Java-based, it takes a while to build screens, “so you’re notsure if you have the most current information,” he says.Sunrise/TDC Switzerland uses a reporting tool fromAgite Software AG called backupVisual, which Hürli-mann says creates more detailed reports and presentsdata more graphically than NOM.

Symantec is aware of the need for more detailedreporting on Veritas NetBackup and is adding more

experts advise users to delay upgradingSymantec Corp. isn’t the first company to rearchitect its prod-

uct from a flat-file to a database architecture. Microsoft Corp.

attempted a similar feat with WinFS, in which it was integrat-

ing unstructured data into a relational database. WinFS

reached beta after several years of development, but Microsoft

canned the project in June 2006. In a blog posted on the Mi-

crosoft Web site, the company announced it would no longer

be “pursuing a separate delivery of WinFS,” instead choosing

to integrate it into the next release of SQL Server.

“Converting a basic file system into a more organized structure

like a database is no mean feat,” says Ash Ashutosh, CTO of

Hewlett-Packard Co.’s storage management software group and

founder of AppIQ. “Everyone understands files, but databases are

a whole different beast,” he says. “The tools are a lot geekier.”

In Symantec’s case, the company picked the Sybase database

because other Symantec products use it, but also because

Sybase made its product look like a file system to the app, ac-

cording to Ashutosh. He says this would have meant fewer

changes to the Veritas NetBackup code, but significant testing

between NetBackup and the Sybase database. Ashutosh says

Symantec should have kept Veritas NetBackup Version 6.0 in “QA

and test for a good 24 months.” Symantec shipped NetBackup 5.1

on June 7, 2004, and NetBackup 6.0 on October 3, 2005. (HP’s

OpenView Storage Data Protector product competes with

Symantec’s Veritas NetBackup.)

According to industry analysts, all software companies,

especially the larger players, are under considerable internal

pressure to meet release dates that have more to do with

meeting quarterly earnings than releasing a solid product.

There’s also pressure from customers demanding new fea-

tures. The upshot is buggier software. Experts advise not to

upgrade to a new release until absolutely necessary.

26 Storage September 2006

functionality to NOM. Furthermore, Symantec plansto improve the logging feature that records all of theevents happening in Veritas NetBackup. Version 6.0adds more detail to the logging records to aid trou-bleshooting, according to Symantec. But beware: Thelogs take up much more disk space because they’regathering more information.

“There have been some isolated issues of customersrunning out of disk space and that’s causing particu-lar issues,” says Symantec’s Kixmoeller.

In these instances, Symantec offers two options: Cre-ate a larger disk pool for the logs if you need that levelof detail or wait for Maintenance Pack 4 (MP4), whichwill let you turn down the verbosity of the logs to cre-ate more space. “It comes down to whether the userwants more information or to be more conservative withdisk space,” says Kixmoeller. MP4 is expected in the fall.

Some users have seen their EMM databases cor-rupted when logging is turned on and the disk fills up.The MP3 alert from Symantec advises users to run log-ging on a dedicated file system to prevent this cor-ruption. Users report that reading the logs takes daysnow rather than hours because they’re so long.

Help arrives onsiteOne user, who requested anonymity because his com-pany has a policy of not endorsing or critiquing productsfrom vendors with which it has relationships, spent daysgoing through logs to figure out where his Veritas Net-Backup implementation was failing. “Nothing gets done[to fix the problem] until you send the logs. In the end,Veritas couldn’t replicate our problems in their envi-ronment, so they sent people to our place,” he says.

The company has 170 master servers—60 runningVeritas NetBackup 6.0 MP2—and almost 2 petabytes(PB) of storage under management. After running intomajor catalog consistency and corruption problemsduring the upgrade process, the company opted tospend $20,000 for a Veritas consultant who found allthe discrepancies on its master servers and was thenable to proceed with the upgrade. “That was a specialdeal for us as we are a large customer,” notes the user.

According to Symantec, a free tool called NetBackupCatalog Consistency (NBCC) can be used prior to anupgrade to Veritas NetBackup 6.0 to check the catalogconsistency. That tool wasn’t available when this userupgraded in mid-March (see “Top five best practicesfor upgrading to Veritas NetBackup 6.0,” p. 22).

The user plans to install MP3, and hopes it will fix ashopping list of other problems, including an issue withmultistreaming, “which was the premier feature ofNetBackup and now we can’t even use it anymore.”

Symantec’s response: When using Veritas NetBackup6.0, 6.0 MP1 and 6.0 MP2, if a multiplexed backupreceives an end of media message—and before it canget new media—another backup sent to the samedrive will fail and there’s the potential for data loss.The company says MP3 fixes this problem.

The user also went through considerable pain wheninstalling Veritas CommandCentral agents on VeritasNetBackup servers, which caused NetBackup to stoprecognizing the company’s tape drives. They had to com-pletely uninstall (rather than disable) the agents to fixthe problem, which caused backups to fail for a day.

“The biggest concern that we want to see fixed is theerror-code 200 problems that we’re having,” says theuser. “If a parent job fails or hangs, nothing else runs.Also, if a manual backup is kicked off, the policy willnot resume running the next day for some reason.”

Growing bug listThe list of bugs goes on and on. So far, Symantec hasfixed 100 bugs in MP3, which a spokesperson says “isvery typical for a quarterly maintenance pack and aproduct that has as many capabilities and supportedcomponents as NBU [NetBackup].” It’s no surprise manyusers are holding off upgrading to Version 6.0 on theadvice of Veritas NetBackup resellers, who are tellingpeople to wait until Version 6.5 (expected in the firstquarter of next year) for the product to become stable(see “Experts advise users to delay upgrading,” p. 24).

Symantec declined to comment on how many bugsit has discovered in Veritas NetBackup Version 6.0. “Wedon’t get an accurate report on the number of bugs,”says the firm’s Adams. “When any new release comesout, there will be any number of bugs … We nevergot any highlight that there were any more [bugs] with[Version] 6 than with previous versions of NetBackup.”

“There are substantial leaps forward [in VeritasNetBackup 6.0],” adds Symantec’s Kixmoeller, “butit’s important to realize they’re substantial changes, andit takes some investment on the customer’s part to learnabout the new tools and properly plan for the upgrade.”

A former Veritas engineer who quit Symantec inMarch because of the Veritas NetBackup problems,says Version 6.0 was probably released too early andwas still essentially a beta product.

“There were several Sybase bugs, with config files get-ting spontaneously corrupted; the worst part was thatthe techs weren’t adequately trained on Sybase to un-derstand the problems,” she says, preferring to remainanonymous because she still works in the industry. 2

Jo Maitland is the News Director at SearchStorage.com.

“IT TAKES SOME INVESTMENT ON THE CUSTOMER’S PART TO LEARN ABOUT THE NEW TOOLS AND PROPERLY PLAN FOR

THE UPGRADE,” SAYS A SYMANTEC SPOKESPERSON.