1
IBM Software Group
August 2003 © 2003 IBM Corporation
Considerations for using TSM in a SAN
Steve Strutt, Tivoli Software, IBM [email protected]
Sept 2002Revision 5
IBM Software Group | Tivoli software
Considerations for using TSM in a SANl © 2003 IBM Corporation2
AgendaSAN Exploitation - LAN-Free backupPerformance characteristics
RequirementsLAN, Hardware, Software, Device, dependencies
SAN considerationsDevice fail-overHBA considerationsSAN design considerationsDevice addressing considerations
Going LiveTesting, Diagnosing ProblemsHints and Tips
Question and Answer
2
IBM Software Group | Tivoli software
Considerations for using TSM in a SANl © 2003 IBM Corporation3
LAN-free BackupAdvantages:- client data can be local or SAN-attached- transparent to application/database- takes backup traffic off the LAN- reduces CP cycles on backup server (no I/O)- faster speed (usually)- only one backup server needs administration
Disadvantages:- still requires CP cycles on client for backup I/O- careful scheduling to avoid tape drive contention
(or exploit disk pooling)
TAPE
FC device
DISK
LAN
D A TAD A TA
DATA
Direct to tape
Disk pool staging
IBM Software Group | Tivoli software
Considerations for using TSM in a SANl © 2003 IBM Corporation4
Performance characteristicsLAN-Free is not necessarily faster
Only network eliminated as bottleneck
Could be other bottlenecksTape drives, Disk subsystemData types
good performance for large files and databasessmall files, performance limited by file system and TSM
architectureLAN-Free to tape
Potentially better performance for large files, as bottleneck becomes file system or tape device
Small files cause tape drives to stop-start more and drives drop out of streaming mode.
LAN-Free to diskIdeal for small files, no stop-start overhead
3
IBM Software Group | Tivoli software
Considerations for using TSM in a SANl © 2003 IBM Corporation5
Customer Performance FiguresLarge UK High Street Retailer
TDP for SQL Server on ESS to 3584 LTO Backup 61.4GB hour (17MB/s) to single driveRestore 44.2GB hour (12.3MB/s) from single drive
NT Filesystem on ESS disk to 3584 LTO - small filesBackup 10GB hour to LTO Restore 6.2GB hour from LTO Could be slower than LAN if tape drives do more stop/start operations
Large UK BankTDP for Exchange to 3583 LTO
Backup 52GB/h (14.4M B/s) to single driveRestore 51GB/h (14.2MB/s) from single drive
IBM Software Group | Tivoli software
Considerations for using TSM in a SANl © 2003 IBM Corporation6
Agenda - Requirements
Hardware – LAN
– Library support for LAN-Free
– SAN device supportSoftware
– Evolving TSM support for LAN-Free
– TSM code dependencies
4
IBM Software Group | Tivoli software
Considerations for using TSM in a SANl © 2003 IBM Corporation7
Requirements - LAN
LAN-Free still requires LAN for meta dataFor Large files and databases
Minimal usage Small files
Maybe the same if not more meta data on LAN than data on SAN if files are very small.
LAN performance and loading still important
IBM Software Group | Tivoli software
Considerations for using TSM in a SANl © 2003 IBM Corporation8
SAN Device support
Initially hardware configurations were certified by Tivolimany different combinations, not all could be tested
Common configurations now testedThe Tivoli view is that hardware is transparent to TSM
If the hardware vendors have validated the devices’ use in a SAN and TSM supportsthe device, then it is supported in a SAN
configuration with Tivoli Storage Manager
5
IBM Software Group | Tivoli software
Considerations for using TSM in a SANl © 2003 IBM Corporation9
Library support for LAN-FreeNative TSM LAN-Free support for:
– 3494 Libraries
– SCSI Libraries– Controlled via SCSI control path
– SCSI connect– Fibre Channel Connect
NO Native TSM LAN-Free support for– STK and ADIC AML libraries
– Require Gresham EDT on every TSM server and Storage Agent. – STK - ACSLS, Library Station (s390)– ADIC – DAS for AML/2, AML/J
IBM Software Group | Tivoli software
Considerations for using TSM in a SANl © 2003 IBM Corporation10
TSM evolving support for LAN-FreeLAN-Free supports
BA Client, file level
BA Client, Volume level/Image BackupPrior to 5.1.5, the tape was rewound between each volume
No support for Backup Set restoreNT/W2K
Supported from 4.1.0, NT/W2K server and NT/W2K TDPs
4.2.1 for Backup Archive ClientSun Solaris and AIX
Supported from 4.2.0, TDPs and Backup Archive ClientHP-UX
Supported from 5.1.0, Full TSM device driver supportLinux (x86, zLinux and zOS)
Supported from 5.2.0
6
IBM Software Group | Tivoli software
Considerations for using TSM in a SANl © 2003 IBM Corporation11
TSM code dependencies
TSM 4.2 and 5.1– TSM Server and Storage Agent code MUST be at the same PTF and
patch level
– Restricts ability to roll out new codeTSM 5.2
– TSM Server and Storage Agent code only dependant at version and release level
– Independent of PTF level
– Easier to deploy and install maintenance
IBM Software Group | Tivoli software
Considerations for using TSM in a SANl © 2003 IBM Corporation12
Agenda - SAN considerationsHigh Availability considerations
– SANs offer high availability
– Tape high availability options
– Fail over and tapeHBA and tape setupHBA SharingSAN design considerationsAttaching tape drives to SANsSAN Operational considerationsDevice addressing considerations
– Managing device addressing– Persistent Naming– Static Device Naming
– Mapping device names to devices
7
IBM Software Group | Tivoli software
Considerations for using TSM in a SANl © 2003 IBM Corporation13
SAN’s offer high availabilityWorks well for disk
– Arrays designed to have two access paths
– Device drivers exploit multiple paths– Failover of paths
– Clustering software fails over applications and disk storage
– Effective TSM support for fail-over– DB and Log Mirrors, DB page shadowing– support for MSCS and HACMP
Does not work well for tape– Tape drives usually single connection
– SCSI Controlled tape libraries usually have a single control point
– Drivers only exploit single path
– Failover issues
Clustering
Redundancy Multiple paths
IBM Software Group | Tivoli software
Considerations for using TSM in a SANl © 2003 IBM Corporation14
Tape high availability options
3590 Dual SCSI/Fibre ports– AIX ATAPE driver can failover to second pathIf the primary path, say rmt0 is not available, the driver will use other paths/device
names via an alternative HBA/device port to access the same device. This is described in the 3590 manual, ‘IBM TotalStorage Tape Installation and User
Guide’. ftp://ftp.software.ibm.com/storage/devdrvr/Doc/
358x LTO libaries– New option to have two library controller LUNs on different paths
– Use two different HBA’s/Switches– Only supported with AIX ATAPE driver– Still single path to drives
– Divide drives between two HBA’s and switches– Still access library, reduced number of drives
Good news for AIX
8
IBM Software Group | Tivoli software
Considerations for using TSM in a SANl © 2003 IBM Corporation15
Fail-over and TapeSCSI Reserve/Release used to serialize access to drives
– Maintains integrity if device definitions are wrongOutstanding SCSI Reserves remain if server using drives go downDrives only released if:
Failed server restarted
Drive powered off and on.
SCSI Reset issued
Potential Impact on other users
SCSI Attached Tape drives can be reset using SCSI Reset
Fibre attached tape drives do not honor SCSI Resets
IBM Software Group | Tivoli software
Considerations for using TSM in a SANl © 2003 IBM Corporation16
Supported SCSI Fail-over configurationsTSM 4.2+ with MSCS on W2K
– Devices registered in wizard, SCSI reset on fail-overTSM 5.1.5 with HACMP on AIX
– Devices defined in ‘startserver’ script, issues verifydevice to reset device
SharedDisk
Shared Tape
Shared SCSIBus
TSM Serverin Cluster
TSM Clients
IP network
9
IBM Software Group | Tivoli software
Considerations for using TSM in a SANl © 2003 IBM Corporation17
Supported Fibre Failover configurations
TSM Serverin HACMP cluster
TSM 5.2.0 with HACMP on AIX– Devices defined in ‘startserver’ script, issues verifyfcdevice to reset device
IBM Software Group | Tivoli software
Considerations for using TSM in a SANl © 2003 IBM Corporation18
Fail-over and LAN-Free and Tape Library SharingOutstanding SCSI Reserves remain if storage agents using drives go downDrives become unavailable to other LAN-Free/Tape Sharing users
Only released if:Failed server restartedDrive powered off and on.
Potential drive mapping issuesFC Reset issued
Potential Impact on other users
Potentially need more tape drives
Some horrible messages if TSM server tries to dismount tapes belonging to failed storage agents.
10
IBM Software Group | Tivoli software
Considerations for using TSM in a SANl © 2003 IBM Corporation19
Failure scenario
ANR8925W Drive DRIVE0 in library ATLP1000 has not been confirmed for use by server UKSAN1_SA for over 1200 seconds.Drive will be reclaimed for use by others.ANR8336I Verifying label of DLT volume 00157D in drive DRIVE0 (MT6.1.0.1).ANR8311E An I/O error occurred while accessing drive DRIVE0 (MT6.1.0.1) for SETMODE operation, errno = 1.ANR8355E I/O error reading label for volume 00157D in drive DRIVE0 (MT6.1.0.1).ANR8311E An I/O error occurred while accessing drive DRIVE0 (MT6.1.0.1) for OFFL operation, errno = 1.ANR8469E Dismount of DLT volume 00157D from drive DRIVE0 (MT6.1.0.1) in library ATLP1000 failed.ANR9999D mmsscsi.c(1647): ThreadId<48> Volume may still be in the drive DRIVE0 (MT6.1.0.1).ANR8446I Manual intervention required for library ATLP1000.
Server running Storage Agent goes down (hardware failureFibre loss), when using a tape device
TSM makes drive ‘unavailable’
IBM Software Group | Tivoli software
Considerations for using TSM in a SANl © 2003 IBM Corporation20
Supporting LAN-Free fail-overNo TSM support for fail-over of Storage AgentSupporting Storage Agent Fail-over
Configure two storage agents, one on each server
Different Storage Agent NamesNo dependency between TSM clients and Storage Agent
BA client and TDP failover, Storage Agent does not
BA client and TDP automatically use new SA on fail-over server
No device issues if LAN-Free client machine fails-over when not using tape devices
Devices lost if LAN-Free client machine fails-over when using tape drives
Require more tape drives
Procedures to recover lost drives
11
IBM Software Group | Tivoli software
Considerations for using TSM in a SANl © 2003 IBM Corporation21
HBA and tape Setup considerationsCheck configuration recommendations of tape hardware vendor
– IBM recommendations
NT/W2KMax Scatter-Gather must be set to 65 or greater
Unable to write to new tapes on Storage AgentW2K creates tapes which cannot be readTSM will check for this in 4.2.1.11 and higher levels.
Issues message, unable to use drive. W2K and LTO use 5.0.2.4 or higher level of Ultrium LTO driver.
IBM Software Group | Tivoli software
Considerations for using TSM in a SANl © 2003 IBM Corporation22
HBA SharingSharing of disk and tape on same HBA not always supported by hardware vendors
Disk OK
Access to tape drives lost under high workload conditionsdrives timeout, go offline
IBM supports disk and 3590 on AIX with 6227/8 adapter under moderate workloads
IBM-SSG do not recommend sharing disk and tape in other configurations.i.e. LTO some evidence that it is OK in low workload environments, such as
previous NT/SQL server LAN-Free environment.
Possible solution, with new generation of adapters being certified to support dual use without restrictions
12
IBM Software Group | Tivoli software
Considerations for using TSM in a SANl © 2003 IBM Corporation23
SAN design
Most SANs designed for disk accessdata flow is optimized for hosts <==> disks
stovepipe design, separate SAN ‘islands’
Tape backup requires flow across SANMay need additional Inter-Switch Links
More ports because of HBA Sharing issues
DiskArray
TAPE
IBM Software Group | Tivoli software
Considerations for using TSM in a SANl © 2003 IBM Corporation24
Attaching tape drives to SANs
Most tape drives are Arbitrated Loop devicesNote some directors do not support direct attachment of FC-AL devices
Brocade switches support Public and Private Loop devices in fabric
InRange OK
McData directors do not support Arbitrated Loop Devicesrequire use of Sphereon 4300 or similar departmental switch.Or SCSI attached via Fibre-to-SCSI Gateway/Router
TAPETAPE
ED5000
4300 etc
TAPE
13
IBM Software Group | Tivoli software
Considerations for using TSM in a SANl © 2003 IBM Corporation25
SAN Operational considerationsUsing fibre channel arbitrated loops (Hubs)
During boot up sequence LIP will interrupt tape operations for shared devices.
Rebooting servers can cause tape failures
When devices and servers are on the same loop, devices should not be rebooted while tape is in use by a TSM server or Storage Agent
Power up sequenceSAN, tape devices, then TSM Server and Storage Agents
Scheduling to avoid drive contentionStorage Agents must wait if no drives available.
Use MountWait parameter to avoid backup failuresManaging device access
Device Class, Mount Limit
Device PATHs
IBM Software Group | Tivoli software
Considerations for using TSM in a SANl © 2003 IBM Corporation26
Device Addressing ConsiderationsAll device addresses are defined centrally on TSM server for LAN-Free
Library, Device and Path statements.Each host knows devices by different device names
Changes in device names or SCSI addresses can cause failuresRequirement to manage device addressing
All definitions must point to same physical deviceQuestion: How to map all device names to the same physical device?
Device Namelb1.0.1.3//./tape0//./tape1/dev/rmt0/dev/rmt1
/dev/rmt/0st/dev/rmt/1st
TSM DefinitionLibrary Lib1 lb1.0.1.3Drive Drive0 //./tape0Drive Drive1 //./tape1Path Drive0 /dev/rmt0Path Drive1 /dev/rmt1
Path Drive0 /dev/rmt/0stPath Drive1 /dev/rmt/1st
TAPE
TA PE
W2KTSM Server
AIX Storage Agent
Solaris Storage Agent
F C d ev ic eWWN1
WWN2
14
IBM Software Group | Tivoli software
Considerations for using TSM in a SANl © 2003 IBM Corporation27
TapeDrives
SANGateway/Router
OS DeviceName to
TSM DeviceName
SCSI IDto
OS DeviceName
DeviceWWN
to SCSI ID
Device WWN
SCSI ID to LUN
ID1
ID2
ID3
SCSI Bus
WWN
WWN
Host
TSMOS
DeviceDriver
HBA
Managing Device Addressing in SANs - 1Host Device Names and SCSI addresses can change
devices added or removeddevices failing
A number of address mappings occur.WWN to SCSI addressSCSI address to device addressGateway/Router - device SCSI address to LUN
Described in Redpiece: Managing device addressing of SAN attached tape for use with Tivoli Storage Manager, REDP-0150-00
IBM Software Group | Tivoli software
Considerations for using TSM in a SANl © 2003 IBM Corporation28
Managing Device Addressing in SANs - 2
Changes are more likely to happen if there are large numbers of devices and hosts in SAN.
– New devices being added, or removedSolution
– Use HBA Persistent Naming– Fixes SCSI address to device WWN
– Static device name mapping– Device Names remain unchanged– Fixed device name to SCSI address mapping
– TSM 5.2– Automatic device tracking
15
IBM Software Group | Tivoli software
Considerations for using TSM in a SANl © 2003 IBM Corporation29
HBA persistent naming supportHBA maintains a fixed WWN to SCSI address relationshipSupport matrix
Platform Emulex Qlogic
AIX Not Applicable (use 6227/8 adapter)
Not Applicable (use 6227/8 adapter)
Windows NT/W2K
YES Yes (from 8.1.3 with
SANblade Manager) Solaris YES YES
IBM Software Group | Tivoli software
Considerations for using TSM in a SANl © 2003 IBM Corporation30
Persistent Naming with Emulex on Windows
16
IBM Software Group | Tivoli software
Considerations for using TSM in a SANl © 2003 IBM Corporation31
Qlogic HBA on WindowsQlogic from 8.1.3 onwards with SANBlade Manager/SANSurfer
Download management utility from Qlogic Website
IBM Software Group | Tivoli software
Considerations for using TSM in a SANl © 2003 IBM Corporation32
Static device naming with Microsoft WindowsTSM device driver uses a naming convention which does not change as devices are added or removed
mtx.y.z.nLTO device driver uses default Windows device naming
\\.\tape0Can change if new devices added or removedWith W2K can change in-flight if devices added or removed from SANRecommendation to use mtx.y.z.n,
Use information from TSM Device Information screen to determine mtx.y.z.n device addresses
In TSM 5.1, the mtx.y.z.n name is shown.
17
IBM Software Group | Tivoli software
Considerations for using TSM in a SANl © 2003 IBM Corporation33
Static device naming on AIX and Solaris
AIX– Device drivers automatically track devices using serial numbers
– This mapping is staticSolaris
– Static device naming convention
– Uses symbolic link to map device name to SCSI address
– Requires HBA Persistent Naming to be configured – HBA configuration file
ls –l /dev/rmt/*lrwxrwxrwx 1 root other 45 Jan 3 14:22 /dev/rmt/0mt -> ../../devices/pci@1f,0/pci@1/scsi@2/mt@5,1:mt
IBM Software Group | Tivoli software
Considerations for using TSM in a SANl © 2003 IBM Corporation34
TSM 5.2 Automatic device trackingTSM tracks devices by serial number
– Specify S/N when defining devices– Automatic discovery of S/N when device defined
At start of each operation TSM server and SA will check the device is the one it expects it to be:
– Windows– Initiates a search for the device and changes mapping to point to
new device and then continues operation.– UNIX
– Issues message and fails operation on that device
Avoids need for persistent binding on Windows– Qlogic complex to set up. – Extra mgmt utilities. – Persistent binding options not tested by hardware vendors!
18
IBM Software Group | Tivoli software
Considerations for using TSM in a SANl © 2003 IBM Corporation35
Mapping device names to devices -1
In a SAN all definitions for a shared device must point to the same physical device.
– Each host usually has a different device name for the device depending on the hardware and configuration.
– Only common information available on each platform is the deviceWWN and serial number.
Device Namelb1.0.1.3//./tape0//./tape1/dev/rmt0/dev/rmt1
/dev/rmt/0st/dev/rmt/1st
TSM DefinitionLibrary Lib1 lb1.0.1.3Drive Drive0 //./tape0Drive Drive1 //./tape1Path Drive0 /dev/rmt0Path Drive1 /dev/rmt1
Path Drive0 /dev/rmt/0stPath Drive1 /dev/rmt/1st
TA PE
TA PE
W2KTSM Server
AIX Storage Agent
Solaris Storage Agent
F C d ev ic eWWN1
WWN2
IBM Software Group | Tivoli software
Considerations for using TSM in a SANl © 2003 IBM Corporation36
Mapping device names to devices -2
1. Define drives on TSM server and determine relationship between TSM device name, host device name and device WWN/Serial No.
Also determine the Element Number of the device in the library
2. For each (Storage Agent) host determine WWN/Serial # and device name for each tape device.
3. Use WWN’s/Serial # to relate device names on each host to TSM device name.
4. Define drives on TSM server using PATH statements
19
IBM Software Group | Tivoli software
Considerations for using TSM in a SANl © 2003 IBM Corporation37
Mapping device names to devices-3TSM Device Name Drive0 Drive1 Drive…
WWN/Serial No.
Library Element Number
TSM Server
Storage Agent1
Storage Agent2
Storage Agent3
Host Device Names
……
IBM Software Group | Tivoli software
Considerations for using TSM in a SANl © 2003 IBM Corporation38
Mapping device names to devices – Windows 1
TSM V5.1 – TSM Utilities, Device InformationShows a devices WWN and Serial Number as well as device address (mtx.y.z.n)
Also allows central discovery of Storage Agent device mappings
Requires SNIA HBA SAN Mgmt API to be installed
20
IBM Software Group | Tivoli software
Considerations for using TSM in a SANl © 2003 IBM Corporation39
Mapping device names to devices – Windows 2 TSM Server, Device Information display with Qlogic SNIA SAN Mgmt API InstalledDevice information can also be displayed from this screen for Storage Agents in the same Windows Domain
IBM Software Group | Tivoli software
Considerations for using TSM in a SANl © 2003 IBM Corporation40
Mapping device names to devices -5
AIX lsattr –El mtx/rmtx shows devices WWNlscfg –vl mtx/rmtx shows devices Serial Number
SolarisRelate device name to WWN using SCSI and LUN addresses
ls -l shows device name and SCSI/LUN mapping
dmesg output shows SCSI Target address to WWN mapping
ls –l /dev/rmt/*lrwxrwxrwx 1 root other 45 Jan 3 14:22 /dev/rmt/0mt -> ../../devices/pci@1f,0/pci@1/scsi@2/mt@5,1:mt
dmesg (/var/adm/messages)......qla2200-hba0-SCSI-target-id-5-fibre-channel-name="100000e00201d0d7";
21
IBM Software Group | Tivoli software
Considerations for using TSM in a SANl © 2003 IBM Corporation41
Agenda - Going Live
Testing– testing
Diagnosing Problems– Storage agent messages
– TSM device utilities
IBM Software Group | Tivoli software
Considerations for using TSM in a SANl © 2003 IBM Corporation42
Testing
Test after every hardware change– Changes can introduce errors which can cause failure to recover data.
Proving data integrity– TSM 5.1 includes end to end CRC checking
– use only during testing
22
IBM Software Group | Tivoli software
Considerations for using TSM in a SANl © 2003 IBM Corporation43
Testing
Check tape hardware works reliably with TSM server in LAN configuration firstCheck TDP’s and B/A Client work on LAN firstTest each drive with each Storage Agent to check they are properly defined and accessible.
Use BA ClientANR8779E (Session: 7, Origin: UKSAN4_SA) Unable to
open drive /dev/mt1, error number=2.
Invalid device specifiederror number=16, Device Busy (SCSI Reserved to another
system)
IBM Software Group | Tivoli software
Considerations for using TSM in a SANl © 2003 IBM Corporation44
Diagnosing problems
Storage Agent can be run in foreground session, to see all messages.
– All Storage Agent messages should be logged centrally in the server Activity Log
– Can issue commands from TSM server consolestorage_agent1: QUERY SESSION
mttest and lbtest utilities– Provided in utilities or devices directory
– test operation of tape devices or library operations.– Shows device serial numbers
23
IBM Software Group | Tivoli software
Considerations for using TSM in a SANl © 2003 IBM Corporation45
Question and Answer