vxvm training 1
TRANSCRIPT
VERITAS Confidential
Welcome
To
VERITAS Presentation
VERITAS Confidential
Volume Manager for Solaris
VERITAS Confidential
Suite of Products
• VERITAS Volume Manager (VM)• VERITAS FlashSnap• Dynamic Multipathing Software (DMP)• VERITAS Volume Replicator (VVR) Support• VERITAS Cluster Server (VCS) Support• Microsoft Cluster Service (MSCS) Support
VERITAS Confidential
VERITAS Volume ManagerGeneral features are:• Centralized Storage Management across
Entire Domain– Supports concurrent local and remote client
connections to the server for browsing and update capabilities; thus, it’s possible to administer online storage remotely.
– Provides asynchronous notification to all connected clients whenever any change to the storage configuration takes place.
VERITAS Confidential
VM – General Features• Configuration Solutions
– Ability to configure and manage different volume layouts: concatenated, striped, mirrored, mirrored striped, and RAID-5 volumes. Supports up to 32-way mirrors on a mirrored volume.
– Automatic detection of failed disks and the ability to repair fault-tolerant volumes on those disks without interrupting applications using the volumes.
VERITAS Confidential
VM – General Features- Provides storage migration path with the ability
to upgrade existing Windows NT 4 or “basic disk” partitions to the new “dynamic” volumes based on VERITAS Volume Manager technology.
- Supports and manages FT (fault-tolerant) disks and volumes previously created in a Windows NT system with Disk Administrator.
- Supports online extending of all volume types.
VERITAS Confidential
VM – General Features- Supports a cluster environment that runs
under VERITAS Cluster Server software or Microsoft Cluster Server software.
- Supports mounting a volume without requiring the use of a drive letter.(windows only)
- Supports moving of storage between computers with the Import and Deport Dynamic Disk Group functions.
- Command line support for administrators who prefer this method over a GUI.
VERITAS Confidential
VM – General Features• Optimized System Performance
- I/O statistics are used to identify high-traffic areas, known as “hot spots.” You can use the Volume Manager Move Subdisk command to resolve these hot spots online.
- Data is assigned to physical drives to evenly balance the I/O load among the disk drives. This is known as storage load balancing.
- Event logging of errors and important information is provided.
VERITAS Confidential
GUI vs. Command Line
• Easy to use• Multi Platform• Makes hard tasks easy• Easy error detection• Alert logs
• Lots of options• Works remotely from
nearly any place.• Makes very hard tasks
possible.• Steep learning curve
GUI Command Line
VERITAS Confidential
VERITAS Enterprise AdministratorVERITAS Enterprise Administrator (VEA), provides access to applications on managed hosts.
VERITAS Confidential
Storage ObjectsStorage Objects• Disk Group
– A set of disks that are an administrative domain for creating and managing volumes. Volumes can’t span disk groups.
– Each disk stores a copy of the configuration data for the whole group. Dividing all disks into smaller groups reduces the amount of configuration data that have to be stored on each disk.
• Disk– Volume Manager disk objects persist even
when the physical disk they represent are removed from the system.
VERITAS Confidential
VEA – Disk View tab
VERITAS Confidential
Storage ObjectsStorage Objects• Volume
– A virtual disk without the size limitations, and with better performance and reliability than the underlying physical disks.
• Subdisk– A contiguous region on a disk that is part of the
physical storage for a volume. Subdisks can’t overlap.
• Plex– A component of a volume that contains one
complete copy of the data or a log.
VERITAS Confidential
VEA – Volume tab
VERITAS Confidential
VEA – Subdisks and Plex
VERITAS Confidential
Storage Object Relationships in General
Disk group
Disk
Plex
Subdisk
File System
Volume
1
NN
N
N N
1 1
1
1
1
1
VERITAS Confidential
VRTSexplorer• Allows support to check the health of the System
•Provides us with the understanding of the system, so that support can make right recommendation
•Gives support a true glance of the problem or issue being resolved.
VERITAS Confidential
How to get VERITAS VRTSexplorer• # ftp ftp.veritas.com• # Login: anonymous• # Passwd: <email address>• # cd /pub/support• # bin• # get vxexplore.tar.Z
• In Windows Explorer• ftp://ftp.veritas.com/pub/support/
vxexplore.tar.Z
• VERITAS Volume Manager 3.5 “Ships with vxexplorer and other tools”
• Trouble Shooting Tools are located ins package “VRTSspt”
VERITAS Confidential
How to Run VRTSexplorer#cd VRTSspt#cd VRTSexplorer./VRTSexplorerVRTSexplorer: Initializing.VRTSexplorer: Please enter case number, or just hit enter:100000001
VRTSexplorer: Please select a destination directory (default: /tmp): VRTSexplorer: Collecting system configuration information for SunOS system.VRTSexplorer: Collecting VERITAS package version information.VRTSexplorer: Collecting loadable module information.VRTSexplorer: Collecting ISIS configuration information.VRTSexplorer: Collecting SIG licensing information.VRTSexplorer: Determining current VxVM operating mode. VRTSexplorer: Collecting VxVM configuration information.
VRTSexplorer: Collecting DMP configuration information.
NOTICE: This section will stop and restart the VxVM Configuration Daemon,
vxconfigd. This may cause your VxVA, VMSA and/or VEA session to exit.
This may also cause a momentary stoppage of any VxVM configuration
actions. This should not harm any data; however, it may cause some
configuration operations (e.g. moving subdisks, plex
resynchronization) to abort unexpectedly. Any VxVM configuration
changes should be completed before running this section.
If you are using EMC PowerPath devices with VERITAS Volume Manager,
you must run the EMC command(s) 'powervxvm setup' (or 'safevxvm
setup') and/or 'powervxvm online' (or 'safevxvm online') if this
script terminates abnormally.
Restart VxVM Configuration Daemon? [y,n] (default: n)y
VRTSexplorer: Collecting VRAS configuration information.
VRTSexplorer: Script finished.
VRTSexplorer: Please ftp /tmp/VRTSexplorer_100000001_aisia.tar.Z to ftp.veritas.com:/incoming
VERITAS Confidential
What is in VRTSexplorer
arp_a ifconfig_a nfs uname_a cron iostat_En pam uptime dev isainfo_v pkginfo usr df_klaV isis pkginfo_l var df_klag kernel pkgparam_patchlist vexplore_version
df_klat license prtconf vmstateeprom modinfo prtdiag vmstat_s env mount_v ps_elf vras etc netstat_i reboot vxfshosted netstat_r showrev vxvm---
Directory
VERITAS Confidential
VXVM Directory
Dmp-----Directoryvxdctl_c_modevxdctl_modevxdg_listvxdg_list_debdgvxdg_list_deqvxdg_list_dg1vxdg_list_rootdgvxdisk_listvxdisk_list_c0t0d0s2
For Every Diskvxdisk_s_listvxkprintvxprint
vxprint_htvxprint_m_debdgvxprint_m_deqvxprint_m_dg1vxprint_m_rootdgvxprint_mpvshr_debdgvxprint_mpvshr_deqvxprint_mpvshr_dg1vxprint_mpvshr_rootdgvxstat_g_debdgvxstat_g_deqvxstat_g_dg1vxstat_g_rootdg
VERITAS Confidential
Common Issues on Solaris
DMP Device FailuresPackage Install/UpgradeDisaster Recovery
(Duplicate Disk Group Copies)
VERITAS Confidential
DMP• DMP Functionality• When is DMP installed• DMP Flow Chart• DMP Error Handling
– DMP Supported Arrays (Appendix A)– Array Types (Appendix B)
VERITAS Confidential
DMP Functionality• Provide a single path abstraction from each disk
device to VxVM– Handles I/O across these paths
• Load balancing of I/Os across the multiple path– Feature currently available only for Active/Active(A/A) DMP
Arrays
• Transparent fail over from one path to another
VERITAS Confidential
DMP Flow Chart
VERITAS Confidential
When is DMP installed• The DMP driver is installed during the VxVM
pkgadd• This can be verified by doing looking at the
modinfo file in the explorer– #more modinfo | grep VxVM
• 23 10202043 1188e8 263 1 vxio (VxVM 3.5s_p2.8 I/O driver)• 25 10305b3e 1b23c 262 1 vxdmp (VxVM 3.5s_p2.8 DMP Driver)• 26 1031e81a 80f 264 1 vxspec (VxVM 3.5s_p2.8 control/status)
VERITAS Confidential
Error Handling in DMP• Error daemon is awakened whenever I/O returns
with an error
• Error daemon then checks the status of the path in a disk array by doing SCSI inquiry on the path
• If path is not alive, error daemon disables the path and retries the I/O on the other path before returning error
VERITAS Confidential
Error Handling in DMP(contd)
• If path is alive, the error daemon retries the I/O, a default of five times. If I/O fails even after the retries DMP assumes it is a media
VERITAS Confidential
Disk Failure• Determine the disk(s) failing
– Use command vxdisk list
• Determine the disk group the failure is involved with
• Check the volume,plexes and sub-disk states – Use command vxprint –ht
• Check message log to determine cause of failure
VERITAS Confidential
DEVICE TYPE DISK GROUP STATUSc1t13d0s2 sliced testdg04 testdg onlinec1t14d0s2 sliced testdg03 testdg onlinec1t15d0s2 sliced - - error c2t10d0s2 sliced rootdisk rootdg onlinec2t13d0s2 sliced testdg01 testdg online- - testdg00 testdg failed was:c1t15d0s2
vxdisk list
VERITAS Confidential
vxprint –g testdg -ht
v vol02 - ENABLED ACTIVE 2097216 RAID - raid5pl vol02-01 vol02 ENABLED ACTIVE 2098272 RAID 4/32 RWsd testdg04-02 vol02-01 testdg04 524356 699426 0/0 c1t13d0 ENAsd testdg03-02 vol02-01 testdg03 524356 699426 1/0 c1t14d0 ENAsd testdg00-02 vol02-01 testdg00 524356 699426 2/0 - NDEVsd testdg01-02 vol02-01 testdg01 524356 699426 3/0 c2t13d0 ENA
/var/adm/messages• Sep 16 13:19:34 aisia disk not responding to selection• Sep 16 13:19:34 aisia vxdmp: [ID 997040 kern.notice] NOTICE:
vxvm:vxdmp: disabled path 32/0x160 belonging to the dmpnode 239/0x20
• Sep 16 13:19:34 aisia vxdmp: [ID 148046 kern.notice] NOTICE: vxvm:vxdmp: disabled dmpnode 239/0x20
• Sep 16 13:19:48 aisia vxio: [ID 686135 kern.warning] WARNING: vxvm:vxio: object testdg00-02 detached from RAID-5 vol02 at column 2 offset 0
• Sep 16 13:19:48 aisia vxio: [ID 354480 kern.warning] WARNING: vxvm:vxio: RAID-5 vol02 entering degraded mode operation
0x20 = 32 Decimal ---This is the DMP node made by Veritas DMP to match the node you must convert the Hex number
/dev/vx/rdmp & dmpbrw------- 1 root other 239, 32 Sep 14 21:18 c1t15d0s0
VERITAS Confidential
With VERITAS VRTSexplorer review the following
• /vxvm/vxprint_ht• /vxvm/vxdisk_list• /var/adm/messages
– search for degraded
• Solution (Appendix E)
VERITAS Confidential
Package Installation/Upgrade• Make sure packages are installed completely
– VXVM requires packages to be installed in correct order– If unsure of what order refer to the manual– VXVM3.5 provides the (installer) script to assist in installation
only
• Check modules are correctly loaded for the Solaris Version loaded on the system
• Check license keys are correctly listed
VERITAS Confidential
pkginfo –l VRTSvxvmPKGINST: VRTSvxvm NAME: VERITAS Volume Manager, Binaries CATEGORY: system ARCH: sparc VERSION: 3.5,REV=06.21.2002.23.14 BASEDIR: / VENDOR: VERITAS Software DESC: Virtual Disk Subsystem PSTAMP: VERITAS-3.5s:21-Jun-2002 INSTDATE: Aug 20 2003 10:47 HOTLINE: 800-342-0652 EMAIL: [email protected] STATUS: completely installed FILES: 603 installed pathnames 22 shared pathnames 9 linked files 76 directories 340 executables 158087 blocks used (approx)
pkgadd -d /cdrom/CD_name/volume_manager3.5/pkgs
1-VRTSvlic 2-VRTSvxvm 3-VRTSvmdoc4-VRTSvmman 5-VRTSob 6-VRTSobgui 7-VRTSvmpro 8-VRTSfspro
VERITAS Confidential
/uname_aSunOS aisia 5.9 Generic_112233-03 sun4u sparc SUNW,Sun-Blade-1000
/kernel# more /kernel/drv/sparcv9/ls_l_vx-rw-r--r-- 1 root sys 435312 Jun 21 2002 /kernel/drv/sparcv9/vxdmp-rw-r--r-- 1 root sys 412304 Jun 21 2002 /kernel/drv/sparcv9/vxdmp.SunOS_5.7-rw-r--r-- 1 root sys 427536 Jun 21 2002 /kernel/drv/sparcv9/vxdmp.SunOS_5.8-rw-r--r-- 1 root other 435312 Aug 20 10:48 /kernel/drv/sparcv9/vxdmp.SunOS_5.9-rw-r--r-- 1 root sys 2953480 Jun 21 2002 /kernel/drv/sparcv9/vxio-rw-r--r-- 1 root sys 2866112 Jun 21 2002 /kernel/drv/sparcv9/vxio.SunOS_5.7-rw-r--r-- 1 root sys 2908152 Jun 21 2002 /kernel/drv/sparcv9/vxio.SunOS_5.8-rw-r--r-- 1 root other 2953480 Aug 20 10:48 /kernel/drv/sparcv9/vxio.SunOS_5.9-rw-r--r-- 1 root sys 15336 Oct 31 2002 /kernel/drv/sparcv9/vxportal-rw-r--r-- 1 root sys 19216 Jun 21 2002 /kernel/drv/sparcv9/vxspec-rw-r--r-- 1 root sys 17840 Jun 21 2002 /kernel/drv/sparcv9/vxspec.SunOS_5.7-rw-r--r-- 1 root sys 18536 Jun 21 2002 /kernel/drv/sparcv9/vxspec.SunOS_5.8-rw-r--r-- 1 root other 19216 Aug 20 10:48 /kernel/drv/sparcv9/vxspec.SunOS_5.9
VERITAS Confidential
/opt/VRTSvlic/bin#./vxlicrep -e License Key =
528222690330627075168549872296 Product Name = VERITAS Volume Manager Lic Key Lib Ver = ELM Key = Valid License Type = PERMANENT_NODE_LOCK Node Lock Type = 3 (Hostid and Architecture ID)
Features := RAID = Enabled
VERITAS Confidential
Disaster Recovery• Disaster Recovery(DR)
– Any event that will require a rebuild of a Disk Group or Volume configuration database.
• Example Errors:Duplicate Disk Group RecordsCorrupted Disk Group ConfigurationNo Valid Disk Group Configuration Found
• When preparing for a disaster it is always a good plan to have the following:
VERITAS Confidential
Disaster Recover (contd)• Layouts of the Diskgroup and Volumes
– vxprint –ht or vxprint -htr– vxprint –g <diskgroup> -mpvshr
• If the above outputs cannot be provided due to a “duplicate disk record” error. It is important to gather the following before making any volume record changes. (Example in: Appendix F)– /etc/vx/diag.d/vxprivutil dumpconfig
/dev/rdsk/c#t#d#s<private> > dump.out
VERITAS Confidential
Tools for Windows
• VxExplorer – gathers most of the necessary information to troubleshoot a problem.
• Vxdisksetid – used to verify diskset id consistency.
• Vxtool – advanced tools used by support and engineering to diagnose and correct problems.
VERITAS Confidential
Disk Group ConfigurationDisk Group Configuration
• VxConfig stores a copy of a disk group’s configuration on each disk in the group.
• A disk group’s configuration is described in terms of configuration records.
• There is a record type for each object type.
VERITAS Confidential
Configuration Record TypesConfiguration Record Types•The types of configuration records
correspond to the VM object model– Disk group record– Volume record– Plex record– Subdisk record– Two kinds of disk records
VERITAS Confidential
Disk Media v.s. Disk Access RecordsDisk Media v.s. Disk Access Records• Disk Media Configuration Record
– Identifies a disk by disk id and gives it an administrative name, e.g. “disk1”.
– A disk group contains dm recs for all member disks.– Subdisk recs are associated with dm recs.– Dm recs can be disassociated from a failed physical
disk and re-associated with a replacement disk.• Disk Access Record
– Identifies an O/S access path to a disk,e.g. \device\Harddisk0.
– Not stored in the on-disk configuration database because it can easily change after rebooting.
VERITAS Confidential
Missing DisksMissing Disks
• If a disk group contains a disk media record for a disk that Windows doesn’t know about (i.e. no disk access record), the disk is a Missing Disk.
VERITAS Confidential
Missing Disk
VERITAS Confidential
VM Alert MessageAlert MessageAlert Message
VERITAS Confidential
Event Viewer Alert
VERITAS Confidential
Replace Disk
VERITAS Confidential
Foreign Disks•A disk is foreign if the hostid in it’s
configuration doesn’t match the server,
or
• If it’s disk group id matches an existing disk group but it’s diskset id doesn’t
VERITAS Confidential
Foreign Disk
VERITAS Confidential
Merge Foreign
VERITAS Confidential
Dynamic Disk HeaderDynamic Disk Header
•A disk header records the disk’s VM disk id, identifies the disk’s disk group, and describes the layout of the private region.
VERITAS Confidential
Disk Header ContentsDisk Header Contents• diskid: 8998ae08-f2a6-4264-97dc-8f5ceeda43de• group: name=NewGrp id=a51fb942-4fcd-4a0f-bc88-0dd0b07a64c9• flags: private autoimport• import: bootsig=130 hostid=d0120cc1-8d21-11d7-aade-806d6172696f• diskset: id=b95a984f-aeb0-4855-ab15-0b39e05145fe• version: 2.12• iosize: 512• public: slice=0 offset=63 len=35551782• private: slice=0 offset=35564430 len=2048• update: seqno=0.36 time=1635821074• headers: 2047 1856• configs: count=1 len=1481• logs: count=1 len=224
VERITAS Confidential
Configuration RecordsConfiguration Records• Disk: Disk2 rid=0.1029 updated=0.1155• assoc: diskid=8998ae08-f2a6-4264-97dc-8f5ceeda43de
• Group: NewGrp rid=0.1025 version=30 update=0.1154• id: dgid=a51fb942-4fcd-4a0f-bc88-0dd0b07a64c9• diskset: id=b95a984f-aeb0-4855-ab15-0b39e05145fe• • Volume: volB rid=0.1054 update=0.1151 mountname=F:• info: len=1536000 guid=dda5ab3f-ac13-4915-8353-51d1a3615fd9• state: state=ACTIVE• • Plex: volB-01 rid=0.1056 update=0.1151• type: layout=STRIPE columns=3 width=128• state: state=ACTIVE• • Subdisk: Disk2-01 rid=0.1049 updated=0.1155• info: disk=0.1029 offset=0 len=256000 hidden=0• assoc: plex=0.1041 (column=3 offset=0)
VERITAS Confidential
Diskset id mismatch
•Diskset id mismatch means the disk was removed from the disk group while the disk was missing.
VERITAS Confidential
Disketset Id• Diskset id is used to make sure that a disk that’s
removed and then reinserted is not modified.• A new diskset id is created when a disk is
missing.• Diskset id is stored in the disk header and also in
the dg record in the config database.• Each dm record also keeps track of the last
diskset id when its da is detached.
VERITAS Confidential
Diskset Id Checking• A disk cannot be onlined if the diskset id does
not match in its header and dg record in config.• A disk can be brought online automatically in
rescan operation if the diskset id in the header matches the “lastset” id in a detached dm record.
Disk: Disk2 rid=0.1029 updated=0.1158 assoc: diskid=8998ae08-f2a6-4264-97dc-8f5ceeda43de detached lastset: id=b95a984f-aeb0-4855-ab15-0b39e05145fe
VERITAS Confidential
Vxdisksetid output
harddisk4header diskset ID=47b91238-1db1-4943-9d63-6264c7b667b7config diskset ID=47b91238-1db1-4943-9d63-6264c7b667b7
harddisk5 has a mismatched diskset IDheader diskset ID=b45dd49a-10aa-43f8-aa21-7c3fc4b3e1ffconfig diskset ID=98a808a8-07cf-4175-8334-3419ed27b396
VERITAS Confidential
Vxtoolvxtool - VERITAS Volume Manager diagnostic operations
Usage: • vxtool disk <sub-operation> [-<options>] [<arguments>] • vxtool kernel <sub-operation> [-<options>] [<arguments>] • vxtool verify [-v] <volume-name> [[<offset>] <length>] • vxtool getioparms <driveletter>:
VERITAS Confidential
Vxtool disk setheaderFields that can be modified:• Diskid• Hostid• Dgid• Dg_name• Disksetid• newdisksetid
VERITAS Confidential
Vxtool disk setheader
The below command sets the diskset id entry in the header record to match that of the config record. After a rescan the disk can be brought back into the diskgroup successfully.
vxtool disk setheader harddisk5 diskset=98a808a8-07cf-4175-8334-3419ed27b396
VERITAS Confidential
DMP Supported Arrays Appendix A
ARRAY TYPE VID PIDA/A SUN AP_NODESA/A VERITAS
ATFNODESA/AECCS allA/A EMC SYMMETRIXA/A FUJITSU GR710, GR720,
GR730, GR740, GR820, GR840A/A HITACHI OPEN-*A/PG HITACHI DF350,DF400,
DF500A/P LSI INF-01-00A/A NEC DS1200,
DS1200F, DS3000SL,DS3000SM,DS3001
A/A NEC DS3001, DS3002, DS1000, DS1000F, DS1100,DS1100F, DS3011, DS1230, DS450, DS450F, iStorage 1000, iStorage 2000, iStorage 4000
A/P SUN T300A/A VERITAS
RDACNODESA/A SENA allA/AIBM 2105A/ASSA SSAA/A StorComp
OmniForceA/PF VERITAS allA/A HP OPEN-*
VERITAS Confidential
Array Types Appendix B
• A/A
• A/P
• A/PF
• A/PG
•Active/Active disk array; I/Os could be simultaneously active on more than one path to the device without any performance degradation
•Active/Passive disk array; I/Os can not be issued to more than one path to the device without severe performance penalty; when such arrays are con.figured in auto-trespass mode,the path failover occurs with a normal I/O command to the passive (standby) path
•Active/Passive disk array in explicit failover mode; A/P array that will require special failover command; support for such arrays can not be added dynamically
•Active/Passive disk array with LUN group failover. A special case of A/P array where a group of LUN’s fail-over
VERITAS Confidential
vxdg_list Appendix C
VERITAS Confidential
vxdisk_list_c1t15d0s2 Appendix D
VERITAS Confidential
Solution Appendix E
• Vxdiskadm (Appendix G)– Option 4 Remove a disk for replacement– Option 5 Replace a failed or removed disk
• Volume should start rebuilding parity on new device– Use vxtask list to view status
• Example: vxtask listTASKID PTID TYPE/STATE PCT PROGRESS171 171 R5RECOVER/R 32.09% 0/699426/224448
RECOV vol02 testdg00-02
VERITAS Confidential
vxprivutil Appendix F
• Appendix C shows what disk have enabled configuration copies
Example:./vxprivutil dumpconfig /dev/rdsk/c2t15d0s3 >
/tmp/dump.out
• Appendix D show what slice the private region is in.
• Verify the dump.out file with the following:cat /tmp/dump.out | vxprint –D - -ht
VERITAS Confidential
vxdiskadm Appendix G
Volume Manager Support OperationsMenu: Volume Manager/Disk
1 Add or initialize one or more disks 2 Encapsulate one or more disks 3 Remove a disk 4 Remove a disk for replacement 5 Replace a failed or removed disk 6 Mirror volumes on a disk 7 Move volumes from a disk 8 Enable access to (import) a disk group 9 Remove access to (deport) a disk group 10 Enable (online) a disk device 11 Disable (offline) a disk device 12 Mark a disk as a spare for a disk group 13 Turn off the spare flag on a disk
SO ON……….