module 4: tips, tuning and troubleshooting
TRANSCRIPT
Module 4: Tips, Tuning and Troubleshooting
Learning Objectives
At the end of this module, you will understand:
Minimizing initial synchronization time
DataKeeper™ registry settings
The DataKeeper™ command line interface
User-reported issues and resolutions
Known restrictions
August 10 Copyright © 2000-2008 by SteelEye Technology, Inc. All Rights Reserved. 2
DataKeeper™ Tips, Tuning and Troubleshooting
Local Initial Synchronization
August 10 Copyright © 2000-2008 by SteelEye Technology, Inc. All Rights Reserved. 3
DataKeeper™ Tips, Tuning and Troubleshooting
Local Initial Synchronization
Addresses issue of limited bandwidth
Much faster than WAN-based resynchronization
Not a panacea
August 10 Copyright © 2000-2008 by SteelEye Technology, Inc. All Rights Reserved. 4
DataKeeper™ Tips, Tuning and Troubleshooting:
Local Initial Synchronization
Setup Final IP Configuration on Source and Target
August 10 Copyright © 2000-2008 by SteelEye Technology, Inc. All Rights Reserved. 5
DataKeeper™ Tips, Tuning and Troubleshooting:
Local Initial Synchronization
August 10 Copyright © 2000-2008 by SteelEye Technology, Inc. All Rights Reserved. 6
Create Static Routes and Test via Ping
DataKeeper™ Tips, Tuning and Troubleshooting:
Local Initial Synchronization
Local synch, then relocate target
Connect to all servers.
Create job and mirror(s) with appropriate compression setting(s).
Complete initial synchronization.
Shutdown target.
Delete route to target from source.
August 10 Copyright © 2000-2008 by SteelEye Technology, Inc. All Rights Reserved. 7
DataKeeper™ Tips, Tuning and Troubleshooting:
Local Initial Synchronization
Local synch, then relocate target
August 10 Copyright © 2000-2008 by SteelEye Technology, Inc. All Rights Reserved. 8
DataKeeper™ Tips, Tuning and Troubleshooting
Performance Optimization
August 10 Copyright © 2000-2008 by SteelEye Technology, Inc. All Rights Reserved. 9
DataKeeper™ Tips, Tuning and Troubleshooting:
Performance Optimization
Compression
August 10 Copyright © 2000-2008 by SteelEye Technology, Inc. All Rights Reserved. 10
DataKeeper™ Tips, Tuning and Troubleshooting:
Performance Optimization
Replication Bandwidth Throttle
August 10 Copyright © 2000-2008 by SteelEye Technology, Inc. All Rights Reserved. 11
DataKeeper™ Tips, Tuning and Troubleshooting:
Performance Optimization
Registry Settings
August 10 Copyright © 2000-2008 by SteelEye Technology, Inc. All Rights Reserved. 12
DataKeeper™ Tips, Tuning and Troubleshooting:
Performance Optimization
August 10 Copyright © 2000-2008 by SteelEye Technology, Inc. All Rights Reserved. 13
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\ExtMirr\Parameters
Parameter Default Value(s)
DontFlushAsynchQueue none drive letter(s) of replicated volume(s)
WriteQueueHighWater 9600 decimal
WriteQueueLowWater 50 decimal
MaxResyncPasses 200 decimal
Registry Settings
DataKeeper™ Tips, Tuning and Troubleshooting:
WAN Performance Optimization
August 10 Copyright © 2000-2008 by SteelEye Technology, Inc. All Rights Reserved. 14
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\ExtMirr\Parameters
Parameter Default (decimal) Recommended (Decimal)
PingInterval 3000 10000 (max = 40000)
MaxPingMisses 1 12
NetworkRetries 1 10
NetworkRetryDelay 5000 10000
NetWriteTimeout 10000 30000
ValidateTargetRetries 9 9
Registry Settings
DataKeeper™ Tips, Tuning and Troubleshooting:
WAN Performance Optimization
Registry Settings
Global Setting: HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\Tcpip\Parameters
GlobalMaxTcpWindowSize (REG_DWORD) = Window Size
Tcp1323Opts (REG_DWORD) = 1
For NIC being used for replication: HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\Tcpip\Parameters
\Interfaces\{GUID}
TcpWindowSize (REG_DWORD) = Window Size
Where: Window Size = Bandwidth * Latency
August 10 Copyright © 2000-2008 by SteelEye Technology, Inc. All Rights Reserved. 15
DataKeeper™ Tips, Tuning and Troubleshooting
Command Line Interface (EMCMD)
August 10 Copyright © 2000-2008 by SteelEye Technology, Inc. All Rights Reserved. 16
DataKeeper™ Tips, Tuning and Troubleshooting
Command Line Interface (EMCMD)
Service Related
STOPSERVICE
GETSERVICEINFO
Job Related
CREATEJOB
DELETEJOB
UPDATEJOB
GETJOBINFO
GETJOBINFOFORVOL
Volume Related
LOCKVOLUME
UNLOCKVOLUME
GETVOLUMEINFO
GETMIRRORVOLINFO
GETEXTENDEDVOLUMEINFO
GETCOMPLETEVOLUMELIST
UPDATEVOLUMEINFO
ISPOTENTIALMIRRORVOL
RESTARTVOLUMEPIPE
GETVOLUMEDRVSTATE
August 10 Copyright © 2000-2010 by SteelEye Technology, Inc. All Rights Reserved. 17
DataKeeper™ Tips, Tuning and Troubleshooting
Command Line Interface (EMCMD)
Volume Related
GETCONFIGURATION
SETCONFIGURATION
GETSOURCEMIRROREDVOLUMES
GETTARGETMIRROREDVOLUMES
ISBREAKUSERREQUESTED
Mirror Related
CREATEMIRROR
RESYNCMIRROR
BREAKMIRROR
PAUSEMIRROR
CONTINUEMIRROR
Mirror Related
DELETEMIRROR
DELETELOCALMIRRORONLY
GETMIRRORTYPE
GETRESYNCSTATUS
READREGISTRY
CLEARSWITCHOVER
SWITCHOVERVOLUME
August 10 Copyright © 2000-2010 by SteelEye Technology, Inc. All Rights Reserved. 18
DataKeeper™ Tips, Tuning and Troubleshooting:
Command Line Interface
EMCMD (used by LifeKeeper)
Bitmap Related
GETREMOTEBITMAP
MERGETARGETBITMAP
Rewind Related
ENABLEREWIND
DISABLEREWIND
SUSPENDREWIND
GETREWINDENABLED
GETREWINDSUSPENDED
GETREWINDLOGFILE
REWINDTAG
FLUSHREWINDLOGFILE
August 10 Copyright © 2000-2010 by SteelEye Technology, Inc. All Rights Reserved. 19
DataKeeper™ Tips, Tuning and Troubleshooting:
Command Line Interface
Usage:
EMCMD <ip address> GETMIRRORVOLINFO <letter>
Command: emcmd 192.168.251.98 getmirrorvolinfo e
Output: E: 1 192.168.251.98 192.168.252.98 1
Parameter Description
E Mirrored volume
1 Current ROLE of the system specified 0 = None 1 = Source 2 = Target
192.168.251.98 SOURCE system
192.168.252.98 TARGET system
1 Current STATE of the mirror 0 – None 1 – Mirroring 2 – Resyncing 3 – Broken 4 – Paused 5 – Resync Pending
August 10 Copyright © 2000-2010 by SteelEye Technology, Inc. All Rights Reserved. 20
DataKeeper™ Tips, Tuning and Troubleshooting:
Command Line Interface
Use Case: Manual Switchover, Source Out of Service EMCMD <target ip> SWITCHOVERVOLUME <letter>
Use Case: Manual Switchover, Source Destroyed
EMCMD <target ip> DELETELOCALMIRRORONLY <letter>
EMCMD <target ip> CLEARSWITCHOVER <letter>
Source must be off-line, otherwise a mirror split-brain condition may be created.
August 10 Copyright © 2000-2010 by SteelEye Technology, Inc. All Rights Reserved. 21
DataKeeper™ Tips, Tuning and Troubleshooting:
Command Line Interface
August 10 Copyright © 2000-2010 by SteelEye Technology, Inc. All Rights Reserved. 22
Use Case: Target as Tape Backup Source
Tape backup tools may be used without application-aware agents
Scripted/automated
Description Command
Pause the mirror on
the source system
specifying IP
address of target
EMCMD <source ip> PAUSEMIRROR <letter> <target ip>
Unlock volume on
target system. EMCMD <target ip> UNLOCKVOLUME <letter>
Backup or copy
data from target
system.
copy; xcopy; ntbackup; backup; …
DataKeeper™ Tips, Tuning and Troubleshooting:
Command Line Interface
August 10 Copyright © 2000-2010 by SteelEye Technology, Inc. All Rights Reserved. 23
Description Command
Lock volume on
target system. EMCMD <target ip> LOCKVOLUME <letter>
Continue mirror on
source system EMCMD <source ip> CONTINUEMIRROR <letter> <target ip>
When mirror paused and unlocked, writes to source are flagged in bitmap. Sent to target when mirror continued (partial resynchronization).
When target unlocked, it is writable. Any blocks written on target overwritten by source blocks when mirror continued (partial resynchronization).
Use Case: Target as Tape Backup Source
DataKeeper™ Tips, Tuning and Troubleshooting:
Command Line Interface
Use Case: Disk-to-Disk Copy
Mirror Target
Mirror Source
Replication
August 10 Copyright © 2000-2010 by SteelEye Technology, Inc. All Rights Reserved. 24
DataKeeper™ Tips, Tuning and Troubleshooting
Troubleshooting, Known Issues and Restrictions
August 10 Copyright © 2000-2008 by SteelEye Technology, Inc. All Rights Reserved. 25
DataKeeper™ Tips, Tuning and Troubleshooting
“Split Brain” Scenario
Communications between systems lost
Target volume brought on-line
Communications between systems re-established
Event logs indicate there are two sources
Mirror paused on both servers
August 10 Copyright © 2000-2008 by SteelEye Technology, Inc. All Rights Reserved. 26
DataKeeper™ Tips, Tuning and Troubleshooting
Split Brain
August 10 Copyright © 2000-2008 by SteelEye Technology, Inc. All Rights Reserved. 27
DataKeeper™ Tips, Tuning and Troubleshooting
Split Brain Recovery
Determine which server should be source
Choose mirror source
Click “OK”
August 10 Copyright © 2000-2008 by SteelEye Technology, Inc. All Rights Reserved. 28
Mirror partially resynchronized
Disconnect/Reconnect GUI from/to servers if status incorrect
Source
Server
Mirror Source
Target
Server
Mirror Target
Resynch
Volume X: Volume X:
DataKeeper™ Tips, Tuning and Troubleshooting
Issues and Resolutions
Component Message/Issue Cause Suggested
Action
User Interface Cannot connect to
servers
NetBIOS (137-139)
and/or SMB (445) ports
blocked by firewall
Open ports
User Interface
Can connect to servers,
create job and select
volumes for source and
target, but mirror
creation fails
DataKeeper dispatch
port (9999) or mirror
ports (10000-10025)
blocked by firewall
Open ports
User Interface No volumes available
for mirroring
Only C: available or
volume contains page
file
Create new volume or
remove page file from
volume
August 10 Copyright © 2000-2008 by SteelEye Technology, Inc. All Rights Reserved. 29
DataKeeper™ Tips, Tuning and Troubleshooting
Issues and Resolutions
Component Message/Issue Cause Suggested
Action
User Interface
Failed to create mirror.
Target System event
log entry contains error
code: 0xC0000055
Distributed Link
Tracking Client service
enabled and active
Stop and disable
service on both source
and target
User Interface
Mirror created, but not
stored in job. Event log
entry contains “File:
.\GuiThread.cpp Line
3099 …”
Local security policy for
“Network Access: Let
Everyone permissions
apply to anonymous
users” is disabled.
Enable policy on all
servers.
System Event Log Create mirror failed.
vmms.exe program has
volume open
preventing DataKeeper
from obtaining volume
lock
Stop vmms.exe
program
August 10 Copyright © 2000-2008 by SteelEye Technology, Inc. All Rights Reserved. 30
DataKeeper™ Tips, Tuning and Troubleshooting
Issues and Resolutions
Component Message/Issue Cause Suggested
Action
Windows Clustering
Newly created resource
appears off-line but is
unlocked
Expected behavior Bring resource on-line
Windows Clustering
DataKeeper Volume
resource not available for
use
DataKeeper installed
before cluster name
resource created
Perform repair install of
DataKeeper
Windows Clustering
DataKeeper Volume
resource not available for
use. Event log contains
“DataKeeperVolume.dll,
Error: 70”
All cluster nodes were
not on-line
Bring all cluster nodes
on-line
August 10 Copyright © 2000-2008 by SteelEye Technology, Inc. All Rights Reserved. 31
DataKeeper™ Tips, Tuning and Troubleshooting
Known Issues and Workarounds
Component Message/Issue Cause Suggested Action
Performance Monitor
on Windows Server
2008 64-bit
No DataKeeper counter
logs available in
Performance Monitor
DataKeeper counters
not compatible with
64-bit Performance
Monitor
Use 32-bit Performance
Monitor (under
%WINDIR%\SysWow64\)
Performance Monitor
on Windows Server
2003
Counter logs do not
work on Windows
Server 2003
Known issue to be
resolved in future
release
August 10 Copyright © 2000-2008 by SteelEye Technology, Inc. All Rights Reserved. 32
DataKeeper™ Tips, Tuning and Troubleshooting
Known Restrictions and Recommendations
DataKeeper home directory path name must be 32 characters or less
Pause mirror before running CHKDSK on source volume
Pause mirror before running defragmentation on source volume
Heavy write activity when using synchronous mirror negatively effects application performance
Resizing mirrored Dynamic Disks not supported
Create bitmap directory before relocating bitmap files
Bitmaps on Dynamic Disks that are not “System” are not supported
August 10 Copyright © 2000-2008 by SteelEye Technology, Inc. All Rights Reserved. 33
DataKeeper™ Tips, Tuning and Troubleshooting
Summary
August 10 Copyright © 2000-2008 by SteelEye Technology, Inc. All Rights Reserved. 34
DataKeeper™ Tips, Tuning and Troubleshooting
Summary
Minimizing initial synchronization time for WAN deployments
DataKeeper™ registry keys and settings
The DataKeeper™ command line interface and several use cases
How to recover from a “split-brain” situation
User configuration oversights and errors
Known restrictions and explanations
August 10 Copyright © 2000-2010 by SteelEye Technology, Inc. All Rights Reserved. 35
End Module 4
Lab: Tips, Tuning and Troubleshooting
Copyright © 2000-2008 by SteelEye Technology, Inc. All Rights Reserved. 36