b9: success with openedge® replication david eddy senior solution consultant what you dont know can...
TRANSCRIPT
B9: Success with OpenEdge® Replication
David EddySenior Solution Consultant
What you don’t know can hurt you
© 2008 Progress Software Corporation2
Obligatory (gratuitious) Quotation
An ounce of prevention is worth a pound of cure.
28.35 grams of prevention is worth .45 kilograms of cure
© 2008 Progress Software Corporation3
Success With Replication
Understand the architecture Know your requirements Analyze your system Plan, configure, deploy,monitor Succeed
© 2008 Progress Software Corporation4
Agenda
Terminology and architecture – Visual Replication performance hotspots Replication availability hotspots Source and target management
© 2008 Progress Software Corporation5
Replication Terminology
Primary – original production machine Secondary – original recovery machine Source – production database Target – recovery database Replication server – rpserver (source) Replication agent – rpagent (target) DBService queue – stores IPC messages Network pipe – TCP/IP rpserver -> rpagent
© 2008 Progress Software Corporation6 !
Architecture of Replication – Primary/Source
Primary Machine
R/WriteClients
AI Extents
DB Service
Q
Repl Server
AI ExtentsSource
Database
Database Brk/Server Processes
© 2008 Progress Software Corporation7
Architecture of Replication – Secondary/Target
Secondary Machine
Repl Agent
Database Brk/ServerProcesses
TargetDatabase
R/OClients
© 2008 Progress Software Corporation8
Architecture of Replication
Primary Machine Secondary Machine
Repl Agent
Database Brk/ServerProcesses
TargetDatabase
R/OClients
R/WriteClients
AI Extents
DB Service
Q
Repl Server
AI ExtentsSource
Database
Database Brk/Server Processes
© 2008 Progress Software Corporation9
Agenda
Terminology and architecture - Visual Replication performance hotspots Replication availability hotspots Source and target management
© 2008 Progress Software Corporation10
Performance Hotspots
Primary Machine Secondary Machine
Repl Agent
Database Brk/ServerProcesses
TargetDatabase
R/OClients
R/WriteClients
AI Extents
DB Service
Q
Repl Server
AI ExtentsSource
Database
Database Brk/Server Processes
© 2008 Progress Software Corporation11
Source DB Performance Considerations
Speed/power of machine Replication Plus – offload read only clients to
target database AI files – fixed extents
AI/BI blocksizes should be the same• 16 K is the most efficient
• Need to truncate ai and bi
© 2008 Progress Software Corporation12
Target DB Performance Considerations
System should not be underpowered by comparison to the source system
No need for after-imaging Read only clients
© 2008 Progress Software Corporation13
Network Performance
WAN vs. LAN The bigger the pipe the better Determine size of pipe Whitepaper available on PDSN http://tinyurl.com/6xqp78
© 2008 Progress Software Corporation14
Sample Network Bandwidth Calculation
Hourly After Image Blocks from 5 production databases = 713mb
By calculation:• Replication size = 1.5 * AI size
– 713mb*1.5 = 1069.5mb
• Add replication overhead (1.1)– 1069.5mb*1.1 = 1176.45mb
Throughput per second:• 1176.45mb/3600=.32679mb/b
Change to kilobytes /s:• .32679mb/s * 1000 = 326.79kb/b needed to sustain
transfer
© 2008 Progress Software Corporation15
Replication Performance – DBService Buffer
DBService Buffer Queue must be sized appropriately – Solution P121969• Failure to do so may result in source
slowdown and target falling behind
Analyze AI activity prior to implementing• Use PROMON -> R&D -> Status Display (#1)-
> DB Service Manager (#16).
© 2008 Progress Software Corporation16
Determining Optimal -Pica
The larger the –pica, the further behind the target may get
Monitor AI writes during busiest period for one hour
-pica = (TAIW/Blockcount) * 1.25 Blockcount depends on version
• OpenEdge 10 – blockcount is 9.16
• Progress® 9.1x – blockcount is 18.2
© 2008 Progress Software Corporation17
Sample –Pica Calculation
Values up to 8192k for 10.1B01 and later 34560 TAIW over one hour Formula for OE 10:
• (34560/9.16) * 1.25 = 4716.1572052401746724890829694323
Why not just use 8192???
© 2008 Progress Software Corporation18
“Houston, we have a problem” - Promon
03/03/08 Status: Database Service Manager
17:15:02
Communication Area Size : 2049.00 KB
Total Message Entries : 18733
Free Message Entries : 4
Used Message Entries : 18729
Registered Database Service Objects
Name Rdy Status Messages Locked by
OpenEdge Replication Server Y RUN 18729
OpenEdge RDBMS Y REG 0
OpenEdge DB Agent Y RUN 0
© 2008 Progress Software Corporation19
Agenda
Terminology and Architecture - visual Replication performance hotspots Replication availability hotspots Source and target management
© 2008 Progress Software Corporation20
Availability Hotspots
Primary Machine Secondary Machine
Repl Agent
Database Brk/ServerProcesses
TargetDatabase
R/OClients
R/WriteClients
AI Extents
DB Service
Q
Repl Server
AI ExtentsSource
Database
Database Brk/Server Processes
© 2008 Progress Software Corporation21
Availability Hotspots
Secondary Machine
Repl Agent
Database Brk/ServerProcesses
TargetDatabase
R/OClients
R/WriteClients
AI Extents
DB Service
Q
Repl Server
AI ExtentsSource
Database
Database Brk/Server Processes
DOWN!
Severed
Primary Machine Loss of Database TCP or Process Failure
Locked
© 2008 Progress Software Corporation22
Availability Hotspots – AI Management
Replication DOES NOT manage AI files AI Files must be emptied and backed up
• 10.1A AI archiver became available.
Database crashes when no empty AI extents.
© 2008 Progress Software Corporation23
Availability Hotspots – Locked AI Files
AI “locked” when AI notes not replicated to the target db• Common issue – often caused by
simple maintenance routines and failure to monitor status of replication
Configure the rpserver and rpagent to detect and handle outages
All about the *.repl.properties file!! Configure, test well and sleep easier at night
© 2008 Progress Software Corporation24
Availability – Many Locked AI Files
Use dsrutil monitor against source and target db. Check for replication shared memory
If rpagent is running, restart the replication server on the source
If rpagent is not running, restart target database and restart replication server
© 2008 Progress Software Corporation25
“Houston, we have a problem” – part 2
Extent: 1 Extent: 3Status: Busy Status: Locked Type: Variable Length Type: Variable Length Path: C:\wrk101c\repl\source.a1 Path: C:\wrk101c\repl\source.a3 Size: 3192 Size: 6264 Used: 3180 Used: 6200 Start: Wed May 14 14:09:34 2008 Start: Wed May 14 14:08:04 2008 Seqno: 5 Seqno: 3
Extent: 2 Extent: 4Status: Locked Status: Locked Type: Variable Length Type: Variable Length Path: C:\wrk101c\repl\source.a2 Path: C:\wrk101c\repl\source.a4 Size: 230008 Size: 25208 Used: 229674 Used: 25063 Start: Wed May 14 13:45:30 2008 Start: Wed May 14 14:08:24 2008 Seqno: 2 Seqno: 4
© 2008 Progress Software Corporation26
Availability – All AI Files Locked
Source database activity comes to a halt Can you afford to bring the db down?
• If not - disablesitereplication
• If yes – stop database– add new ai files– prostrct reorder– restart.
© 2008 Progress Software Corporation27
Prostrct add sourcedb addai.st
Extent: 1 Extent: 4Status: Busy Status: Locked Type: Variable Length Type: Variable Length Path: C:\wrk101c\repl\source.a1 Path: C:\wrk101c\repl\source.a4 Size: 3192 Size: 25208 Used: 3180 Used: 25063 Start: Wed May 14 14:09:34 2008 Start: Wed May 14 14:08:24 2008 Seqno: 5 Seqno: 4
Extent: 2 Extent: 5Status: Locked Status: Empty Type: Variable Length Type: Variable Length Path: C:\wrk101c\repl\source.a2 Path: C:\wrk101c\repl\source.a5 Size: 230008 Size: 120 Used: 229674 Used: 0Start: Wed May 14 13:45:30 2008 Start: N/ASeqno: 2 Seqno: 0
Extent: 3 Extent: 6Status: Locked Status: Empty Type: Variable Length Type: Variable Length Path: C:\wrk101c\repl\source.a3 Path: C:\wrk101c\repl\source.a6 Size: 6264 Size: 120 Used: 6200 Used: 0Start: Wed May 14 14:08:04 2008 Start: N/ASeqno: 3 Seqno: 0
© 2008 Progress Software Corporation28
Prostrct reorder ai sourcedb
Extent: 1 Extent: 4Status: Busy Status: Locked Type: Variable Length Type: Variable Length Path: C:\wrk101c\repl\source.a1 Path: C:\wrk101c\repl\source.a4 Size: 3192 Size: 230008 Used: 3180 Used: 229674Start: Wed May 14 14:09:34 2008 Start: Wed May 14 13:45:30 2008 Seqno: 5 Seqno: 2
Extent: 2 Extent: 5Status: Empty Status: Locked Type: Variable Length Type: Variable Length Path: C:\wrk101c\repl\source.a2 Path: C:\wrk101c\repl\source.a5 Size: 120 Size: 6264 Used: 0 Used: 6200Start: N/A Start: Wed May 14 14:08:04 2008 Seqno: 0 Seqno: 3
Extent: 3 Extent: 6Status: Empty Status: Locked Type: Variable Length Type: Variable Length Path: C:\wrk101c\repl\source.a3 Path: C:\wrk101c\repl\source.a6 Size: 1 20 Size: 25208 Used: 0 Used: 25063Start: N/A Start: Wed May 14 14:08:24 2008 Seqno: 0 Seqno: 4
© 2008 Progress Software Corporation29
Source and Target DBs Won’t Synchronize
Try restarting a few times. -Ma, -Mn, -n match? Changes to .properties files? If it continues to fail, contact support
• Source/target db log files
• AI files
• pmmgr.properties file– arguments=-logging 2
© 2008 Progress Software Corporation30
Agenda
Terminology and architecture - visual Replication performance hotspots Replication availability hotspots Source and target management
© 2008 Progress Software Corporation31
Managing Replication – DSRUTIL Monitor and Status
Not immediately obvious when a failure occurs
Proactively verify replication performance and status
© 2008 Progress Software Corporation32
DSRUtil Monitor
DSRUTIL source/target –C monitor• Attaches to replication shared memory.
– Cannot connect to replication shared memory. Status = -1
• Checks status of Server and Agent– Server status– Agent status
© 2008 Progress Software Corporation33
DSRUtil source –C monitor
OpenEdge Replication Monitor Page 1 Database: C:\wrk101c\repl\source
Database is enabled as OpenEdge Replication: Source
Server is: In Normal Processing Number of configured agents: 1 Delay Interval (current / min / max): 5 / 5 / 500 Recovery information: State: No recovery being performed
Agents needing recovery: 0
Agents connected: 0 Agents in synchronization: 0 Transition information: Type: Manual
© 2008 Progress Software Corporation34
DSRUtil target –C monitor (page 1)
Database: C:\wrk101c\repl\target
Database is enabled as OpenEdge Replication: Target
Agent: Name: agent1 ID: 1 Host name: State: Normal Processing Ready: Yes Critical: No Method: Asynchronous Agent is waiting for: Nothing Maximum bytes in TCP/IP message: 8500 Server/Agent connection time: Wed May 14 13:48:43 2008 Delay Interval (current / min / max): 5 / 5 / 500 Transition information: Type: Manual The last block received at: Wed May 14 13:51:14 2008 Activity information: Blocks received: 2084 Blocks processed: 2084
© 2008 Progress Software Corporation35
DSRUtil target –C monitor (page 2)
Blocks acknowledged: 0 Notes processed: 207774 Transactions started: 10269 Transactions ended: 10269 Synchronization points: 33 AI Block Information: Source RDBMS Block (Seq / Block): 2 / 2516 Last Processed Block (Seq / Block): 2 / 2494 Latency Information: Repl Server behind Source DB by: 1 second(s) Current Source Database Transaction: 13906 Last Transaction Applied to Target: 13793 Target Current as of (Target, Source): Wed May 14 13:51:13
2008, Wed May 14 13:51:13 2008 with delta of 000:00:00
© 2008 Progress Software Corporation36
DSRUTIL Status
DSRUTIL source/target –C status Return code indicates current state of
replication server or agent. Good for automated scripts Grep for return code Replication User Guide provides details
© 2008 Progress Software Corporation37
Managing Replication – OE Management
OpenEdge Management 10.1B02/3.1B02+ remotely monitor log files
10.1B02 AdminServer running on the remote system
Attach OE_DB_Replication Log File Ruleset to the log file monitor
Alert/email thrown when message violating rule set appears.
© 2008 Progress Software Corporation38
OE Management – Replication Log File Rule Set
© 2008 Progress Software Corporation39
Summary
Terminology and Architecture Replication performance hotspots Replication availability hotspots Source and target management
© 2008 Progress Software Corporation40
Success With Replication
Understand the architecture Know your requirements Analyze your system Plan, Configure, Monitor Succeed
© 2008 Progress Software Corporation41
For More Information, go to…
PSDN• www.psdn.com/library/kbcategory.jspa?categoryID=21• www.psdn.com/library/kbcategory.jspa?categoryID=334
Knowledge Centrum (esupport.progress.com)• 3.1B01/10.1B01 upgrade: P122926, P123418, P123420,
P123424, P123426, P123427, P123676• sizing –pica: P121969
Documentation• OpenEdge Replication 10.1C docs (www.psdn.com)• OpenEdge Management 3.1C docs (www.psdn.com)
© 2008 Progress Software Corporation42
Questions?
© 2008 Progress Software Corporation43
Thank You
© 2008 Progress Software Corporation44