data replication implementation

What is New in IBM Replication Solution

Richard SinnIBM Santa Teresa LabAS/400 Tech Conference 2000

IBM Santa Teresa Lab

Presentation Goals Learn introduction materials to replication Learn IBM’s solution and where to order Understand the concept of Capture/Apply Pickup useful details of Capture/Apply Quick Overview of Advanced Features and

Future Enhancement

IBM DB2 Data Propagator

Product ID: – 5769DP2 for V4R1+– 5769DPX for future release (X is a number)

Product Home Page at: http://www.software.ibm.com/data/dpropr

You could order online OR try it free for 70 days in future release

Agenda What is Replication IBM Data Propagator Overview

– Capture Side– Apply Side

AS/400 Data Propagator Useful Internal Advance Replication Features Future Enhancement Conclusion

What is Replication Movement of Data from a

database source system to a target database

Copy only the "Changes" vs Copy "the whole thing"

Redefining Application Development Model

Build Data Warehouse Making Data Available

Replication Requirements

Copy - aCopy - bPrimary - c

Copy - aPrimary - bCopy - c

Primary

Copy Copy

Copy

Primary Primary

Primary Primary

Primary

Data dissemination Data consolidation

Replication of distinct fragments Update anywhere

Primary - aCopy - bCopy - c

Why Replication ? Making Data Available Use Information

Intelligently and improve decision making

Stay competitive with both Data access and Data movement

Speedup application deployment

Reengineer business processes

Increase online throughput

Improve system availability

Support audit requirements

Support Data Warehousing

IBM Data Propagator Overview

Journal

ADMINISTRATION (e.g DJRA)

COPY

CONTROL

COPYCOPY

Base TablesColumn SelectionAfter Image or Before & After image

APPLY

CONTROL

CAPTURE

BASE

Source/Operational System

Target

IBM Data Propagator Overview Capture works on "Source

System" Apply works on "Target

System" To tell Capture what to

work on: Registration To tell Apply what to work

on: Subscription Registrations/

Subscriptions are created by DJRA

IBM Data Propagator Overview DPR runs on multiple

platforms (AIX, NT, AS/400, 95, HP-UNIX, OS/2, etc)

All control info is SQL based

Published Architecture: works with NotesPump, etc

Well established technology

Overview: Admin with DJRA

Overview: Administration

Administration: Registration

Select AS/400 database Select table Select necessary

column only Pick capture policy Pick update capture

policy Pick conflict detection

level

Overview: Capture

Journal

ADMINISTRATION

COPY

CONTROL

COPYCOPY

Base TablesColumn SelectionAfter Image or Before & After image

APPLY

UNIT OF WORK

CHANGE DATA

CONTROL

CAPTURE

BASE

Operational System

Overview: Capture DJRA creates

registration information in the control tables associated with Capture/400

"Changes" are actually put into "Change Data" table

When there is a commit, a row will be put into "Unit of Work" table

Captures base table changes from Journal

Timestamps changes Maintains transaction

consistency Automatically maintains

staging tables Pruning of CD tables

are automatically as well

Overview: Subscription Set

Overview: Subscription Member

Overview: Subscription

Subscription is bounded by set

Set can have multiple members

Event based, or Time based

Blocking factor Can select necessary

columns only

Overview: Apply

Journal

Operational System CONTROL

APPLY

BASE

UNIT OF WORK

CHANGE DATA

CONTROL

CAPTURE

ADMINISTRATION

PIT/User CopyHISTORY

STAGINGREPLICA

Base and Copy TablesInterval and RepetitionColumn and Row SelectionComputed ColumnsAggregationsAppend or Replace

Target

Overview: Apply

DJRA creates control information associated with Apply/400

According to the control info, Apply– Runs at user-specified

intervals or events– Refreshes, updates, and

enhances copies– Do distribution

optimizations

Apply picks up "changes" from the "Change Data" table and "Unit of Work" table

Apply communicates with Capture via Syn point

Apply is a big SQL application

Apply can be "Push" or "Pull"

Useful Internal: Capture

Journal

CONTROL

CAPTURE JOURNALJOB

CHANGE DATA CHANGE DATACHANGE DATAUNIT OF WORKCHANGE DATA CHANGE DATACHANGE DATA

BASEBASE

CAPTURE JOURNALJOB

Journal

BASEBASE

CAPTURE CONTROLJOB

Capture: Control Job Control Job - QDPRCTL5

– Analyze the user options and new registration– Check replication source validity– Sort out all selected replication sources by journals– Determine eligibility– Determine journal job starting time– Start up journal jobs– Prune - CD tables, UOW and AUTHTKN– Handle messages (from commands and journal jobs)

Capture: Journal Jobs Journal Job(s) - Top Program

QZSNCAP2– Issue RCVJRNE command

• Use time to specify which journal entries to process

• Give a list of up to 300 file/members• Give a list of 24+ specific journal entry types• Specify a delay of 60 seconds

– Handles messages from exit program

Capture: Exit Program Journal Job(s) - Exit Program QZSNCAP3

– Receive and analyze changes from the journal– Advance SYNCHPOINT in PRUNCNTL table on a set

basis when necessary– Maintain CD_OLD_SYNCHPOINT and

CD_NEW_SYNCHPOINT in register table– Maintain the global row in register table– Support application commitment control– Handle special journal types (end journal, renamed, etc)– Cancel handling (Warm start table), etc

Capture: Performance Tuning

Latency is 4 to 5 minutes

CPU overhead is about 6 to 8%

Ways to tune performance:– Job priority– Num of replication

sources per journal job– COMMIT_INTERVAL in

CCPPARMS

Apply: Job Information STRDPRAPY APYQUAL(ABC) TRACE(*ALL)

Work with Subsystem Jobs STL400F 01/24/00 15:30:03 Subsystem . . . . . . . . . . : QZSNDPR Opt Job User Type -----Status----- Function ABC SINN BATCH MSGW PGM-QZSNAPV2 KSCHIA KSCHIA BATCH ACTIVE PGM-QZSNAPV2 QDPRCTL5 KSCHIA BATCH MSGW PGM-QZSNCAP1 QSQJRN KSCHIA BATCH ACTIVE PGM-QZSNCV72

Apply: Trace Information

STRDPRAPY APYQUAL(ABC) TRACE(*ALL)

Work with Job Spooled Files Job: ABC User: SINN Number: 048177

Device or Total Current Opt File Queue User Data Status Pages Page Copies QPZSNATRC PRT01 ABC OPN 1818 1

Apply: Apply Trail Table

SQL format trail info is also available STRSQL

select * from asn/ibmsnap_applytrail

APPLY_QUAL SET_NAME WHOS_ON_FIRST ASNLOAD MASS_DELETE EFFECTIVE_MEMBERS STL400F STL400F S N Y 1 STL400F STL400F S N N 0 STL400F STL400F S N N 0 STL400F STL400F S N N 0 STL400F STL400F S N N 1 STL400F STL400F S N N 0 STL400F STL400F S N N 0 STL400F STL400F S N N 1

Apply: Apply Trail Table

Some of the available information are:

APPLY_QUAL, SET_NAME, ASNLOAD, MASS_DELETE, EFFECTIVE_MEMBERS, SET_INSERTED, SET_DELETED, SET_UPDATED, SET_REWORKED, STATUS, LASTRUN, LASTSUCCESS, SOURCE_SERVER, SOURCE_TABLE, SOURCE_VIEW_QUAL, TARGET_SERVER, TARGET_TABLE, SQLSTATE, SQLCODE, SQLERRP, SQLERRM

Advance Replication Features AS/400 Relative Record Number (RRN) Support Remote Journal Replication Support Automated Deletion of Journal Receivers Performance Improvement to Apply/400 Programming with ASNLOAD and ASNDONE Set Subscriptions Update Anywhere IBM Replication Solution Extendibility

RRN Replication Support

Journal

CAPTURE

SOURCE TARGET

COPY

RRN("base-owner".base-table")

TARGET

COPY

Normal differential processing (contents of IBM_QSQRRN)

Differential Refresh

APPLY

UNIT OF WORK

CHANGE DATA

Full Refresh

Base

COL1 COL2 COL3 COL4

I RRN etc.D RRN etc.

APPLY

RRN Replication Support

Allow user to have a unique key for CCD and Point-in-time target tables when no combination of target columns is unique (no SQL key)

DJRA support administration of RRN replication on AS/400

Remote Journal Replication

Remote Journal can be setup using AS/400 commands (V4R3+)

Capture and Apply run on the same system

Free up resource on operational (source) system

Better performance (no network overhead, etc)

Remote Journal Replication

Operational System Target System

Journal

BASE

Remote Journal

CONTROL

CAPTURE

UNIT OF WORK

CHANGE DATA

CONTROL

COPY

COPYCOPY

APPLY

Reg. Ext.

Automated Deletion of JR ReceiversSource System

Journal

CONTROL

CAPTURE

Journal Receivers QZSNDREP

exit program Detached Receiver

Delete Receiver

Keep Receiver

Yes Vote

No Vote

Automated Deletion of JR Receivers Exit program registered at

installation time (QIBM_QJO_DLT_JRNRCV)

DLTJRNRCV calls the exit program before deleting JR receivers

If Capture needs the receiver, it votes no on delete

When receivers are managed by system, it can then be deleted automatically

Performance Improvements to Apply Blocked reading and writing to work files Use of OS/400 API QSQPRCED to perform blocked

insert into local AS/400 target tables during full refresh

Use of CLRPFM (Clear Physical File Member) CL command to clear local AS/400 target tables

Use of Static SQL (instead of Dynamic SQL) when possible

"Direct insert" in remote journal replication

Programming with Data Propagator ASNDONE exit routine is a program that Apply can

optionally call after subscription processing completes, regardless of success or failure

Divided by Apply qualifier Other done logic can be added

Language Source File NameSubscription notify programmember name

Full refresh program member name

C QDPR/QCSRC ASNDONE ASNLOADCOBOL QDPR/QCBLLESRC ASNDONE ASNLOAD

RPG QDPR/QRPGLESRC ASNDONE ASNLOAD

Programming with Data Propagator ASNLOAD full-refresh exit routine is called by Apply

– when a full refresh of a target table is necessary OR – if you specify the name of a full refresh program on the

FULLREFPGM parameter when you start Apply Can be used to do "offline, non-SQL" full refresh Other pre-load logic can also be added ASNDONE/ASNLOAD can be named anything,

located in any library ASNDONE/ASNLOAD can be coded in C, COBOL,

RPG, etc.

Features: Set Subscriptions

Journal

SOURCE CONTROL

BASE

UNIT OF WORK

CHANGE DATA

CONTROL

CAPTURE

TARGET

ADMINISTRATION

APPLY

SET 1, TBL 1

SET 1, TBL 2

SET 1, TBL 3SET 2, TBL 1

SET 2, TBL 2

Features: Set Subscriptions

Maintains referential constraints over replication

Replicates tables within a set in a single unit of work

Coordinates the components of a view subscription

Limits the boundary for cascade rejections due to RI violation or update collision

Features: Update Anywhere

USERTABLE

CDCA

PTU

RE

CD

REPLICA

CA

PTU

RE

AP P

LY

CD

REPLICA

CA

PTU

RE

APP

LY

CD

REPLICA

CA

PTU

RE

APP

LY

Collision detection levelsNoneStandardEnhanced

Reference site

Features: Update Anywhere

Hierarchical Architecture

Three levels of conflict detection

Automatic transaction compensation

Compensation of dependent transactions



AS/400 Data Propagator Useful Internal Advance Replication Features: Extendibility Future Enhancement Conclusion

Extendibility: DataJoiner + Data PropagatorOS/2

AIX

DOS

Windows

HP-UX

Solaris

Transparency

Full SQL

Global OptimizationSingle-DBMS

Image

DB2 for VSE & VMor DB2 for AS/400

DB2 for OS/2DB2 for AIX

Oracle

Sybase

MicrosoftSQL Server

Informix

DB2 Common Server

NT IPCS

DataJoiner

DB2 for MVS,

DB2 for PE

other Relationalor Non-Relational

DBMS

DB2 for NT

DB2 for HP-UXor DB2 for Solaris

VSAM

IMS

APPLY

Extendibility: DataJoiner + Data Propagator

GLOBALCATALOG

COPY

Journal

TargetCONTROL

APPLY

BASE

UNIT OF WORK

CHANGE DATA

CONTROL

CAPTURE

ADMINISTRATION

DataJoiner

Oracle SybaseInformixSQL Server

Operational System

DB2

Extendibility: DataJoiner + Data Propagator

COPY

Target

BASE

STAGING

Triggers

CONTROL

APPLY

ADMINISTRATION

GLOBALCATALOG

DataJoiner

Oracle SybaseInformixMicrosoft SQL Server

Operational System

DB2

Extendibility: Lotus Notes, IMS, Flat Files

Enterprise Data Propagation to NotesDB2 Family

Source Data

DPropR Staging Tables

Updates

Refresh

Trigger

Multi-vendor

DataRefresher

Notes DBNotesPump

IMS

IMS, VSAM, flat files

DataPropagatorRelational

DataPropagatorNon-relational

Updates

Future Enhancement Enhanced Remote Journal Replication

– Administration– Performance

CD Row Changed Only Support– Generate CD Row when there is change in the

captured column– DASD vs CPU

LOB replication Support– Easy management of LOB object across the network

Future Enhancement Delivered as Keyed Stamped Media

– Every new OS/400 customers will have a copy– Try and Buy– Manual available online

BIG INTEGER Replication Support Support for greater than 18 characters object DataLinks Replication

– Integrated with your database and file system– Customer programming for replication

Conclusion: IBM Data Propagator Solution Versatile Infrastructure

– Operational application– Informational application– Mobile & occasionally connected systems– Update anywhere with conflict detection

and automatic compensation Leverages Current Assets

– Legacy Sources– Standard SQL

Efficient & Automated Operations– Journal/Log-based capture for DB2– Trigger-based capture for non-DB2

sources – Network optimizations

Scaleable Design– Servers– Networks

Flexible Distribution– Subsetting with join views– Fan-out distribution

Integrated Administration– DB2 UDB Control Center GUI for

administration for V5– CL commands for pre-V5 function– Automated initialization– Tailorable

Robust Data Enhancements– Derivation, Summarization, Translation...

Multi-vendor Interoperability– SQL based architecture– Architected data staging area– Replication among DB2, Oracle, Sybase,

SQL Server, Informix, IMS, IDMS and VSAM

Conclusion: IBM Data Propagator Solution

DB2 Family

LegacyData

Source Data

Staging Tables

Copies

DataPropagatorRelational

DataRefresher

Capture DB2

Capture IMS

DB2 Family orMulti-vendor

RDBMS

Multi-vendorRDBMS

Lotus Notes

Capture Triggers

NotesPump

DataPropagatorRelationalApply DB2

DataPropagatorNonRelational

Refresh for IMS,VSAM, flat files

DataJoiner Multi-vendor

Access

MobileComputing

Apply for Notes or ODBC

Reference

http://www.software.ibm.com/data/dpropr http://www.software.ibm.com/data/datajoiner http://www.software.ibm.com/data http://as400bks.rochester.ibm.com http://www.as400.ibm.com

THANKS MUCH FOR COMING !!!

Trademark Information AIX, AS/400, DataHub, DataJoiner, DataPropagator, DataRefresher, DB2, DProp,

DRDA, IBM, IMS, MVS, OS/2, OS/400, RS/6000, VM/ESA and VSE/ESA are trademarks or registered trademarks of the IBM Corpopration in the United States or other countries or both.

Microsoft, Windows, and the Windows 95, and the Windows 98, and the Windows NT logos are trademarks or registered trademarks of Microsoft Corporation.

UNIX is a registered trademark in the United States and other countries licensed exclusively through X/Open Company Limited.

Lotus Notes is a trademark of the Lotus Development Corporation.

Other company, product, and service names may be trademarks or service marks of

others.

data replication implementation

Documents