8398 goldengate integrated capture 1888658

8/12/2019 8398 Goldengate Integrated Capture 1888658

1/54

Copyright 2012, Oracle and/or its affiliates. All ri ghts reserved.1


2/54


Extracting Data in OracleGoldenGate IntegratedCapture Mode

Joydip KunduDirector, Software DevelopmentEnterprise ReplicationOracle Corp


3/54


The following is intended to outline our general product

direction. It is intended for information purposes only,and may not be incorporated into any contract.It is not a commitment to deliver any material, code, orfunctionality, and should not be relied upon in makingpurchasing decisions. The development, release, and

timing of any features or functionality described forOracles products remains at the sole discretion ofOracle.


4/54


Program Agenda

Introduction to Oracle GoldenGate

Overview of Integrated Capture

Benefits of Integration

Deployment Options

Operational Aspects of Integrated Capture

Summary


5/54Copyright 2012, Oracle and/or its affiliates. All ri ghts reserved.5

Introduction to OracleGoldenGate


6/54Copyright 2012, Oracle and/or its affiliates. All ri ghts reserved.6

Log-based

Changed Data

Data Integrator

New DB/HW/OS/APP

Fully ActiveDistributedDB

ReportingDatabase

Data

Warehouse

ODS

Zero

Quer

Data S

Rea

Even

High

MessageBus

Oracle & Non-OracleDatabase(s)

Message Bus

Legacy Systems

Oracle GoldenGateLow-Impact, Real-Time Data Integration & Transactional R

GlobalDataCenters


7/54


Oracle GoldenGateDifferentiators

FlexiblePerformance R

Low latency for datamovement

Low-impact capture,

routing, transformation,and delivery oftransactional data

Modern, open, modulararchitecture

Supports heterogeneous

sources and targets

Agile and adaptable tomodern data integrationarchitectures

Maintains integrity

Resilient a

interruptio

Fast Recoprocess in


8/54


How Oracle GoldenGate Works

Source

Oracle & Non-Oracle

Database(s)

Capture: committed transactions are captured (and can be filtered) as they occur by readilogs. New with V.11.2.1, GoldenGate offers two options for capture for Oracle; Classic &

Trail: stages and queues data for routing.

Pump: distributes data for routing to target(s).

Route: data is compressed, encrypted for routin

Delivery: applies data with trtransforming the data as req

Capture Trail

Files

Pump Delive

LAN / WA N / Internet

Over TCP/IP

Trail

Files


9/54


How Oracle GoldenGate Works

Source

Oracle & Non-Oracle

Database(s)

Capture: committed transactions are captured (and can be filtered) as they occur by readilogs. New with V.11.2.1, GoldenGate offers two options for capture for Oracle; Classic &

Trail: stages and queues data for routing.

Pump: distributes data for routing to target(s).

Route: data is compressed, encrypted for routin

Delivery: applies data with trtransforming the data as req

Capture

Delivery

Trail

Files

Pump

Trail

FilesPump

Delive

Cap

Bi-directional

LAN / WA N / Internet

Over TCP/IPTrail

Files

TrailFiles


10/54


Program Agenda




Deployment Options


Summary


11/54


Integrated Capture:What is it?


12/54


Classic Capture

Redo Parsing

Oracle GoldenGate: Classic Capture

Memory Manager

SQL Interface

FilteringFormattingTransformation and

Mapping

Checkpoint Manager

TransformationEngine

TGe

Messaging Subsystem

Database Access

Parser

MetadataManager

GoldenGateParameter File

DatabaseArchive/Redo Logs


13/54


Oracle GoldenGate: Integrated Capture

Integrated Capture

Memory Manager

SQL Interface

FilteringFormattingTransformation and

Mapping

Checkpoint Manager

TransformationEngine

TGe

Messaging Subsystem

Database Access

Parser

MetadataManager

LCRProcessor

DatabaseLogMining

Server

GoldenGateParameter File


14/54


Program Agenda




Deployment Options


Summary


15/54


16/54


Oracle GoldenGate (Benefits of Integra


OGG-11.2-CC OGG-11.2-

DataType

Compression No Full Support

(Basic-, OLTPPDML/XA on RACDistributed Txn

No Yes

Basicfile LOBs DMLs from redoDBMS_LOB via fetch

Yes

Securefile LOBs Fetch Based Full Support

Fetch: DEDUP

XML XML-CLOB : RedoBinary XML: FetchXML-OR: NoXMLType Table: No

FullSupport


17/54


Oracle GoldenGate (Benefits of Integra


OGG-11.2-CC OGG-11.2-

IntegrationwithDatabasefeatures

ASM Yes (Setup required:DBLOGREADER)

Transparent

RMAN Yes (Setup: RegisterExtract)

Transparent

Data Guard Limited Transparent

RAC User intervention requiredfor addition/removal ofinstances

Transparent


18/54


Program Agenda




Deployment Options


Summary


19/54


Supports multiple deployment configuration

On-Source : Source database and IntegratedCapture process are in the same machine

Downstream : Integrated Capture runs ondifferent database typically on differentmachines

Easy transitions for existing GoldenGatecustomers

Customers may choose which option they preferbased on their requirements.

Integrated Capture: Deployment Overv

A hit t R l t i d


20/54


Architecture Real t ime mode

Source database and Integrated Capture process are in the

User/Application changes

Recordchanges

Capturechanges

Redo Logs

Capture

Integrated

ExtractOracle Database

TrailFile

..

Source Machine

LCR1

LCR2LCR3.

.

.

Logical ChangeRecords


21/54


On-Source Deployment of Integrated C

Register with database GGSCI> DBLOGIN USER dbuser PASSWORD dbpasswd

GGSCI> REGISTER EXTRACT ext1 DATABASE

Must do this before you can add Extract

Can take a few minutes on large database

Add Extract GGSCI> ADD EXTRACT ext1 INTEGRATED TRANLOG, BEGIN NOW

All other steps remain the same

No change required parameter file

O l D t b V i S t d i OGG


22/54


Supported FeatDatabase Version

Oracle Database Versions Supported in OGG

C a p t u r e( a k a C l a s s i c

C a p t u r e )

10.2 Supported

11.1.0.7 TDE/TSE, Fetch support for SecureF11.2.0.2 TDE/TSE, Fetch support for Secure

11.2.0.3 TDE/TSE, Fetch support for SecureF

I n t e g r a t e dC a p t u r e

10.2 * Supported

11.1.0.7 * Supports TDE/TSE

11.2.0.2 * Supports TDE/TSE,XA RAC, CompreFetch support for Securefiles, ADTs, Tables, Object Tables

11.2.0.3 Supports TDE/TSE, XA RAC, ComprBinaryFetch support for Securefiles, ADTs, Tables, Object Tables

A hit t R l Ti D t d


23/54


Architecture Real Time Downstream mode


Recordchanges

Redo Logs

Captu

reOracle Database

Source Machine

Source database and Integrated Capture process are in diff

StandbyRedo Logs

Oracle Database

Downstream Machine

Continuous REDOShipping

Capturechanges

LCR1

LCR2

LCR3.

.

.


Architecture Downstream Archive Log mode


24/54


Architecture Downstream Archive Log mode


Recordchanges

Redo Logs

Captu

reOracle Database

Source Machine

Source database and Integrated Capture process are in diff

Archive Logs

Oracle Database

Downstream Machine

Capturechanges

LCR1

LCR2

LCR3.

.

.


I d C D D l


25/54


Integrated Capture: Downstream Deplo

Database

Release

Redo

Compatibility

On-Source

Deployment

Downstream

Deployment

10.2.0.5+ 10.2+ No No support for

XML/Objects/Compr

TDE/TSE

11.1.0.7+ 11.0+ No No support for

XML/Objects/Secure

Compression

11.2.0.1/2 11.2+ No No support for XML/

11.2.0.3 11.2+ All features All features

N C t (D t D l


26/54


New Concepts (Downstream Deployme

GGSCI

Source Database (Existing)

GGSCI> DBLOGIN userid PASSWORD pwd

Mining Database (New) MININGDBLOGIN USERID mdbuser, PASSWORD mpwd

Defaults to DBLOGIN if not specified

Used for creating logmining server

Mandatory for downstream deployment

Must have privileges granted through

SQL > EXEC DBMS_GOLDENGATE_AUTH.GRANT_ADMIN_PRgrantee => 'gg_admin', -privilege_type => capture, grant_select_privileges=> tru

do_grants => TRUE);



27/54



Extract

Source Database (Existing)

USERID user PASSWORD password

Mining Database (New) TRANLOGOPTIONS MININGUSER USERID mdbuser, MININGPASSWORD mp

Used for interacting with the logmining server

Mandatory for downstream deployment

Must have privileges granted through

SQL > EXEC DBMS_goldengate_AUTH.GRANT_ADMIN_PRIVILgrantee => 'gg_admin', -privilege_type => capture, grant_select_privileges=> tru

do_grants => TRUE);

D t D l t f I t t d


28/54


Downstream Deployment of Integrated

Set up Mining User Account

Use dbms_goldengate_auth.grant_privileges

GRANT SELECT ON V_$DATABASE to

Configure archival of local redo

Make sure downstream mining database is in archivelog mo

SQL> ALTER SYSTEM SET LOG_ARCHIVE_DEST_1='LOCATION=/home/arc_dest/local

VALID_FOR=(ONLINE_LOGFILE,PRIMARY_ROLE) ' ; SQL> ALTER SYSTEM SET LOG_ARCHIVE_DEST_1 ENABLE;

D t D l t f I t t d


29/54


Downstream Deployment of IntegratedArchived Log Only Mode

Prepare Source Database to Send Redo

Set DG_CONFIG SQL> ALTER SYSTEM SET LOG_ARCHIVE_CONFIG

= ' DG_CONFIG=(dbms1, dbmscap) '

Set up redo transport

SQL> ALTER SYSTEM SET LOG_ARCHIVE_DEST_2 =

'SERVICE=DMBSCAP.EXAMPLE.COM ASYNC NOREGISTEVALID_FOR=(ONLINE_LOGFILES, PRIMARY_ROLE)

TEMPLATE=/usr/oracle/log_for_dbms1/dbms1_arch_%t_%s

DB_UNIQUE_NAME = dbmscap

SQL> ALTER SYSTEM SET LOG_ARCHIVE_DEST_STATE_

Do nstream Deplo ment of Integrated


30/54


Downstream Deployment of IntegratedArchived Log Only Mode

Set up Integrated Capture on DBMSCAP

Register Extract with Downstream Database GGSCI> DBLOGINuserid ggasadm1 password ggs123

GGSCI> MININGDBLOGINuserid ggadmcap password ggs12


Create Extract at the downstream mining database

GGSCI> ADD EXTRACT ext1 INTEGRATED TRANLOG BEG

Edit Extract Parameter file (ext1.prm)

USER ggadm1@dbms1 password

MININGUSER ggsadmcap@dbmscap MININGPASSWORD



31/54


Downstream Deployment of IntegratedReal Time Mode

Prepare Mining Database to Receive Redo

Create Standby Redo Logs (SRL) SQL> ALTER DATABASE ADD STANDBY LOGFILE GROUP

Must have 1 more SRL per redo thread

Use V$STANDBY_LOGto verify

Set archival for SRLs

SQL> ALTER SYSTEM SET LOG_ARCHIVE_DEST_2='LOCATION=/home/arc_dest/srl_dbms1

VALID_FOR=(STANDBY_LOGFILE, PRIMARY_ROLE) ';

SQL> ALTER SYSTEM SET LOG_ARCHIVE_DEST_STATE_



32/54


Downstream Deployment of IntegratedReal Time Mode

Prepare Source Database to Send Redo

Set DG_CONFIG SQL> ALTER SYSTEM SET LOG_ARCHIVE_CONFIG

= ' DG_CONFIG=(dbms1, dbmscap) '

Set up redo transport

SQL> ALTER SYSTEM SET LOG_ARCHIVE_DEST_2 =

'SERVICE = DBMSCAP.EXAMPLE.COM ASYNC NOREGIST

VALID_FOR=(ONLINE_LOGFILES, PRIMARY_ROLE)

DB_UNIQUE_NAME=dbmscap ';



33/54


Downstream Deployment of IntegratedReal Time Mode Set up Integrated Capture on DBMSCAP

Register Extract with Downstream Database

GGSCI> DBLOGIN userid ggasadm1 password ggs123

GGSCI> MININGDBLOGINuserid ggadmcap password ggs12


Create Extract at the downstream mining database

GGSCI> ADD EXTRACT ext1 INTEGRATED TRANLOG BEG

Edit Extract Parameter file (ext1.prm) USER ggadm1@dbms1 password

MININGUSER ggsadmcap@dbmscap MININGPASSWORD g

TRANLOGOPTIONS INTEGRATEDPARAMS (downstream_re



34/54


Downstream Deployment of IntegratedRedo Log Setup

Things to Remember

Foreign Archived Logs should not be archived in the FRA (Fast R

Real time mining can be active for only onesource database

Each SRL must be as large as the largest redo log file at the sou(Recommendation: Use redo logs of uniform size)

If you have nredo threads each with mredo log groups, you wiredo log groups at the downstream database

DBIDs of the source databases must be unique

New Parameters for Integrated Captur


35/54



TRANLOGOPTIONS FETCHPARTIALLOB:

Use it when replicating to non-Oracle databases

Replicat is older than 11.2.1.0.0

Fetches full LOB for partial LOB operations

TRANLOGOPTIONS FETCHPARTIALXML

Use it when replicating to non-Oracle databases

Replicat is older than 11.2.1.0.0

Fetches full XML instead of Oracle proprietary XML

Program Agenda


36/54


Program Agenda




Deployment Options


Summary



37/54



TRANLOGOPTIONS INTEGRATEDPARAMS:

Example: INTEGRATEDPARAMS (max_sga_size 1024,

max_sga_size

Amount of shared memory (in MB) to use in the logm

Comes from streams_pool

Add up max_sga_size from all Extracts running on a g

Make sure you leave 25% of streams_pool for other dactivities like AQ, Datapump

We default it to 1024 MB



38/54



TRANLOGOPTIONS INTEGRATEDPARAMS:

parallelism

Controls the number of preparer processes

Defaults to 2

TRANLOGOPTIONS CHECKPOINTRETENTIONTIME

Defaults to 3 days

how long checkpoints taken by the logmining server

Unneeded Parameters for Integrated C


39/54


Unneeded Parameters for Integrated C

THREADS (RAC Integration)

ASMUSER, ASMBUFSIZE, DBLOGREADER (ASM Inte

DECRYPTPASSWORD (TDE/TSE Integration)

Integrated Capture:


40/54


Integrated Capture:Operational Things to Remember

Cannot alter SCN or Timestamp of Extract process befo

Logmining Server checkpoints are removed periodically RDBMS alert log is a rich source of information

Important MOS Notes

Specific patches for Integrated Capture: 1411356.1

Integrated Capture health check script 1448324.1

OGG Best Practices: Configuring Downstream Integrated Captur

Performance Tuning for OGG 1488668.1

Looking at the alert log
https://bug.oraclecorp.com/pls/bug/webbug_edit.edit_info_top?rptno=1411356https://bug.oraclecorp.com/pls/bug/webbug_edit.edit_info_top?rptno=1448324https://bug.oraclecorp.com/pls/bug/webbug_edit.edit_info_top?rptno=1488668https://bug.oraclecorp.com/pls/bug/webbug_edit.edit_info_top?rptno=1488668https://bug.oraclecorp.com/pls/bug/webbug_edit.edit_info_top?rptno=1448324https://bug.oraclecorp.com/pls/bug/webbug_edit.edit_info_top?rptno=1411356


41/54


Looking at the alert log

GoldenGate CAPTURE CP01 for OGG$CAP_EXTWELL started with pid=11

Starting persistent Logminer Session with sid = 6for GoldenGate Capture

OGG$CAP_EXTWELL

LOGMINER: Parameters summary for session# = 6


42/54


Looking at the alert log (contd)

LOGMINER: session#=6(OGG$CAP_EXTWELL), readerMS03 pid=149 OS id=27806 sid=3

LOGMINER: session#=6(OGG$CAP_EXTWELL), builderMS06 pid=158 OS id=27808 sid=6

LOGMINER: session#=6(OGG$CAP_EXTWELL), preparerMS04 pid=181 OS id=27810 sid=

LOGMINER: session#=6(OGG$CAP_EXTWELL),preparerMS05 pid=229 OS id=27812 sid=

LOGMINER: Begin mining logfile during dictionary loadfor session6thread 1 sequence 250, +

LOGMINER: End mining logfileduring dictionary loadfor session6thread 1 sequence 250, +

LOGMINER: Begin mining logfile for session 6 thread 1 sequence 259, +FRA/

LOGMINER: End mining logfile for session 6 thread 1 sequence 259, +FRA/

RMAN & Recovery Area Integration


43/54


RMAN & Recovery Area Integration If mining on local database

Archived logs can be staged in Fast Recovery Area

RMAN will not delete any archived logs that Integrated needs unless space is exhausted

Note we have to delete otherwise database availabcompromised

(alert log): WARNING: The following archived logs needed by

Streams/GoldenGate capture process are being deleted to frerecovery area....

If mining on downstream database

Cannot use Fast Recovery Area to stage foreign archiv

Program Agenda


44/54


Program Agenda




Deployment Options


Summary

Integrated Capture: Summary


45/54


Additional data type support via redo

Compression (Basic, OLTP- and EHCC)

PDML/XA on RAC and Distributed Transaction

Basicfile and Securefile LOBs (including compression and

XML (CLOB, Binary and OR)

Integrated with Database features

RACASM

Data Guard

TDE/TSE

g p y

Join the Data Integration Community


46/54


g y

Twitter

twitter.com/ORCLGoldenGate

Facebookfacebook.com/OracleDataIntegration

Oracles Data GoldenGate YouTube channelyoutube.com/oraclegoldengate

Oracle.com/goto/datain

LinkedInOracle Data Integra

Oracles Data Integblogs.oracle.com/da

Key Oracle GoldenGate Sessions at OO


47/54


y

Monday October 1st

Real World Zero Downtime Operations Customer Panel

Oracle GoldenGate 11gR2 New Features

1:45pm Moscone

3:15pm Moscone

Tuesday October 2nd

Oracle GoldenGate and Oracle Data Guard: Working Together Seamlessly

Real World Operational Reporting Customer Panel

High-Volume OLTP with Oracle GoldenGate: Best Practices from Comcast

Everything You Need to Know about Monitoring Oracle GoldenGate

10:15am Moscon

11:45am Moscon

1:15pm Moscon

5pm Moscone W

Wednesday October 3rd Best Practices for High Availability with GoldenGate on Oracle Exadata

Oracle GoldenGate Case Study: Real-Time Operational Reporting Deployment

Oracle GoldenGate Best Practices for Conflict Detection and Resolution

Tuning and Troubleshooting Oracle GoldenGate on Oracle Database

11:45am Moscon

1:15pm Moscon

3:30pm Moscone

5pm Moscone S

Oracle GoldenGate Hands-On-Labs


48/54


Wednesday October 3rd

Introduction to Oracle GoldenGate Veridata 10:15am Marriot

Thursday October 4th

Deep Dive into Oracle GoldenGate 11:15am Marriott

Key HA Sessions and Demos by Oracle Dev


49/54

Copyright 2012, Oracle and/or its affiliates. All ri ghts reserved.49 After OpenWorld, visit oracle.com/g

Monday, 1 October Moscone South

12:30pOracle Data Guard Zero-Data-Loss Protection at Any Distance, 300

12:30p Future of Exadata: OLTP, Warehousing, and Consolidation, 104

1:45pAutomating ILM with the Latest Database Technology, 300

1:45p Extracting Data in Oracle GoldenGate Integrated Capture Mode, 102

3:15pMaximize Availability with the Latest Database Technology, 303

3:15p Maximize Enterprise Availability with the Latest DB Technology, 3034:45pMission-Critical Oracle Exadata OLTP Deployment at PayPal, 300

4:45p Temporal Database Capabilities with the Latest DB Technology, 300

Tuesday, 2 October Moscone South

10:15aDatabase Tables to Storage Bits: Data Protection Best Practices, 300

10:15a GoldenGate & Data Guard: Working Together Seamlessly, 305

11:45aActive Data Guard Zero-Downtime Database Maintenance, 300

11:45a Using Automatic Storage Mgmt with the Latest DB Technology, 301

1:15pThe Four Ts of RMAN: Tips, Tuning, Troubleshooting, and ?, 102

5:00pMaximum Availability Architecture Best Practices for Exadata, 303

Wednesday, 3 October Moscone South

10:15aOperational Best Practices for Orac

10:15a Maximize Availability by Minimizing

and Application, 301

11:45aWhats New in the Latest Generatio

11:45a Best Practices for HA w/ GoldenGa

1:15pOracle Secure Backup: Integration B

Engineered Systems, 300

1:15p Application MAA Best Practices on O

5:00pTuning &Troubleshooting Oracle G

Thursday, 4 October Moscone South

11:15aIntegrate Your Globally Distributed

Cloud Computing Benefits, 300

12:45pBackup and Recovery of Oracle Ex

and Best Practices, 300

Demos Mon 10:00a-6:00p - Tue 9:45a-6:00p- Wed 9:45a-4:00pOracle Maximum Availability Architecture, S-011

Oracle GoldenGate 11gR2 New Features, S-239

Oracle Database 12c: Global Data Services, S-010

Oracle Database 12c Application Continuity - S-009

Oracle Secure Backup, S-014

Oracle Active Data Guard, S-007

Oracle Recovery Manager and Oracle Flash

Oracle Real Application Clusters and Oracle

Oracle Database 12c Xstream, Streams, Ad


50/54



51/54


Integrated Capture: Overview


52/54


Brings GoldenGate extract closer to the RDBMS engine

Leverages Oracle Database Logmining API

Seamless traversal of different branches of redo

Parallelism in mining and transforming redo records

Transparent mining of different threads in a RAC

Future enhancement in the RDBMS area will be supported quicke

Multi Threaded

Pipelined architecture.

Performance gain from parallel processing

Integrated Capture: Deployment Option


53/54


On-Source or Local Deployment

Capture runs on the same database

Real time capture of changes

Source database must be at 11.2.0.3+

Downstream or Remote Deployment

Use redo transport to ship redo logs

No overhead on source

Can ship redo from multiple source databases Only one source can be mined in real time

Source can be in 10.2.0.5+

Downstream database must be at 11.2.0.3+ , same platform


54/54

8398 goldengate integrated capture 1888658

Documents