1 copyright © 2012, oracle and/or its affiliates. all rights reserved. · with v.11.2.1,...


Extracting Data in Oracle GoldenGate Integrated Capture Mode Joydip Kundu Director, Software Development Enterprise Replication Oracle Corp


The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracle’s products remains at the sole discretion of Oracle.


Program Agenda

Introduction to Oracle GoldenGate

Overview of Integrated Capture

Benefits of Integration

Deployment Options

Operational Aspects of Integrated Capture

Summary


Log-based Changed Data

Data Integrator

New DB/ HW/OS/APP

Fully Active Distributed DB

Reporting Database

Data Warehouse

ODS

Zero Downtime Upgrade & Migration

Query & Report Offloading

Data Synchronization within the Enterprise

Real-time BI, Operational Reporting, MDM

Event Driven Architecture, SOA

High Availability/ Disaster Recovery

Message Bus

Oracle & Non-Oracle Database(s)

Message Bus

Legacy Systems

Oracle GoldenGate Low-Impact, Real-Time Data Integration & Transactional Replication

Global Data Centers


Oracle GoldenGate Differentiators

Flexible Performance Reliable

• Low latency for data movement

• Low-impact capture, routing, transformation, and delivery of transactional data

• Modern, open, modular architecture

• Supports heterogeneous sources and targets

• Agile and adaptable to modern data integration architectures

• Maintains transactional integrity

• Resilient against interruptions and failures

• Fast Recovery after process interruptions.


How Oracle GoldenGate Works

Source Oracle & Non-Oracle Database(s)

Target Oracle & Non-Oracle

Database(s)

Capture: committed transactions are captured (and can be filtered) as they occur by reading the transaction logs. New with V.11.2.1, GoldenGate offers two options for capture for Oracle; Classic & Integrated Capture

Trail: stages and queues data for routing.

Pump: distributes data for routing to target(s).

Route: data is compressed, encrypted for routing to target(s).

Delivery: applies data with transaction integrity, transforming the data as required.

Capture Trail Files

Pump Delivery

LAN / WAN / Internet Over TCP/IP

Trail Files


How Oracle GoldenGate Works

Source Oracle & Non-Oracle Database(s)

Target Oracle & Non-Oracle

Database(s)

Capture: committed transactions are captured (and can be filtered) as they occur by reading the transaction logs. New with V.11.2.1, GoldenGate offers two options for capture for Oracle; Classic & Integrated Capture

Trail: stages and queues data for routing.

Pump: distributes data for routing to target(s).

Route: data is compressed, encrypted for routing to target(s).

Delivery: applies data with transaction integrity, transforming the data as required.

Capture

Delivery

Trail Files

Pump

Trail Files

Pump

Delivery

Capture

Bi-directional

LAN / WAN / Internet Over TCP/IP

Trail Files

Trail Files


Program Agenda




Deployment Options


Summary


Integrated Capture: What is it?


Classic Capture

Redo Parsing

Oracle GoldenGate: Classic Capture

Memory Manager

SQL Interface

Filtering Formatting Transformation and Mapping

Checkpoint Manager

Transformation Engine

Trail File Generation

Messaging Subsystem

Database Access

Parser

Metadata Manager

GoldenGate Event Log

GoldenGate Checkpoint File

GoldenGate Trail Files

GoldenGate Parameter File

Database Archive/Redo Logs


Oracle GoldenGate: Integrated Capture

Integrated Capture

Memory Manager

SQL Interface

Filtering Formatting Transformation and Mapping

Checkpoint Manager

Transformation Engine

Trail File Generation

Messaging Subsystem

Database Access

Parser

Metadata Manager

LCR Processor

Database LogMining

Server

GoldenGate Event Log

GoldenGate Checkpoint File

GoldenGate Trail Files

GoldenGate Parameter File


Program Agenda




Deployment Options


Summary


Integrated Capture: Enhancing Redo-Based Capture Support


Oracle GoldenGate (Benefits of Integration)

Flexible Performance Reliable

OGG-11.2-CC OGG-11.2-IC Data Type

Compression No Full Support (Basic-, OLTP- and EHCC)

PDML/XA on RAC Distributed Txn

No

Yes

Basicfile LOBs DMLs from redo DBMS_LOB via fetch

Yes

Securefile LOBs

Fetch Based Full Support Fetch: DEDUP/NOLOGGING

XML XML-CLOB : Redo Binary XML: Fetch XML-OR: No XMLType Table: No

Full Support


Oracle GoldenGate (Benefits of Integration)

Flexible Performance Reliable OGG-11.2-CC OGG-11.2-IC

Integration with Database features

ASM Yes (Setup required: DBLOGREADER)

Transparent Support

RMAN Yes (Setup: Register Extract)

Transparent Support

Data Guard Limited Transparent Support RAC User intervention required

for addition/removal of instances

Transparent Support


Program Agenda




Deployment Options


Summary


Supports multiple deployment configuration – On-Source : Source database and Integrated

Capture process are in the same machine – Downstream : Integrated Capture runs on

different database – typically on different machines

Easy transitions for existing GoldenGate

customers – Customers may choose which option they prefer

based on their requirements.

Integrated Capture: Deployment Overview “On Source”

“Downstream”


Architecture – Real time mode

Source database and Integrated Capture process are in the same machine

User/Application changes

Record changes

Capture changes

Redo Logs

Capture

Integrated Extract

Oracle Database

Trail File …..

Source Machine

LCR1 LCR2 LCR3 …. …. ….

Logical Change Records


On-Source Deployment of Integrated Capture

• Register with database – GGSCI> DBLOGIN USER dbuser PASSWORD dbpasswd – GGSCI> REGISTER EXTRACT ext1 DATABASE – Must do this before you can add Extract – Can take a few minutes on large database

• Add Extract – GGSCI> ADD EXTRACT ext1 INTEGRATED TRANLOG, BEGIN NOW

• All other steps remain the same • No change required parameter file


Supported Features Database Version

Oracle Database Versions Supported in OGG 11gR2

Capture ( a k a C l a s s i c

C a p t u r e )

10.2 Supported

11.1.0.7 TDE/TSE, Fetch support for SecureFiles



I n tegrated

Capture

10.2 * Supported

11.1.0.7 * Supports TDE/TSE

11.2.0.2 * Supports TDE/TSE,XA RAC, Compression Fetch support for Securefiles, ADTs, VARRAYS, Nested Tables, Object Tables

11.2.0.3 Supports TDE/TSE, XA RAC, Compression, XML OR, XML Binary Fetch support for Securefiles, ADTs, VARRAYS, Nested Tables, Object Tables


Architecture – Real Time Downstream mode


Record changes

Redo Logs

Capture

Integrated Extract

Oracle Database

Trail File

Source Machine

Source database and Integrated Capture process are in different machine

Standby

Redo Logs

Oracle Database

Downstream Machine

….. Continuous REDO Shipping

Capture changes

LCR1 LCR2 LCR3 …. …. ….


Capture


Architecture – Downstream Archive Log mode


Record changes

Redo Logs

Capture

Integrated Extract

Oracle Database

Trail File

Source Machine

Source database and Integrated Capture process are in different machine

Archive Logs

Oracle Database

Downstream Machine

….. Capture changes

LCR1 LCR2 LCR3 …. …. ….


Capture


Integrated Capture: Downstream Deployment

Database Release

Redo Compatibility

On-Source Deployment

Downstream Deployment

10.2.0.5+ 10.2+ No No support for XML/Objects/Compression/ TDE/TSE

11.1.0.7+ 11.0+ No No support for XML/Objects/Securefiles/ Compression

11.2.0.1/2 11.2+ No No support for XML/Objects

11.2.0.3 11.2+ All features All features


New Concepts (Downstream Deployment)

• GGSCI – Source Database (Existing)

– GGSCI> DBLOGIN userid PASSWORD pwd

– Mining Database (New) – MININGDBLOGIN USERID mdbuser, PASSWORD mpwd – Defaults to DBLOGIN if not specified – Used for creating logmining server – Mandatory for downstream deployment – Must have privileges granted through

SQL > EXEC DBMS_GOLDENGATE_AUTH.GRANT_ADMIN_PRIVILEGE( - grantee => 'gg_admin', - privilege_type => ‘capture’, grant_select_privileges=> true, do_grants => TRUE);


New Concepts (Downstream Deployment)

• Extract – Source Database (Existing)

– USERID user PASSWORD password

– Mining Database (New) – TRANLOGOPTIONS MININGUSER USERID mdbuser, MININGPASSWORD mpassword

– Used for interacting with the logmining server – Mandatory for downstream deployment – Must have privileges granted through

SQL > EXEC DBMS_goldengate_AUTH.GRANT_ADMIN_PRIVILEGE( - grantee => 'gg_admin', - privilege_type => ‘capture’, grant_select_privileges=> true, do_grants => TRUE);


Downstream Deployment of Integrated Capture

• Set up Mining User Account • Use dbms_goldengate_auth.grant_privileges • GRANT SELECT ON V_$DATABASE to <user>

• Configure archival of local redo • Make sure downstream mining database is in archivelog mode • SQL> ALTER SYSTEM SET LOG_ARCHIVE_DEST_1='

LOCATION=/home/arc_dest/local VALID_FOR=(ONLINE_LOGFILE,PRIMARY_ROLE) ' ;

• SQL> ALTER SYSTEM SET LOG_ARCHIVE_DEST_1 ENABLE;


Downstream Deployment of Integrated Capture: Archived Log Only Mode • Prepare Source Database to Send Redo

– Set DG_CONFIG – SQL> ALTER SYSTEM SET LOG_ARCHIVE_CONFIG = ' DG_CONFIG=(dbms1, dbmscap) '

– Set up redo transport – SQL> ALTER SYSTEM SET LOG_ARCHIVE_DEST_2 =

'SERVICE=DMBSCAP.EXAMPLE.COM ASYNC NOREGISTER VALID_FOR=(ONLINE_LOGFILES, PRIMARY_ROLE)

TEMPLATE=/usr/oracle/log_for_dbms1/dbms1_arch_%t_%s_%r.log DB_UNIQUE_NAME = dbmscap‘ – SQL> ALTER SYSTEM SET LOG_ARCHIVE_DEST_STATE_2 = ENABLE


Downstream Deployment of Integrated Capture: Archived Log Only Mode • Set up Integrated Capture on DBMSCAP

– Register Extract with Downstream Database – GGSCI> DBLOGIN userid ggasadm1 password ggs123 – GGSCI> MININGDBLOGIN userid ggadmcap password ggs123 – GGSCI> REGISTER EXTRACT ext1 DATABASE

– Create Extract at the downstream mining database – GGSCI> ADD EXTRACT ext1 INTEGRATED TRANLOG BEGIN NOW

– Edit Extract Parameter file (ext1.prm) – USER ggadm1@dbms1 password – MININGUSER ggsadmcap@dbmscap MININGPASSWORD


Downstream Deployment of Integrated Capture: Real Time Mode • Prepare Mining Database to Receive Redo

– Create Standby Redo Logs (SRL) – SQL> ALTER DATABASE ADD STANDBY LOGFILE GROUP 3 … – Must have 1 more SRL per redo thread – Use V$STANDBY_LOG to verify

– Set archival for SRLs – SQL> ALTER SYSTEM SET LOG_ARCHIVE_DEST_2 =

'LOCATION=/home/arc_dest/srl_dbms1 VALID_FOR=(STANDBY_LOGFILE, PRIMARY_ROLE) ';

– SQL> ALTER SYSTEM SET LOG_ARCHIVE_DEST_STATE_2 = ENABLE


Downstream Deployment of Integrated Capture: Real Time Mode • Prepare Source Database to Send Redo

– Set DG_CONFIG – SQL> ALTER SYSTEM SET LOG_ARCHIVE_CONFIG = ' DG_CONFIG=(dbms1, dbmscap) '

– Set up redo transport – SQL> ALTER SYSTEM SET LOG_ARCHIVE_DEST_2 = 'SERVICE = DBMSCAP.EXAMPLE.COM ASYNC NOREGISTER VALID_FOR=(ONLINE_LOGFILES, PRIMARY_ROLE) DB_UNIQUE_NAME=dbmscap ';


Downstream Deployment of Integrated Capture: Real Time Mode • Set up Integrated Capture on DBMSCAP

– Register Extract with Downstream Database – GGSCI> DBLOGIN userid ggasadm1 password ggs123 – GGSCI> MININGDBLOGIN userid ggadmcap password ggs123 – GGSCI> REGISTER EXTRACT ext1 DATABASE

– Create Extract at the downstream mining database – GGSCI> ADD EXTRACT ext1 INTEGRATED TRANLOG BEGIN NOW

– Edit Extract Parameter file (ext1.prm) – USER ggadm1@dbms1 password – MININGUSER ggsadmcap@dbmscap MININGPASSWORD ggs123 – TRANLOGOPTIONS INTEGRATEDPARAMS (downstream_real_time_mine Y)


Downstream Deployment of Integrated Capture: Redo Log Setup

• Things to Remember • Foreign Archived Logs should not be archived in the FRA (Fast Recovery Area)

• Real time mining can be active for only one source database

• Each SRL must be as large as the largest redo log file at the source database (Recommendation: Use redo logs of uniform size)

• If you have n redo threads each with m redo log groups, you will need n*(m+1) redo log groups at the downstream database

• DBIDs of the source databases must be unique


New Parameters for Integrated Capture

• TRANLOGOPTIONS FETCHPARTIALLOB: • Use it when replicating to non-Oracle databases • Replicat is older than 11.2.1.0.0 • Fetches full LOB for partial LOB operations

• TRANLOGOPTIONS FETCHPARTIALXML • Use it when replicating to non-Oracle databases • Replicat is older than 11.2.1.0.0 • Fetches full XML instead of Oracle proprietary XML difference


Program Agenda




Deployment Options


Summary



• TRANLOGOPTIONS INTEGRATEDPARAMS: • Example: INTEGRATEDPARAMS (max_sga_size 1024, parallelism 2) • max_sga_size

• Amount of shared memory (in MB) to use in the logmining server • Comes from streams_pool • Add up max_sga_size from all Extracts running on a given instance • Make sure you leave 25% of streams_pool for other database

activities like AQ, Datapump • We default it to 1024 MB



• TRANLOGOPTIONS INTEGRATEDPARAMS: • parallelism

• Controls the number of preparer processes • Defaults to 2

• TRANLOGOPTIONS CHECKPOINTRETENTIONTIME • Defaults to 3 days • how long checkpoints taken by the logmining server are saved


Unneeded Parameters for Integrated Capture

• THREADS (RAC Integration) • ASMUSER, ASMBUFSIZE, DBLOGREADER (ASM Integration) • DECRYPTPASSWORD (TDE/TSE Integration)


Integrated Capture: Operational Things to Remember • Cannot alter SCN or Timestamp of Extract process before REGISTER • Logmining Server checkpoints are removed periodically • RDBMS alert log is a rich source of information • Important MOS Notes

• Specific patches for Integrated Capture: 1411356.1 • Integrated Capture health check script 1448324.1 • OGG Best Practices: Configuring Downstream Integrated Capture 1485620.1 • Performance Tuning for OGG 1488668.1

https://bug.oraclecorp.com/pls/bug/webbug_edit.edit_info_top?rptno=1411356�





Looking at the alert log GoldenGate CAPTURE CP01 for OGG$CAP_EXTWELL started with pid=112, OS id=27802 Starting persistent Logminer Session with sid = 6 for GoldenGate Capture

OGG$CAP_EXTWELL LOGMINER: Parameters summary for session# = 6 <= This remains the same LOGMINER: StartScn: 2883760144139 (0x029f.6d7fbf0b) LOGMINER: SkipScn: 2883751645398 (0x029f.6cfe10d6)


Looking at the alert log (contd) LOGMINER: session#=6 (OGG$CAP_EXTWELL), reader MS03 pid=149 OS id=27806 sid=3417 started LOGMINER: session#=6 (OGG$CAP_EXTWELL), builder MS06 pid=158 OS id=27808 sid=6798 started LOGMINER: session#=6 (OGG$CAP_EXTWELL), preparer MS04 pid=181 OS id=27810 sid=456 started LOGMINER: session#=6 (OGG$CAP_EXTWELL), preparer MS05 pid=229 OS id=27812 sid=3431 started LOGMINER: Begin mining logfile during dictionary load for session 6 thread 1 sequence 250, +FRA/… LOGMINER: End mining logfile during dictionary load for session 6 thread 1 sequence 250, +FRA/… LOGMINER: Begin mining logfile for session 6 thread 1 sequence 259, +FRA/… LOGMINER: End mining logfile for session 6 thread 1 sequence 259, +FRA/…


RMAN & Recovery Area Integration • If mining on local database

• Archived logs can be staged in Fast Recovery Area • RMAN will not delete any archived logs that Integrated Capture

needs unless space is exhausted • Note we have to delete otherwise database availability will be

compromised • (alert log): WARNING: The following archived logs needed by

Streams/GoldenGate capture process are being deleted to free space in the recovery area....

• If mining on downstream database • Cannot use Fast Recovery Area to stage foreign archived logs


Program Agenda




Deployment Options


Summary


Additional data type support via redo Compression (Basic, OLTP- and EHCC) PDML/XA on RAC and Distributed Transaction Basicfile and Securefile LOBs (including compression and encryption) XML (CLOB, Binary and OR)

Integrated with Database features RAC ASM Data Guard TDE/TSE

Integrated Capture: Summary


Join the Data Integration Community

Twitter twitter.com/ORCLGoldenGate

Facebook facebook.com/OracleDataIntegration

Oracle’s Data GoldenGate YouTube channel youtube.com/oraclegoldengate

Oracle.com/goto/dataintegration

LinkedIn “Oracle Data Integration”

Oracle’s Data Integration blog blogs.oracle.com/dataintegration


Key Oracle GoldenGate Sessions at OOW

Monday – October 1st Real World Zero Downtime Operations Customer Panel Oracle GoldenGate 11gR2 New Features

1:45pm Moscone West – 3005 3:15pm Moscone West – 3005

Tuesday – October 2nd Oracle GoldenGate and Oracle Data Guard: Working Together Seamlessly Real World Operational Reporting Customer Panel High-Volume OLTP with Oracle GoldenGate: Best Practices from Comcast Everything You Need to Know about Monitoring Oracle GoldenGate

10:15am Moscone South–305 11:45am Moscone West – 3005 1:15pm Moscone West–3022 5pm Moscone West – 3005

Wednesday – October 3rd Best Practices for High Availability with GoldenGate on Oracle Exadata Oracle GoldenGate Case Study: Real-Time Operational Reporting Deployment Oracle GoldenGate Best Practices for Conflict Detection and Resolution Tuning and Troubleshooting Oracle GoldenGate on Oracle Database

11:45am Moscone South – 102 1:15pm Moscone West– 2003 3:30pm Moscone West – 3005 5pm Moscone South –102


Oracle GoldenGate Hands-On-Labs

Wednesday – October 3rd • Introduction to Oracle GoldenGate Veridata 10:15am Marriott Marquis - Salon 1/2

Thursday – October 4th • Deep Dive into Oracle GoldenGate 11:15am Marriott Marquis - Salon 1/2

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 49 After OpenWorld, visit oracle.com/goto/availability

Key HA Sessions and Demos by Oracle Development Monday, 1 October – Moscone South

12:30p Oracle Data Guard Zero-Data-Loss Protection at Any Distance, 300 12:30p Future of Exadata: OLTP, Warehousing, and Consolidation, 104 1:45p Automating ILM with the Latest Database Technology, 300 1:45p Extracting Data in Oracle GoldenGate Integrated Capture Mode, 102 3:15p Maximize Availability with the Latest Database Technology, 303 3:15p Maximize Enterprise Availability with the Latest DB Technology, 303 4:45p Mission-Critical Oracle Exadata OLTP Deployment at PayPal, 300 4:45p Temporal Database Capabilities with the Latest DB Technology, 300 Tuesday, 2 October – Moscone South 10:15a Database Tables to Storage Bits: Data Protection Best Practices, 300 10:15a GoldenGate & Data Guard: Working Together Seamlessly, 305 11:45a Active Data Guard Zero-Downtime Database Maintenance, 300 11:45a Using Automatic Storage Mgmt with the Latest DB Technology, 301 1:15p The Four Ts of RMAN: Tips, Tuning, Troubleshooting, and … ?, 102 5:00p Maximum Availability Architecture Best Practices for Exadata, 303

Wednesday, 3 October – Moscone South 10:15a Operational Best Practices for Oracle Exadata, 102 10:15a Maximize Availability by Minimizing Disruption for End Users and Application, 301 11:45a What’s New in the Latest Generation of Oracle RAC, 301 11:45a Best Practices for HA w/ GoldenGate on Oracle Exadata, 102 1:15p Oracle Secure Backup: Integration Best Practices with Engineered Systems, 300 1:15p Application MAA Best Practices on Oracle Private Clouds, 200 5:00p Tuning &Troubleshooting Oracle GoldenGate on Oracle, 102 Thursday, 4 October – Moscone South 11:15a Integrate Your Globally Distributed IT for Key Cloud Computing Benefits, 300 12:45p Backup and Recovery of Oracle Exadata: Experiences and Best Practices, 300

Demos – Mon 10:00a-6:00p - Tue 9:45a-6:00p - Wed 9:45a-4:00p Oracle Maximum Availability Architecture, S-011 Oracle GoldenGate 11gR2 New Features, S-239 Oracle Database 12c: Global Data Services, S-010 Oracle Database 12c Application Continuity - S-009

Oracle Secure Backup, S-014 Oracle Active Data Guard, S-007 Oracle Recovery Manager and Oracle Flashback Technologies, S-019 Oracle Real Application Clusters and Oracle RAC One Node - S-008 Oracle Database 12c Xstream, Streams, Advanced Queueing, S-018


Brings GoldenGate extract closer to the RDBMS engine Leverages Oracle Database Logmining API

– Seamless traversal of different branches of redo – Parallelism in mining and transforming redo records – Transparent mining of different threads in a RAC – Future enhancement in the RDBMS area will be supported quicker in the product

Multi Threaded – Pipelined architecture. – Performance gain from parallel processing

Integrated Capture: Overview


Integrated Capture: Deployment Options • On-Source or Local Deployment

– Capture runs on the same database – Real time capture of changes – Source database must be at 11.2.0.3+

• Downstream or Remote Deployment – Use redo transport to ship redo logs – No overhead on source – Can ship redo from multiple source databases – Only one source can be mined in real time – Source can be in 10.2.0.5+ – Downstream database must be at 11.2.0.3+ , same platform as source


Description Category

Integrated Capture - What You Need To Know

Exadata Capture support for EHCC compression

Compression Capture support for OLTP, Segment compression

Distributed Transactions XA-RAC, PDML

RAC Dynamic Node Awareness

New Datatypes XML OR, XML Binary

LOB Full and Partial read from REDO log

Deployment Online and Downstream capture support

DDL Captures tables created with column level password specification

1 copyright © 2012, oracle and/or its affiliates. all rights reserved. · with v.11.2.1,...

Documents