------------------------------------------------------------General information about DDL replication------------------------------------------------------------

Below are the operations which will be replicated onto Target as part of the DDL replication.

###########################################################################################Operations Object------------ --------CREATE, TABLE (includes AS SELECT),INDEX,TRIGGER,SEQUENCE,MATERIALIZED VIEW,VIEW,FUNCTION,PACKAGE,PROCEDURE,SYNONYM,PUBLIC SYNONYMALTER,DROP,RENAME,COMMENT ON2

GRANT,REVOKE TABLE,SEQUENCE,MATERIALIZED VIEW

ANALYZE TABLE,INDEX,CLUSTER

Oracle-reserved schemasThe following schema names are considered Oracle-reserved and must be excluded fromthe Oracle GoldenGate DDL configuration. Oracle GoldenGate will ignore these schemas.

Schema ANONYMOUSAURORA$JIS

$UTILITY$AURORA$ORB$UNAUTHENTICATEDCTXSYSDBSNMPDMSYSDSSYSEXFSYSMDSYSODMODM_MTROLAPSYSORDPLUGINSORDSYSOSE$HTTP$ADMINOUTLNPERFSTATPUBLICREPADMINSYSSYSMANSYSTEMTRACESVRWKPROXYWKSYSWMSYSXDB##########################################################################################

################################Source side golden gate settings################################Server -- nycprdporadb01.qualityhealth.com - 72.251.220.225database -- QHOPRD

GG_HOME -- /ora_acfs01/goldengate/product/11.1.1/gg1GG DB user -- GGADMIN >>>>>> password is available in PMPGG OS user -- grid

################################Target side golden gate settings################################Server -- nycprdprepdb03.qualityhealth.com - 72.251.220.224database -- REPPRDGG_HOME -- /ora_acfs01/goldengate/product/11.1.1/gg1GG DB user -- GGADMIN >>>>>> password is available in PMPGG OS user – grid

Based on previous experiences we have noted few incidents which we need to take care going forward , below are those DDL operations we need to be carefulwhile performing on Source database.

###########################################################################################NOTE:- To enable ddl replication we are using "DDL INCLUDE MAPPED" in both the extract parameters i.e E_QH1 and E_QH2 , that means any DDL operation on the tables which are mentioned in these extract parameters will be replicated to the target side.###########################################################################################

1) If any new table is being created as part of the email client send generally with subject line – “QH Release DDLs for review” then we need to check with client if they want replication to be enabled for any new table which is being created as part of the

release , if so then follow below steps to enable the replication for new table created.

SOURCE- GGSCI>Stop extract E_QH1 Create new table to source and target database from SQL prompt SOURCE- GGSCI>Add new table to the extract parameter file E_QH1 at

the end of the file using “edit params E_QH1” it ‘ll open the parameter file in vi editor.

SOURCE- GGSCI>dblogin userid ggadmin, password <password> -- password is available in PMP

SOURCE- GGSCI>add trandata MTSPROD.<newtable> SOURCE- GGSCI>info trandata MTSPROD.<newtable> SOURCE- GGSCI>start extract E_QH1

2) On Source If any new trigger is added on any tables for which replication is already enabled.

We disabled the trigger replication using "DBOPTIONS SUPPRESSTRIGGERS" on target side , so there is no impact if trigger is added on any table for which replication is running.

3) If Replication on Target side get abandoned due to below errors .

=========================================2013-03-04 03:27:45 WARNING OGG-01154 Oracle GoldenGate Delivery for Oracle, rep1.prm: SQL error 1403 mapping SCOTT.EMP to SCOTT.EMP OCI Error ORA-01403: no data found, SQL <DELETE FROM "SCOTT"."EMP" WHERE "EMPNO" = :b0>.2013-03-04 03:27:45 WARNING OGG-01003 Oracle GoldenGate Delivery for Oracle, rep1.prm: Repositioning to rba 2320 in seqno 3.2013-03-04 03:27:45 ERROR OGG-01296 Oracle GoldenGate Delivery for Oracle, rep1.prm: Error mapping from SCOTT.EMP to SCOTT.EMP.2013-03-04 03:27:45 ERROR OGG-01668 Oracle GoldenGate Delivery for Oracle, rep1.prm: PROCESS ABENDING.=========================================

Then check the table SCOTT.EMP if it has any FKeys enabled , identify the parent table and check if parent table is being replicated or not thru GG , if parent table is not being replicated then possible cause of replication failure is parent data Is not available on target side which is needed by child table , ask client if we can copy the parent table data from source to target , if client allows then copy data and start replication , if client denies then disable FK constraint on child table And start replication.

4) If QHOPRD database get crash on host “nycprdporadb01” and we are having errors while starting GG on Source server.

Errors will look like blow on source server "nycprdporadb01.qualityhealth.com" when starting the "PUMP PROCESSES" =========================================ERROR OGG-01031 Oracle GoldenGate Capture for Oracle, p_qh1.prm: There is a problem in network communication, a remote file problem, encryption keys for target and source do not match (if using ENCRYPT) or an unknown error. (Reply received is Unable to open file "/i01/app/goldengate/product/11.1.1/gg1/dirdat/QH/r1028240" (error 11, Resource temporarily unavailable)).2013-01-27 04:41:49 ERROR OGG-01668 Oracle GoldenGate Capture for Oracle, p_qh1.prm: PROCESS ABENDING.

2013-01-27 04:42:10 ERROR OGG-01031 Oracle GoldenGate Capture for Oracle, p_qh2.prm: There is a problem in network communication, a remote file problem, encryption keys for target and source do not match (if using ENCRYPT) or an unknown error. (Reply received is Unable to open file "/i01/app/goldengate/product/11.1.1/gg1/dirdat/QH/r2000849" (error 11, Resource temporarily unavailable)).2013-01-27 04:42:10 ERROR OGG-01668 Oracle GoldenGate Capture for Oracle, p_qh2.prm: PROCESS ABENDING=========================================

Reason :- If QHOPRD database get crash on host “nycprdporadb01” , it ‘ll start on "nycprdporadb02.qualityhealth.com” due to server level clustering , After that it ‘ll failback to original server "nycprdporadb01.qualityhealth.com" but it left running “PUMP PROCESSES” on "nycprdporadb02” which causes further GoldenGate issues , Solution :- To resolve the issue we need to cleanly stop all GoldenGate processes on failover server "nycprdporadb02” and then start GG on “nycprdporadb01”

PUMP processes on failover server can be found like below.

"PUMP PROCESSES on the failover server "nycprdporadb02.qualityhealth.com":-=====================================[grid@nycprdporadb02 ~]$ ps -ef|grep -i extractgrid 12627 12617 0 Jan26 ? 00:00:29 /ora_acfs01/goldengate/product/11.1.1/gg1/extract PARAMFILE /ora_acfs01/goldengate/product/11.1.1/gg1/dirprm/p_qh1.prm REPORTFILE /ora_acfs01/goldengate/product/11.1.1/gg1/dirrpt/P_QH1.rpt PROCESSID P_QH1 USESUBDIRSgrid 12628 12617 0 Jan26 ? 00:00:28 /ora_acfs01/goldengate/product/11.1.1/gg1/extract PARAMFILE /ora_acfs01/goldengate/product/11.1.1/gg1/dirprm/p_qh2.prm REPORTFILE /ora_acfs01/goldengate/product/11.1.1/gg1/dirrpt/P_QH2.rpt PROCESSID P_QH2 USESUBDIRSgrid 18250 18113 0 09:57 pts/0 00:00:00 grep -i extract =====================================

Kill above two processes and failover server nycprdporadb02 and then start GG processes on nycprdporadb01.

5) Adding partitions to partitioned tables on source.

While adding partitions we need to do additional steps for below table , apart from this all changes will be replicated to target automatically.

MTSPROD.MTS_SITE_PARTICIPATION

Perform below steps

===========================1. GGSCI (dev1) 1> edit params defgen

DEFSFILE /i01/app/goldengate/product/11.1.1/gg1/dirsql/mtsprodactv.sqlUSERID ggadmin, PASSWORD <password is available in PMP >TABLE MTSPROD.MTS_ACTV_RESP_PARTICIPATION;

2. ./defgen paramfile /i01/app/goldengate/product/11.1.1/gg1/dirprm/defgen.prm

3. Copy /ora_acfs01/goldengate/product/11.1.1/gg1/dirsql/mtsprodactv.sql from source to target on same location.

4. start REPLICAT R_QH1===============================