OUAG

SANGAM

AIOUG

DEPLOY A DISASTER RECOVERY SOLUTION AND SETUP STRESS TESTING CLUSTER USING CASCADED STANDBY

SESSSION# 201572

PR E PAR E D B Y :

RANDHIR KUMAR

O R AC L E D BA , G A L LU P

2

Oracle DBA at Gallup, an American research-based, global performance-management consulting organization.

Over 5yrs Experience in Oracle products.

Managed Oracle RAC database on RHEL6 server, handful experience in MYSQL

Done R&D projects

in.linkedin.com/in/imrandhirkumar

Speaker Introduction

3

Introduction

Why Cascaded Standby?

Setup Cascaded Standby

Setup Snapshot standby

Setup Stress Cluster

Conversion

Questions

Agenda

RANDHIR KUMAR

PHYSICAL+SNAPSHOT SNAPSHOT PHYSICAL STANDBY

4

Oracle Data Guard : One of the most efficient DR solution using Physical Standby.

Snapshot Standby: A hidden feature of oracle data guard is actually a fully updatable database can be used for stress testing.

INTRODUCTION

PRIMARY

;

How to combine both technology?

RANDHIR KUMAR

5

Why Cascaded Standby?

Allows usage of one Staging Standby as Snapshot Standby Database while other as a DR site.

No extra Bandwidth required, No Production performance overhead.

Also Receives redo from production in the background and ships it to Cascaded Standby.

PRIMARY STANDBY CASCADED STANDBY

SNAPSHOT

RANDHIR KUMAR

6

http://krandhir.wordpress.com/create-a-physical-standby-with-data-guard-broker/

Data Guard Physical Standby Setup

IVRD IVRDSB

PRIMARY PHYSICAL

DATA GURAD BROKER

RANDHIR KUMAR

7

11g Enterprise Edition Database (11.2.0.3) s/w with PSU11

Restrictions with 11.2.0.3 : A Physical Standby Database is the only Standby Database type that can cascade

redo.

The Data Guard broker does not support Cascaded Destinations in 11g.

A Cascaded Standby can’t be created using Active Duplication with active standby as target (Bug 18970361).

Environments

IVRD IVRDDR IVRDSB

PRIMARY STAGING CASCADED

8

Data Guard Architecture

SGA

ORL

ORL

Primary Database

Cascaded Standby Database

Use

r C

om

mit

REDO BUFFER

LGWR

SRL

SRL

ARCH

Database datafile

APPLY

Staging Standby Database

ARCH

LNS RFS APPLY Database datafile

ARCH

RFS ARCH

Database datafile

SRL

SRL

ARCH

RANDHIR KUMAR

9

Second line of defense.

Maintained at geographically remote place to meet DR aspect.

Part of 11g Enterprise Edition License.

Doesn’t supported by Data Guard broker.

Receives archives from either primary or staging standby database

ARCH to ARCH shipping. ASYNC always.

Cascaded Standby Database Cascaded Standby Database

RANDHIR KUMAR

10

Take a backup of Init.ora file and stop archive deletion on staging standby.

Copy password files from primary to respective $OH on Stress Cluster

Add TNS on Primary and staging standby for cascaded standby.

Add static entry for new standby in listener.ora file

Set parametes on both primary and standing standby. Log_Archive_Config=‘dg_config=(IVRD,IVRDSB,IVRDDR)’

Log_Archive_Dest_3='service=IVRDDR VALID_FOR=(STANDBY_LOGFILE,ALL_ROLES) DB_UNIQUE_NAME=IVRDDR'

Log_Archive_Dest_state_3=DEFER

On STAGING DATABASE:

FAL_SERVER=‘IVRD’ and FAL_CLIENT=‘IVRDSB’

CASCADED STANDBY SETUP Cascaded Standby Setup

RANDHIR KUMAR

11

Using RMAN “DUPLICATE FROM ACTIVE DATABASE” to create the Standby Database

Replace IVRD with IVRDDR in init file backed from primary.

Change few required parameters as per cascaded standby. CONTROFILE

LOCAL LISTENER and REMOTE LISTERERS

log_archive_dest_1

LOG_ARCHIVE_DEST_2='SERVICE=IVRD ASYNC VALID_FOR=(ONLINE_LOGFILES,PRIMARY_ROLE) DB_UNIQUE_NAME=IVRD’

LOG_ARCHIVE_DEST_STATE_2=DEFER

Log_Archive_Config==‘dg_config=(IVRD,IVRDSB,IVRDDR)’

FAL_CLIENT=IVRDDR

FAL_SERVER=IVRDSB,IVRD

*.log_file_name_convert='+DISKGROUP1/IVRD/ONLINE','+DISKGROUP6/IVRDDR/ONLINE'

*.db_file_name_convert='+DISKGROUP1/IVRD','+DISKGROUP1/IVRDSB‘

Enable redo transport from staging standby to cascaded standby

Start recovery on cascaded standby.

Cascaded Standby Setup Cascaded Standby Setup

RANDHIR KUMAR

12

Read-Write Mode(fully updatable)

Internally creates a Guaranteed Restore Point(GRP)

Receives REDO from primary but doesn’t not apply, ships to cascaded standby if enabled.

Single DGMGRL command creates GRP, cancels recovery, converts into snapshot and open in read right mode.

Flashback needs to be enabled if converting through DGMGRL

Stop archive deletion on Staging standby.

Snapshot Standby Database

SQL>ALTER SYSTEM SET DB_RECOVERY_FILE_DEST_SIZE=100G;

SQL>ALTER SYSTEM SET DB_RECOVERY_FILE_DEST='+FLASHBACK';

SQL> alter database flashback on;

RANDHIR KUMAR

13

Convert the staging standby into snapshot mode.

New Incarnation on cascaded standby due to conversion=> Terminate the sync up with Primary due to incarnation mismatch.

Setup Snapshot Standby Database

PRIMARY STANDBY CASCADED STANDBY SNAPSHOT

INCRANTION NO 2

INCRANTION NO 2

INCRANTION NO 2

INCRANTION NO 3

INCRANTION NO 3

open resetlogs

RANDHIR KUMAR

14

Solution Stop MRP on Cascaded Standby

Do the conversion

Restore the old incarnation on cascaded standby

Setup Snapshot Standby Database

SQL> alter database recover managed standby database cancel;

DGMGRL> connect sys/pwd

DGMGRL> CONVERT DATABASE 'IVRDSB' TO SNAPSHOT STANDBY;

DGMGRL> SHOW CONFIGURATION;

RMAN> list incarnation;

List of Database Incarnations

DB Key Inc Key DB Name DB ID STATUS Reset SCN Reset Time

------- ------- -------- ------------ ---------- --------- ----------

2 2 IVRD 2093128578 PARENT 2412534122835 22-JUN-14

3 3 IVRD 2093128578 CURRENT 2419817575199 16-AUG-14

RMAN> reset database to incarnation 2;

RANDHIR KUMAR

15

Alert log info

Sun Oct 19 09:58:54 2014

MRP0: Background Media Recovery process shutdown (IVRDSB1)

Created guaranteed restore point

SNAPSHOT_STANDBY_REQUIRED_10/19/2014 09:58:54

Sun Oct 19 09:58:57 2014

RESETLOGS after complete recovery through change

2430651513612

Resetting resetlogs activation ID 2093143682 (0x7cc2d682)

Standby became primary SCN: 2430651513610

Setting recovery target incarnation to 8

CONVERT TO SNAPSHOT STANDBY: Complete - Database mounted as

snapshot standby

Completed: alter database convert to snapshot standby

Sun Oct 19 09:59:03 2014

ALTER DATABASE OPEN

Data Guard Broker initializing...

Data Guard Broker initialization complete

This instance was first to open

Picked broadcast on commit scheme to generate SCNs

Sun Oct 19 09:59:03 2014

Assigning activation ID 2103267323 (0x7d5d4ffb)

Sun Oct 19 09:59:07 2014

Completed: ALTER DATABASE OPEN

Sun Oct 19 09:59:37 2014

RFS[10]: Selected log 11 for thread 1 sequence 685 dbid

2093128578 branch 850929569

sequence 685 for destination LOG_ARCHIVE_DEST_3

Sun Oct 19 09:59:37 2014

Sun Oct 19 09:58:27 2014 Completed: alter database recover managed standby database cancel Sun Oct 19 09:59:08 2014 RFS[1]: New Archival REDO Branch(resetlogs_id): 861357539 Prior: 850929569 RFS[1]: Archival Activation ID: 0x0 Current: 0x7cc2d682 RFS[1]: Effect of primary database OPEN RESETLOGS RFS[1]: Incarnation entry added for Branch(resetlogs_id): 861357539 (IVRDDR1) Sun Oct 19 09:59:08 2014 Setting recovery target incarnation to 8 Archived Log entry 428 added for thread 2 sequence 1 rlc 861357539 ID 0x0 dest 3: RFS[1]: Opened log for thread 1 sequence 1 dbid 2093128578 branch 861357539 Archived Log entry 429 added for thread 1 sequence 1 rlc 861357539 ID 0x7d5d4ffb dest 3: Sun Oct 19 10:00:23 2014 Setting recovery target incarnation to 2 Sun Oct 19 09:59:30 2014 alter database recover managed standby database using current logfile disconnect Sun Oct 19 09:59:37 2014 RFS[2]: Selected log 10 for thread 1 sequence 685 dbid 2093128578 branch 850929569 Sun Oct 19 10:00:47 2014 Media Recovery Log +DISKGROUP6/ivrddr/1_685_850929569.arc Media Recovery Log Sun Oct 19 11:01:56 201Archived Log entry 485 added for thread 2 sequence 585 rlc 850929569 ID 0x7cc2d682 dest 3:

IVRDSB IVRDDR

RANDHIR KUMAR

16

START MRP ON Cascaded standby

Verify the redo shipping to standby even the Standby Database is in open mode. Check the block# column if its changing

Setup Snapshot Standby Database

SQL> alter database recover managed standby database using current

logfile disconnect;

STAGING STANDBY(SNAPSHOT STANDBY) SQL>SELECT STATUS, SEQUENCE#, BLOCK# FROM V$MANAGED_STANDBY WHERE

CLIENT_PROCESS in ('LGWR', 'ARCH');

STATUS SEQUENCE# BLOCK#

------------ ----------------- ------------

IDLE 564 376

IDLE 650 521

CASCADED STANDBY SQL>SELECT STATUS, SEQUENCE#, BLOCK# FROM V$MANAGED_STANDBY WHERE

CLIENT_PROCESS in ( 'ARCH');

STATUS SEQUENCE# BLOCK#

------------ ---------- ----------

CLOSING 518 1

CLOSING 627 1

RANDHIR KUMAR

17

Check the recovery on Cascaded standby when Staging standby is in snapshot mode.

Setup Snapshot Standby Database

On IVRDDR

SQL>select to_char(current_scn)

from v$database;

TO_CHAR(CURRENT_SCN)

-------------------------------

2428647321575

ON IVRD

SQL>select

scn_to_timestamp(2428647321575)

from dual;

SCN_TO_TIMESTAMP(2428647321575)

-------------------------------

02-OCT-14 07.02.23.000000000 AM

Alert log IVRDDR Sun Oct 19 10:02:19 2014 RFS[2]: Selected log 10 for thread 1 sequence 690 dbid 2093128578 branch 850929569 Sun Oct 19 10:02:19 2014 Archived Log entry 442 added for thread 1 sequence 690 ID 0x7cc2d682 dest 1: Archived Log entry 443 added for thread 2 sequence 4 rlc 861357539 ID 0x7d5d4ffb dest 3: RFS[1]: Opened log for thread 2 sequence 575 dbid 2093128578 branch 850929569 Archived Log entry 444 added for thread 2 sequence 575 rlc 850929569 ID 0x7cc2d682 dest 3: Archived Log entry 447 added for thread 1 sequence 6 rlc 861357539 ID 0x7d5d4ffb dest 3: Sun Oct 19 11:06:30 2014 alter database recover managed standby database using current logfile disconnect Sun Oct 19 11:06:31 2014 MRP0 started with pid=42, OS id=27661 MRP0: Background Managed Standby Recovery process started (IVRDDR1) started logmerger process Sun Oct 19 11:06:37 2014 Managed Standby Recovery starting Real Time Apply Sun Oct 19 11:06:37 2014 Media Recovery Log +DISKGROUP6/ivrddr/1_690_850929569.arc Media Recovery Log +DISKGROUP6/ivrddr/2_574_850929569.arc Media Recovery Log +DISKGROUP6/ivrddr/2_575_850929569.arc

RANDHIR KUMAR

18

Create and start Cluster Services for Snapshot Standby Databases

Change Database Links on Snapshot Standby’s to Standby Service names.

Change password of Application and User accounts to be different from production.

Comment the code for SMTP Server to prevent any unwanted emails.

Job Classes will not run as Job services point to primary services. So modify the job class attribute with new services

Change the DATABASE_ROLE to SNAPSHOT STANDBY for the jobs that you want to run on Standby Stress Cluster

Disable Materialized views not needed or dependent upon anything outside of Stress Cluster

Add Snapshot Standby databases to OEM for management

Enable Scheduler Jobs

Setup Stress Cluster

RANDHIR KUMAR

19

Job Classes will not run as Job services will be different. Follow the below to enable jobs

Change the DATABASE_ROLE to SNAPSHOT STANDBY for the jobs that you want to run on Standby Stress Cluster.

Scheduler Jobs in Snapshot Standby Mode

Alter Job Class to attach to new Job Services BEGIN

DBMS_SCHEDULER.SET_ATTRIBUTE (

name =>'IVR_JOB_CLASS ',

attribute => 'SERVICE',

value => 'IVRSBJOB');

end;

BEGIN

DBMS_SCHEDULER.SET_ATTRIBUTE (

name => ‘’TEST_REMINDER_EMAIL',

attribute => 'database_role',

value =>'SNAPSHOT STANDBY');

END;

END;

RANDHIR KUMAR

20

Conversion(Back to Physical) Delete snapshot standby archivelogs

Convert into Physical Standby

DISABLE FLASHBACK on Staging Database

Enable archive deletion on Staging Database.

Select 'delete noprompt archivelog '||''''||name||''';' from

v\$archived_log a where registrar='ARCH' and STATUS = 'A' order by

first_time desc;

DGMGRL> connect sys/pwd

CONVERT DATABASE 'IVRDSB' TO PHYSICAL STANDBY;

DGMGRL> SHOW CONFIGURATION

Alter database flashback off;

RANDHIR KUMAR

21

DBMS Scheduler for SNAPSHOT STANDBY bug 13399711

Switchover Bug 11719886

Cascaded Standby can’t be created using Active Duplication with active standby as target

18970361

Bugs and Patches

22

References

Oracle Data Guard 11g Handbook

Oracle Documentation

THANKS FOR ATTENDING!