Configuring Standby Database(Dataguard) on R12 using RMAN Hot Backup Configuring Standby Database(Dataguard) on R12 using RMAN Hot Backup

Applies to:Oracle Applications Technology Stack - Version: 12.0.4 to 12.0.4 Linux x86

PurposeThe purpose of this note to provide case study on setting up Dataguard (Standby Database) Configuration on R12 using RMAN Hot Backup

Case Study : Configuring Standby Database(Dataguard) on R12 using RMAN Hot BackupChecked for relevance by ATG-ICM Subject Matter Expert (SME): 28-APR-2009 Implementing Dataguard Physical Standby on R12 on 10g Database Using RMAN Hot Backup ================================================== ===================== This case study was implemented based on steps from the official Note 452056.1 Business Continuity for Oracle Applications Release 12 on Database Release 10gR2 and should be used as a guide only along with the steps in aforementioned document. STEPS: ===== ==> Machine 01 has an E-Business Suite Release 12.0.4 (PROD) DB_NAME=PROD01 ==> Machine 02 is clean and will be used as the Standby machine DB_NAME=PROD02 ==> Machine 03 will be clean and used as the Observer Prepare the Existing Database as PRIMARY ==> Machine 01

1. Create Database Password Files cd $ORACLE_HOME/dbs orapwd file=orapw 2. Create an initialization parameter include file (IFILE) and add ifile entry in the pfile Create IFILE as follows: [oracle@machine01 dbs]$ vi PROD_machine01_ifile.ora Add the following Entry at end of pfile IFILE=$ORACLE_HOME/dbs/PROD_ 3. Create directory called " PROD_machine01" and place all network files under this folder [oracle@machine01 PROD_machine01]$ mkdir PROD_machine01 Add IFILE entry on both TNSNAMES.ORA and LISTENER.ORA to use IFILE at the end of file In Listener.ora IFILE=$ORACLE_HOME/network/admin/PROD_machine01/listener_ifile.ora In Tnsnames.ora IFILE=$ORACLE_HOME/network/admin/PROD_machine01/PROD_machine01_ifile.ora Create common IFILE for both Listener and Tnsnames entry [oracle@machine01 PROD_machine01]$ vi PROD_machine01_ifile.ora 4. Add the following entries to the tnsnames IFILE for standby configuration PROD01= (DESCRIPTION= (ADDRESS_LIST= (LOAD_BALANCE=YES) (FAILOVER=YES) (ADDRESS=(PROTOCOL=tcp)(HOST=machine01.oracle.com)(PORT=1533)) ) (CONNECT_DATA= (SID=PROD) ) )

PROD02= (DESCRIPTION= (ADDRESS_LIST= (LOAD_BALANCE=YES) (FAILOVER=YES) (ADDRESS=(PROTOCOL=tcp)(HOST=machine02.oracle.com)(PORT=1533)) ) (CONNECT_DATA= (SID=PROD) ) )

5. On PRIMARY , set the following parameters in the initialization parameter include file (IFILE) LOG_ARCHIVE_DEST_1='LOCATION=USE_DB_RECOVERY_FILE_DEST MANDATORY' LOG_ARCHIVE_FORMAT='%t_%s_%r.dbf' DB_RECOVERY_FILE_DEST_SIZE = 107374182400 DB_RECOVERY_FILE_DEST= '/oracle/archive' DB_UNIQUE_NAME=PROD01 LOG_ARCHIVE_CONFIG='dg_config=(PROD01,PROD02)' LOG_ARCHIVE_DEST_2='service=PROD02 valid_for=(online_logfiles,primary_role) db_unique_name=PROD02 LGWR ASYNC=20480 OPTIONAL REOPEN=15 NET_TIMEOUT=30' LOG_ARCHIVE_DEST_STATE_2=enable FAL_SERVER='PROD02' FAL_CLIENT='PROD01' #standby_archive_dest='LOCATION=USE_DB_RECOVERY_FILE_DEST' standby_file_management=AUTO parallel_execution_message_size=8192 Refer the following URL (10g Documentation to understand the above parameters) http://download.oracle.com/docs/cd/B19306_01/server.102/b14239/create_ ps.htm#i63561 6. Enable Archivelog mode in PRIMARY and enable Flashback (see Failover section for more information) SQL>archive log list >The output for above command will show whether archive is enabled or Not < >If output shows as follows,then proceed for next step in this section to ENABLE ARCHIVE LOG Mode < SQL> archive log list Database log mode No Archive Mode Automatic archival Disabled

Archive destination /oracle/PROD/10.2.0/db_1/RDBMS/ Oldest online log sequence 386 Current log sequence 387 SQL> shutdown immediate SQL> startup mount SQL> alter database force logging; SQL> alter database archivelog; SQL> alter database flashback on; >> 7. Create standby redolog >>Create standby redo logs on the primary database to support the standby role. The standby redo logs must be the same size as the primary database online redo logs. It is recommended that the number of standby redo logs is one more than the number of online redo logs. alter database add standby logfile thread 1 group 3 ( '/oracle/PROD/db/apps_st/data/stdbylog03a.dbf', '/oracle/PROD/db/apps_st/data/stdbylog03b.dbf') size 1G; SQL> alter database add standby logfile thread 1 group 4 ( '/oracle/PROD/db/apps_st/data/stdbylog04a.dbf', '/oracle/PROD/db/apps_st/data/stdbylog04b.dbf') size 1G; SQL> alter database add standby logfile thread 1 group 5 ( '/oracle/PROD/db/apps_st/data/stdbylog05a.dbf', '/oracle/PROD/db/apps_st/data/stdbylog05b.dbf') size 1G; Clone the Database Software to Standby ----------------------------------------8. Prepare the Database Tier for Cloning As ORACLE user on one of the production database servers, run the adpreclone.pl script. For Example: cd $ORACLE_HOME/appsutil/scripts/

9. Copy Database Tier to Standby Use following command to tar the source file , copy and untar in target ==> The following command will automatically tar,copy and untar in the target system $ tar czvf - . |ssh -l oracle machine2.oracle.com "cd /oracle/PROD/db/tech_st; tar xvzf -" >> It is recommended to use the same directory names on the standby site. $HOME/.ssh/authorized_keys Verify that the .ssh directory permission is set to 700 $ chmod 700 $HOME/.ssh Verify that the authorized_keys file permission is set to 600 $ chmod 600 $HOME/.ssh/authorized_keys Verify that the $HOME directory does not have write permissions for group and others. $ chmod go-w $HOME Ensure that the ssh daemon is running $ ps -ef |grep -i ssh root 2418 22004 0 10:25 ? 00:00:00 sshd: oracle [priv] oracle 2420 2418 0 10:25 ? 00:00:00 sshd: oracle@pts/1 oracle 2690 2421 0 10:37 pts/1 00:00:00 grep -i ssh root 22004 1 0 Feb29 ? 00:00:00 /usr/sbin/sshd Test the installation at both Node(s): ------------------------------------- Login to the Remote Node as oracle user (Machine 2). Execute the following command, replacing [user] with the username on the Remote Node, and [SERVER] with the host name of the Remote Node: ssh -l [user] [SERVER] $ uname -n machine02.oracle.com $ ssh -l oracle machine02 (connect to same machine) Last login: Tue Mar 4 10:41:52 2008 from machine02.oracle.com $ uname -n machine02.oracle.com $ exit Connection to machine02 closed $ ssh -l oracle machine01 (connect to machine 1) Last login: Tue Mar 4 10:41:52 2008 from machine02.oracle.com $ uname -n

machine01.oracle.com When prompted to confirm the identity, respond "Yes". Disconnect from the server. $ exit Connection to machine01 closed Execute the following commands: $ uname -n machine02.oracle.com $ ssh oracle@machine01 uname -n machine01.oracle.com The remote hostname should appear, followed by the prompt, and you should not be asked to enter and ID or password. If you are prompted for a password, verify the following: The content of the authorized_keys file contains the content of id_rsa.pub of the client The permissions of the authorized_keys file are set to 600. The permissions of the .ssh directory are set to 700. The permissions of the $HOME directory do not include execute for group and other.

18. Establish the Standby Database Create a hot backup using Recovery Manager (RMAN) Take Backup of Primary Database and Copy the backup to Standby(Machine02) using RMAN HOT BACKUP a)set RDBMS_ORACLE_HOME env file b) $ rman target / RMAN> run { allocate channel c1 type disk; allocate channel c2 type disk; allocate channel c3 type disk; backup database plus archivelog; } >>For RMAN: Monitoring Recovery Manager Jobs ==> Ref Note 144640.1 To monitor the Rman Backup job, Please execute the following in SQLPLUS as sysdba

SQL> SELECT sid, spid, client_info FROM v$process p, v$session s WHERE p.addr = s.paddr AND client_info LIKE '%id=rman%';

SQL> SELECT sid, serial#, context, sofar, totalwork,round(sofar/totalwork*100,2) "% Complete" FROM v$session_longops WHERE opname LIKE 'RMAN%' AND opname NOT LIKE '%aggregate%' AND totalwork != 0 AND sofar totalwork / SQL> SELECT sid, seconds_in_wait AS sec_wait, event FROM v$session_wait WHERE wait_time = 0 ORDER BY sid; Take backup of current controlfile at primary for standby site in RMAN For Backup as Backupset: RMAN> run { allocate channel c1 type disk; backup current controlfile for standby; } For Backup as COPY : RMAN> run { allocate channel c1 type disk; backup as copy current controlfile for standby; } Once backup is done, copy the backup with archivelogs & Standby controlfile to remote location(standby) rsync -av /oracle/archive oracle@machine02.oracle.com:/oracle/archive 19. At Standby (Machine02), Startup mount the Standby database The following command will automatically duplicate the database using backup taken at primary and

mount the database $ rman target sys/oracle@prod_in01 auxiliary sys/oracle@prod_in02 RMAN> DUPLICATE TARGET DATABASE FOR STANDBY DORECOVER NOFILENAMECHECK; Run the following command to check the standby database details SQL> select database_role,switchover_status,checkpoint_change#,switchover#,dataguard_brok er,guard_status,current_scn,flashback_on from v$database The output should be similar to the following: DATABASE_ROLE SWITCHOVER_STATUS CHECKPOINT_CHANGE# SWITCHOVER# DATAGUAR GUARD_S CURRENT_SCN FLASHBACK_ON --------------- ------------------ ---------------------- ----------- --------------- -----------------------------------PHYSICAL STANDBY NOT ALLOWED 5965111847227 83208404 ENABLED NONE 5965112465873 YES 20. On the primary database enable the previously deferred remote destination by executing this command from SQL*Plus connected as sysdba: SQL> alter system set log_archive_dest_state_2=enable SID='*'; 21. Place the standby database in managed recovery by executing this command from SQL*Plus connected as sysdba: SQL> recover managed standby database using current logfile disconnect; 22. Verify whether the standby is correctly applying redo from the primary. On the primary database, archive the current log using the following SQL statement: Find the current Sequence and last archived log sequence: SQL> archive log list; >Archive the current redolog sequence: SQL> alter system archive log current;

> On the standby database, query the v$archived_log view to verify that the logs are received and applied: SQL> select thread#, sequence#, applied, to_char(first_time, 'mm/dd/yy hh24:mi:ss') first, to_char(next_time, 'mm/dd/yy hh24:mi:ss') next, to_char(completion_time, 'mm/dd/yy hh24:mi:ss') completion from v$archived_log order by first_time; You can also execute the following query to find the shipping details on both primary and standby SQL> select process,status,sequence# from v$managed_standby; 23. If incase the logs are not shipping, Please check the following a) Check alert.log file on Primary and see whether log_archive_dest_2 is reachable and check any TNS error, if the standby destination not reachable, fix the TNS issue b) Check the status of the log_archive_dest_2 SQL> select status,error from v$archive_dest; > If any error from above statement, you shall defer it temporarily SQL> alter system set log_archive_dest_state_2=defer; c) Once the error is rectified on log_archive_dest_2, you can execute the following query to enable it SQL> alter system set log_archive_dest_state_2=enable; d) If logs are not shipping and if you want to apply manually, you shall move/copy the missing archivelog from primary to standby manually using OS scp command and run the following command to apply the changes SQL> alter database register logfile '/oracle/archive/PROD_IN02/archivelog/2008_11_18/o1_mf_1_34_4l5bb7b4_.arc'; SWITCHOVER ============

1. Switchover Primary Database to Remote Standby Database a) Shutdown Production E-Business Suite Shut down E-Business Suite application tier. Ensure the application is completely shutdown. b) On the PRIMARY database , Issue the following command to commit to switchover to standby: SQL> alter database commit to switchover to standby with session shutdown; c) Shutdown the PRIMARY database SQL> shutdown immediate d) Stop the PRIMARY database listener: on all database nodes: lsnrctl stop e) On the STANDBY database, verify that it is ready to be converted to the new primary: SQL> select switchover_status from v$database; You should see the following value: SWITCHOVER_STATUS ----------------TO PRIMARY f) On the STANDBY database, execute the following command to convert it to be the new primary: SQL> alter database commit to switchover to primary; g) Enable Flashback on the STANDBY database (Optional) SQL> alter database flashback on; h) Open the STANDBY Database (New Primary database) SQL> alter database open; i) Remove the Old Application Topology Connect to the new primary database using SQL*Plus as user APPS and execute the following commands:

SQL> EXEC FND_CONC_CLONE.SETUP_CLEAN; SQL> commit; j) Configure the Standby Database Tier Run AutoConfig on STANDBY database(New Primary) node to configure the Oracle home for use by the E-Business Suite. $ cd $ORACLE_HOME/appsutil/scripts/ $ ./adautocfg.sh

------------------k) Start Original PRIMARY as Standby Start the database listener on database node at the original primary site: $ lsnrctl start Start and mount database instance and then start managed recovery: SQL> startup mount; SQL> recover managed standby database using current logfile disconnect; On the New PRIMARY database at the standby site enable remote archiving by executing this command from SQL*Plus connected as sysdba: SQL> alter system set log_archive_dest_state_2=enable SID='*'; Verify whether logs are switching from New PRIMARY to this Standby database as described in Step 20. l) Perform the Cloning Finishing Tasks as per Note 406982.1 FAILOVER ========= The filover can be done only if flashback is enabled on both databases. If following Note 452056.1, flashback would only be enabled on one of the databases after following the Switchover section. Flashback should be enabled at the start on Primary when archive log mode was enabled, and before the Standby is built. 1. Failover to Standby Database On the standby database, execute the following command to convert it to be the new primary:

SQL> recover managed standby database cancel; SQL> recover managed standby database finish force; SQL> alter database commit to switchover to primary; 2. Enable Flashback if not enabled already SQL> alter database flashback on; 3. Open the Standby Database SQL> alter database open; 4. Remove the Old Application Topology Connect to New PRIMARY database as APPS User and execute the following commands: SQL> EXEC FND_CONC_CLONE.SETUP_CLEAN; SQL> commit; 5. To Configure Database Tier, Run AutoConfig on the New PRIMARY database node to configure the Oracle home cd $ORACLE_HOME/appsutil/scripts/ 6. To Configure Application Tiers, Run AutoConfig on New PRIMARY site application tier nodes. cd $ADMIN_SCRIPTS_HOME ./adautocfg.sh 7. To Establish Original Primary as Standby Using Flashback On the standby (New PRIMARY)site, note the SCN when the database became primary: select to_char(standby_became_primary_scn) from v$database; 8. On the Original PRIMARY site, flashback and start managed recovery: SQL> shutdown immediate; SQL> startup mount; SQL> flashback database to scn

Verify whether standby operation on Original Primary and Perform Cloning Finishing Task as per Note 406982.1 Open the Standby Database for Testing Using Flashback Database =================================================

1. On the standby site, cancel managed recovery: SQL> recover managed standby database cancel; 2. On the standby site, enable flashback if it has not been enable before: SQL> alter database flashback on; 3. On the standby site, create a guaranteed restore point (named testing _starts in this example): SQL> create restore point testing_starts guarantee flashback database; 4. On the Primary Site , switch the current redo log and then defer the archive destination: SQL> alter system archive log current; SQL> alter system set log_archive_dest_state_2=defer SID='*'; 5. On the standby site, activate and open the database: SQL> alter database activate standby database; SQL> alter database set standby database to maximize performance; SQL> alter database open; 6. As Standby Database is now OPEN, Now You can Perform Testing and any changes made to this database will be rolled back after Flashback Note : This standby site will be getting behind on redo application during the testing period and so make sure that you do not get too far behind. 7. Flashback the Database and Resume Standby Operation On the standby site , Use flashback and start managed recovery: SQL> startup mount force; SQL> flashback database to restore point testing_starts; SQL> drop restore point testing_starts; SQL> alter database convert to physical standby; SQL> startup mount force; SQL> alter database recover managed standby database using current logfile disconnect; 8. On the Primary site , enable the archive destination: SQL> alter system set log_archive_dest_state_2=enable SID='*';