3-node data guard / broker / rman...

Lowell Noodle Company

Oracle 11g Data Guard3-Node/Broker/RMAN

Implementation ProjectGroup 4Minh Vo, Susan Champigny,

Ganapathi S. Santhana

10/27/2012

1 University AvenueLowell, MA 01854

Phone: 978 934-4000


1

IntroductionIn this project we’ll be setting up a 3-Node Data Guard implementation with Fast Start Failover controlled by a Broker/Observer. Adding upon the work done last month, we’ll put in place another standby in Active mode (read only with redo logs applied).

Then, we’ll be ready to integrate a dedicated RMAN database into your environment. RMAN will provide enhancements such as automatic parallelization of backup, restore, and recovery operations. A new recovery catalog will be created. Using a recovery catalog scenario will allow generation of reports while allowing storing RMAN scripts in a repository.

Benefits of using a Recovery Catalog versus just using RMAN is the availability of metadata about the production (target) database, which can contain multiple databases if needed in the future. The use of storing information about more than one incarnation database allows reporting on the production database (target) from a non-current incarnation.

Finally, we’ll test Flashback functionality and enable row movement to possibly run Flashback Queries.


2

Plan of Action SETUP AND VERIFY CURRENT 2-NODE CONFIGURATION

o Preparing our Server for a Second Standby

ADDING 3 RD NODE o Adding Another Standby to our Configuration

o Configure Listener and TNSNAMES

o Create/Copy Password Files

o Modify Init Files and Enable DG_BROKER

o Configure DG_BROKER

TESTING DATA GUARD INSTALLING RMAN DATABASE

o Creating a New Database for RMAN

o Building RMAN Catalog Repository

o Implementing our Backup Strategy

o Set Retention Policy

o Calling Scripts from CRONTAB

TESTING RMAN APPENDIX

o DGMGRL Warnings

o Proper Data Guard Configuration and Shutting Down

o Enabling/Disabling Fast Start Failover

o Add Disk/Volume Group

o


3

Setup and Verify Current 2-Node ConfigurationPREPARING OUR SERVER FOR A SECOND STANDBY (LNCSB2) ACTIVE DATA GUARD

Check if we have enough space in the file system. Looks like 11 GB will be sufficient for a second standby and RMAN implementation.

Verify sequence numbers match in our current 2-node setup.

Set DG_BROKER_START to TRUE on all databases.


4

Now configure Broker on Primary server.


5

On Primary, add 3 more Standby log file groups

They should appear in v$standby_log; now.


6

Do the same thing for Standby database.

Show configuration again…


7


8

/u01/app/oracle/product/11.2.0/db_2/network/admin/listener.ora# listener.ora Network Configuration File: /u01/app/oracle/product/11.2.0/db_2/network/admin/listener.ora[…] (SID_DESC = (GLOBAL_DBNAME=lnc_fc12ora112_DGMGRL) (SID_NAME=ora11) (SERVICE=lnc_fc12ora112) (ORACLE_HOME = /u01/app/oracle/product/11.2.0/db_2) ) (SID_DESC = (GLOBAL_DBNAME=lncsb_fc12ora112_DGMGRL) (SID_NAME=lncsb) (SERVICE=lncsb_fc12ora112) (ORACLE_HOME = /u01/app/oracle/product/11.2.0/db_2) )

Add these two listener entries so that Data Guard Broker can connect to the right DB_UNIQUE_NAME services when managing our setup. If not, we get the ugly

Unable to connect to databaseORA-12514: TNS:listener does not currently know of service requested in connect descriptor


9

Turn on Active Data Guard on standby using alter database open read only;

Now we’ll enable Flashback on primary and standby databases, as it’s a prerequisite for setting fast start failover feature (FSFO). Then we’ll enable fast start failover feature.

Lets see if Flashback is enabled on our databases. They are.

Now enable fast_start failover and check its status:


10

DGMGRL> START OBSERVER


11

Now check verbose configuration.

Test some switchovers with

DGMGRL> switchover to lncsb_fc12ora112

This initially gave us some TNS errors, so we doublechecked listener.ora and tnsnames.ora and made the changes seen in the final files shown above. Works great now.


12

Our two-database (Primary, Standby 1) Data Guard Broker setup looks pretty good at this point. Now let’s create and add a secondary standby (Standby 2) to DGMGRL’s collection of databases.


13

Adding 3rd NodeADDING SECOND STANDBY (LNCSB2) TO OUR DATA GUARD CONFIGURATION

Make the directory structure for lncsb2 as follows:

Change permissions to 775 for these directories, then do the same for /u05/oradata/lncsb and /u05/oradata/lncsb/arch directories.


14

Create the lncsb2_create_standby.sh script used to copy necessary files over to standby location. Ours looked like this:

LNCSB2_CREATE_STANDBY.SH

-- mikec - mv modified 10/27/2012-- execute in primary database (ora11) as SYS user-- ensure SID is set to ora11 (we're using this as primary db)-- Set SQL*Plus variables to manipulate outputset feedback on heading off verify offset pagesize 0 linesize 200-- Set SQL*Plus user variables used in script-- Linux User variablesdefine dir = '/u05/oradata/lncsb2'define fil = '/tmp/lncsb2_coldbkup.sql'define pdir = '/u01/app/oracle/admin/ora11/pfile'alter database backup controlfile to trace;prompt *** Spooling to &filspool &filselect 'host cp '|| name ||' &dir' from v$datafile order by 1;select 'host cp '|| member ||' &dir' from v$logfile order by 1;-- select 'host cp '|| name ||' &dir' from v$controlfile order by 1;select 'host cp '|| name ||' &dir' from v$tempfile order by 1;spool off;-- Shutdown the database cleanly!echo Database: $ORACLE_SID shutting down.shutdown exitimmediate;!echo .!echo Instance shutdown...-- Run the copy file commands!echo Copying database files...@&fil-- Start the database again!echo Database Copied...Starting Up in MOUNT Mode Now


15

startup mount;alter database create standby controlfile as '/u05/oradata/lncsb2/stdby.ctl';!echo Standby Database Control File Created for lncSB!echo!echo ora11 Database Shutting Down...shutdown immediate!echo ora11 Database Shutdown...!echo DBA Should Startup lncSB2 in Standby Mode First!echo Then Startup ora11 in Normal Mode Second!echo Manually Switch Logfile in lnc To Activate Standby Recovery Modeset feedback offexit


16

Stored in /tmp to be run inside sqlplus as sysdba.

Verify permissions for the standby database data files are set to 775.


17

CONFIGURE LISTENER AND TNSNAMES

Now we proceed to set up communication between 3 nodes. Key values in listener.ora here are SID_NAME and GLOBAL_DBNAME… making sure each SID_LIST item is set correctly.

# listener.ora Network Configuration File: /u01/app/oracle/product/11.2.0/db_2/network/admin/listener.ora

LISTENER = (DESCRIPTION_LIST = (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = fc12ora112.uml.edu)(PORT = 1521)) ) )

ADR_BASE_LISTENER = /u01/app/oracle

SID_LIST_LISTENER = (SID_LIST = (SID_DESC = (GLOBAL_DBNAME=ora11) (SID_NAME=ora11) (ORACLE_HOME = /u01/app/oracle/product/11.2.0/db_2) ) (SID_DESC = (GLOBAL_DBNAME=lncsb2_fc12ora112) (SID_NAME=lnc_fc12ora112) (ORACLE_HOME = /u01/app/oracle/product/11.2.0/db_2) ) (SID_DESC = (GLOBAL_DBNAME=lncsb_fc12ora112)


18

(SID_NAME=lnc_fc12ora112) (ORACLE_HOME = /u01/app/oracle/product/11.2.0/db_2) ) (SID_DESC = (GLOBAL_DBNAME=lnc_fc12ora112) (SID_NAME=lnc_fc12ora112) (ORACLE_HOME = /u01/app/oracle/product/11.2.0/db_2) ) (SID_DESC = (GLOBAL_DBNAME=rman) (SID_NAME=rman) (ORACLE_HOME = /u01/app/oracle/product/11.2.0/db_2) ) (SID_DESC = (GLOBAL_DBNAME=lnc_fc12ora112_DGMGRL) (SID_NAME=ora11) (SERVICE=lnc_fc12ora112) (ORACLE_HOME = /u01/app/oracle/product/11.2.0/db_2) ) (SID_DESC = (GLOBAL_DBNAME=lncsb_fc12ora112_DGMGRL) (SID_NAME=lncsb) (SERVICE=lncsb_fc12ora112) (ORACLE_HOME = /u01/app/oracle/product/11.2.0/db_2) ) (SID_DESC = (GLOBAL_DBNAME=lncsb2_fc12ora112_DGMGRL) (SID_NAME=lncsb2) (SERVICE=lncsb2_fc12ora112) (ORACLE_HOME = /u01/app/oracle/product/11.2.0/db_2) ) )

TNSNAMES.ORA

ORA11 = (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = fc12ora112.uml.edu)(PORT = 1521)) (CONNECT_DATA = (SERVER = DEDICATED) (SERVICE_NAME = ora11) ) )

LNCSB = (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = fc12ora112.uml.edu)(PORT = 1521)) (CONNECT_DATA = (SERVER = DEDICATED) (SERVICE_NAME = lncsb) ) )


19

LNCSB2 = (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = fc12ora112.uml.edu)(PORT = 1521)) (CONNECT_DATA = (SERVER = DEDICATED) (SERVICE_NAME = lncsb2) ) )

LNCSB2_FC12ORA112 = (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = fc12ora112.uml.edu)(PORT = 1521)) (CONNECT_DATA = (SERVER = DEDICATED) (SERVICE_NAME = lncsb2_fc12ora112) ) )

LNCSB_FC12ORA112 = (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = fc12ora112.uml.edu)(PORT = 1521)) (CONNECT_DATA = (SERVER = DEDICATED) (SERVICE_NAME = lncsb_fc12ora112) ) )

LNC_FC12ORA112 = (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = fc12ora112.uml.edu)(PORT = 1521)) (CONNECT_DATA = (SERVER = DEDICATED) (SERVICE_NAME = lnc_fc12ora112) ) )

RMAN = (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = fc12ora112.uml.edu)(PORT = 1521)) (CONNECT_DATA = (SERVER = DEDICATED) (SERVICE_NAME = rman) ) )

Reflected Listener status:


20

COPY PASSWORD FILES FOR STANDBY 2


21

Copy lncsb pfile for lncsb2 and create a link to it:

Make sure /etc/oratab has home for lncsb2 set

Now make sure DG_CONFIG and DB_FILE_NAME_CONVERT and such parameters set correctly on all pfiles.


22

IN STANDBY2 (INITLNCSB2.ORA):

audit_file_dest="/u01/app/oracle/admin/lncsb2/adump"audit_trail=NONEcompatible=11.2.0.0.0control_files=/u05/oradata/lncsb2/stdby.ctldb_block_size=8192db_cache_size=243269632db_domain=""db_name="ora11"db_recovery_file_dest_size=4196401152db_recovery_file_dest="/u01/app/oracle/fast_recovery_area"diagnostic_dest=/u01/app/oraclejava_pool_size=67108864large_pool_size=37748736open_cursors=300pga_aggregate_target=127926272processes=150shared_pool_size=247463936undo_tablespace=UNDOTBS1event=""## add parameters missing, but ignore deprecated onesdb_file_multiblock_read_count=8db_flashback_retention_target=1440db_file_name_convert=('/u02/oradata/ora11','/u05/oradata/lncsb2')job_queue_processes=2utl_file_dir=/u01/app/oracle/admin/lncsb2/utlundo_management=AUTO## Data Guard configlog_archive_config='DG_CONFIG=(lnc_fc12ora112,lncsb_fc12ora112,lncsb2_fc12ora112)'log_archive_dest_1='LOCATION=/u05/oradata/lncsb2/arch/ VALID_FOR=(ALL_LOGFILES,ALL_ROLES) DB_UNIQUE_NAME=lncsb2_fc12ora112'log_archive_dest_2='SERVICE=lnc_fc12ora112 VALID_FOR=(ONLINE_LOGFILES,PRIMARY_ROLE) DB_UNIQUE_NAME=lnc_fc12ora112 LGWR ASYNC REOPEN=10'log_archive_dest_state_1=ENABLElog_archive_dest_state_2=ENABLEdb_unique_name='lncsb2_fc12ora112'service_names=lncsb2_fc12ora112log_archive_format=lncsb2_%s_%t_%r.arclog_file_name_convert=('/u02/oradata/ora11','/u05/oradata/lncsb2')FAL_CLIENT='lncsb2_fc12ora112'FAL_SERVER='lnc_fc12ora112'## standby_archive_dest='/u04/oradata/lncsb/arch'## deprecated...standby_file_management='AUTO'remote_login_passwordfile='SHARED'

INITLNCSB.ORA

+ log_archive_config='DG_CONFIG=(lnc_fc12ora112,lncsb_fc12ora112,lncsb2_fc12ora112)'


23

INITORA11.ORA

+ log_archive_config='DG_CONFIG=(lnc_fc12ora112,lncsb_fc12ora112,lncsb2_fc12ora112)'+ log_archive_dest_3+ log_archive_dest_3_enable

Standby 2:

Primary:

SQL> alter system set log_archive_config='DG_CONFIG=(lnc_fc12ora112,lncsb_fc12ora112,lncsb2_fc12ora112)' scope=both;SQL> alter system set log_archive_dest_3='SERVICE=lncsb2_fc12ora112 VALID_FOR=(ONLINE_LOGFILES,PRIMARY_ROLE) DB_UNIQUE_NAME=lncsb2_fc12ora112 LGWR ASYNC AFFIRM' scope=both;SQL> alter system set log_archive_dest_state_3=ENABLE scope=both;


24

Standby 1:SQL> alter system set log_archive_config='DG_CONFIG=(lnc_fc12ora112,lncsb_fc12ora112,lncsb2_fc12ora112)' scope=both;

Create SPFILE for the second standby database. Then, start it up in standby managed recovery mode.

Enable Flashback.


25

Enable DG Broker in second standby lncsb2.


26

Test by switching some log files on Primary.

Now check if they’ve been applied on Standby 1 and 2. select sequence#, applied from v$archived_log order by sequence#;

Note: Here we see Sequence# 214 applied as “IN-MEMORY”. Seems to be an 11G “feature”. We thought this was fixed by turning on Flashback on Standby 2, which was left off by accident or by cycling the DBs seemed to have worked too… but not true. The steps on this paper have it fixed.


27

Check DelayMins for lncsb and set DelayMins for lncsb2 to 5 minutes.

DGMGRL> edit database 'lncsb2_fc12ora112' set property 'DelayMins'=5

Then check:


28

Alert trace file for lncsb2:

For good measure, we’re setting MaxConnections to 5 for all databases in our DG configuration.


29

Throughout everything, OBSERVER was still up and running on one of my instances. (Same server).

SHUTDOWN ABORT The primary database!!! To trigger failover automatically.

Wait to see if OBSERVER detects blip in heartbeat and will initiate fast-start failover to standby 1. We see here that it does!

Show configuration shows that primary is dead and that lncsb_fc12ora112 is now the acting Primary database. It’s also showing a Warning: ORA-16829: fast-start failover configuration is lagging. Again, we had fixed this when we found that Flashback was disabled for one of the standby. Enabling it, and cycling (shutdown/startup) the databases fixed it.


30

Now we’ll try to bring actual Primary back up…

Notice ORA-16649, but OBSERVER immediately sees this and reinstates the Primary. Way cool.


31


32

Setting Up RMAN GOING THROUGH THE PROCESS OF SETTING UP A NEW RMAN DATABASE AND ADDING REPOSITORY

RMAN_OFA.SH – create the directory structure for RMAN in /03/oradatacd /cd /u03/oradatamkdir rmanchmod -R 770 rmancd /cd /u01/app/oracle/oradatamkdir rmancd rmanmkdir arch bkupcd /u01/app/oracle/oradatachmod -R 775 rmancd /u01/app/oracle/adminmkdir rmanchmod -R 775 rmancd rmanmkdir adump dpdump create pfile utlchmod 775 *cd /u01/app/oracle/diag/rdbmsmkdir rmanchmod 775 rmancd rmanmkdir rmanchmod 775 rmancd rmanmkdir alert cdump hm incident incpkg ir lck metadata stage sweep tracechmod 775 *cd $HOME/TOOLS/setup_rmancp initrman.ora /u01/app/oracle/admin/rman/pfilecp spfilerman.ora /u01/app/oracle/admin/rman/pfilecp oratab /etccp build_rman.sql /u01/app/oracle/admin/rman/createcd /u01/app/oracle/product/11.2.0/db_1/dbsln -s /u01/app/oracle/admin/rman/pfile/initrman.ora initrman.oraln -s /u01/app/oracle/admin/rman/pfile/spfilerman.ora spfilerman.oracd /home/oracle/TOOLS/setup_rmanechoecho RMAN OFA Structure Setup: Completedecho

INITRMAN.ORA############################################################################### Copyright (c) 1991, 2001, 2002 by Oracle Corporation##############################################################################

############################################ Cache and I/O###########################################db_block_size=8192


33

############################################ Cursors and Library Cache###########################################open_cursors=300

############################################ Database Identification###########################################db_domain=fc12ora112.uml.edudb_name=rman

############################################ File Configuration###########################################control_files=("/u03/oradata/rman/rman_ctrl01.ctl", "/u03/oradata/rman/rman_ctrl02.ctl")db_recovery_file_dest=/u01/app/oracle/fast_recovery_areadb_recovery_file_dest_size=4039114752

############################################ Miscellaneous###########################################compatible=11.2.0.0.0diagnostic_dest=/u01/app/oracle

############################################ Processes and Sessions###########################################processes=150

############################################ Security and Auditing###########################################audit_file_dest=/u01/app/oracle/admin/rman/adumpaudit_trail=noneremote_login_passwordfile=EXCLUSIVE

############################################ Shared Server###########################################

############################################ System Managed Undo and Rollback Segments###########################################undo_tablespace=UNDOTBS1

BUILD_RMAN.sql-- connect as sysdba then execute this script

spool build_rman.log

startup pfile=/u01/app/oracle/admin/rman/pfile/initrman.ora nomount

-- pfile="/u01/app/oracle/admin/rman/pfile/initrman.ora";

CREATE DATABASE "rman"MAXINSTANCES 2MAXLOGHISTORY 226MAXLOGFILES 16MAXLOGMEMBERS 4


34

MAXDATAFILES 40DATAFILE '/u03/oradata/rman/rman_system01.dbf'SIZE 500M REUSEAUTOEXTEND OFFEXTENT MANAGEMENT LOCALSYSAUX DATAFILE '/u03/oradata/rman/rman_sysaux01.dbf'SIZE 500M REUSEAUTOEXTEND OFFDEFAULT TEMPORARY TABLESPACE TEMP TEMPFILE '/u03/oradata/rman/rman_temp01.dbf'SIZE 100M REUSEAUTOEXTEND OFFUNDO TABLESPACE "UNDOTBS1"DATAFILE '/u03/oradata/rman/rman_undotbs01.dbf'SIZE 100M REUSEAUTOEXTEND OFFCHARACTER SET WE8MSWIN1252NATIONAL CHARACTER SET AL16UTF16LOGFILE GROUP 1 ('/u03/oradata/rman/rman_redo01a.log', '/u03/oradata/rman/rman_redo01b.log') SIZE 8M, GROUP 2 ('/u03/oradata/rman/rman_redo02a.log', '/u03/oradata/rman/rman_redo02b.log') SIZE 8M, GROUP 3 ('/u03/oradata/rman/rman_redo03a.log', '/u03/oradata/rman/rman_redo03b.log') SIZE 8MUSER SYS IDENTIFIED BY "oracle"USER SYSTEM IDENTIFIED BY "oracle";CREATE TABLESPACE USERSDATAFILE '/u03/oradata/rman/rman_users01.dbf' SIZE 10M REUSEEXTENT MANAGEMENT LOCAL UNIFORM SIZE 128K;-- CREATE TABLESPACE EXAMPLE-- DATAFILE '/u03/oradata/rman/rman_example01.dbf' SIZE 150M REUSE-- EXTENT MANAGEMENT LOCAL UNIFORM SIZE 128K;-- LOAD CATALOGspool off@$ORACLE_HOME/rdbms/admin/catalog.sql;@$ORACLE_HOME/rdbms/admin/catproc.sql;@$ORACLE_HOME/sqlplus/admin/plustrce.sql;connect system/oracle@$ORACLE_HOME/sqlplus/admin/pupbld.sql;host echo ""host echo ""host echo "RMAN Database Build complete"connect / as sysdba

Switch . oraenv to rman and login to sqlplus / as sysdba. Then run build_rman.sql.SQL > @/home/oracle/TOOLS/setup_rman/build_rman.sql


35

My final initrman.ora

audit_file_dest="/u01/app/oracle/admin/rman/adump"audit_trail=NONEcompatible=11.2.0.0.0control_files=("/u03/oradata/rman/rman_ctrl01.ctl", "/u03/oradata/rman/rman_ctrl02.ctl")db_block_size=8192db_cache_size=243269632db_domain=""db_name="rman"db_recovery_file_dest_size=4196401152db_recovery_file_dest="/u01/app/oracle/fast_recovery_area"diagnostic_dest=/u01/app/oraclejava_pool_size=67108864large_pool_size=37748736open_cursors=300pga_aggregate_target=127926272processes=150shared_pool_size=247463936undo_tablespace=UNDOTBS1event=""db_file_multiblock_read_count=8db_flashback_retention_target=1440job_queue_processes=2utl_file_dir=/u01/app/oracle/admin/rman/utl


36

undo_management=AUTO## Data Guard configdb_unique_name='rman'service_names=rmanlog_archive_format=rman_%s_%t_%r.arcremote_login_passwordfile='SHARED'

TNSNAMES and LISTENER.ora all have this configured already. See section 2B.

rman_ofa.shcd /cd /u03/oradatamkdir rmanchmod -R 775 rmancd /cd /u01/app/oracle/oradatamkdir rmancd rmanmkdir arch bkupcd /u01/app/oracle/oradatachmod -R 775 rmancd /u01/app/oracle/adminmkdir rmanchmod -R 775 rmancd rmanmkdir adump dpdump create pfile utlchmod 775 *cd /u01/app/oracle/diag/rdbmsmkdir rmanchmod 775 rmancd rmanmkdir rmanchmod 775 rmancd rmanmkdir alert cdump hm incident incpkg ir lck metadata stage sweep tracechmod 775 *echo RMAN OFA Structure Setup: Completedecho

Run it.

Now as sysdba in rman:

CREATE DATABASE rman USER SYS IDENTIFIED BY change_on_install USER SYSTEM IDENTIFIED BY change_on_install LOGFILE GROUP 1 ('/u03/oradata/rman/rman_redo01.log') SIZE 100M BLOCKSIZE 512, GROUP 2 ('/u03/oradata/rman/rman_redo02.log') SIZE 100M BLOCKSIZE 512, GROUP 3 ('/u03/oradata/rman/rman_redo03.log') SIZE 100M BLOCKSIZE 512 MAXLOGFILES 5 MAXLOGMEMBERS 5 MAXLOGHISTORY 1 MAXDATAFILES 100 CHARACTER SET US7ASCII NATIONAL CHARACTER SET AL16UTF16 EXTENT MANAGEMENT LOCAL DATAFILE '/u03/oradata/rman/rman_system01.dbf' SIZE 325M REUSE SYSAUX DATAFILE '/u03/oradata/rman/rman_sysaux01.dbf' SIZE 325M REUSE


37

DEFAULT TABLESPACE users DATAFILE '/u03/oradata/rman/rman_users01.dbf' SIZE 500M REUSE AUTOEXTEND ON MAXSIZE UNLIMITED DEFAULT TEMPORARY TABLESPACE temp TEMPFILE '/u03/oradata/rman/rman_temp01.dbf' SIZE 20M REUSE UNDO TABLESPACE undotbs1 DATAFILE '/u03/oradata/rman/rman_undotbs01.dbf' SIZE 200M REUSE AUTOEXTEND ON MAXSIZE UNLIMITED;

Create Additional Tablespaces To make the database functional, you must create additional tablespaces for your application data. The following sample script creates some additional tablespaces:

CREATE TABLESPACE apps_tbs LOGGING DATAFILE '/u03/oradata/rman/rman_apps01.dbf' SIZE 500M REUSE AUTOEXTEND ON NEXT 1280K MAXSIZE UNLIMITED EXTENT MANAGEMENT LOCAL;-- create a tablespace for indexes, separate from user tablespace (optional)CREATE TABLESPACE indx_tbs LOGGING DATAFILE '/u03/oradata/rman/rman_indx01.dbf' SIZE 100M REUSE AUTOEXTEND ON NEXT 1280K MAXSIZE UNLIMITED EXTENT MANAGEMENT LOCAL;

Run Scripts to Build Data Dictionary Views@?/rdbms/admin/catalog.sql


38

Creates the views of the data dictionary tables, the dynamic performance views, and public synonyms for many of the views. Grants PUBLIC access to the synonyms.

@?/rdbms/admin/catproc.sql Runs all scripts required for or used with PL/SQL.

@?/sqlplus/admin/pupbld.sql Required for SQL*Plus. Enables SQL*Plus to disable commands by user.

BUILDING A RMAN CATALOG REPOSITORY:


39

Make directory rman11 in /u01/app/oracle/oradata and apply 775 permissions.--- NO we might want to put it in /u03/oradata/rman instead!!!

Login to sqlplus as sysdba and create new tablespace for RMAN. Also create RMAN user and grant privileges.

Now login to the rman catalog rman/change_on_install, and create catalog;


40

Now startup ora11 while deferring log_archive_dest_state_2 and log_archive_dest_state3… effectively stopping our dataguard configuration.

RMAN into the catalog with target=sys/change_on_install@ora11 (Primary). Then register the database. RMAN should show the DBID being ORA11 (DBID=681353050) and confirm that database is registered and full resync complete.

Cool. Now lets check if everything is working. We’ll run a couple list commands:


41


42

Note that flashback is already set up with recovery_file_dest= /u01/app/oracle/fast_recovery_area Defer dest_states 2 and 3, effectively turning off our dg config.

Oracle Notes:

When the flash recovery area is used, RMAN automatically uses Oracle Managed Files (OMF) for its backup files. By default, RMAN also automatically places backup files into the flash recovery area, if there is one configured. To enable the flash recovery area, you must set the two initialization parameters DB_RECOVERY_FILE_DEST_SIZE (which specifies the disk quota, or maximum space to use for flash recovery area files for this database) and DB_RECOVERY_FILE_DEST (which specifies the location of the flash recovery area)

Setting Up a Flash Recovery Area for RMAN maximum number of days into the past for which you can recover to 5:

ALSO SHOULD WE SET Flashback with alter system set DB_FLASHBACK_RETENTION_TARGET=7200; for 5 days?? This is separate from RMAN recovery window.

RMAN> CONFIGURE RETENTION POLICY TO RECOVERY WINDOW OF 5 DAYS;

RMAN> CONFIGURE RETENTION POLICY TO REDUNDANCY 3;


43

CONFIGURE BACKUP OPTIMIZATION ON;CONFIGURE CONTROLFILE AUTOBACKUP ON;

Now comes the hard part. We decided for a strategy that:

RUN { RECOVER COPY OF DATABASE WITH TAG 'incr_update'; BACKUP INCREMENTAL LEVEL 1

FOR RECOVER OF COPY WITH TAG 'incr_update' DATABASE; }

Saved to be scheduled by cron nightly.

Since LNC wants max retention to be 5 days, we thought a nightly script which checks for a full backup and otherwise creates incremental type 1 backup would work best. (As seen above). A weekly script with type 0 backup on Sunday for instance, and incremental every other day would not work.

One sample run:


44


45

rman TARGET / @target catalog rman/rman@rman SCRIPT 'script_name'; to call from cron. Set it to run every night at Midnight.

After the third run, it’s doing the incremental level 1s as seen below.

With this solution at most we’ll have 24 hours of redo to apply.

OPTIONAL: Turn on Standby 1 and 2 and reinitiate DataGuard Broker. Standby 1 is currently our failover standby with redo logs applied instantly. Standby 2’s purpose is to act as a reporting server (real time query enabled) for internal use (note with a 5 minute data delay) while also syncing redo logs with Primary with a transport delay of 5 mins. We may want to shorten this time using the DelayMins DGMGRL property.


46

Startup Primary and check log_archive_config is correct and log_archive_dest (and _state) properly set and enabled.


47

Startup the Observer in DGMGRL.

Check DataGuard configuration. Things are looking good. RMAN is up with 5 day retention and backup script scheduled to run every midnight. DataGuard configuration is concurrently running with two standbys in which one can/is converted to Active.

Set lncsb2_fc12ora112 to Active (read-only with Redo applied). In DGMGRL edit the standby to set apply-off, then in sqlplus (on lncsb2)

SQL > alter database open read only;

Finally, set database state to apply-on again.


48

PLAYING WITH RMAN

RMAN > list backup;

RMAN > report obsolete;


49

Update the status of backups in the repository compared to their status on disk:


50

Report on schema of backup:RMAN> report schema;


51

Validate that a database can be backed up using RMAN:RMAN> backup validate database archivelog all;

Your options are either to restore the missing file(s), or to perform a crosscheck. To perform a crosscheck, run the following command from within RMAN: change archivelog all crosscheck; then retry validating.


52

Backup database using one datafile per rman backup fileset (for quicker restores):RMAN> backup database filesperset 1 plus archivelog delete input;


53

Backup database using a LEVEL 0 INCREMENTAL (baseline backup):RMAN> backup incremental level 0 database;

Delete expired backups after executing crosscheck:RMAN> crosscheck backup;RMAN> delete expired backup;

RMAN> delete obsolete;

ADDING SOME MORE SCRIPTS TO THE REPOSITORY:

list script names;

Rman_backup (already exists – does full backup if none exists, then incrementals if it does.)

printing stored script: rman_backup{ RECOVER COPY OF DATABASE WITH TAG 'incr_update'; BACKUP INCREMENTAL LEVEL 1 FOR RECOVER OF COPY WITH TAG 'incr_update' DATABASE; }

Cold_backupprinting stored script: ora11_coldbkup{shutdown immediate;startup mount;backup device type disk filesperset = 10 database include current controlfiletag ora11_cold_backup;alter database open;


54

delete noprompt expired backup;delete noprompt obsolete;}

Hot Backupprinting stored script: ora11_hotbkup{CROSSCHECK BACKUP;CROSSCHECK archivelog all;CROSSCHECK COPY;CROSSCHECK backup of database;CROSSCHECK backup of controlfile;delete noprompt obsolete;SQL 'ALTER SYSTEM SWITCH LOGFILE'; delete noprompt expired archivelog all;backup device type disk filesperset = 10 database include current controlfiletag ora11_hot_backup;delete noprompt expired backup;delete noprompt obsolete;}

Recover from Hot Backupprinting stored script: recover_hotbkup{host 'date';startup nomount;restore controlfile;host 'date';sql' alter database mount';host 'date';restore database;host 'date';restore archivelog all;host 'date';recover database;host 'date';sql 'alter database open resetlogs';}

Recover from Cold Backupprinting stored script: recover_coldbkup{host 'date';startup nomount;restore controlfile;host 'date';sql' alter database mount';


55

host 'date';restore database;host 'date';sql 'alter database open resetlogs';}

TEST FLASHBACK TABLE FUNCTIONALITY

First we’ll create a new table test2 and fill it with pretty random data.

Notes: SQL> select oldest_flashback_scn, oldest_flashback_time from v$flashback_database_log;Create new tablespace tbs_minh and create a new user.


56

grant create session, create trigger, create sequence, create type, create procedure, create cluster, create operator, create indextype, create table to minh;

Connect as minh/minh and create table test in minh schema. Don’t forget to enable row movement.


57


58

Now check if we can Flashback table to 5 minutes prior:SQL> flashback table minh.test to timestamp (systimestamp-interval '5' minute);

Great! We were able to flashback right before our “minute begin…” tests. After a few minutes, running the SQL again gets us the updated set at whatever point in time.


59

USING FLASHBACK QUERY

select * from minh.test as of timestamp (systimestamp-interval '10' minute); nets us the same kind of output.

To flashback the entire database to a restore point or before a specific SCN,RMAN> list backup;

Then, RMAN> flashback database to before scn 2063021;

If it complains about “ORA-38757: Database must be mounted and not open to FLASHBACK.” Switch Primary to mount state, being sure to kill connected non-SYS/non-PUBLIC users if needed.

spool kill_user_sessions.outselect 'alter system kill session '''||sid||','||serial#||''' immediate;'from v$session where username not in ('SYS','PUBLIC');spool off@kill_user_sessions.out


60

AppendixAFTER ENABLING LNCSB2_FC12ORA112, WE GOT DGMGRL WARNINGS

MORE DGMGRL WARNINGS:


61

Edit database set property for each of the “inconsistent” settings. Check Primary for what the values should be. Show database verbose <Primary>.

DGMGRL> show database lncsb2_fc12ora112 statusreport

DISABLING/ENABLING FAST START FAILOVER


62

DGMGRL> edit database 'lnc_fc12ora112' set property 'LogXptMode'='SYNC' ;Property "LogXptMode" updatedDGMGRL> edit database 'lncsb_fc12ora112' set property 'LogXptMode'='SYNC' ;Property "LogXptMode" updatedDGMGRL> edit database 'lncsb2_fc12ora112' set property 'LogXptMode'='SYNC' ;Property "LogXptMode" updatedDGMGRL> edit database 'lnc_fc12ora112' set property 'FastStartFailoverTarget'='lncsb_fc12ora112' ;Property "FastStartFailoverTarget" updatedDGMGRL> edit database 'lncsb_fc12ora112' set property 'FastStartFailoverTarget'='lnc_fc12ora112' ;Property "FastStartFailoverTarget" updatedDGMGRL> EDIT CONFIGURATION SET PROTECTION MODE AS MAXAVAILABILITY;


63

Succeeded.

Oracle Notes:

To enable fast-start failover for a broker configuration with multiple standby databases, the FastStartFailoverTarget property on the primary database and one standby database must be set to point to each other. Both the primary database and the target standby database must have:

Standby redo logs configured The LogXptMode property set to SYNC Flashback Database enabled on both the primary and standby databases

In addition, the configuration protection mode must be set to MAXAVAILABILITY.


64

PROPER CONFIGURATION


65

PROPER SHUTDOWN:

DGMGRL> connect sys/change_on_installConnected.DGMGRL> stop observer;Done.Observer stopped


66

Shut down the observer and wait for the FS_FAILOVER_OBSERVER_PRESENT column in the V$DATABASE fixed view to contain the value "NO" for both the primary and target standby databases.

Now shutdown immediate the Primary followed by standby databases.


67

ADD DISK / VOLUME GROUP

Running out of space again, with all the backups… added 10 GB drive.

3-node data guard / broker / rman...

Documents