03/31/04

Disaster Recovery Using TDP Disaster Recovery Using TDP with RMANwith RMAN

Wylie Merritt

ORACLE DBA

Oklahoma Dept. of Human Services

GoalsGoals

Determine viability of TDP for z/LinuxDevelop a ‘hot’ backup strategyTest various recovery scenarios,

including (especially) disaster recovery

Make a recommendation by YE 2003

BackgroundBackground

OKDHS had previously always done file-level backups of a cold database

Migration to a new environment meant new tools

Institutional memory: Warm backups = trouble

BackgroundBackground

RMAN had never been implemented, much less used

IBM was already under contract and Tivoli was already in widespread use on the mainframe side

The BIG Question….The BIG Question….

Can we trust this mission-critical, high-visibility database to RMAN/TDP?

Plan of AttackPlan of Attack

Tivoli Storage Manager was already installed and in use (cold backups)

Tivoli Data Protector (beta) installed in late July ‘03

RMAN hooked up to TDP – it works!Now – How do we use RMAN?

Plan of Attack, Cont.Plan of Attack, Cont.

Preliminary testing promising – Backing up and restoring tablespaces,

data files, etc. works fine– Not excessively complex (though not

graphical)– Ability to do warm backups = a good

thing

BUT...

What about disaster recovery?

Replication as Disaster Recovery POPReplication as Disaster Recovery POP

‘Need to replicate’ already existed Identical Linux guest created for

replication – or so we thought! Lesson learned #1– Tivoli has to be ‘tricked’ into seeing the new

host as the original one– Some critical parameters:

• SErvername – in dsm.sys and dsm.opt• TDPO_NODE – in tdpo.opt

Replication as Disaster Recovery POPReplication as Disaster Recovery POP (Cont.) (Cont.)

What if your RMAN Catalog is gone?Lesson learned #2– Always use ‘controlfile autobackup’• Allows use of backup file (if accessible) to

determine which is the latest full backup, which log files will be required (and need restoring), etc.• As well as insuring you always have a

current control file!

Replication as Disaster Recovery POPReplication as Disaster Recovery POP (Cont.) (Cont.)

How do you connect to a non-existent database instance? (Hint: look at your listener.ora)

Lesson learned #3– Some ‘obscure’ listener.ora parameters come

in handy during ‘disaster recovery’• GLOBAL_DBNAME• SID_NAME• ORACLE_HOME

– In the SID_LIST_<lsnr> section

Replication as Disaster Recovery POPReplication as Disaster Recovery POP (Cont.) (Cont.)

Lesson learned #3 (cont.)– Also need to have defined (in init.ora)• DB_NAME• DB_DOMAIN• DB_NAME + DB_DOMAIN =

GLOBAL_DBNAME (from listener.ora)

Allows you to connect remotely to a non-existent (or down) database instance

Bottom Line – Must HavesBottom Line – Must Haves

Backup or some other way to replicate the environment, including:– Operating system– Oracle (including RMAN)– Tivoli (including Storage Manager and

Data Protector files)

Don’t forget...

All relevant (and relatively static) parameter files, such as tdpo.opt, dsm.opt, dsm.sys, init.ora, listener.ora & tnsnames.ora

Bottom Line – Must Haves Bottom Line – Must Haves (Cont.)(Cont.)

Connection to the new host– Local is good, but remote will work also

(if you’re prepared)A recent RMAN backup that includes– Controlfile autobackup– Log files

ScheduleSchedule

You are here

SO...

We have satisfied ourselves that this is a viable approach, and that TDP will perform as advertised

Aug Sep Oct Nov Dec Jan Feb Mar Apr May Jun

Proof of PrincipleProof of PrincipleProof of PrincipleProof of Principle

TestingTestingTestingTesting

ImplementationImplementationImplementationImplementation

Current PlanCurrent Plan

High-availability database will be backed up weekly using RMAN/TDP combination. Est. planned downtime at < 30 min./wk. (currently 1 hr.)

Monthly (?) Flash Copy All other z/Linux Oracle databases will be

backed up weekly using Storage Manager (cold backup, file level)

Go-live date: April 10th

Contact InformationContact Information

DBA – Wylie Merritt– Wylie.Merritt@okdhs.org– 405.522.1364

Sys Admin (OS) – Chris Little– Chris.Little@okdhs.org– 405.522.1306

Sys Admin (Tivoli) – Frank Hubble– Frank.Hubble@okdhs.org– 405.522.1314