oracle active data guard zero-downtime database …...• working as senior database administrator...

Copyright © 2012, Oracle and/or its affiliates. All rights reserved Confidential – Oracle Restricted. 1

Oracle Active Data Guard Database Rolling Maintenance Larry M. Carpenter Distinguished Product Manager - Oracle Kalyan Kumar Kallepally Senior Database Administrator - Wellcome Trust Sanger Institute Joydip Kundu Director, Software Development - Oracle


A look at Rolling Upgrades today

A customer’s viewpoint

Simple Database Upgrades for the future

In Closing

Agenda

3 Copyright © 2011, Oracle and/or its affiliates. All rights reserved.

Database Rolling Upgrades

PROD Physical STBY

synchronize

Install new Oracle version in separate homes on A & B, set guaranteed

restore point (GRP) on A

synchronize Logical STBY

PROD Temporarily convert B to use SQL

Apply, upgrade and sync

Database A Database B

SWITCHOVER Switchover, flashback A to GRP,

mount in new/upgraded home PROD Physical STBY

synchronize PROD Upgrade A via redo stream

and synchronize Physical STBY release n

release n+1

My Oracle Support Note 949322.1





In Closing

Agenda


Introducing Kalyan Kallepally Senior Database Administrator

Upgrade Databases With Minimal downtime

Oracle Open World - 2012

by Kalyan Kallepally

Senior Database Administrator

Wellcome Trust Sanger Institute Cambridge, UK

[email protected]

mailto:[email protected]�

Who am I • Working as Senior Database Administrator at

– Wellcome Trust Sanger Institute • 10+ years of experience in Oracle (8-11gr2) • 5+ years of experience in MySQL, SQL Server • Recently started blogging

– www.dbakalyan.wordpress.com – You can reach me on [email protected]

http://www.dbakalyan.wordpress.com/�

mailto:[email protected]�

Wellcome Trust Sanger Institute

• One of the leading genomics centres in the world • Located in Hinxton, south of Cambridge, UK • Founded in 1993 by the Wellcome Trust and the UK Medical Research Council

(MRC) • Primarily funded by the Wellcome Trust • The institute is named after Dr Fred Sanger – The double Nobel prize-winning

biochemist • Responsible for the completion of 1/3 of the human genome

• Current computational capacity of 16k cores: – 12k cores on blade format – 4k cores in racked servers

• Current usable storage capacity 16 Petabytes: – Estimated growth 3

Petabytes/year • Data shared globally across

the world with many scientific research institutes

Database Layout • Primary database (300G)

– ASM – 2 Physical standbys (local and remote)

• Local standby is within same site as of the primary but placed in a different quadrant of the datacentre

• DR standby is 10 miles away from the Primary with dedicated private 10Gb/s WAN to DR datacentre

Upgrade Process • Upgraded database from 11.1.0.6 to 11.2.0.3

using Transient Logical standby – Down time reduced by 98%

• Upgrade process is split into 2 phases: – Phase 1 (Upgrade Primary Database) – Phase 2 (Upgrade Standbys & Switchback)

Phase - 1

Upgrade Primary Database

Physical Standby Database

11.1.0.6

Production Database

11.1.0.6

11.1.0.6

Sync or Async Redo shipping


Local Site

DR Site

Read Write Clients Read Only Clients

Redo Logs

Redo Logs

ASM

ASM

ASM

Broker

Backup

Redo apply

Redo apply

Oracle Home (11.1.0.6)




Upgrade Pre-requisites • Disable Data Guard Broker • Protection level must be either Maximum

availability or Maximum performance • Ensure Flashback is enabled on all databases in

the Data Guard configuration • Create guaranteed restore points • Parameter “compatible” must be set to same on

all the databases


11.1.0.6

Primary Database

11.1.0.6

11.1.0.6

Local Site

DR Site


Redo Logs

ASM

ASM

ASM

Broker

Backup

Redo apply

Oracle Home (11.1.0.6) Physical Standby

Database

• Disable Broker • Enable Flashback • Create Guaranteed Restore points

Redo Logs

Redo apply





11.1.0.6

Primary Database

11.1.0.6

11.1.0.6

Local Site

DR Site


Redo Logs

Redo Logs

ASM

ASM

ASM

Broker

Backup

Physical Standby Database Redo

apply




Transform Redo to SQL

Logical Standby Database

SQL apply

• Convert the DR physical standby to Transient Logical Standby • Check if Logical standby is in sync


Redo apply




11.1.0.6

Primary Database

11.1.0.6

11.2.0.3

Local Site

DR Site


Redo Logs

Redo Logs

ASM

ASM

ASM

Broker

Backup

Redo apply

SQL apply






• Install new version of Oracle database Software • Upgrade the LSTBY using DBUA from the new ORACLE HOME • After successful upgrade check if LSTBY is in sync again




11.1.0.6

Redo Logs Online Redo Logs


11.1.0.6

Primary Database

11.1.0.6

11.2.0.3



Local Site

DR Site


Redo Logs

ASM

ASM

ASM

Broker

Backup

Redo apply

SQL apply






• Plan for switch over to LSTBY • Stop the redo transmission to PSTBY • Up to this point absolutely there is no impact on the availability of the primary database. • Switch over • Re-point the clients, services



SQL apply

Phase - 2

Upgrade Standbys & Switch Back


11.1.0.6

Primary Database

11.1.0.6

11.2.0.3



Local Site

DR Site


Redo Logs

ASM

ASM

ASM

Broker

Backup

Redo apply




• LSTBY (previous Primary) is in a Transient state • It is Running lower version than primary • Cannot understand/apply changes received from the higher version of primary database • Cannot provide disaster recovery


Online Redo Logs



SQL apply


11.1.0.6

Primary Database

11.1.0.6

11.2.0.3



Local Site

DR Site


Redo Logs

ASM

ASM

ASM

Broker

Backup

Redo apply




• Flashback LSTBY to guaranteed restore point and shut down • Install new oracle version • Mount database from new home and convert LSTBY to PSTBY • When Redo Apply starts, it registers a new incarnation from the primary database


Online Redo Logs



SQL apply


Redo apply



11.1.0.6

Primary Database

11.2.0.3



Local Site

DR Site


ASM

ASM

ASM

Broker

Backup

Redo apply




• Wait for physical standby to be upgraded via the redo stream • At this point you have a second interruption to the database availability • Switch Back Primary to its Original location • Re-point the clients accordingly


Online Redo Logs



11.2.0.3

Redo Logs

Redo apply

Redo Logs

Redo apply

Redo Logs

Online Redo Logs

11.1.0.6


11.1.0.6

Primary Database

11.2.0.3



Local Site

DR Site


ASM

ASM

ASM

Broker

Backup

Redo apply




• Install new database home on other standbys • Mount PSTBY from new home • Flashback database to guaranteed restore point Oracle Home (11.2.0.3)

Oracle Home (11.2.0.3) Physical Standby Database

11.2.0.3 Redo Logs

Redo apply

Redo Logs

Online Redo Logs



11.1.0.6

Primary Database

11.2.0.3



Local Site

DR Site


ASM

ASM

ASM

Broker

Backup

Redo apply




• Enable the redo transmission to local PSTBY to catch up to new incarnation • PSTBY will be upgraded via redo stream • Enable Broker



11.2.0.3 Redo Logs

Redo apply

Redo Logs

Online Redo Logs


11.2.0.3


Primary Database

11.2.0.3



Local Site

DR Site


ASM

ASM

ASM

Broker

Backup

Redo apply




• Drop guaranteed restore points • Rolling upgrade has Finished and databases are back in the original positions • With very minimal interruptions Oracle Home (11.2.0.3)


11.2.0.3 Redo Logs

Redo apply

Redo Logs

Online Redo Logs


11.2.0.3

Pros & Cons Regular /Conventional Upgrade • Upgrade using DBUA/CLI

(50mins) • Upgrade using datapump -

Complete Down time required for exports, scp & imports (approx 3 hrs) – Manually recreate standbys

after the upgrade • Errors at any stage can

increase down time

Rolling Upgrade using Transient LSTBY

• Down time is required for only switchover (reduced by 98% - 5mins)

• Standbys are automatically upgraded

• Fail safe options at each step • Down time is independent of

database size

Caveats • Need to upgrade grid/ASM before the database upgrade • Need to manually update the OCR (srvctl) registries • If Upgrading from 11gR1 to 11gR2 then move the database

spfile from $OH/dbs to ASM - if not, DBUA will fail • If the “compatible” parameter is changed after LSTBY upgrade

- database cannot be rolled back • To change the “compatible” parameter after the upgrade - another downtime is needed

References • http://www.oracle.com/technetwork/database/features/availability/maa-

wp-11g-transientlogicalrollingu-1-131927.pdf • http://www.oracle.com/technetwork/database/features/availability/maa-

wp-11g-upgrades-made-easy-131972.pdf

http://www.oracle.com/technetwork/database/features/availability/maa-wp-11g-transientlogicalrollingu-1-131927.pdf�

http://www.oracle.com/technetwork/database/features/availability/maa-wp-11g-transientlogicalrollingu-1-131927.pdf�

http://www.oracle.com/technetwork/database/features/availability/maa-wp-11g-upgrades-made-easy-131972.pdf�













Over to Larry

Thank you


Just How Many Steps Is That?

1. Disable the Broker configuration 2. Stop the Broker at the Primary 3. Stop the Broker at the Standbys 4. Configure the Redo transport parameters

on the target standby 5. Configure the Redo transport parameters

on bystander standbys 6. Set the LOG_FILE_NAME_CONVERT

parameter to clear logfiles 7. Create a guaranteed restore point on the

primary database

8. Create a guaranteed restore point on the standby database

9. Cancel Managed Recovery 10. Execute DBMS_LOGSTDBY.BUILD at

primary 11. Convert the standby to a Transient

Logical standby using KEEP IDENTITY 12. Open the Transient Logical standby 13. Start SQL Apply 14. Turn off automatic archive log delete 15. Synchronize


But Wait! There’s More!

16. Stop SQL Apply 17. Create a 2nd guaranteed restore point 18. Stop Redo Transport 19. Upgrade the Transient Logical standby 20. Enable Redo Transport 21. Restart SQL Apply 22. Synchronize 23. Start Switchover at Primary 24. Complete Switchover at standby 25. Open new Primary

26. Stop redo transport 27. Flashback the original Primary 28. Mount original Primary in new home 29. Convert it to a physical standby 30. Shutdown 31. Mount new Physical Standby 32. Start Redo Apply 33. Enable redo transport 34. Synchronize 35. Start switchover on new primary


And finally

36. Complete switchover at standby 37. Open new Primary 38. Restart old Primary as a standby 39. Start Managed Recovery 40. Drop 3 guaranteed restore points 41. Enable the Broker Configuration





In Closing

Agenda


DBMS_ROLLING: Concepts

Rolling changes can be applied on the whole Data Guard configuration – Three stages (Specification – Compilation – Execution) – Execution has three stages: Start, Switchover, Finish

Two key groups – Leading Group

Changes are applied first (before Switchover) Has a Master Database (the future primary database)

– Trailing Group Changes are applied last (after Switchover) Has a Master Database (the original primary)


DBMS_ROLLING: Concepts

Leading Group – Need to identify the Master Database (LGM) during Specification – LGM starts as a Physical Standby, converted into a Logical Standby

(START), and then becomes the primary database (SWITCHOVER) – Other databases in the Leading Group protect the LGM – LGM responsibility transferrable on failure

Trailing Group – Contains the original primary database (Trailing Group Master) – Other databases in the Trailing Group protect the TGM – TGM responsibility transferrable on failure


Data Guard Simple Rolling Upgrades Simple, Reliable, Repeatable Database Rolling Upgrades

Start Switchover Finish


Goal

Early problem detection Dedicated interface: DBMS_ROLLING PL/SQL package Centralized, simplified, and uniform execution Fault tolerance Configuration rollback Centralized monitoring: DBA_ROLLING views

Simple, Reliable, Repeatable


Upgrading The Database Software

The Data Guard Simple Rolling Upgrades procedure will be usable for database version upgrades starting with the first patchset of Oracle Database12c.

– To upgrade from previous Oracle Database11g releases to Oracle Database12c and to move from the first release of Oracle Database 12c to the first patchset you will need to use the classic Transient Logical Standby upgrade procedure.

The procedure can be used for other database maintenance tasks with the first release


What Kind Of Changes Can Be Made?

Add Partitioning to Non-Partitioned Tables Change Basicfile LOBs to Securefile LOBs Change XML-CLOB to Binary XML Alter table to be OLTP-compressed

First Release Examples


New Data Type Support

Advanced Data Type Support – XML OR – Binary XML – Securefile Deduplication – Objects/Collections (includes SDO_GEOMETRY, DICOM) – New Support for Oracle-Supplied Features

DBFS AQ Oracle Text XDB

Eliminating Barriers to Utilizing Database Rolling Upgrade


Process – Planning & Setup Phase

Generate an upgrade plan – Call DBMS_ROLLING.INIT_PLAN

Generates an upgrade plan with a configuration specific set of instructions to guide the administrator through the upgrade process

– Call DBMS_ROLLING.SET_PARAMETER Modify parameters of the rolling upgrade

Prepare your changes to the database


Process - Execution Phase

Call DBMS_ROLLING.START_PLAN – Configures primary and standby databases participating in the upgrade

Make changes to the standby database Call DBMS_ROLLING.SWITCHOVER

– Swaps roles between current primary and new primary with the changes, switchover is only downtime required

Call DBMS_ROLLING.FINISH_PLAN – Completes upgrade of the old primary and bystanders and resynchronizes with the new primary


A Sample Walkthrough - Planning Phase

Primary database in Boston Physical standby database in

Cambridge (SYNC) 2nd Physical standby database in

Chicago (ASYNC) Broker Controlled

Current Configuration

DGMGRL> SHOW CONFIGURATION Configuration – OOW2012 Protection Mode: MaxAvailability Databases: bos - Primary database cam - Physical standby database chi - Physical standby database Fast-Start Failover: DISABLED Configuration Status: SUCCESS DGMGRL>



Connect to the Primary Execute the INIT_PLAN

procedure Sets up default parameters

– 2 for each database – 25 general parameters

Start the Planning Process

SQL> exec dbms_rolling.init_plan(‘cam’); PL/SQL procedure successfully completed. SQL> select name, curval from 2 dba_rolling_parameters where scope is null; NAME CURVAL ------------------------ ------------------------ ACTIVE_SESSIONS_TIMEOUT 3600 ACTIVE_SESSIONS_WAIT 0 BACKUP_CTLFILE rolling_change_backup.f DICTIONARY_LOAD_TIMEOUT 3600 DICTIONARY_LOAD_WAIT 0 EVENT_RECORDS 10000 FAILOVER 0 GRP_PREFIX DBMSRU_ … UPDATED_TGS_WAIT 1 25 rows selected.



Create Redo Transport parameters at target database

– In our example, on CAM to point to BOS and CHI

Required to complete the process after the switchover as the Broker is disabled.

Reconfigure Redo Transport

SQL> alter system set log_archive_dest_5='service="BOS", SYNC reopen=30 db_unique_name="BOS" valid_for=(PRIMARY_ROLE, ONLINE_LOGFILE)'; System altered. SQL> alter system set log_archive_dest_6='service="CHI", ASYNC reopen=30 db_unique_name="CHI" valid_for=(PRIMARY_ROLE, ONLINE_LOGFILE)'; System altered. SQL>



Example “INVOLVEMENT” Each database protects the

Primary when in the Standby role – BOS Primary

CAM and CHI protect BOS – CAM Primary

BOS and CHI protect CAM – CHI Primary

BOS and CAM protect CHI

Examining Parameters of the Plan

SQL> select scope, name, curval 2 from dba_rolling_parameters 3 where scope is not null order by scope; SCO NAME CURVAL --- --------------- --------------- bos INVOLVEMENT FULL bos PROTECTS NONE cam INVOLVEMENT FULL cam PROTECTS PRIMARY chi INVOLVEMENT FULL chi PROTECTS PRIMARY 6 rows selected. SQL>



Validates plan parameters Creates an upgrade plan

– Successful build required in order to perform a rolling upgrade

Parameter changes require that the procedure is called again Complete plan can be reviewed in

DBA_ROLLING_PLAN

Build the Plan when ready

SQL> select revision, status, phase 2 from dba_rolling_status; REVISION STATUS PHASE ---------- ------------ -------------- 0 READY BUILD PENDING SQL> exec dbms_rolling.build_plan(); PL/SQL procedure successfully completed. SQL> select revision, status, phase 2 from dba_rolling_status; REVISION STATUS PHASE ---------- ------------ -------------- 1 READY START PENDING


A Sample Walkthrough - Start Phase

Must be executed on the primary database

– Formally starts the rolling upgrade operation.

– When complete, the configuration possesses a fully configured transient logical standby database.

Still no outage

Start the Process

SQL> exec dbms_rolling.start_plan(); PL/SQL procedure successfully completed. SQL> select dbun, role, engine_status, 2 update_progress from dba_rolling_databases; DBUN ROLE ENGINE_STATUS UPDATE_PROG ---------- --------- -------------- ----------- bos PRIMARY NOT APPLICABLE NOT STARTED cam LOGICAL RUNNING NOT STARTED chi PHYSICAL RUNNING NOT STARTED SQL> select revision, status, phase 2 from dba_rolling_status; REVISION STATUS PHASE ---------- ------------ -------------- 1 READY SWITCH PENDING


A Sample Walkthrough - Start Phase

Make your changes to the database – For example, move a table from LOBs to Securefile LOBs

Stop SQL Apply Make the changes to the Transient Logical Standby Restart SQL Apply

Perform the changes


A Sample Walkthrough - Switchover Phase

Prior to the switchover you can move read clients to the transient logical standby to avoid an outage Update users will be stalled and

eventually must reconnect to the new primary using standard client failover best practices. Outage should be no more than

90-120 seconds for the database

Switchover!

SQL> exec dbms_rolling.switchover(); PL/SQL procedure successfully completed. SQL> select dbun, role, engine_status, 2 update_progress from dba_rolling_databases; DBUN ROLE ENGINE_STATUS UPDATE_PROG ---------- --------- -------------- ----------- bos LOGICAL STOPPED NOT STARTED cam PRIMARY NOT APPLICABLE NOT STARTED chi PHYSICAL STOPPED NOT STARTED SQL> select revision, status, phase 2 from dba_rolling_status; REVISION STATUS PHASE ---------- ------------ -------------- 1 READY FINISH PENDING


A Sample Walkthrough - Finish Phase

Manually restart and mount the all other databses Execute the FINISH Procedure to

finalize the rolling upgrade operation.

– Configures the former primary as a physical standby, and configures remaining physical standbys to recover the change redo from the new primary.

Finish the change

SQL> exec dbms_rolling.finish_plan(); PL/SQL procedure successfully completed. SQL> select dbun, role, engine_status, 2 update_progress from dba_rolling_databases; DBUN ROLE ENGINE_STATUS UPDATE_PROG ---------- --------- -------------- ----------- bos PHYSICAL RUNNING FINISHED cam PRIMARY NOT APPLICABLE FINISHED chi PHYSICAL RUNNING FINISHED SQL> select revision, status, phase 2 from dba_rolling_status; REVISION STATUS PHASE ---------- ------------ -------------- 1 READY DONE


A Sample Walkthrough - Finish Phase

Enable the Broker – It reconfigures itself to the new

Primary database ‘CAM’. Execute a Broker switchover

– If required to return to the original configuration.

Re-enable the Broker and switchover

DGMGRL> enable configuration; Enabled. DGMGRL> show configuration Configuration – OOW2012 Protection Mode: MaxAvailability Databases: cam - Primary database bos - Physical standby database chi - Physical standby database Fast-Start Failover: DISABLED Configuration Status: SUCCESS DGMGRL> switchover to bos; Performing switchover NOW, please wait... … Switchover succeeded, new primary is "bos"





In Closing

Agenda


DBMS_ROLLING: Key Features

Specify – Compile – Execute Protocol – Catch configuration errors at Compilation step – Run-time errors are detected during Execution

State is kept in the database – Provides robustness

Run-time steps are constant – Regardless of how many databases are involved

Handles failure at the original primary database Allows Data Protection for the upgraded primary right from the start


Behind the Scenes

PROD Physical STBY

Synchronize – redo apply Sets guaranteed restore point

(GRP) on A and B

Synchronize – sql apply Logical STBY

PROD Converts B to transient logical

Database A Database B

PROD SWITCHOVER Switchover Physical

STBY

Synchronize – redo apply PROD

Flashback A to GRP, upgrade via redo stream

Physical STBY

Current Structure New Structure


Now How Many Steps?

Prepare for maintenance – Disable the Broker, INIT_PLAN, BUILD_PLAN, Configure Redo transport

START_PLAN Perform Maintenance SWITCHOVER Mount original Primary FINISH_PLAN

– Enable the Broker, Switchover

Resources • OTN HA Portal:

http://www.oracle.com/goto/availability

• Maximum Availability Architecture (MAA): http://www.oracle.com/goto/maa

• MAA Blogs: http://blogs.oracle.com/maa

• Exadata on OTN: http://www.oracle.com/technetwork/database/exadata/index.html

• Oracle HA Customer Success Stories on OTN: http://www.oracle.com/technetwork/database/features/ha-casestudies-098033.html


Graphic Section Divider

oracle active data guard zero-downtime database …...• working as senior database administrator...

Documents