how to upgrade hundreds or thousands of databases

How to Upgrade Hundreds or Thousands of Databases In a Reasonable Amount of Time

Copyright © 2015, Oracle and/or its affiliates. All rights reserved. |

Roy Swonger Senior Director & Product Manager Database Upgrade & Utilities Oracle Corporation

Mike Dietrich Master Product Manager Database Upgrade Oracle Corporation


Safe Harbor Statement

The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracle’s products remains at the sole discretion of Oracle.

How to Upgrade Hundreds or Thousands of Databases 2


$> whoami

3

20 years w/Oracle Previously with DEC Rdb

11+ years managing Data Pump Database Upgrade SQL*Loader Transportable Tablespaces

How to Upgrade Hundreds or Thousands of Databases

Roy Swonger


6 years RDBMS Core & Mission Critical Support

5.5 years Technology Presales for DataGuard, Upgrades

>7.5 years ST Upgrade Development Team

50% Reference Projects

50% Workshops Worldwide

+ x% Development

Work

$> whoami

4 How to Upgrade Hundreds or Thousands of Databases

Mike Dietrich Master Product Manager Database Upgrades & Migrations

MikeDietrichDE

http://blogs.oracle.com/UPGRADE

Copyright © 2015 Oracle and/or its affiliates. All rights reserved. |

Download Slides from the Database Upgrade Blog





Supported Versions

2007

2008

2009

2010

2011

2012

2013

2014

2015

2016

2017

2018

2019

2020

2021

2022

2023

2024

2025

Oracle 10.1 (GA: Jan 2004)

Oracle 10.2 (GA: Jul 2005)

Oracle 11.1 (GA: Aug 2007)

Oracle 11.2 (GA: Sep 2009)

Oracle 12.1 (GA: Jun 2013)

Oracle 12.2 (GA: xxx 2016)

today


Premier Support Waived Extended Support Paid Extended Support Sustaining Support

Jan 2009 Jan 2012

Jul 2010 Jul 2013

Aug 2012 Aug 2015

Jan 2015

Jun 2018 Jun 2021

Dec 2020 May 2017


We fully understand that things often are complicated and business critical



And things can become really complicated ...


Oracle 8i to 11.2

26 TB

Oracle EBS

5-6 hours max downtime

Good luck!!!


Or more work ...

Triple hops


↘ Oracle 9.2.0.1

↘ Oracle 9.2.0.4

↘ Oracle 10.2.0.5

↘ Oracle 12.1.0.2


But even if it does not look complicated at first sight ...


0

50

100

150

200

250

300

350

400

10.2.0.3.0 10.2.0.4.0 10.2.0.5.0 11.2.0.1.0 11.2.0.2.0 11.2.0.3.0 11.2.0.4.0

2 21 9 10

113

292

382

Databases

54% (447 out of 829 databases) have no bug fixing support anymore


Another example ...


0% 4% 5%

2% 3%

17%

35%

32%

2%

Databases

9.2.0.7.0

10.2.0.3.0

10.2.0.4.0

10.2.0.5.0

11.2.0.1.0

11.2.0.2.0

11.2.0.3.0

11.2.0.4.0

12.1.0.2.0

828


Don't maintain a zoo!


135 Oracle databases

– 7 different Oracle releases

– 13 different Oracle patch levels

6 different OS vendors – 15 different OS versions

2% of all databases

under bug fixing support


Recommendation

Don't sit it out

– It will just get more complicated and risky

Beware of application dependencies

Establish constant database inventory monitoring



Where should you start?



Project Approach?

Planning Data

Gathering Testing

Upgrade/Migration

Go-Live Quarterly Patching



You need a plan


Week T-10 Migration Initiation, & Plan

Baseline & Migration

Migration & Unit test initiation

Unit Test Complete

Application & Performance Test Start

Application & Performance Tests

Application & Performance Tests

Application Testing Complete and Application Freeze

Week T- 9

Week T-8

Week T-7

Week T- 6

Week T-5

Week T-4

Week T-3

Week T-2

Week T-1

Cutover Plan & Tests

Migrate Go/No go

Migration Weekend


You need to know Which / Where / Who / When

Database inventory

OS inventory

Depending applications, certification

Responsibilities

Desired time frame



Build a tracking app – Use APEX



Build Workstreams and Database Groups


Estate Analysis

Workstream 1

Workstream 2

Workstream 3

Upgrade/Migration Group 1



Post Migration Monitoring,

Troubleshooting, and Tuning Team

Production

Production with additional monitoring


General Test Guidelines

Upgrade/Migration Process

Post-Upgrade Functionality

Post-Upgrade Performance

Pro

d L

oa

d



Testing Effort

Relation between test completeness and costs

Costs low high

Co

mp

lete

nes

s 0

%

10

0%



Use the Right Tools: AWR Diff Reports

AWR Diff Report

Migrate

R2

10.2.0.4 12.1.0.2

Snapshot Interval Snapshot Interval

IBM AIX SOLARIS

Compare



Use the Right Tools: Database Replay

Database Replay

Migrate

R2

10.2.0.4 12.1.0.2

Capture Workload Replay Workload

Compare – then restore and run again with changes

IBM AIX SOLARIS

Preprocess Workload



Use the Right Tools: SQL Performance Analyzer

SQL Performance Analyzer

STS

Capture SQLs into a STS

Load STS

STS

Evaluate, compare and tune – over and over again!

Migrate

R2

10.2.0.4 12.1.0.2 IBM AIX SOLARIS



Use the Right Tools: SPA Resulting Reports

Example: Report showing regressed statements and plans



Now RELAX ... Stay Calm ... And open an SR with Oracle Support in case of real trouble


Picture taken from: https://aycw.wordpress.com/2015/06/10/young-peoples-attitudes-toward-work/


And don't forget: The next patching day is right in front of you! Oracle PSUs get delivered on a quarterly basis: 20-OCT-2015 / 19-Jan-2016 / 19-Apr-2016 / 19-Jul-2016



Use Oracle EM Lifecycle Management Pack



With a little help from ... Real Application Testing Upgrade over 300 databases at CSX



Real World Checkpoint

CSX

– One of the nation‘s leading transportation suppliers Encompasses 21,000 miles of tracks

in 23 states, from Florida to Ontario

HQ in Jacksonville, Florida

Customer

Project

Constraints

Preparation

Upgrade

Success?

Remarks



Upgrade and migration of 304 databases to 11.2.0.2 over a 24 month period (2011-2012)

– 1/3 production, 2/3 test and development systems

– Source versions: Oracle 10.2.0.4 and a few Oracle 8i/9i

– Single Instance on IBM AIX RAC on Linux

Plan to use Real Application Testing

– Guarantee quality and mitigate the risk of upgrading

Define standards

– Install 11.2.0.2 and patches and apply latest PSU


Customer

Project

Constraints

Preparation

Upgrade

Success?

Remarks



Limited downtime on some business critical systems

Testing cycles vary from 1-6 months depending on complexity and criticality

Tight on available testing resources

– Time

– People

– Systems

New to Real Application Testing


Customer

Project

Constraints

Preparation

Upgrade

Success?

Remarks



Close alliance with Oracle's Upgrade Development Group and Oracle's RAT Pack

– CSX agreed to become an official Oracle Database 11.2 Reference Customer Monthly status meetings

Onsite training

Live upgrade and RAT demo

– Cloning databases methods for use with RAT


Customer

Project

Constraints

Preparation

Upgrade

Success?

Remarks



Upgrade and migration methods used:

Policy:

– Install Oracle SW with latest patches and PSU

– Combine HW refresh and OS upgrades

Internal application to track owner, status, success

80%

20% Data Pump for OS Migration

DBUA for Upgrades


Customer

Project

Constraints

Preparation

Upgrade

Success?

Remarks



Yes –project took 2 years

– 88% of all databases upgraded within 18 months

– Some systems took longer due to business and/or resource constraints

Performance comparison mechanisms are very helpful

Regressions found and fixed before upgrade

RAT helped a lot to predict workload performance and ensure careful testing


Customer

Project

Constraints

Preparation

Upgrade

Success?

Remarks



“The Real Application Testing tool provided a comprehensive and flexible solution for assessing the impact of the Oracle 11g database upgrade into CSX systems. At CSX we were able to capture real production workloads, replay it in the 11g environment, identify poor performing queries and, fine tune these queries in a test environment before the production implementation.“

Maritza Gonzalez, Technical Director

CSX Corporation


Customer

Project

Constraints

Preparation

Upgrade

Success?

Remarks



More Information? https://blogs.oracle.com/UPGRADE/entry/csx_corporation_upgrades_databases_2x


Customer

Project

Constraints

Preparation

Upgrade

Success?

Remarks



A small team and an excellent plan Upgrade 300 databases at Mobiliar Insurance, Switzerland




Swiss Mobiliar

– Switzerland's most personal insurer

– Founded 1826 in Bern, oldest Swiss insurance

– Legal form:

Cooperative association (mutual company)

– Over 1.7 million persons and firms insured

– Switzerland's no.1 insurer for:

Businesses, term life insurances, households

– 80 general agencies at 160 locations

– Over 4,400 employees and 325 trainees

– Awarded Most Trusted Insurance Brand in Switzerland for 13 consecutive years

39

Customer

Project

Constraints

Preparation

Upgrade

Success?

Remarks




Upgrade 289 databases

– Oracle Database 11.2.0.3 to Oracle Database 12.1.0.2

– 80 production databases

Oracle Cloud Control

AWR Warehouse (home-made)

Shell scripts

Oracle Restart with ASM

Oracle In Memory

OID

40

Customer

Project

Constraints

Preparation

Upgrade

Success?

Remarks




Project timeline: 24 months

– Goal: 18 months

41

Customer

Project

Constraints

Preparation

Upgrade

Success?

Remarks


0

5

10

15

20

25

30

35

40

45

Jan-15 Feb-15 Mrz-15 Apr-15 Mai-15 Jun-15 Jul-15 Aug-15 Sep-15 Okt-15 Nov-15 Dez-15

Migrations per month



Small DBA Team

– Only 3.8 FTE + 1 newbie

2 major software releases of Mobiliar Applications per year resulting in a full month of code freeze

Bug fixing support for Oracle 11.2.0.3 ended Aug 27, 2015

Highly complex applications

– Queries with more than 1000 bind variables and 250 outer joins

One core team member broke his leg in September 2015 and did not return to work yet

42

Customer

Project

Constraints

Preparation

Upgrade

Success?

Remarks




All queries, execution time and execution paths are tracked on a self made AWR Performance DWH

Possibility to compare performance before and after the upgrade on query level

Performance tests performed by application owner

Regression tests done during the testing phase of the Mobi Software Release

– PSU RDBMS 12.1.0.2.4, PSU GRID 12.1.0.2.2

– Two Oracle Homes on the same server

– Upgrade with catctl.pl embedded into home-built shell script

43

Customer

Project

Constraints

Preparation

Upgrade

Success?

Remarks




Project Monitoring with Excel Sheet

44

Customer

Project

Constraints

Preparation

Upgrade

Success?

Remarks


0

50

100

150

200

250

300

350



DBs 12c 0 7 10 18 26 62 105 126 136 152 157 170

DBs 11g 289 282 279 271 263 227 184 163 153 137 132 119

Oracle Database Swiss Mobiliar

DBs 11g DBs 12c



Automated upgrade via home-built scripts:

45

Customer

Project

Constraints

Preparation

Upgrade

Success?

Remarks




Live? And alive?

– Yes! First system went live on Feb 4, 2015

– Current status (Oct 12, 2015):

152 out of 289 databases live on Oracle Database 12.1.0.2

53% upgraded already

Project perfectly on schedule

46

Customer

Project

Constraints

Preparation

Upgrade

Success?

Remarks


137

152

No. Databases

Oracle. 11.2.0.3 Oracle 12.1.0.2



No major issues after go live

– Issues caught during testing, e.g.:

– Bug 20776435: PARSE OF INMEMORY QUERY IS SPINNING ON CPU Query with 250 Left Joins

Workaround: alter session set "_optimizer_inmemory_table_expansion"=false;

– Bug 19677469: NO QUERY REWRITE Workaround: CREATE MATERIALIZED VIEW …. SELECT /*+ MV_MERGE */ …

or: alter system set "_fix_control"='10145667:OFF';

47

Customer

Project

Constraints

Preparation

Upgrade

Success?

Remarks




Oracle In-Memory boosts performance

– Read on: http://tinyurl.com/Mobi12c

48

Customer

Project

Constraints

Preparation

Upgrade

Success?

Remarks


http://tinyurl.com/Mobi12c



"The entire upgrade project of our 300 Oracle databases at Die Mobiliar is running very well.

When we catch issues, we fix them before going live. And features such as Oracle In-Memory add a lot of performance boost to some of our applications.

Especially the Oracle Upgrade Reference Program was a great help and added a lot of value to our project's progress.

We are very happy and satisfied with the upgrade process and the reliability and performance of Oracle Database 12c". Paolo Kreth, Group Manager Databases, Die Mobiliar

49

Customer

Project

Constraints

Preparation

Upgrade

Success?

Remarks



Famous Last Words Conclusion



Conclusion

If you have a complicated environment, you are not alone!

But, you can do it:

– Plan your effort

– Use the right tools

– Track and report progress



Download Slides from the Database Upgrade Blog




Copyright © 2015 Oracle and/or its affiliates. All rights reserved. | Oracle Confidential – Internal 53

Classroom Training

Learning Subscription

Live Virtual Class

Training On Demand

Keep Learning with Oracle University

education.oracle.com

Cloud

Technology

Applications

Industries