ebs_reducedowntime_onupgrade

Oracle E-Business Suite Release 12.1 Upgrade Best Practices -

Technical Insight

Udayan Parvate, Director, EBS Release Engineering

Lester Gutierrez, Senior Architect, EBS Applications Performance

The following is intended to outline our general product

direction. It is intended for information purposes only,

and may not be incorporated into any contract. It is not a

commitment to deliver any material, code, or functionality,

and should not be relied upon in making purchasing

decisions.

The development, release, and timing of any features or

functionality described for Oracle’s products remains at

the sole discretion of Oracle.

Objectives

• Provide an overview of the R12.1 release, the technology stack

and supported upgrade paths

• Outline best practices for the R12.1 technical upgrade, with a

focus on downtime reduction

• Review runtime performance monitoring and tuning tips

• Provide sample customer upgrade profiles and share Oracle’s

GSI upgrade experience

Agenda

• R12.1 Upgrade Overview

• Upgrade Best Practices

• Customer Upgrade Snapshots

• References

• Additional Resources

• Q&A

R12.1 Upgrade Overview

R12.1 Rapid Install (RI)

R12.1 Maintenance Pack (MP)

• R12.1 was generally available (GA) in May 2009

– Via Oracle Electronic Product Delivery (EPD) and Oracle Store

– For all the supported platforms and languages

• Can be used by new and upgrading customers (11.5.9 and

above) to go directly to R12.1

– If you are on R11i, use R12.1 RI from EPD. Follow instructions from

the “Upgrade Guide: 11i to 12.1” and “12.1.1 Release notes”

– If you are on R12.0.X, use R12.1 MP (7303030) from My Oracle

Support (MOS). Follow instructions from “R12.1 Maintenance Pack

Install Instructions (752619.1)”

R12.1 Technology Stack

For details “Oracle E-Business Suite Release 12.1 Maintenance Pack Installation Instructions” (752619.1)

TECHNOLOGY

COMPONENT

VERSION

INCLUDED

11i10CU2

VERSION

INCLUDED

12.0.4 RI

VERSION

INCLUDED

12.1 RI

VERSION

CERTIFIED

MINIMUM

REQUIRED

VERSION

• Apps Mid tier-

Forms/Reports 6.0.8.25 10.1.2.2 10.1.2.3 - 10.1.2.3

• Apps Mid tier-

Java Oracle

• Apps Mid tier-

1.0.2.2/1.4.2 10.1.3.0/1.5 10.1.3.4/1.6.0 10.1.3.5 10.1.3.4/1.6

• Database/

JDBC drivers 9.2.0.6/9.2.0.6 10.2.0.3 11.1.0.7/11.1.0.7 10.2.0.5/11.2.0.1 10.2.0.4/11.1.0.7

R12.1 Upgrade Paths

• Minimum EBS suite level for direct upgrade to R12.1

– 11i9, 11i9CU1, 11i9CU2 or above

– 11i10, 11i10CU1, 11i10CU2 or above

– R12.0 and above

• Minimum EBS suite level required for database versions

– 10.2.0.4 11i9CU2, 11i10CU2

– 10.2.0.5/11.0.7/11.2.0.1 11i10CU2, R12.0.4

• Based on R12.1 DB guide (761570.1), certified upgrade path options can

be categorized into :

A. Upgrade database and EBS level in a single downtime

B. Upgrade database and EBS level in separate downtimes

C. Apply upgrade interoperability DB patches and then upgrade EBS level

R12.1 Upgrade Paths Continued.. 11.0/11i.X => 12.1

< = 11.5.8

11.5.9/cu1

11.5.10/cu1

10.2.0.4

10.2.0.5

11.2.0.1

11.1.0.7

11i10cu2

C. Apply upgrade interoperability DB patches and then upgrade EBS

SOURCE

TARGET

11i9cu2

R12.1 Upgrade Paths Continued.. 12.0.X => 12.1

10.2.0.4

> = 12.0.4

10.2.0.5

11.2.0.1

11.1.0.7

< = 12.0.3 A

C. Apply upgrade interoperability DB patches and then upgrade EBS

SOURCE

TARGET

R12.1 .1 Key Facts

11i10CU2 R12.0.4 RI R12.1 RI

# of Product Schemas 209 195 (25 removed, 11 added

when compared with

11.5.10 CU2)

201 ( 25 removed, 17 added

when compared with

11.5.10 CU2 )

#file calls in DB

portion of the U driver 104242 144940 156622

PROD database size

File system size

#files shipped in RI 268359 357778 385755

#of Changed+ New

files in DB portion of

the U driver

NA ~95488 ( Vs 11.5.10.2 ) ~104057 ( Vs 11.5.10.2 )

~31843 ( Vs 12.0.4 )

Upgrade Best Practices

• Identification of required patches

– Prepare a complete list of pre and post patches and recommended

code levels

• Keep the system current on AD/ATG/OAM code e.g. latest AD/ATG

RUPs on 11i/R12.0 and once on R12.1

• High priority patches from MOS.

• Consolidated Upgrade Patches (CUP) and required one-offs that

need to be applied in pre-install mode. R12.1 CUP1 (7303029:12.1.0)

is available now

• Review “Known-issues” sections from key documents to determine

additional pre or post upgrade patches: Release notes, MP Install

Instructions

Upgrade Best Practices Continued..

• Patch merging and sequencing

– Take advantage of patch merge, hot patching and apply patches in the

correct sequence

• Merge patches

• Merge NLS patches per language. Apply NLS patches as “hot”

patches

• Perform uptime maintenance when possible e.g. hot patching of

iHelp or NLS patches, upload patch history

• HRGLOBAL step before NLS ( Document 145837.1 & 414434.1 )

• Stage patches locally accessible by mid-tier machines instead of on

nfs mounted or remote servers

• Adpatch options and other reporting tools

– Use applicable adpatch options to reduce downtime and track key

upgrade aspects using in-built tools

• Use non-interactive patching

• Use adpatch options such as nomaintainmrc, phtofile, nocompiledb,

nolink, nogenform, nogenrep, nocompile jsp, noautoconfig, novalidate

(1078973.1)

• Use optimal values for batch size and #workers

• Use OAM reports and Patch Wizard functionality to track patch

timing, files applied, functionality patched, impacted customized files

etc. (976188.1, 976688.1)

• Downtime reduction procedures – Use “Shared APPL_TOP” with “Distributed AD” for upgrades and

regular maintenance for multi-node instances

• No need to apply the same patch on multiple tiers

• Distributed AD adds to the degree of parallelism by distributing AD workers across application tier nodes and improves D/G portion of the patch driver.

– Use “Staged APPL_TOP” for regular maintenance and upgrade

• Saves time to patch the file system (C/G portion) by using a patched up copy of production instance file system

• Use in 11i => R12.1 upgrade to avoid applying NLS C/G portion

• Can use for R12.0.X => R12.1 upgrade and once on R12.1

• Downtime reduction procedures continued..

– For the duration of the upgrade, consider…

• Disabling custom triggers and business events

• Disabling auditing if enabled

• If possible run in noarchivelog mode

• Disabling flashback DB

• Removing TDE from volume tables

• Pre-upgrade activities

– Identify and execute tasks that could be completed in a

separate downtime period, prior to the production upgrade

• Use applicable steps mentioned in the "Downtime reduction" and “Upgrade By Request” appendices E and G of the R12.1 upgrade guide

• Upgrade RDBMS version to latest certified for the current APPS level ( 11.2.0.2 / 11.1.0.7 / 10.2.0.5 )

• Convert to Oracle Applications Tablespace Model (OATM)

– Compacts data, optimizes storage settings and I/O

• Other planned HW or OS upgrades

• Pre-upgrade activities continued...

• Use TUMS (“The Upgrade Manual Script”) to avoid running

tasks that are not relevant for your system

• Convert to multiple organization architecture (Document 210193.1)

• Drop MRC schema ( 11.5.10 and above )

• Assign post upgrade jobs to specialized CM queue (by

request_type, see Document 399362.1)

• Flush all the interfaces, such as Autoinvoice, Journal entry

import, order import etc..

• Review and disable all debug, logging at site, responsibility,

user level

• Gather stats with GATHER_AUTO option for all schemas

close to the start of downtime

• Pre-create new large indexes created by upgrade (constrained

to indexes on existing columns)

• Minimize historical data to be upgraded as per business

requirements – “Upgrade By Request”

• Purge old and/or transient data before upgrading

– Over 260 standard purge programs in R12

– Use new “Purge Portal” in OAM to administrate

• Purging – Use OAM to configure, initiate and monitor purge programs

• Set the execution frequency and view program history

• Programs tagged with the “Purge” program type

System Administrator >Oracle Applications Manager >Purging/Critical Activities

• Key recommended fixes

– Affecting Upgrade Performance

• AD-Parallel improved scalability: 8917381 (Part of 9179588 AD

• Forms/reports generation regression with 11g: 8557019

– General

• Review document 244040.1 for latest EBS recommended

performance fixes

• Database tier configuration

– Maximize SGA and PGA sizing

• An upgrade only involves 10's of concurrent sessions;

starting rules of thumb ...

– log buffer = 30 to 100 Mb

– shared pool = 1 to 4 Gb

– pga target = 3 to 20 Gb

– buffer cache = multi Gb, be generous without causing

excessive paging or swapping

• Adjust with help from AWR pool advisories

• Database tier configuration

– Other upgrade specific init.ora changes

• If specified, remove db_file_multiblock_read_count

– Maximize multiblock I/O sizes

• Set job_queue_processes = # of CPUS

– adobjcmp.sql (Phases : plb+90 and last+63)

• Set parallel_max_servers = 2 X CPUs

– Helps with large index creation, stats gathering and some

large upg+ phase jobs

– Need to test with production-like DB server and I/O subsystem

– Shutdown other RAC instances

• Batch size and #workers

– Batch size

• 10K is suitable for most installs, you can test other values from 1K

up to 100K if time allows

– # Workers

• Starting rule-of-thumb is between 1 and 1.5 x #CPUs

– It is critical to do multiple rounds of testing, adjusting above settings to

maximize server utilization, but constrained by factors such as

• Memory utilization (no swapping/ excessive paging)

• CPU utilization (scale down if at 100%)

• I/O response times (scale down if averages > 20 ms)

• Performance testing, monitoring and additional

optimizations...

– Analyze long runners via timing report, ad_task_timing analysis

– Mine ad_task_timing to identify low worker utilization due to

phasing waits and review responsible culprits

– Review targeted AWR Instance and SQL reports

– awrrpt.sql and awrsqrpt.sql

– Use My Oracle Support to check for known issues and

workarounds for the longest running jobs

optimizations continued... – Outside-the-box optimizations – Requires Support Approval*

• Long running index creation and Statistics gathering

– As these tasks run with little or no concurrent tasks, consider using alter system commands to increase parallel_max_servers and pga_aggregate_target

(Need to test so as to not exhaust machine resources)

– Large indexes: Pre-create, but do not alter column definitions

– For stats (adsstats.sql ; phase: last+63) : On RAC, consider using all nodes when gathering stats or use exp/imp of stats

optimizations continued... – Outside-the-box optimizations – Requires Support Approval*

• Long running mv related xdf’s (“en” phases)

– Cleanup and truncate large MV logs (requires MV complete refresh)

• Look more closely at long running jobs that you suspect may not be required for your system

– Thousands of jobs and customer product/data/setup configurations

– Some of the heavy lifting work may not be required for a specific site

– Check with Oracle Support to validate

– A fix maybe available or possible to optimize for your case

– New 12.1.3 AD timing report by module

• Runtime Performance Testing Tips – Use Automated, scripted tools

• EBS Test Started Kits (Winrunner/QTP)

– Bundled QA based automated scripts for EBS testing - Patch 8408886

• Oracle Applications Testing Suite (Accelerators for EBS)

– Web and Forms based flows

– Complement with user participation tests and batch load tests with frequent

and critical jobs

– References

http://blogs.oracle.com/stevenChan/2009/10/oats_ebs_certified.html

http://blogs.oracle.com/stevenChan/2009/08/evolutionary_steps_automated_testing_ebs.html

• Runtime Performance Testing Tips – Review the white paper: Upgrade to Oracle 11g - Performance Best

Practices

http://www.oracle.com/apps_benchmark/html/white-papers-e-business.html

• Points to note include11g features for

– Testing of 11g or 10g workloads with Real Application Testing

– Managing SQL Execution Plans and minimizing performance degradation with11g SQL Plan Management

– Session OOW2010: Tuning the Oracle E-Business Suite Environment

– Review EBS Benchmark Publications • http://www.oracle.com/apps_benchmark/html/white-papers-e-business.html

Customer Upgrade

Snapshots

Oracle E-Business Suite Release 12.1 Live

Customers

Customer Upgrade Snapshots

11i to 12.1

• CPS (Chicago Public Schools) – Release: 11.5.10.2 to 12.1

– DB size: 900GB

– #Workers and batch size: 32, 10000

– #CPUs on DB server: 2 node RAC, 8 CPUs per node

– Downtime reduction measures

• Distributed AD with shared APPL_TOP

• Upgrade RDBMS to 10.2.0.4 in a separate downtime

• # hrs for the D driver: 22 hrs

– Customer snapshot http://www.oracle.com/customers/snapshots/chicago-public-schools-ebs-snapshot.pdf

Customer Upgrade Snapshots Continued...

11i to 12.1

• Cisco

– Release: 11i to R12

– DB size: 600GB

– #CPUs on DB server: 16

• Distributed APPL_TOP

– #hrs for the D driver: 5.5 hrs

11i to 12.1

• Dell

– Release: 11i10 to R12.1.2

– DB size: 15TB , 16 node RAC Cluster

• Pre-create large indexes

– #hrs for the D driver: 30 hrs (latest test results)

12.0 to 12.1

• Zebra Technologies Corporation – Release: 12.0.6 to 12.1

– DB Size: 106GB

• Staged APPL_TOP

– #hrs for the U driver: 12 hrs

– Customer snapshot http://www.oracle.com/customers/snapshots/zebra-technologies-corporation-ebs-snapshot.pdf

• Oracle GSI – Release: 12.0.3+ to R12.1

– DB size: 17TB

– #CPUs on DB server: 88 processors

• Staged APPL_TOP for US and ten languages

• Ran data fixes for problems found in test upgrades prior to production upgrade to minimize stoppages

• Distributed APPL_TOP (4 servers - 15 workers each)

• Pre-create large indexes (6 hrs savings)

– #hrs for the D driver: 14 hrs

12.0 to 12.1

• Oracle GSI – Release: 12.1+ to R12.1.3 -- (Latest Testing)

– DB size: 17TB

– #CPUs on DB server: 150 processors

• Ran data fixes for problems found in test upgrades prior to production upgrade to minimize stoppages

• Distributed APPL_TOP (4 servers - 50 workers each)

• Pre-create large indexes

12.1 to 12.1.3

• AT&T – Release: 12.0+ to R12.1.2

– DB size: 10 TB

– #CPUs on DB server: 32 Processors

12.0 to 12.1

References

• R12.1 documentation roadmap (790942.1)

• “Oracle E-Business Suite Release 12.1 Info center” (806593.1)

• Database preparation guidelines for R12.1 upgrade (761570.1)

• Patching FAQs (459156.1, 225165.1)

• Using staged or shared APPL_TOP and distributed AD (734025.1, 384248.1, 236469.1)

• OAM “Patch Wizard” overview and FAQ (976188.1, 976688.1)

• AD Command Line Options for Release R12 (1078973.1)

• Recommended Performance Fixes (244040.1)

• R12 Upgrade Sizing & Best Practices (399362.1)

Additional Resources

• White paper

– Planning Your Oracle E-Business Suite Upgrade from Release 11i to

Release 12.1 (987516.1)

– R12 Upgrade considerations by product: Financials (889733.1)

• Have upgrade questions ?

– Post on OTN R12 upgrade forum

http://forums.oracle.com/forums/forum.jspa?forumID=395&start=0

Finding Additional Information Accelerate your evaluation and planning

Contains

• Presentations

• RCDs

• RVPs

• Videos

• Customer

Stories

• White Papers

• etc

http://launch.oracle.com/?OOW

Oracle Products Available Online

Oracle Store

Buy Oracle license and support online today at

oracle.com/store

We encourage you to use the newly minted corporate tagline

“Software. Hardware. Complete.” at the end of all your

presentations. This message should replace any reference to

our previous corporate tagline “Oracle Is the Information

Company.”

Oracle OpenWorld

Latin America 2010

December 7–9, 2010

Oracle OpenWorld

Beijing 2010

December 13–16, 2010

EBS 12.1 UPGRADE DEMOPOD

MOSCONE SOUTH, S-089

MON : 9:45 AM – 5:30 PM

TUE : 9:45 AM– 5:30 PM

WED : 9:00 AM – 4:00 PM

SUMMARY

DO’s..

•Read and follow official documentation and have a project plan

•Engage Support early on to validate your upgrade plan

•Identify and execute tasks you can do today

•Test ! Test ! Test !

•Keep track of patches applied in test upgrades

•Be smart about using the right tools and explore published downtime reduction techniques

•Optimize patch runs to suite your H/W

DON’Ts..

• Ignore including relevant functional SMEs in the planning process from the beginning

• Skip failed or long running jobs as a regular practice

• Drop indexes

• Run adpatch in serial mode

• Perform workarounds without approval from support/DEV

• Insufficient time and/or # of rounds of upgrade and critical functional flow testing

ebs_reducedowntime_onupgrade

upgrade database

technical upgrade

upgrade guide

direct upgrade

supported upgrade paths

ebs level9 r12

agenda r12

certified upgrade path

Documents

one flew over the cuckoo's nest

xcdc14a

(tesla) - the tesla magnetic car engine

aesops fables

iron mills essay

life is just a dream - or is it?

the dutch republic in international trade

jan van eyck and the man in a red turban

barclays1

introduction to six sigma

fortran

star wars trivia!

venture capital

disclaimer

compressing and decompressing folders

tragic heroes

chapter 24

explore the levels of creation

oedipus the king: the ideal tragic play

who killed god