© 2010 oracle corporation 1 · • custom scripts developed around opatch sqlplus etc opatch •...

Session #S316967Session #S316967 Patching Enterprise-Wide Databases: Automation Techniques & Real World Insights

Hari Srinivasan, Pr. Product Manager, OracleHari Srinivasan, Pr. Product Manager, OracleTim Misner, Sr. Director Product Development, OracleDeepa Nambiar, Sr. DBA, Johns Hopkins Applied Physics Lab

Safe Harbor Statement

The following is intended to outline our general

product direction. It is intended for information

purposes only, and may not be incorporated into any

contract It is not a commitment to deliver any materialcontract. It is not a commitment to deliver any material,

code, or functionality, and should not be relied upon in

making a purchasing decision. The development, g p g p ,

release, and timing of any features or functionality

described for Oracle’s products remains at the sole

discretion of Oracle.


Oracle Enterprise ManagerSession # S316975

Patching Enterprise-Wide Databases: Automation Techniques & Real World Insights

You had almost given up on Patching thousands of databases. And you are not alone to be bugged by the cumbersome,

Automation Techniques & Real World Insights

manual processes involved in database patching. Till Enterprise Manager 11g arrived with its rich MOS integration. In this session, you will discover how customers have realized high y gROI from Enterprise Manager's patch management solution.


Business-Driven IT Management


Program Agenda

• Common Patch Management ChallengesPatch Management Options• Patch Management Options– Contrast between various manual and

automated options

• Enterprise Manager 11g driven Patch Management– Features and BenefitsFeatures and Benefits– Best Practices

• Customer Case Study – Johns Hopkins Laboratory


Top 4 Patching Challenges

Identifying affected systems

Maintaining scripts

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

Identifying and resolving conflicts

Identifying right patches

• Huge overhead in maintaining Home grown scripts• Identifying vulnerable targets among possible• Identifying vulnerable targets among possible

thousands• Identifying right patches to be applied• Resolving patch conflicts and filing merge requests if

needed


* Patching Survey conducted by Oracle across customers, Mar 2010

Patching HabitsFrequency & Complexityeque cy & Co p e y

Frequency

More than 4 times

Number of patches applied per downtime

Twice

Thrice

Four times (quarterly)

One

Multiple

0% 10% 20% 30% 40%

Once

0% 20% 40% 60% 80%

One

• Most customers patch multiple times a year– Quarterly patching most common

M t t l lti l t h d ti• Most customers apply multiple patches per downtime– Some large customers apply 15-20 patches in a single

downtime session


* Patching Survey conducted by Oracle across customers, Mar 2010

Typical Database Patch Management ProcessComplex, manual, error-prone…Co p e , a ua , e o p o e

1 Identifypatches

2 Download patchesto local system

3 Identify targets

4 Apply ontest system

5 Create/update(optional)

scripts

• Quarterly Security Patch from Oracle

• Service Requests• Patch from other

• Oracle Support ServicesO Based onPatch from other

sources• Oracle Technology Network• My Oracle Support

HelpD k

6 Conflictsdetected?

No

7 File SR, monitor,replace with merge

Yes

C t

Based on heterogeneous environments (RAC, single instance, DR)

Desk

8 Apply across test systems

replace with mergepatch

Create an incident

Request for downtime / t i l

9 Test patch in QA environment

10 Apply patch in productionManual processes

FTPpatches

retrieve approval

Request for downtime / retrieve approval


11 Verify patch application

processes by one or many DBAsUpdate / close RFC

Typical Database Patch Management Process…and time-consuming too (experience of large customer w/1000+ databases)a d e co su g oo (e pe e ce o a ge cus o e / 000 da abases)

1 Identifypatches

2 Download patchesto local system

3 Identify targets

4 Apply onTest system

5 Create/update(optional)

scripts3 hrs.

• Quarterly Security Patch from Oracle

• Service Requests• Patch from other

• Oracle Support ServicesO

Based on

3 hrs. 8 hrs. 3 hrs.2 days to1 week

Patch from other sources

• Oracle Technology Network• My Oracle Support

HelpD k


No

7 File SR, monitor,replace with merge

Yes

C t

ased oheterogeneous environments (RAC, single instance, DR)

1 day to2 weeks

Desk

8 Apply across Test systems

replace with mergepatch

Create an incident


1 week

9 Test patch in QA environment

10 Apply patch in ProductionManual processes

FTPpatches

retrieve approval


2 weeks5 hrs.


11 Verify patch application

processes by one or many DBAsUpdate / close RFC

10 weeks

Total Rollout Effort (Per Quarter): 25 weeks!!!

Current Patching Tools

• Command Line tool installed with the base database product• Scope of execution limited to an Oracle Home

• Custom scripts developed around opatch SQLPlus etc

opatch• Supports interactive and silent modes• Limited capability to handle pre and post patching steps

Custom

• Custom scripts developed around opatch, SQLPlus, etc.• Usually limited to a single server• Significant maintenance overhead to meet new version and

configuration needs

• Out of box procedures to handle end-to-end patch deployment• Can span multiple servers

scripts

EnterpriseManager

• Can span multiple servers• Can handle pre- and post- patching steps• Oracle-provided updates to handle new version and

configuration needs


Comparison of Patching Options

OPATCH CUSTOM SCRIPT ENTERPRISE MANAGER

Automatic advisory No No Yes

Patch conflict check Deployment time only Deployment time only Yes (Up-front)

Community information review No No Yes

Automated download No No Yes

Multiple instances support per HOME

No Yes YesHOME

Pre and post Shutdown and startup

No Yes Yes

Multiple databases on No No Yesdifferent machines

Multiple patches in single run Yes Yes Yes

Post patching dictionary changes

No Yes Yeschanges

Rolling Clusterware and RAC patching

Limited Limited Yes

Guaranteed support for new database versions

Yes No Yes


database versions

Enterprise Manager – Patch Management ProcessReliable Scalable AutomatedReliable, Scalable, Automated

1 Identifypatches

2 Consolidate listof patches

3 Create patch Plan

4 ValidatePatch Plan

• Patch advisory• Quarterly Security

Patch from Oracle• Service Requests• Patch search

NY

Siebel Help Desk


NoPlace merge patchrequest; monitor SRfor a merge patch

Yes

C t

6

7 Create Policies to test patch application

Create an incident


BMCRemedy

HPCompare Oracle

f8 Test patch in QA environment

9 Apply patch in production

retrieve approval


HPServiceCenter

Homes forcompliancevalidation


10 Verify patch application (re-run UDP)Update / close RFC

13*MOS@EM - My Oracle Support Integrated within EM

Enterprise Manager Patch ManagementEnd-to-End Change Lifecycle

• Advise/recommend patches based on configuration

• Let users research the patch by providing readme and Patch community informationPatch

Advise

• Plan downtime• Detect and verify

patching success

PatchVerification

PatchPlanning

• File change request• Identify patch conflicts,

impact and dependencies

• Detect drift from existing gold images and rebuild them for

PatchR ll t

dependencies• Plan rollback strategy• Test patch in test

environment

future software rollouts

• Close change request Rollout • Test application in test

environment

• Roll out the patches within stipulated downtime

eques


stipulated downtime

Database Patch ManagementPatch Advise

• Generate advisories based onC & O O– Critical patch updates & Other Oracle Recommendations

– Incident (SR) based one-off recommendations

PatchAdvise

P h• Community information (from tens

of thousands of customers)• Advisories are ranked based on

PatchVerification

PatchRollout

PatchPlanning

criticality• File exception violations with

Remedy, HP Service Center, Siebel CRMCRM


Database Patch ManagementPatch Advisea c d se


“Enterprise Manager enabled systems

What APL Is Saying

administrators to use personalized dashboards and power views to quickly identify issues related to their areas of responsibility and take timely action”Johns Hopkins University, Applied Physics Laboratory


Database Patch ManagementPatch Planninga c a g

• Bundle multiple changes for rollout under a single downtimeunder a single downtime

• Upfront conflict and pre-requisite analysis

– Integrate with Oracle Support to request and get Merge patches

PatchAdvise

P hget Merge patches• Understand environments and

“Analyze” patchability before the actual deployment

PatchVerification

PatchRollout

PatchPlanning

• Design change workflow using Plans• Deploy the changes with Best Practices

– Deployment Procedures• Activate Policies for ongoing tracking of• Activate Policies for ongoing tracking of

the patches• Bi-directional integration with Remedy,

HP Service Center, Siebel


Database Patch ManagementPatch Planninga c a g


“The availability of Oracle Real Application Testing

What Starwood Is Saying

for Oracle Database 10g and Oracle9i Database, is key to accelerating our migration of mission critical applications to Oracle Database 11g in an efficient manner.”Starwood Hotels & Resorts Worldwide, Inc.


Database Patch ManagementPatch Rollouta c o ou

• Patch multiple installations with multiple patches in a single windowpatches in a single window

– Support one-offs, patchsets, CPUs and PSUs (incl. overlay patches)

– Support patching of RAC, Clusterware, ASM,

PatchAdvise

P hDatabase – Zero-downtime Rolling patching of RAC

clusters– Support patching Databases on Physical

PatchVerification

PatchRollout

PatchPlanning

Support patching Databases on Physical Standby

• Support impersonation methods for deployment in multi-admin environments

• Support Software Library for Release Mgmt

• Flexible error handling, retry and rollback


Database Patch ManagementPatch Rollouta c o ou


“We manage thousands of databases and application ith E t i M d h b

What Bayer Is Saying

servers with Enterprise Manager, and we have been able to reduce the patch application time from one hour down to one minute per database. Enterprise Manager allows us to automate this process which translates intoallows us to automate this process, which translates into huge savings in time and money.”Bayer


Database Patch ManagementPatch Verificationa c e ca o

• Post-change validation and gverification– Deep out-of-box checks for

Databases

PatchAdvise

P h• Create User Defined Policies for ongoing compliance tracking

• Out-of-box patch reports

PatchVerification

PatchRollout

PatchPlanning

• Check compliance to the Patch policy


Database Patch ManagementPatch Verificationa c e ca o


“Enterprise Manager helps us efficiently address li d l t i t S ifi ll

What CERN Is Saying

compliance and regulatory requirements. Specifically, Configuration Management Pack provides proactive change detection, policies and reporting capabilities that simplify and automate processes enabling us to furthersimplify and automate processes, enabling us to further reduce costs and risk and improve overall efficiencies.”CERN


Best PracticeRecommendations


Commonly Asked Questions

How do I ensure that I find and fix all possible errors b f t d i d ti i d ?before, not during, my downtime window?

Can I reduce the downtime involved in the patching process?process?

Can I make the patching operation more secure and auditable?

Can the procedure notify me during execution so I can let it run completely unattended (“lights out”)?

Can I use patching features in my security restrictedCan I use patching features in my security restricted environment?


Make Patch Rollouts ReliableBest Practice Recommendation 1es ac ce eco e da o

How do I ensure that I found and fix all possible errors before, not during, my downtime window?– Is my environment patchable?

Do the targets meet the pre-requisites?– Do the targets meet the pre-requisites?

Can I reduce the downtime involved in the patching process?



Can I use patching features in my security-restricted environment?( g )


Check for Patchability Using ReportsBest Practice Recommendation 1es ac ce eco e da o

Here’s How


Run Patchability Report from Deployments >> Reports >> ‘EM Target Patchability Report’

Check for Patchability Using Pre-Flight ChecksBest Practice Recommendation 1es ac ce eco e da o

Here’s How


Run the Deployment Procedure in Analyze Mode at the end of the procedure interview

Make Patch Rollouts ReliableBest Practice Recommendation 1es ac ce eco e da o

How do I ensure that I found and fix all possible errors before, not during, my downtime window?– Is my environment patchable?

• EM Target Patchability Report: Identify patchable• EM Target Patchability Report: Identify patchable targets under the environment monitored and mitigate issues to improve patching coverage

– Do the targets meet the pre-requisites?– Do the targets meet the pre-requisites?• Pre-flight checks with Analyze Mode: Run

comprehensive checks on Patches and Targets checking for applicability of the patch and targetchecking for applicability of the patch and target sanity to ensure success during patch deployment


Stage Patches Pre-Downtime Best Practice Recommendation 2es ac ce eco e da o

How do I ensure that I found and fix all possible errors before, not during, d ti i d ?my downtime window?

Can I reduce the downtime involved in the patching process?p– Can I pre-stage the patch before patching process?– Can I use a shared location to stage the patches and apply

patches from there?Can I make the patching operation more secure and auditable?

patches from there?

Can the procedure notify me during execution so I can let it run completely tt d d (“li ht t”)?

Can I use patching features in my security-restricted environment?unattended (“lights out”)?


Pre-Stage Patches to Shared LocationBest Practice Recommendation 2es ac ce eco e da o

Here’s How

Pick a procedure >> Specify the location to stage the patch >> Execute the Procedure in Analyze mode – Patch gets staged in the location specified during Analysis.

Disable the Stage step in the Procedure >> Select the previously specified patch stage


Disable the Stage step in the Procedure >> Select the previously specified patch stage location – Patch is selected from the pre-staged location rather than being restaged.

Save Downtime and Bandwidth with Pre-StagingBest Practice Recommendation 2es ac ce eco e da o

How do I ensure that I found and fix all possible errors before, not during, d ti i d ?


my downtime window?

process?– Can I pre-stage the patch before patching process?– Can I use a shared location to stage the patches and

apply patches from there?apply patches from there?• Save Downtime and Bandwidth: Stage the patch

once to a shared staging location. Patch the local hub of systems from the pre-staged locationy p g


Make Patch Rollouts Secure and AuditedBest Practice Recommendation 3es ac ce eco e da o 3

Can I avoid identifying errors that are based on myi t d i th d l t ?


environments during the deployment process?


Can I avoid using shared OS accounts while patching?– Can I avoid using shared OS accounts while patching? Can I lock access to ‘Oracle’ account?

Can the procedure notify me during execution so I can let it run completely tt d d (“li ht t”)?

Can I use patching features in my security-restricted environment?unattended (“lights out”)?


Use Impersonated Accounts for RolloutsBest Practice Recommendation 3es ac ce eco e da o 3

Pick a procedure >> Do a ‘Create Like’ >>

Here’s How


Pick a procedure >> Do a Create Like >> >> Specify the User Impersonation settings in Privilege Delegation Settings

Make Patch Rollouts Secure and AuditableBest Practice Recommendation 3es ac ce eco e da o 3

Can I avoid identifying errors that are based on my environments during the d l t ?


deployment process?

Can I make the patching operation more secure and auditable?– Can I lock access to the Oracle account?Can I lock access to the Oracle account?

• User Impersonation: Lock Oracle accounts and setup authentication through SUDO or Power Broker. Update EM Deployment Procedures to executeUpdate EM Deployment Procedures to execute through SUDO or Power broker


“Light Outs” Tracking Through Notifications Best Practice Recommendation 4es ac ce eco e da o

Can I avoid identifying errors that are based on myi t d i th d l t ?


environments during the deployment process?


Can the procedure notify me during execution so I can let it run completely unattended (“lights out”)?can let it run completely unattended (“lights out”)?– Can I track procedure executions and step level

executions without being on the system?

Can I use patching features in my security-restricted environment?


Set Up EM Notifications to Patch in “Lights Out” Mode Best Practice Recommendation 4es ac ce eco e da o

Here’s How


Pick a procedure >> Do a ‘Create Like’ >> Enable Notification along with required status

“Light Outs” Tracking Through Notifications Best Practice Recommendation 4es ac ce eco e da o

Can I avoid identifying errors that are based on my environments during th d l t ?

C I k th t hi ti d dit bl ?


the deployment process?


Can the procedure notify me during execution so I can let it run completely unattended (“lights out”)?it run completely unattended ( lights out )?– Can I track procedure executions and step level executions

without being on the system?Remote Monitoring: Enable Notifications for different• Remote Monitoring: Enable Notifications for different status of procedures to receive email/pager alerts for procedure execution. This provides user with handle to monitor the execution even when it runs unattended


monitor the execution even when it runs unattended

‘Offline’ Datacenters – No Connectivity to MOSBest Practice Recommendation 5es ac ce eco e da o 5






Can I use patching features in my security-restricted environment?

How do I get to use patching my OMS doesn’t have– How do I get to use patching – my OMS doesn’t have connectivity to My Oracle Support?


‘Off-Line’ Data Centers – Feature LimitationsBest Practice Recommendation 5es ac ce eco e da o 5

1. Limited availability of Patch Management Features associated with My Oracle Support 2. Patch Search, Patch Activity and Patch Plans are not available in this mode3. Patch Recommendations and Patch Rollout through Deployment Procedures

can be exercised still


can be exercised still

‘Off-Line’ Data Centers – Patch RecommendationsBest Practice Recommendation 5es ac ce eco e da o 5

Here’s How

1. Setup >> Patching Setup >> Online and Offline Settings >> Set ‘Connection’ mode to Offline

2. Download Patch recommendations and related XMLs from MOS separately3 Upload Metadata XMLs and compute patch recommendations by running ‘Refresh


3. Upload Metadata XMLs and compute patch recommendations by running Refresh from My Oracle Support’ job

‘Off-Line’ Data Centers – Patch DeploymentBest Practice Recommendation 5es ac ce eco e da o 5

Here’s How

1. Download patches separately from MOS to local system2. Deployments >> View/Upload Patches >> “Upload” – Mass Upload Patches to

Software Library

How


Software Library3. Select from Software Library during Deployment

‘Off-Line’ Datacenters – No Connectivity to MOSBest Practice Recommendation 5es ac ce eco e da o 5





Can I use patching features in my security restricted


Can I use patching features in my security-restricted environment?– How do I get to use patching if my OMS doesn’t have

ti it t M O l S t?connectivity to My Oracle Support?• Offline Mode Support: Download recommendation

XMLs and patches separately. Select from Software Lib d i d l t


Library during deployment.

Best Practice RecommendationsMake Automation Efficienta e u o a o c e

Make patch rollouts reliable with ‘Patchability’ report yand pre-flight checks

Save downtime and bandwidth by pre-stagingy p g g

Set up impersonated accounts for secure rollouts

Set up notifications for Lights Out execution

For “offline” datacenters, fetch recommendations and o o e datace te s, etc eco e dat o s a dmass upload patches to Software Library for rollouts


Andreas StephanSenior DBA Consultant

S

“We manage thousands of databases and application servers

Bayer Business Services

Patch Management C St d B with Enterprise Manager, and we

have been able to reduce the time for database upgrades from 4 hours

down to 1 hour, as well as reduce

Case Study: Bayer

2,000 databases, 5 full time DBAs, 1 h t h down to 1 hour, as well as reduce

patch application time from 1 hour down to 1 minute per database.

Enterprise Manager Grid Control ll t t t thi

1 hour per patch, 4 times a year

Before After allows us to automate this process, which translates into huge savings

in time and money.”

Before Enterprise Manager

After Enterprise Manager

8,000 person h

133 person hhours

$160,000hours$2,666


Reduce Operations Costs with Automation

• Reduction in costs of

Forrester Total Economic Impact of Configuration Management

and Provisioning Packscosts of managing IT

• Increase staff productivity

• Increased agility to businessto business needs

• Reduction on capital spending 130% ROI

over 3 years

14 months payback period


Session: #S316967 Patching Enterprise-Wide Databases: g

Automation Techniques & Real World Insights

Johns Hopkins UniversityApplied Physics Laboratory

- Deepa NambiarDeepa Nambiar

Agenda

01 Infrastructure

02 Ch ll02 Challenges

03 Solution: OEM Patch Automation03 Solution: OEM Patch Automation

04 OEM 11g Patch Automation Featuresg

05 ROI and Results

51

The Johns Hopkins University- Applied Physics Laboratory Celebrating 65 + Years of Service toLaboratory Celebrating 65 Years of Service to the Nation

Not-for-profit research and development laboratory founded in 1942.Division of Johns Hopkins UniversityStaffing: 4,000 employees (69% scientist and engineers)Annual revenue ~ 780MWho do we work for- We do projects for Department of Defense, DARPA, Department of Treasury, NASA, NSA etc.

52

InfrastructureERPs and multiple custom developed applications• ERP (e.g. PeopleSoft, eBusiness Suite)• Other custom applications – Time Keeping, Resource

Management, Data Warehouse etc. 150+ databases across multiple data centers150+ databases across multiple data centersOracle Enterprise Manager(OEM) Grid Control (GC) as central monitoring and administration tool. Production version 10.2.0.5. Currently evaluating new features of OEM11g on a test system. Plan to upgrade soon.Diagnostics, Tuning, Configuration Management and Provisioning Packs among otherso s o g ac s a o g ot e sOracle database versions – 11.2.0.1 RAC and single instance databases,11.1.0.7, 10.2.0.4 & 9.2.0.8 single instance databasesO S Solaris 10 and Oracle Enterprise Linux Rel5O.S. - Solaris 10 and Oracle Enterprise Linux Rel5

53

Agenda

01

Ch ll02

Infrastructure

Challenges02

03 Solution: OEM Patch Automation03 Solution: OEM Patch Automation

04 OEM 11g Patch Automation Featuresg

05 ROI and Results

54

JHU APL Ch llJHU-APL Challenges

APL security guidelines demand implementation of securityAPL security guidelines demand implementation of security fixes within short time frame Oracle Corporation releases PSU/CPUs quarterlyManual patching time consuming and error proneg g150+ databases, 4 times a year, approx 0.5hours/db , assuming 51% multi tasking.

~ 300 hours patching per yearEffectively utilize limited DBA resources (150+ Databases /Effectively utilize limited DBA resources (150+ Databases / 3 DBA‘s)Being on course with PSU/CPU patch cycles.Making sure all required targets are patched at the same g q g plevel.Spent a lot of time on repetitive tasks like patching and did not leave much time for DBA‘s to work on strategic activities.

55

Agenda

01

02 Ch ll

Infrastructure

02 Challenges

03 Solutions- OEM Patch Automation03

04 OEM 11g Patch Automation Features

Solutions- OEM Patch Automation

g

05 ROI and Results

56

Solutions for Patching

Grid Control Patching deployment procedures allowComprehensive Automation- Full scope of patching procedure including blackouts, OPatch update and SQL apply and utlrp on databasesapply and utlrp on databases.Detects databases already shutdown and in blackout.One time Patch Download from My Oracle Support to software library directly made it available across dev testsoftware library directly, made it available across dev, test uat and production environments.Schedule Mass deployment: No more logging into every server to apply the patch.pp y pError Logging, Ability to Customize and Status Notifications through emailsQuick deployment.

57

Pre Rollout

Patch Cycle At JHU-APL

Pre RolloutCheck patchconflicts

Applies on Preprod System InAnalyze Mode

Checks CriticalPatch Advisory

Identifies System Vulnerabilities

Apply on Pre prod Systems

R ll tRollout

GeneratesReports

Selects the Patches from

(Pre-Maintenance)Verifies Production System in

(During Maintenance)Mass deployment in Reports

Software Library Production System in Analyze Mode Production System

58

Patch Management Life Cycle- PreRollout

Before Patch application, Verify Target Patchability‘Target Patchability Report’, lists databases that cannot be patched

using OEMGC.11g Rel2 database cannot be patched with OEMGC 10.2.0.5G d ti t lid t th t ll t t d i ti ithGood practice to validate that all targets and communicating with

OMS and can be patched using the tool.

59

Patch Management Life Cycle- PreRollout

1

g y

Identify the Critical Patch Advisory /system vulnerabilities i kl d il i OEMGCquickly and easily using OEMGC.

Patch Advisories under Compliance, provides complete list of database homes and security patches that need to be applied.y p pp

10.2.0.5 supports CPU and PSU patch application, including overlay patches. EM 10.2.0.5 provides CPU advisories.

11g provides PSU advisories.

60

Patch Management Life Cycle –Pre Rolloutg y

2

Identify vulnerable systems easily 3

61

Patch Management Life Cycle – Pre g yRolloutAnalyze mode ROCKS! - If patch conflicts are detected contact oracle

support for a replacement patch.

62

Patch Management Life Cycle – Pre Rollout

Conflict Detection

63

Patch Management Life Cycle – Pre RolloutPatch Management Life Cycle Pre Rollout

Real world example

Deployment procedure run history available, if need to refer to old conflicts arisesconflicts arises.It is recommend that deployment procedure runs be meaningfully named

64

Patch Management Life Cycle- PreRolloutPatch Management Life Cycle PreRollout

Analyze mode conducts a dry run of the patching process without actually applying the patch.

Upfront patch conflict and pre-requisite analysisUpfront patch conflict and pre requisite analysisIf a conflict is detected, may need to file a merge

patch request.Good practice for DBA’s to r n the deplo mentGood practice for DBA’s to run the deployment

procedure in analyze mode against all the targets, as they patch Development test and production databases.

65

Patch Management Life Cycle –Patch RolloutPatch Management Life Cycle Patch Rollout

Rollout :Select ‘ Patch Oracle Database’ Deployment Procedure1

2

Patch now available in the soft arethe software library

66

Patch Management Life Cycle -Patch Rollout

Rollout :Select Targets and proceed to Deploy3

Patch Management Life Cycle Patch Rollout

67

Patch Management Life Cycle- Patch RolloutPatch Management Life Cycle Patch Rollout

Add database targets applicable to the patch across multiple servers. Patch without having to physically log into all the servers. Can use a

single interface.

68

Patch Management Life Cycle –Patch RolloutPatch Management Life Cycle Patch Rollout

Patch application can be prescheduled, and just check status.Ability to Patch multiple installations with multiple patches in a

single window We take advantage of thissingle window. We take advantage of this.Not a ‘Black Box’

- Lists steps that the deployment procedure goes through.-Logs available for each step of the patching process.g p p g p

‘Applied Interim Patches Report’ to verify overall patch effort.

69

Agenda

01

02 Ch ll

Infrastructure

02 Challenges

03 Solutions –OEM Patch Automation03

04

Solutions OEM Patch Automation

OEM 11g- Patch Automation Features

05 ROI and Results

70

Patch Management Life Cycle- In OEMGC 11gPatch Management Life Cycle In OEMGC 11g

1

View Overall Patch Recommendations

OEM11g has integrated MOS into the EM product. MOS is now a tab within EM.

New process of identifying Security vulnerabilities.Security recommendations under Patches & Updates tab lists securitySecurity recommendations, under Patches & Updates tab lists security

patches that are applicable to a particular database.

71

Patch Management Life Cycle- In OEMGC 11g

2 Select specific Patch Recommendations

Patch recommendations list the patch #, the DB version host name. The new interface Provides ready access to the readme and access toThe new interface Provides ready access to the readme and access to

community reviews. Earlier had to separately log into forums.

72

Patch Management Life Cycle -OEMGC 11gPatch Management Life Cycle -OEMGC 11g3. Add the patch to Patch Plan

Concept of Patch plan : Logical grouping of patches and targets.You can directly add the patches to a new or existing patch plan.Current design enables you to identify vulnerabilities and check for conflicts and file replacement patch requests all on the same page!file replacement patch requests all on the same page!

73

Patch Management Life Cycle -OEMGC 11gg y g

4. Validate for Conflicts

5. Request Replacement patches

Upfront validation for conflict amongst the patches selected and withUpfront validation for conflict amongst the patches selected and with that of the patches in the oracle home. No manual merge patch service requests required.Screenshot shows output from the replacement patch request.O l t t h bt i d it i d f l t d t tOnce replacement patches are obtained it is ready for replacement and status

changes to Conflict free.

74

Patch Management Life Cycle -OEMGC 11gg y g

Replacement Patch Tracking

Once a request is placed, the status can be tracked from the main Patches & Updates page. 75

Patch Management Life Cycle –OEMGC 11gatc a age e t e Cyc e O GC g

6. Schedule Deployment

‘Run Procedure’ is where the new functionality integrates with the old functionality ‘Deployment Procedures’

Clicking ‘Run Procedure on the Patch Plan will present you with

76

Clicking Run Procedure on the Patch Plan will present you with the set of Deployment Procedure screens you have already seen in EM 10.2.0.5.

Patch Management Life Cycle –OEMGC 11g

7. Apply Patches through Deployment Procedures

The Patches from the plan are auto selectedNote: At the current patch level, all the selected targets are not auto populated and you get the above messageTip: The message lists the targets that were part of the patch plan

77

Tip: The message lists the targets that were part of the patch planProceed with the procedure as in 10.2.0.5 EM

Agenda

01

02 Ch ll

Infrastructure

02 Challenges

03 Solutions –OEM Patch Automation03

04

Solutions OEM Patch Automation

OEM 11g- Patch Automation Features

05

g

ROI and Results

78

ROI and ResultsWe have changed from custom script based patchingWe have changed from custom script based patching to using OEMGC for patching.

N i t l l i di l b fNo incremental value in spending a large number of hours on repetitive tasks like patching.

The tool takes the load off and lets DBA’s spend more time on strategic tasks.

OEM, adapted to PSU patches almost seamlessly as opposed to home grown scripts.

79

ROI and ResultsWithout EMGC With EMGCWithout EMGC With EMGC

Patching time fromstart to finish

AdministratorTime involved

Installation time from start to finish

Administrator Time involved

~ 300 hrs (150* .5 hours *4 /deployment)

~ 300 hours 15 min for each mass deployment. Each run may patch one or more of these 150 databases in

Few clicks

parallel

Average time for 150 deployments: 37 hours

We derived great value out of the product and hence we have continued to use it and move forward with it in other areas.

80

ResourcesTip: Fixes are released almost every Quarter p yFollow Metalink Note: 427577.1 : OMS and Agent Patches required for setting up Provisioning, Patching and Cloning in 10.2 to 11.1 GC Enterprise Manager Administrator's Guide for Software and Server Provisioning and Patching –http://download.oracle.com/docs/cd/B16240_01/doc/em.102/e10954/title htm0954/title.htmhttp://www.oracle.com/technetwork/articles/grid-automation-deployment-procedur-130372.pdfOEM 11g-OEM 11ghttp://www.oracle.com/technetwork/oem/automation/twp-em11g-patch-mgmt-167855.pdfOEM11g admin guide-htt //d l d l /d / d/E11857 01/i t ll 111/ 16http://download.oracle.com/docs/cd/E11857_01/install.111/e16847/toc.htm

81

ThanksThanks

Deepa [email protected]

Johns Hopkins University Applied Physics Laboratory11100 Johns Hopkins Rd11100 Johns Hopkins RdLaurel, Maryland 20906

82

Oracle Helps You Maximize Customer Value

Avoids online revenue losses up to 25%

Deploys SOA infrastructure 92% faster

Saves 80% time and effort for managing Databases

Cuts configuration management effort by 90%

Improves IT productivity by 25%

Drives asset utilization up by 70%

Replaces manual tools with automation; saves time by 50%

Saves $1.9 millionwith Oracle Enterprise Manager

Saves $170,000 per year with Oracle Enterprise Manager

Saves weeks on application testing time

Reduces Database testing time by 90%

Reduces provisioning effort by 75%

Delivers 24/7 uptime with Oracle Enterprise Manager

Cuts application testing from weeks tohours

Reduces critical patching time by 80%


Additional Information

• Database Patching with Enterprise Manager 11gg g g– http://www.oracle.com/us/corporate/press/018427

OTN• OTN.com– http://www.oracle.com/technology/products/oem/mgmt_soluti

ons/provisioning.html


© 2010 oracle corporation 1 · • custom scripts developed around opatch sqlplus etc opatch •...

Documents