oow16 - online patching with oracle e-business suite 12.2 [con6710]
TRANSCRIPT
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
Online Patching with Oracle E-Business Suite 12.2
Kevin Hudson Senior Director Applications Technology Oracle E-Business Suite Development September 22, 2016
Confidential – Oracle Internal
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
Related Sessions
Thursday, September 22, 2016
1:15 PM Moscone West – 2004 CON6708-Migrating and Managing Customizations for Oracle E-Business Suite 12.2 Santiago Bastidas, Sr. Principal Product Manager, Applications Technology, Oracle E-Business Suite
Confidential – Oracle Internal 3
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
Program Agenda: Patching with E-Business Suite 12.2
Solution Approach
Overview of 12.2 Patching Architecture
Patching EBS 12.2 with Online Patching
Online Patching Cycle – Special Operations
Patching Applications Tier with Online Patching
AD & TXK Delta 8 New Features
1
2
3
4
5
4 Confidential – Oracle Internal
6
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
What if the Following were True During Patching • Critical business operations were not interrupted by maintenance operations
• Revenue generating activities were to stay online
– Web Orders
– Booking Orders
– Service Requests
• Business users were to stay online
– Pay Suppliers
– Pay Employees
– Uninterrupted manufacturing operations
• Downtime required by any patch was predictable
– Measured in minutes not days or hours
• Global Single Instances remained operational 24*7
– Maintenance window easier to allocate as shorter time required
5 Confidential – Oracle Internal
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
Online Patching
• In 12.2 all patching operations are online
– EBS will remain available to users during patching operations
• Different lines of Business are no longer affected by patching
– Run Financials while applying an HR legislative updates
– Run Manufacturing while applying Financials updates
E-Business Suite Remains Online During Patching
6 Confidential – Oracle Internal
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
Oracle E-Business Suite 12.2 Architecture
7
JDB
C/SQ
L Ne
t
Application Database
RAC & ASM
Global Single Data Model
WebLogic JSP
Forms
BI Publisher
BC4J
Web
Lis
ten
er
UIX 11g
WebLogic Server
• Online patching uses the latest features of the technology stack
– Edition-Based Redefinition (EBR)
– Web Logic Server
Edition-Based Redefinition
Confidential – Oracle Internal
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
Patches are Applied to a Copy of Production
• Copy the production code
• Apply patches to the copy
Patches - 12.2.4 - 12.2.6 - …
E-Business Suite
12.2.3
Production Copy of Production
E-Business Suite 12.2.3
8 Confidential – Oracle Internal
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
Copy the Code, NOT the Data
• EBS uses both the file system and the database to store the code and data that make up the application
– Code • Stored both on the File system & In the Database
• Any code object changed in a patch is copied
– Data • Stored both on the File system & In the Database
• Application data is NOT copied by a patch
9 Confidential – Oracle Internal
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
Downtime Redefined as “Cutover”
• Cutover is the time taken to switch users from the production system to the newly patched copy
• Cutover changes the unit of measure for downtime
– Measured in minutes NOT hours or days
• Cutover time is very predictable
– The time taken to bounce the Middle Tiers
Changing the Unit of Measure for Downtime
10 Confidential – Oracle Internal
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
Downtime Limited to Short Cutover
• Users are logged off the production Instance
• When users reconnect they are directed to the newly patched system
E-Business Suite
12.2.3
Patches •12.2.4 •12.2.6 •….
E-Business Suite
12.2.3
Production Patched Production
11 Confidential – Oracle Internal
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
• Downtime Patches
– No online users
– Wall clock time very important
– Consumes all resources
– Upgrade designed to run as fast as possible
• Online Patches – User remain online
– Wall clock time is no longer an overriding concern
– Online users share resources
– Data upgrades designed to not affect the running application
Changing the Patching Paradigm
12 Confidential – Oracle Internal
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
Patching Occurs on a Copy
• File system Patching –All patches are applied to a secondary file system
–Synchronization of the file systems is managed by the patching tools
• Database Patching –A separate copy is maintained of all database code objects that are
changed by a patch
–Utilizes Edition-Based Redefinition feature of the Database
13 Confidential – Oracle Internal
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
Online Patching - High level Architecture
14 Confidential – Oracle Internal
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
Oracle E-Business Suite 12.2 Architecture Dual File System
15
Synchronization managed by patching tools
Edition-Based Redefinition
Non-Editioned File System
Developer 10.1.2
COMMON_TOP
APPL_TOP
INST_TOP
Oracle HTTP Server (OHS)
WebLogic Server (WLS)
File System 1
Developer 10.1.2
COMMON_TOP
APPL_TOP
INST_TOP
Oracle HTTP Server (OHS)
WebLogic Server (WLS)
File System 2
PATCH_TOP
APPL_TOP_NE
LOGS More information in the Online Patching FAQ (Note 1583902.1)
Confidential – Oracle Internal
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
File System Editioning
16 Confidential – Oracle Internal
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
• Run file system
– Used by online users
– Stores a complete copy of all Applications and Middle Tier code
– Logically mapped to either fs1 or fs2
• Patch file system
– Used by patching tools
– Stores a complete copy of all Applications and Middle Tier code
– Logically mapped to either fs1 or fs2
17
• Non-Editioned file system
– Used for data files e.g.: data import/export files, log files, report output files
– Only stores data files
Online Patching uses a Dual File System
fs1 and fs2 switch Run and Patch designation during the cutover phase of an Online Patching cycle
fs1
Run
fs1
Cutover fs1 fs2
Patch Patch
fs2
Run
Confidential – Oracle Internal
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
12.2 E-Business Suite File System Layout File System Laid Down after 12.2 Rapid Install
Install base
fs_ne fs2 EBSapps.env fs1
New file to choose the environment to set: EBSapps.env RUN|PATCH
fs1
inst FMW_HOME EBSapps
10.1.2 appl comn
ar … gl ad
fs2
inst FMW_HOME EBSapps
10.1.2 appl comn
ar … gl ad
File System 1 File System 2
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
12.2 E-Business Suite File System Layout
• Stores external data files • Human readable
• Have a structure that is mostly independent of the application code level
• Not directly patched by EBS Development and they should not be editioned
• Examples • Concurrent report output
• Log files
• Data import/export files
• SSL certificates
• File-based system integration
• Patch downloads
Non-Editioned File System
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
Install base
fs_ne fs2 EBSapps.env fs1
New file to choose the environment to set: EBSapps.env RUN|PATCH
12.2 E-Business Suite File System Layout Non-Editioned File System
fs_ne
inst EBSapps
adop
appl patch log
fs2 fs1
Non-Editioned File System • Subdirectories sparsely populated • Only created as needed
20 Confidential – Oracle Internal
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
R12.2 Environment Overview
• New script to setup the environment: EBSapps.env RUN | PATCH
• New environment variables for R12.2 to help you navigate the system $FILE_EDITION - Current fs edition type sourced, values: run or patch
$RUN_BASE - RUN file system base directory
$PATCH_BASE - PATCH file system base directory
$NE_BASE - Non-edition file system base directory
$APPL_TOP_NE - Application top that stores product files considered data
$PATCH_TOP – Directory where the patch files are unzipped
– On a multi-node system with non-shared file systems, you must copy the patch files to each separate $PATCH_TOP directory, so that the patch files are available from the same location on all nodes
Working with the 12.2 Environment
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
R12.2 Environment Overview
$ source EBSapps.env RUN
E-Business Suite Environment Information
----------------------------------------
RUN File System : /scratch/orauser/im4kevin/fs2/EBSapps/appl
PATCH File System : /scratch/orauser/im4kevin/fs1/EBSapps/appl
Non-Editioned File System : /scratch/orauser/im4kevin/fs_ne
DB Host: rws3270749.us.oracle.com Service/SID: im4kevin
Sourcing the RUN File System ...
Setting the 12.2 Environment
$ echo $FILE_EDITION
$ echo $RUN_BASE
$ echo $PATCH_BASE
$ echo $NE_BASE
$ echo $APPL_TOP_NE
$ echo $PATCH_TOP
: run
: /scratch/orauser/im4kevin/fs2
: /scratch/orauser/im4kevin/fs1
: /scratch/orauser/im4kevin/fs_ne
: /scratch/orauser/im4kevin/fs_ne/EBSapps/appl
: /scratch/orauser/im4kevin/fs_ne/EBSapps/patch
• Echo environment variables
• Source the Run environment
Note: fs2 is the RUN file system
Note: RUN file system is sourced
Note: fs2 is the RUN file system
22 Confidential – Oracle Internal
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
Install base
fs_ne
$NE_BASE
inst EBSapps
adop
appl
$APPL_TOP_NE
patch
$PATCH_TOP
log
fs2
$RUN_BASE
fs1
$PATCH_BASE
Working with the 12.2 Environment
12.2 E-Business Suite File System Layout
• Source the Run environment $ source EBSapps.env RUN
E-Business Suite Environment Information
----------------------------------------
RUN File System : /scratch/orauser/im4kevin/fs2/EBSapps/appl
PATCH File System : /scratch/orauser/im4kevin/fs1/EBSapps/appl
Non-Editioned File System : /scratch/orauser/im4kevin/fs_ne
DB Host: rws3270749.us.oracle.com Service/SID: im4kevin
Sourcing the RUN File System ...
$ echo $FILE_EDITION
$ echo $RUN_BASE
$ echo $PATCH_BASE
$ echo $NE_BASE
$ echo $APPL_TOP_NE
$ echo $PATCH_TOP
: run
: /scratch/orauser/im4kevin/fs2
: /scratch/orauser/im4kevin/fs1
: /scratch/orauser/im4kevin/fs_ne
: /scratch/orauser/im4kevin/fs_ne/EBSapps/appl
: /scratch/orauser/im4kevin/fs_ne/EBSapps/patch
• Echo environment variables
23 Confidential – Oracle Internal
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
File Systems Patching Cycle
• Two complete file systems are always present
– Run Edition - Used by the running Application
– Patch Edition - Either: • Currently being patched
or
• Waiting for the next patching cycle
– The two files systems are rotated between Run and Patch during every patch cycle
Cutover
Prepare
Prepare
Cutover
1
2
1
2
24 Confidential – Oracle Internal
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
Database Edition-Based Redefinition
25 Confidential – Oracle Internal
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
11gR2 Edition-Based Redefinition (EBR)
• Key enabling technology for Online Patching
– Enables the online upgrade of the application definition
– Allows an application to efficiently store multiple copies of its code and schema in the same database
• Provides an isolation mechanism that allows pre-upgrade and post-upgrade schemas to co-exist – Changes to database objects are made in the isolation of an “Edition”
– Changes to database objects do not affect the running application
26 Confidential – Oracle Internal
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
Online Patching is Enabled by 11gR2 Edition-Based Redefinition Multiple Copies of Database Code Objects Can Coexist
• Client code connects to an “edition” of the database
– Run Edition • Used by online users
• Never changed by a patch
– Patch Edition • Used by patching tools
• Changes do not affect the running application
• Patch Edition becomes the Run Edition at Cutover
27
DBA Business Users
Confidential – Oracle Internal
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
12.2 Online Patching Cycle
28 Confidential – Oracle Internal
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
Online Patching Cycle Orchestration
• Formalized Patching Cycle
– New utility to orchestrate the Online Patching cycle • adop replaces adpatch as the 12.2 patching tool
• Supports single and multi-node environments
– Dual file systems are synchronized automatically
– Database Editions are managed automatically
• Users remain online during patching
– Patching CANNOT affect the running application • Data model changes (tables, indexes…)
• Code changes ( pl/sql, views, c code, java code…)
– Running application CANNOT affect the patch
29 Confidential – Oracle Internal
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
Understanding the Online Patching Cycle
Confidential – Oracle Internal 30
• Online Patching is used to apply all patches in 12.2 • Online Patching cycle includes 5 major phases • New patching tool “adop” orchestrates the patching cycle
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
Online Patching Cycle - Overview
Understanding the Online Patching Cycle
• The Basics
• Remove obsolete objects
Cleanup
• Restart application on
Patch Edition
Cutover
• Compile invalid Objects
• Wait for a good downtime window
Finalize
• Apply one or more patches to the Patch Edition
Apply
• Copy the production application code
• Create a new Patch Edition in the database
Prepare
Users Online Users Online Users Offline
• Online Patching is used to apply all patches in 12.2 • Online Patching cycle includes 5 major phases • Application is only offline during the Cutover phase
31 Confidential – Oracle Internal
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
Online Patching Cycle – Prepare Phase
• The Basics Cleanup Cutover Finalize Apply
• Copy the production application code
• Create a new Patch Edition in the database
Prepare
Users Online Users Online Users Offline
Prepare
• Synchronizes Patch Edition and Run Edition on the file system • Create a new Patch Edition in the database
32 Confidential – Oracle Internal
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
Preparing the File System
Online Patching Cycle – Prepare Phase
33 Confidential – Oracle Internal
• Patch Edition & Run Edition of the file systems are synchronized
• All files on the Patch Edition are an exact copy of the files on the Run Edition
• Incremental synchronization is the default – File based
• All changed files are synchronized
– Patch based • Patches applied on last cycle are reapplied
• Full synchronization available if necessary
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
Preparing the Database
Online Patching Cycle – Prepare Phase
34 Confidential – Oracle Internal
• Patch Edition is created in the database
• All code objects in the Patch Edition begin as pointers to code objects in the Run Edition
• Efficient use of space – Objects are only “Actualized” into the Patch Edition when a
new revision of the code is applied
• Storage objects such as tables are not copied – New patching standards are followed for upgrading
storage objects
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
Online Patching Cycle – Prepare Phase
35
• ADOP controls Preparing the system for patching
• Validates system configuration to ensure that the system is ready to start an Online Patching cycle.
• Prepare is run on all nodes in a muti-node configuration
• Synchronizes the Run and Patch File System
• Creates the Database Patch Edition
What to Know
• Execute prepare on the primary node
$ adop phase=prepare
What to Do
Confidential – Oracle Internal
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
Online Patching Cycle - Apply
• Execute patch drivers to update Patch Edition
• Multiple patches can be applied, including customizations
Cleanup
Cutover
Finalize
• Apply one or more patches to the Patch Edition
Apply
• Copy the production application code
• Create a new Patch Edition in the database
Prepare
Users Online
Users Online
Users Offline
36 Confidential – Oracle Internal
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
Patch Driver
– File Driver
–Database Driver
Online Patching Cycle - Apply
• Patches are applied to the copy (Patch Edition)
• The production application is unaffected by the changes
– Users are connected to the Application and can perform their work unaffected by the patch
• Changes are made in the isolation of an Edition
– The running application is unaffected by these changes
Applying Patches
37 Confidential – Oracle Internal
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
Online Patching Cycle – Apply Phase
38
• ADOP controls application of patches to the file system and database
• Mutiple patches can be applied in the same patching cycle
• Supports all options previous supported by adpatch
• Supports fully automated execution via inputs files
What to Know
• Execute adop apply on the primary node $ adop phase=apply patches=12345
$ adop phase=apply patches=1234,5678,9876
$ adop phase=apply patches=1234,5678 workers=4
$ adop input_file=April2016.txt
April2016.txt
phase=prepare,apply,finalize
patches=1234,5678,9876
workers=4
…
What to Do
Confidential – Oracle Internal
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
Online Patching Cycle - Finalize
• Compile invalid objects
• Generate derived objects
• Wait for a good downtime window
Cleanup Cutover
• Compile invalid Objects
• Wait for a good downtime window
Finalize
• Apply one or more patches to the Patch Edition
Apply
• Copy the production application code
• Create a new Patch Edition in the database
Prepare
Users Online
Users Online
Users Offline
39 Confidential – Oracle Internal
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
Online Patching Cycle - Finalize
40
• Performs any remaining processing that is needed to ensure the system is ready for the fastest possible cutover
– Object compilation
• Environment can pause at this stage for as long as required
• Default finalize_mode is quick
– Full mode computes statistics on key data dictionary tables for improved performance
What to Know
• Execute adop finalize on the primary node $ adop phase=finalize
$ adop phase=finalize finalize_mode=quick
$ adop phase=finalize finalize_mode=full
What to Do
Confidential – Oracle Internal
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
Online Patching Cycle - Cutover
• Stop middle-tier services
• Cutover to the Patch Edition of the file system
• Cutover to the Patch Edition of the database
• Restart middle-tier services
Cleanup
• Restart application on Patch Edition
• Compile invalid Objects
• Wait for a good downtime window
Finalize
• Apply one or more patches to the Patch Edition
Apply
• Copy the production application code
• Create a new Patch Edition in the database
Prepare
Users Online
Users Online
Users Offline
Cutover
41 Confidential – Oracle Internal
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
Online Patching Cycle - Cutover
• During cutover a brief downtime is taken –Middle tier services are stopped and users are logged off the system
–The Patch file system is promoted to be the Run file system
–The Patch database edition is promoted to be the Run database edition
–Perform final maintenance operations
–Middle tier services are restarted
–Users are brought back online on the patched system
• The database does NOT need to be and should not be bounced during cutover
42 Confidential – Oracle Internal
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
Cutover Automatically Manages all Services on all Nodes
Online Patching Cycle - Cutover
• Middle-tier processes stopped – Manages all nodes in the configuration
via remote execution
– End Users are disconnected
• File system roles swapped – Patched (FS-2) promoted to Run
– FS-1 available for next patching cycle
• Database Patch Edition promoted to Run Edition
• Middle-tier processes restarted
– Manages all nodes in the configuration via remote execution
– Users reconnect
43 Confidential – Oracle Internal
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
Online Patching Cycle - Cutover
44
• Cutover represents the official downtime for an EBS patch
– Application tier services are automatically shut down and restarted on all nodes
• You must ensure that no users remain on the system during cutover
– Any third-party processes connected to the old run edition of the database should be shut down, otherwise they will be terminated automatically
• Cutover will take longer if it has to wait for long-running concurrent processes to complete
– You can terminate the internal concurrent manager by executing the adcmctl.sh abort command
What to Know
• Execute adop cutover on the primary node
$ adop phase=cutover
What to do
Confidential – Oracle Internal
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
Online Patching Cycle - Cutover
45
Deferring Application Tier Restart
• In some cases, you may need to perform additional manual steps after cutover but before restarting the application tier services
– Apply customizations
– Patch the database tier
• You must start the application tier services manually after performing additional steps
What to Know
• Execute adop cutover on the primary node without starting the application services
$ adop phase=cutover mtrestart=no
• Perform manual steps
• Start application tier services manually
$ source EBSapps.env run
$ adstrtal.sh –mode=allnodes
What to do
Confidential – Oracle Internal
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
Online Patching Cycle - Cleanup
• Remove obsolete objects
• Restart application on Patch Edition
Cutover
• Compile invalid Objects
• Wait for a good downtime window
Finalize
• Apply one or more patches to the Patch Edition
Apply
• Copy the production application code
• Create a new Patch Edition in the Database
Prepare
Users Online
Users Online
Users Offline
Cleanup
• Cleanup old editions to recover space • Cleanup occurs after the users have been brought back online
46 Confidential – Oracle Internal
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
Online Patching Cycle - Cleanup
• Cleanup occurs after users have been brought back online to the newly patched Application
• File system
– The pre-patch file system is now ready for the next patching cycle
– Re-synchronized
• Database
– Code objects patched in the latest patching cycle are deleted from Old Editions
– Once all objects are deleted from an Old Edition it is also deleted
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
Online Patching Cycle - Cleanup
48
• Users are online during the cleanup process
• Removes obsolete objects from the system to recover space
• If you fail to run the cleanup phase explicitly, it will be run automatically on the next prepare cycle
– This will cause a delay in starting your next online patching cycle
• Three cleanup_modes are available
– quick - Performs only essential cleanup tasks
– standard (default) - Removes obsolete code objects
– full - Performs maximum cleanup, which drops all obsolete code and data from earlier editions
What to Know
• Execute adop cutover on the primary node
$ adop phase=cleanup
$ adop phase=cleanup cleanup_mode=quick
$ adop phase=cleanup cleanup_mode=full
What to do
Confidential – Oracle Internal
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
Online Patching Cycle - Overview
Understanding the Online Patching Cycle
• The Basics
• Remove obsolete objects
Cleanup
• Restart application on
Patch Edition
Cutover
• Compile invalid Objects
• Wait for a good downtime window
Finalize
• Apply one or more patches to the Patch Edition
Apply
• Copy the production application code
• Create a new Patch Edition in the database
Prepare
Users Online Users Online Users Offline
• Online Patching is used to apply all patches in 12.2 • Online Patching cycle includes 5 major phases • Application is only offline during the Cutover phase
49 Confidential – Oracle Internal
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
Monitor The Online Patching Cycle
$ adop -status
Display Status of a Single Node Online Patching Cycle
Node Name Node Type Phase Status Started Finished Elapsed
------------- ---------- -------- --------- ------------------------------ ------------------------------ ------------
ebsnode01 Primary PREPARE COMPLETED 11-APR-15 04:10:19 -07:00 11-APR-15 04:13:55 -07:00 0:03:39
APPLY COMPLETED 11-APR-15 04:15:00 -07:00 11-APR-15 04:26:01 -07:00 0:11:01
FINALIZE COMPLETED 11-APR-15 04:27:02 -07:00 11-APR-15 04:28:40 -07:00 0:01:38
CUTOVER COMPLETED 11-APR-15 04:28:26 -07:00 11-APR-15 04:38:05 -07:00 0:09:39
CLEANUP COMPLETED 11-APR-15 04:45:02 -07:00 11-APR-15 04:46:11 -07:00 0:01:09
50 Confidential – Oracle Internal
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
Online Patching Cycle – Multi-Node
• Completely automated in multi-node deployments through adop
• No distinction between environments that are: – Shared file system configuration
– Non shared file system configuration
• Primary node takes control of the patching cycle – Performed by remote command invocations across nodes
– Remote command invocation setup required
51 Confidential – Oracle Internal
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
Online Patching Cycle – Multi-Node
Single Application Tier Node
adop
Primary Node
Application Tier
Database Tier
Multiple Application Tier Nodes
Primary Node
Secondary Node
Secondary Node
adop
Always Execute adop on the Primary Node
Application Tier
Database Tier
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
Online Patching Cycle – Multi-Node
53
• In a multi-node deployment, adop commands are only executed from the primary node.
• The primary adop node uses remote execution to automatically perform required actions on any secondary node
• Remote command invocations from the primary node to all secondary nodes must be enabled.
What to Know
• On the primary node
$ adop phase=prepare
$ adop phase=apply patches=12345
$ adop phase=finalize
$ adop phase=cutover
$ adop phase=cleanup
What to Do
Confidential – Oracle Internal
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
Display Status of a Multi-Node Online Patching Cycle
Muti-Node Configuration Automatically Managed
$ adop -status
Current Patching Session ID: 75
Node Name Node Type Phase Status Started Finished Elapsed
--------------- --------------- --------------- --------------- ------------------------------ ------------------------------ ---------
ebsapi01d Primary PREPARE COMPLETED 14-NOV-14 09:59:33 -05:00 14-NOV-14 10:14:11 -05:00 0:14:38
APPLY NOT STARTED
FINALIZE NOT STARTED
CUTOVER NOT STARTED
CLEANUP NOT STARTED
ebsape01d Secondary PREPARE COMPLETED 14-NOV-14 09:59:45 -05:00 14-NOV-14 10:13:18 -05:00 0:13:33
APPLY NOT STARTED
FINALIZE NOT STARTED
CUTOVER NOT STARTED
CLEANUP NOT STARTED
ebsape02d Secondary PREPARE COMPLETED 14-NOV-14 09:59:38 -05:00 14-NOV-14 10:16:32 -05:00 0:16:54
APPLY NOT STARTED
FINALIZE NOT STARTED
CUTOVER NOT STARTED
CLEANUP NOT STARTED
File System Synchronization Used in this Patching Cycle: Full
54 Confidential – Oracle Internal
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
Online Patching Cycle – Special Operations
55 Confidential – Oracle Internal
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
Online Patching Cycle - fs_clone
56
Re-Cloning the Patch Edition File System
• Patch file system is normally incrementally synchronized with the run edition file system during prepare
• In some special cases a full re-clone is required when
– The previous Online Patching cycle was aborted
– Manual changes were made to the run edition file system
– Middle-tier technology components were patched
• A full re-clone is recommended
– After applying a large EBS RUP
• Re-cloning the patch edition file system
– Ensures that the files systems are correctly synchronized
– Minimizes any synchronization delay that would normally occur on the next prepare
• Cannot be executed when there is an active Online Patching cycle
What to Know
• Execute re-clone on the primary node
$ adop phase=fs_clone
What to Do
Confidential – Oracle Internal
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
Online Patching Cycle - Abort
• The Online Patching Cycle can be aborted at any time prior to Cutover
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
Online Patching Cycle - Abort
58
• Useful when an Online Patching cycle encounters problems that cannot be fixed immediately
• Returns the system to normal runtime operation
• Online Patching cycle cannot be aborted after cutover
• Requires a full cleanup and fs_clone before the next patching cycle to remove the effects of the failed cycle
• If a full cleanup & fs_clone are not run explicitly, they will be run automatically by the next prepare
– This will cause a delay during the prepare phase of the next online patching cycle
What to Know
• Execute adop abort on the primary node
$ adop phase=abort
$ adop phase=cleanup cleanup_mode=full
$ adop phase=fs_clone
What to Do
Confidential – Oracle Internal
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
Online Patching Cycle – actualize_all
59
Full Cleanup as a Periodic Maintenance Operation
• As each Online Patching cycle is completed, the database will accumulate an additional old database edition.
• When the number of database editions reaches 25 or more, you should consider running a special maintenance operation to remove old database editions
• Actualize all is a special adop phase that ensures every database object has a version in the PATCH edition
– This allows all copies in OLD editions to be removed by full cleanup
What to Know
• Execute adop on the primary node to actualize all objects and perform a full cleanup
$ adop phase=prepare
$ adop phase=actualize_all
$ adop phase=finalize finalize_mode=full
$ adop phase=cutover
$ adop phase=cleanup cleanup_mode=full
What to do
Confidential – Oracle Internal
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
Online Patching Cycle - Actualize All
60
Full Cleanup Combined with a Scheduled Online Patching Cycle
• The actualize all and full cleanup can also be executed in conjunction with an Online Patching cycle
• This maintenance operations will take much longer than a typical Online Patching cycle and should only be performed when there is no immediate need to start or complete a patching cycle
What to Know
• Execute adop on the primary node to actualize all objects and perform a full cleanup
$ adop phase=prepare
$ adop phase=apply patches=12345,67890
$ adop phase=actualize_all
$ adop phase=finalize finalize_mode=full
$ adop phase=cutover
$ adop phase=cleanup cleanup_mode=full
What to do
Confidential – Oracle Internal
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
Single File System and Downtime Patching Mode
• Designed for building & testing custom code
– Patches should always be tested on a standard dual file system test instance before being applied to production.
• Single file system
– Allows for a reduced footprint
– Patch file system is not present
– All patches must be applied to the RUN edition using adop apply in downtime mode
• Downtime mode for adop
– Applies patches directly to the RUN edition of the instance
– The downtime mode should only be used on a production when directed by the patch readme or other documentation
61
Designed for Development Instances Only
Confidential – Oracle Internal
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
• Patches must be applied in downtime mode
– adop phase=apply
apply_mode=downtime
patches=12345,9876
• Patches are applied to:
– Run file system
– Run Edition of the database
Single File System for Development Systems
62
Synchronization is managed by patching tools
Developer 10.1.2
COMMON_TOP
APPL_TOP
INST_TOP
Oracle HTTP Server (OHS)
WebLogic Server (WLS)
Patch File System
Non-Editioned File System
PATCH_TOP
APPL_TOP_NE
LOGS
Edition-Based Redefinition
Developer 10.1.2
COMMON_TOP
APPL_TOP
INST_TOP
Oracle HTTP Server (OHS)
WebLogic Server (WLS)
Run File System
Confidential – Oracle Internal
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
Applying Downtime Mode Patches on a Single File System
63
• Only supported on development systems
– Patches should still be applied to a test instance using a full Online Patching cycle
• Patches are applied to the RUN edition of the file system and database
• Oracle does not test patches in Downtime mode unless explicitly stated
• All application tier services must be down before applying a patch in Downtime mode
What to Know
• Execute apply on the development primary node $ adop phase=apply patches=123456 apply_mode=downtime
What to do
Confidential – Oracle Internal
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
Patching Application Tier Technology Stack
64 Confidential – Oracle Internal
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
Applying Technology Stack Updates
65
Oracle E-Business Suite Release Application Tier
• Application tier technology stack updates can be
– Applied to the PATCH file system while EBS is online
– Applied in conjunction with an EBS Online Patching cycle
or
– Applied as a separate Online Patching exercise
• A full re-clone must be performed after applying Application tier patches to synchronize the RUN and PATCH file systems
What to Know
• Prepare the PATCH filesystem.
• Apply technology stack patches to PATCH filesystem.
• Apply EBS patches (optional).
• Coordinate time for CUTOVER and complete the online patching cycle.
• Synchronize the technology stack patches between the RUN and PATCH filesystems.
What to Do
Confidential – Oracle Internal
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. | Oracle E-Business Suite 12.2 Patching Technology Components Guide (Doc 1355068.1)
Applying Technology Stack Updates
66
Oracle E-Business Suite Release Application Tier
• Application tier technology stack updates can be
– Applied to the PATCH filesystem while EBS is online
– Applied in conjunction with an EBS Online Patching cycle
or
– Applied as a separate Online Patching exercise
• A full re-clone must be performed after applying Application tier patches to synchronize the RUN and PATCH file systems
What to Know • Prepare the instance for patching
$ adop phase=prepare
• Apply technology stack patches to the PATCH file system
$ source .EBSapps.env PATCH
$ bsu.sh
$ opatch apply
• Apply EBS patches (optional)
$ adop phase=apply inputfile=myinputfile
• Complete the Online Patching cycle
$ adop phase=finalize
$ adop phase=cutover
$ adop phase=cleanup
• Synchronize technology patches between the RUN and PATCH file systems
$ adop phase=fs_clone
What to Do
Confidential – Oracle Internal
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
EBS Technology Code level Checker (ETCC)
• Ensures you have the necessary database and middle tier bugfixes installed on your Oracle E-Business Suite Release 12.2 system
• Database Code Level Checker
– Verifies database patches required by EBS 12.2
✓
• Application Tier Code Level Checker
– Verifies technology stack patches required by EBS 12.2
Application Tier
Forms 10.1.2
OHS
Oracle Common
WebLogic
✓ ✓ ✓ ✓
Forms 10.1.2
OHS
Oracle Common
WebLogic
✓ ✓ ✓ ✓
fs1 fs2
Application TOPs Application TOPs
67 Confidential – Oracle Internal
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
EBS Technology Code level Checker (ETCC)
• ETCC can be downloaded via Patch 17537119 from My Oracle Support
• Oracle strongly recommends the use of this utility to ensure that all required database and middle tier bugfixes have been installed
• Database EBS Technology Codelevel Checker (DB-ETCC) – checkDBpatch.sh
• Middle Tier EBS Technology Codelevel Checker (MT-ETCC) – checkMTpatch.sh
68 Confidential – Oracle Internal
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
EBS Technology Codelevel Checker (ETCC) New: Automatically maps bug fixes to patches and recommends patches
69 Confidential – Oracle Internal
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
• EBS R12.2 Online Patching
• Simplified UI supports
–Patch deployment procedure
–Cutover deployment procedure
–File system clone
• Send notifications
• Centralized view of patch logs
Change Management - Patch Manager
Confidential – Oracle Internal 70
Application Management Suite for Oracle E-Business Suite
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
Apply Online Patches with Oracle E-Business Suite Plug-In
71
Oracle E-Business Suite Release Patch Manager and Online Patching
Confidential – Oracle Internal
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
Online Patching - Testing
• Patch Edition is not a testing environment
– Application tiers cannot be run against the PATCH edition
• Patches must be tested on a test instance with a complete Online Patching cycle
– Do not used downtime mode
– Do not use a single file system
• Consider new test scenarios
–Users can be online during patch application
72 Confidential – Oracle Internal
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
Online Patching – Resource Considerations
• Database
– SYSTEM tablespace requires double its current space allocation • 25GB increase to 50GB
– SEED tablespace requires double its current space allocation • 5GB increases to 10GB
• File System
– 2 file systems require double the space of 1 file system
• System resources consumed by a patch can be throttled
– ADOP Workers • Smart worker allocation designed to share resources with online users
73 Confidential – Oracle Internal
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
• AD/TXK are the infrastructure components that provide the 12.2 Online Patching technology
• Strongly recommend that you apply the latest patch collections at your earliest convenience
74
Oracle EBS 12.2 AD/TXK Delta 8 Now Available Stay Current with the Latest Updates
Description MOS Doc ID
Applying the Latest AD and TXK Release Update Packs to Oracle E-Business Suite Release 12.2
1617461.1
Oracle E-Business Suite Applications DBA and Technology Stack Release Notes for R12.AD.C.Delta.8 and R12.TXK.C.Delta.8
2159750.1
Confidential – Oracle Internal
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
AD & TXK Delta 8 New Features Overview
• Delta Synchronization
– Seed data
– Applications file system
• Log File Structure
• 16 Character Application Short Name
• Logical Host Name
• Support for 64 bit JDK on Concurrent Manager Tier
Oracle Confidential – Internal/Restricted/Highly Restricted 75
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
AD & TXK Delta 8 – Seed Data Synchronization
• Significant performance improvement in adop Apply phase – Only rows changed since the last synchronization are updated
– The time taken to synchronize seed data is proportional to the number of rows changed since the last synchronization
– Minor changes no longer incur the cost of full data replication
• Dual rowset seed data synchronization
– Two permanent sets of seed data
– Only rows changed by a patch are synchronized on prepare
– No cleanup required
Oracle Confidential – Internal/Restricted/Highly Restricted 76
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
AD & TXK Delta 8 - File System Synchronization
• Significant performance improvement in adop Prepare phase – Configurable callout to file system synchronization logic
– Example implementation provided using unix rsync command
– Customer can replace command with smart/advanced disk utility of their choice
• Automatic support for customizations
– All files in the Applications file system are synchronized
– Customizations no longer need to be registered in the custom synchronization driver
Confidential – Oracle Internal 77
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
AD & TXK Delta 8 - File System Synchronization
• New prepare phase parameter (sync_mode) to control synchronization behaviour – adop phase=prepare sync_mode=patch (default)
– adop phase=prepare sync_mode=delta (new file based)
• Delta style synchronization uses the file system synchronization command specified in: $AD_TOP/patch/115/etc/delta_sync_drv.txt
Oracle Confidential – Internal/Restricted/Highly Restricted 78
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
AD & TXK Delta.8 - Log Directory Structure
• $ADOP_LOG_HOME – <session_id>
‒<execution_id>
–<phase>
‒<node>
Oracle Confidential – Internal/Restricted/Highly Restricted 79
• $ADOP_LOG_HOME – 142
‒20160719_142714
‒prepare
‒rws3270749
• Improved structure of log directories
– Log directories organized in a logical hierarchical structure <session_id>/<execution_id>/<phase>/<node>/
– Consistent naming of top level log file adop.log
– Validation logs in a named directory
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
AD & TXK Delta 8 - 16 character Applications Short Name
• 12.1.3 and prior
– Customer could register custom applications via the forms User Interface
– Forms user interface allowed application short names > 8 characters
– adsplice only supported application short names <= 8 characters and was an Internal Oracle utility
• 12.2 Requires customers to run adsplice for all custom applications as part of the upgrade
– This is a new requirement for 12.2 to ensure custom applications are registered correctly
– Many customers have custom application short names > 8 characters
Oracle Confidential – Internal/Restricted/Highly Restricted 80
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
AD & TXK Delta 8 - Logical Host Name Support
• Decouple EBS installation from the physical host name –Support both Applications and Database tier
–Simplified reconfiguration for: • Cloning
• Disaster recovery
–Configuring Oracle E-Business Suite Release 12.x Using Logical Host Names (Doc ID 1968231.1)
Oracle Confidential – Internal/Restricted/Highly Restricted 81
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
Plan for Database Administrators and Developers Stay Current with Documentation Updates • Oracle® E-Business Suite Concepts Release 12.2
• Oracle® E-Business Suite Maintenance Guide Release 12.2
• Oracle® E-Business Suite Developer's Guide Release 12.2
• Oracle® E-Business Suite Release 12.2 Information Center (Doc 1583110.1)
• Developing and Deploying Customizations in Oracle E-Business Suite Release 12.2 (Doc 1577661.1)
• Guidance for Integrating Custom and Third-Party Products with Oracle E-Business Suite Release 12.2 (Doc 1916149.1)
• Oracle E-Business Suite Release 12.2: Online Patching FAQ (Doc 1583902.1)
82 Confidential – Oracle Internal
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
Online Patching Key Takeaways
• Online Patching is the new patching paradigm for E-Business Suite 12.2
• Online Patching is enabled by the Dual File System and 11gR2 Edition-Based Redefinition
• Online Patching cycle consists of 5 major phases
– Prepare, Apply, Finalize, Cutover, Cleanup
• Online Patching infrastructure can be used to patch the application tier
• Stay current with the latest AD/TXK patching infrastructure
– Currently AD/TXK Delta 8
• Stay current with documentation updates
83 Confidential – Oracle Internal
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. | 84
Comprehensive Training for implementers, technical professionals, consultants and super users
• Applications Technology
• EBS Strategy and Roadmap
• EBS and Oracle Cloud Solutions
• Asset Lifecycle and Service
• Financials
• Human Capital Management
• Logistics
• Manufacturing
• Order Management
• Procurement
• Projects
• Value Chain Planning
Oracle E-Business Suite Learning Subscription Learning Simplified
• High quality training videos and demonstrations delivered by Oracle experts
• Hands-on labs so you can practice what you learn
• 12 months of 24/7 access
• Access to instructors to get answers to your questions
• Dynamic updates for new features and product enhancements direct from Product Development
education.oracle.com/subscriptions/ebs
Confidential – Oracle Internal