rac upgrade from 10.2.0.1 to 10.2.0.4

8/3/2019 RAC Upgrade from 10.2.0.1 to 10.2.0.4

1/24

Real Application Cluster database upgradtion from 10.2.0.1 to 10.2.0.4

Database Version: 10.2.0.1Operating System: Ret Hat linux 4 As & Vmware 5

Storage Type : ASM & RAW Storage

Pre-Request for upgrade:

Before upgrading any database we need take complete cold backup for database. Also we needto take backup of ORACLE_HOME, ORACLE_BASE & ORACLE_CRS_HOME

In my case I taken ORACLE_HOME & ORACLE_CRS_HOME backups using (for allnodes)

cd /u01/before_patch_backup

tar -cvf oracle_base_jul19.tar $ORACLE_BASE

tar -cvf oracle_home_jul19.tar $ORACLE_HOME

tar -cvf oracle_crs_home_jul19.tar $ORACLE_CRS_HOME

Form storage level we need take backups. I used dd command for taking OCR &VOTING disk backups.

Also using I have taken complete RMAN backup for my database & using dd commandI taken RAW devices backups (I mean ASM)

PS: Make sure; before taking any backups you need to sotp your cluster dont forgot. Usingthe below command you down your cluster.

[root@linux1 bin]# ./crsctl stop crs

Stopping resources.Successfully stopped CRS resourcesStopping CSSD.Shutting down CSS daemon.Shutdown request successfully issued.

[root@linux1 bin]# ps -ef | grep pmonroot 27831 27206 0 08:26 pts/2 00:00:00 grep pmon[root@linux1 bin]# ps -ef | grep lsnroot 27833 27206 0 08:26 pts/2 00:00:00 grep lsn

[root@linux2 bin]# ./crsctl stop crsStopping resources.

Successfully stopped CRS resourcesStopping CSSD.Shutting down CSS daemon.Shutdown request successfully issued.

[oracle@linux2 /]$ ps -ef | grep pmonoracle 20613 19349 0 05:54 pts/1 00:00:00 grep pmon[oracle@linux2 /]$[oracle@linux2 /]$ ps -ef | grep lsnoracle 20638 19349 0 05:54 pts/1 00:00:00 grep lsn

1


2/24

PS: In my test environment I dont have separte ASM home; Im using ORACLE_HOMEitslef. If you have ASM home then you may need take backup of ASM home also.

Cluster Upgration:

Now Im going to upgrade my cluster database. Please carefully gothrow below screenshots & SQL Scripts.

Step1:

1. Down 10.2.0.4 patch set (Patch Nr: 6810189)

2. Extract your patch3. From oracle user you need to exeucte runInstaller then you will get below

screen shot.

Step2:

Please confirm your CRS Home. Because first of all you need to upgrade cluster only

(Not Oracle database or ASM Home)

2


3/24

Step3:

This is TWO Nodes cluster; thats why you will get your cluster nodes names list.

3


4/24

Step4:

This is pre-request checking steps; in case any thing wrong here you will come to know

all the details.

Step5:

4


5/24

Once successfully completed your pre-request then continue the patch installation on

cluster.

Step6

After succesfully installation you may need execute the below script from root user (in allnodes you need follow same steps)

5


6/24

/u01/app/oracle/product/10.2.0/crs/bin/crsctl stop crs

/u01/app/oracle/product/10.2.0/crs/install/root102.sh

Form Node1

[root@linux1 bin]# ./crsctl stop crsStopping resources.Error while stopping resources. Possible cause: CRSD is down.Stopping CSSD.Unable to communicate with the CSS daemon.

6


7/24

[root@linux1 bin]# cd ..[root@linux1 crs]# cd install

[root@linux1 install]# ./root102.shCreating pre-patch directory for saving pre-patch clusterware files

Completed patching clusterware files to/u01/app/oracle/product/10.2.0/crsRelinking some shared libraries.Relinking of patched files is complete.WARNING: directory '/u01/app/oracle/product/10.2.0' is not owned by rootWARNING: directory '/u01/app/oracle/product' is not owned by rootWARNING: directory '/u01/app/oracle' is not owned by rootWARNING: directory '/u01/app' is not owned by rootWARNING: directory '/u01' is not owned by rootPreparing to recopy patched init and RC scripts.Recopying init and RC scripts.Startup will be queued to init within 30 seconds.Starting up the CRS daemons.Waiting for the patched CRS daemons to start.This may take a while on some systems.

.

.

.

.10204 patch successfully applied.clscfg: EXISTING configuration version 3 detected.clscfg: version 3 is 10G Release 2.Successfully accumulated necessary OCR keys.

Using ports: CSS=49895 CRS=49896 EVMC=49898 and EVMR=49897.node : node 1: linux1 linux1-priv linux1Creating OCR keys for user 'root', privgrp 'root'..Operation successful.clscfg -upgrade completed successfully

From Node2

[root@linux2 bin]# ./crsctl stop crsStopping resources. This could take several minutes.Successfully stopped CRS resources.

7


8/24

Stopping CSSD.Shutting down CSS daemon.Shutdown request successfully issued.

[root@linux2 bin]# cd ../install

[root@linux2 install]# ./root102.sh >> rootexecteation2.txt &

[1] 10282

[root@linux2 install]# tail -f rootexecteation2.txtWARNING: directory '/u01/app/oracle/product/10.2.0' is not owned by rootWARNING: directory '/u01/app/oracle/product' is not owned by rootWARNING: directory '/u01/app/oracle' is not owned by rootWARNING: directory '/u01/app' is not owned by rootWARNING: directory '/u01' is not owned by rootPreparing to recopy patched init and RC scripts.Recopying init and RC scripts.Startup will be queued to init within 30 seconds./etc/profile: line 55: ulimit: open files: cannot modify limit:Operation not permitted/home/oracle/.bash_profile: line 30: ulimit: open files: cannot modifylimit: Operation not

permittedStarting up the CRS daemons.Waiting for the patched CRS daemons to start.This may take a while on some systems.

.

.10204 patch successfully applied.clscfg: EXISTING configuration version 3 detected.clscfg: version 3 is 10G Release 2.Successfully accumulated necessary OCR keys.

Using ports: CSS=49895 CRS=49896 EVMC=49898 and EVMR=49897.node : node 2: linux2 linux2-priv linux2Creating OCR keys for user 'root', privgrp 'root'..Operation successful.clscfg -upgrade completed successfully

Cluster Status

[root@linux2 install]# cd ../bin

8


9/24

[root@linux2 bin]# ./crs_stat -t Name Type Target State Host------------------------------------------------------------ora.devdb.db application ONLINE OFFLINEora....b1.inst application OFFLINE OFFLINEora....b2.inst application ONLINE OFFLINE

ora...._taf.cs application OFFLINE OFFLINEora....db1.srv application OFFLINE OFFLINEora....db2.srv application OFFLINE OFFLINEora....SM1.asm application OFFLINE OFFLINEora....X1.lsnr application OFFLINE OFFLINEora.linux1.gsd application ONLINE ONLINE linux1ora.linux1.ons application ONLINE ONLINE linux1ora.linux1.vip application ONLINE ONLINE linux1ora....SM2.asm application ONLINE ONLINE linux2ora....X2.lsnr application ONLINE ONLINE linux2ora.linux2.gsd application ONLINE OFFLINEora.linux2.ons application ONLINE ONLINE linux2ora.linux2.vip application ONLINE ONLINE linux2

PS: As per the above cluster status. (Which are highlighted red colors should be onlineform youre all nodes)

I am having some problem in Node2 thats why GSD down in node2.

Oracle Database Upgrade & ASM Upgration

I dont have separate ASM home. Im keeping ASM in ORACLE_HOME itself; So Now

going to continue my database & asm upgrade.

1. From oracle user you need to exeucte runInstaller then you will get below

screen shot.2. Please make sure your ORACLE_HOME in your path.

3. Please check any listener running from all nodes; (Else its will create problem)

4. Please check any database/asm instance running (If yes; pls shutdown thoseinstance)

9


10/24

For example:

[oracle@linux1 bin]$ ps -ef | grep pmonoracle 11993 22392 0 13:16 pts/2 00:00:00 grep pmon

[oracle@linux1 bin]$ ps -ef | grep lsnoracle 5155 1 0 11:06 ? 00:00:08

/u01/app/oracle/product/10.2.0/db_1/bin/tnslsnr LISTENER_LINUX1 -inheritoracle 12000 22392 0 13:16 pts/2 00:00:00 grep lsn[oracle@linux1 bin]$[oracle@linux1 bin]$ lsnrctl stop LISTENER_LINUX1

LSNRCTL for Linux: Version 10.2.0.1.0 - Production on 19-JUL-200913:16:42

10


11/24

Copyright (c) 1991, 2005, Oracle. All rights reserved.

Connecting to

(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=linux1-vip.gen.com)(PORT=1521)(IP=FIRST)))The command completed successfully

[oracle@linux2 bin]$ ps -ef | grep LISTENER_LINUX2oracle 12726 1 0 13:29 ? 00:00:01

/u01/app/oracle/product/10.2.0/db_1/bin/tnslsnr LISTENER_LINUX2 -inheritoracle 14894 8929 0 13:36 pts/1 00:00:00 grep LISTENER_LINUX2[oracle@linux2 bin]$ lsnrctl stop LISTENER_LINUX2



Connecting to

(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=linux2-vip.gen.com)(PORT=1521)(IP=FIRST)))

The command completed successfully[oracle@linux2 bin]$[oracle@linux2 bin]$ ps -ef | grep pmonoracle 14993 8929 9 13:36 pts/1 00:00:00 grep pmon[oracle@linux2 bin]$ ps -ef | grep lsnoracle 15002 8929 0 13:36 pts/1 00:00:00 grep lsn[oracle@linux2 bin]$

Before upgrade cluster status & Listener status

[root@linux1 bin]# ./crs_stat -tName Type Target State Host------------------------------------------------------------ora.devdb.db application OFFLINE OFFLINEora....b1.inst application OFFLINE OFFLINEora....b2.inst application OFFLINE OFFLINEora...._taf.cs application OFFLINE OFFLINEora....db1.srv application OFFLINE OFFLINEora....db2.srv application OFFLINE OFFLINEora....SM1.asm application ONLINE ONLINE linux1ora....X1.lsnr application ONLINE ONLINE linux1ora.linux1.gsd application ONLINE ONLINE linux1ora.linux1.ons application ONLINE ONLINE linux1ora.linux1.vip application ONLINE ONLINE linux1ora....SM2.asm application ONLINE ONLINE linux2ora....X2.lsnr application ONLINE ONLINE linux2ora.linux2.gsd application ONLINE ONLINE linux2

11


12/24

ora.linux2.ons application ONLINE ONLINE linux2ora.linux2.vip application ONLINE ONLINE linux2

[oracle@linux1 bin]$ ps -ef | grep lsnoracle 8117 1 0 03:17 ? 00:00:02

/u01/app/oracle/product/10.2.0/db_1/bin/tnslsnr LISTENER_LINUX1 -inheritoracle 12039 10732 0 03:28 pts/3 00:00:00 grep lsn

[oracle@linux1 bin]$[oracle@linux1 bin]$ lsnrctl stop LISTENER_LINUX1



Connecting to

(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=linux1-vip.gen.com)(PORT=1521)(IP=FIRST)))The command completed successfully[oracle@linux1 bin]$[oracle@linux1 bin]$ ps -ef | grep lsnoracle 12085 10732 0 03:28 pts/3 00:00:00 grep lsn[oracle@linux1 bin]$[oracle@linux1 bin]$ ps -ef | grep pmonoracle 8024 1 0 03:17 ? 00:00:03 asm_pmon_+ASM1oracle 12162 10732 0 03:28 pts/3 00:00:00 grep pmon[oracle@linux1 bin]$

[oracle@linux2 ~]$ cd /u01/app/oracle/product/10.2.0/crs/bin/[oracle@linux2 bin]$ ps -ef | grep lsnoracle 7134 1 0 03:19 ? 00:00:01

/u01/app/oracle/product/10.2.0/db_1/bin/tnslsnr LISTENER_LINUX2 -inheritoracle 10678 10552 0 03:29 pts/2 00:00:00 grep lsn[oracle@linux2 bin]$[oracle@linux2 bin]$ lsnrctl stop LISTENER_LINUX2



Connecting to

(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=linux2-vip.gen.com)(PORT=1521)(IP=FIRST)))The command completed successfully[oracle@linux2 bin]$[oracle@linux2 bin]$ ps -ef | grep pmonoracle 10784 1 21 03:29 ? 00:00:00 asm_pmon_+ASM2oracle 10806 10552 0 03:30 pts/2 00:00:00 grep pmon

Step2:

12


13/24

This is TWO Nodes cluster; thats why you will get your cluster nodes names list.

Step3

If you want register you CSI Nr then you can otherwise ignore this slide

13


14/24

Step4

14


15/24

This is pre-request checking steps; in case any thing wrong here you will come to know

all the details.

15


16/24

Step5

Once successfully completed your pre-request then continue the patch installation oncluster.

16


17/24

Step6:

Now you need to execute below script for all nodes (Using root user)

/u01/app/oracle/product/10.2.0/db_1/root.sh

[root@linux1 bin]# /u01/app/oracle/product/10.2.0/db_1/root.sh

17


18/24

Running Oracle10 root.sh script...

The following environment variables are set as:ORACLE_OWNER= oracleORACLE_HOME= /u01/app/oracle/product/10.2.0/db_1

Enter the full pathname of the local bin directory: [/usr/local/bin]:The file "dbhome" already exists in /usr/local/bin. Overwrite it? (y/n)

[n]: yCopying dbhome to /usr/local/bin ...

The file "oraenv" already exists in /usr/local/bin. Overwrite it? (y/n)[n]: y

Copying oraenv to /usr/local/bin ...The file "coraenv" already exists in /usr/local/bin. Overwrite it?(y/n)[n]: y

Copying coraenv to /usr/local/bin ...

Entries will be added to the /etc/oratab file as needed byDatabase Configuration Assistant when a database is createdFinished running generic part of root.sh script.Now product-specific root actions will be performed.

[root@linux2 bin]# /u01/app/oracle/product/10.2.0/db_1/root.shRunning Oracle10 root.sh script...

The following environment variables are set as:ORACLE_OWNER= oracleORACLE_HOME= /u01/app/oracle/product/10.2.0/db_1

Enter the full pathname of the local bin directory: [/usr/local/bin]:The file "dbhome" already exists in /usr/local/bin. Overwrite it? (y/n)[n]: y

Copying dbhome to /usr/local/bin ...The file "oraenv" already exists in /usr/local/bin. Overwrite it? (y/n)[n]: y

Copying oraenv to /usr/local/bin ...The file "coraenv" already exists in /usr/local/bin. Overwrite it?(y/n)[n]: y

Copying coraenv to /usr/local/bin ...

Entries will be added to the /etc/oratab file as needed byDatabase Configuration Assistant when a database is createdFinished running generic part of root.sh script.Now product-specific root actions will be performed.

[oracle@linux1 bin]$ . oraenvORACLE_SID = [devdb1] ? devdb

[oracle@linux1 bin]$ export ORACLE_SID=devdb1

[oracle@linux1 bin]$ sqlplus '/as sysdba'

SQL*Plus: Release 10.2.0.4.0 - Production on Sat Jul 25 03:38:20 2009

18


19/24

Copyright (c) 1982, 2007, Oracle. All Rights Reserved.

Connected to an idle instance.

SQL> startup nomountORACLE instance started.

Total System Global Area 285212672 bytesFixed Size 1267044 bytesVariable Size 100666012 bytesDatabase Buffers 176160768 bytesRedo Buffers 7118848 bytesSQL>SQL> show parameter clus

NAME TYPE VALUE------------------------------------ -----------------------------------------cluster_database boolean TRUEcluster_database_instances integer 2cluster_interconnects string

SQL> alter system set cluster_database=FALSE scope=spfile;

System altered.

SQL> shutdownORA-01507: database not mounted

ORACLE instance shut down.SQL>SQL> exitDisconnected from Oracle Database 10g Enterprise Edition Release

10.2.0.4.0 - ProductionWith the Partitioning, Real Application Clusters, OLAP, Data Miningand Real Application Testing options

[oracle@linux1 bin]$ sqlplus '/as sysdba'




SQL> startup upgradeORACLE instance started.

Total System Global Area 285212672 bytesFixed Size 1267044 bytesVariable Size 100666012 bytesDatabase Buffers 176160768 bytesRedo Buffers 7118848 bytesDatabase mounted.Database opened.SQL>

19


20/24

SQL> show parameter cluster

NAME TYPE VALUE------------------------------------ -----------------------------------------cluster_database boolean FALSEcluster_database_instances integer 1

cluster_interconnects stringSQL>

SQL>SQL> spool /u01/patch.logSQL>SQL> @$ORACLE_HOME/rdbms/admin/catupgrd.sql...........

No errors.No errors.

TIMESTAMP--------------------------------------------------------------------------------COMP_TIMESTAMP RUL 2009-07-25 11:54:47DBUA_TIMESTAMP RUL VALID 2009-07-25 11:54:47

TIMESTAMP-----------------------------------------------------------------------

---------COMP_TIMESTAMP UPGRD_END 2009-07-25 11:54:50.Oracle Database 10.2 Upgrade Status Utility 07-25-200911:54:50.Component Status VersionHH:MM:SSOracle Database Server VALID 10.2.0.4.001:23:50JServer JAVA Virtual Machine VALID 10.2.0.4.000:15:05Oracle XDK VALID 10.2.0.4.000:02:21Oracle Database Java Packages VALID 10.2.0.4.000:02:39Oracle Text VALID 10.2.0.4.000:01:48Oracle XML Database VALID 10.2.0.4.000:11:57Oracle Real Application Clusters VALID 10.2.0.4.000:00:22

20


21/24

Oracle Workspace Manager VALID 10.2.0.4.300:04:23Oracle Data Mining VALID 10.2.0.4.000:03:23OLAP Analytic Workspace VALID 10.2.0.4.000:03:25OLAP Catalog VALID 10.2.0.4.000:08:07

Oracle OLAP API VALID 10.2.0.4.000:06:20Oracle interMedia VALID 10.2.0.4.000:31:09Spatial VALID 10.2.0.4.000:13:33Oracle Expression Filter VALID 10.2.0.4.000:02:04Oracle Enterprise Manager VALID 10.2.0.4.000:12:57Oracle Rule Manager VALID 10.2.0.4.000:01:11.Total Upgrade Time: 03:24:44DOC>#######################################################################DOC>#######################################################################DOC>DOC> The above PL/SQL lists the SERVER components in the upgradedDOC> database, along with their current version and status.DOC>DOC> Please review the status and version columns and look forDOC> any errors in the spool log file. If there are errors in thespoolDOC> file, or any components are not VALID or not the current version,DOC> consult the Oracle Database Upgrade Guide for troubleshooting

DOC> recommendations.DOC>DOC> Next shutdown immediate, restart for normal operation, and thenDOC> run utlrp.sql to recompile any invalid application objects.DOC>DOC>#######################################################################DOC>#######################################################################DOC>#

SQL> shutdown immediateDatabase closed.Database dismounted.ORACLE instance shut down.SQL>SQL> startupORACLE instance started.

Total System Global Area 285212672 bytesFixed Size 1267044 bytes

21


22/24

Variable Size 146803356 bytesDatabase Buffers 130023424 bytesRedo Buffers 7118848 bytesDatabase mounted.Database opened.

SQL> @$ORACLE_HOME/rdbms/admin/utlrp.sqlTIMESTAMP

--------------------------------------------------------------------------------COMP_TIMESTAMP UTLRP_BGN 2009-07-25 12:12:32

DOC> The following PL/SQL block invokes UTL_RECOMP to recompileinvalid...........

ERRORS DURING RECOMPILATION---------------------------

0

PL/SQL procedure successfully completed.

SQL> show parameter cluster

NAME TYPE VALUE------------------------------------ -----------------------------------------cluster_database boolean FALSEcluster_database_instances integer 1cluster_interconnects stringSQL>

SQL> alter system set cluster_database=TRUE scope=spfile;

System altered.

SQL> alter system set cluster_database=TRUE scope=spfile;

System altered.

SQL> shutdownDatabase closed.Database dismounted.ORACLE instance shut down.SQL>SQL> startupORACLE instance started.

Total System Global Area 285212672 bytesFixed Size 1267044 bytesVariable Size 163580572 bytesDatabase Buffers 113246208 bytesRedo Buffers 7118848 bytesDatabase mounted.Database opened.

22


23/24

SQL>SQL> show parameter cluster

NAME TYPE VALUE------------------------------------ -----------------------------------------cluster_database boolean TRUEcluster_database_instances integer 2

cluster_interconnects string

[oracle@linux1 bin]$ ./srvctl start instance -d devdb -i devdb1

[oracle@linux1 bin]$ ./crs_stat -tName Type Target State Host------------------------------------------------------------ora.devdb.db application ONLINE ONLINE linux2ora....b1.inst application ONLINE ONLINE linux1ora....b2.inst application ONLINE OFFLINEora...._taf.cs application OFFLINE OFFLINEora....db1.srv application OFFLINE OFFLINEora....db2.srv application OFFLINE OFFLINEora....SM1.asm application ONLINE ONLINE linux1ora....X1.lsnr application ONLINE ONLINE linux1ora.linux1.gsd application ONLINE ONLINE linux1ora.linux1.ons application ONLINE ONLINE linux1ora.linux1.vip application ONLINE ONLINE linux1ora....SM2.asm application ONLINE ONLINE linux2ora....X2.lsnr application ONLINE ONLINE linux2ora.linux2.gsd application ONLINE ONLINE linux2ora.linux2.ons application ONLINE ONLINE linux2ora.linux2.vip application ONLINE ONLINE linux2

[oracle@linux2 bin]$ ./srvctl start instance -d devdb -i devdb2

[oracle@linux1 bin]$ ./crs_stat -tName Type Target State Host------------------------------------------------------------ora.devdb.db application ONLINE ONLINE linux2ora....b1.inst application ONLINE ONLINE linux1ora....b2.inst application ONLINE ONLINE linux2ora...._taf.cs application OFFLINE OFFLINEora....db1.srv application OFFLINE OFFLINEora....db2.srv application OFFLINE OFFLINEora....SM1.asm application ONLINE ONLINE linux1ora....X1.lsnr application ONLINE ONLINE linux1ora.linux1.gsd application ONLINE ONLINE linux1ora.linux1.ons application ONLINE ONLINE linux1ora.linux1.vip application ONLINE ONLINE linux1ora....SM2.asm application ONLINE ONLINE linux2ora....X2.lsnr application ONLINE ONLINE linux2ora.linux2.gsd application ONLINE ONLINE linux2ora.linux2.ons application ONLINE ONLINE linux2ora.linux2.vip application ONLINE ONLINE linux2

[oracle@linux2 bin]$ . oraenvORACLE_SID = [devdb] ?[oracle@linux2 bin]$ sqlplus "/as sysdba"

23


24/24




SQL> conn sys@devdb1 as sysdba

Enter password:Connected.SQL>SQL> select * from v$version where rownum=1;

BANNER----------------------------------------------------------------Oracle Database 10g Enterprise Edition Release 10.2.0.4.0 - Prod

SQL> conn sys@devdb2 as sysdbaEnter password:Connected.SQL> select * from v$version where rownum=1;

BANNER----------------------------------------------------------------Oracle Database 10g Enterprise Edition Release 10.2.0.4.0 - Prod

rac upgrade from 10.2.0.1 to 10.2.0.4

Documents