cookbookaixv1r1

Installing Oracle9i RAC on IBM pSeries with AIX 4.3.3 page 1/ 53

Quick Installation Guide

Oracle9i RAC on IBM eServer pSeries

with AIX 4.3.3

Version 1.1

February 2002

Authors : Fabienne Lepetit - Oracle France

Michel Passet - EMEA Oracle/IBM JSC Many thanks to : John McHugh - Oracle Corporate

François Pons - Oracle EMEA PTS EMEA Oracle/IBM JSC team


Installing Oracle 9.0.1 RAC on AIX 4.3.3

A. THE AIM OF THIS DOCUMENT....................................................................................................... 3 B. PREPARING THE SYSTEM.............................................................................................................. 4 B. PREPARING THE SYSTEM.............................................................................................................. 5

Hardware requirements...................................................................................................................... 5 Software requirements ....................................................................................................................... 5 AIX patches requirements .................................................................................................................. 5 SSA external disks microcode............................................................................................................ 5 SSA adapters microcode.................................................................................................................... 6

C. NETWORK CONFIGURATION ........................................................................................................ 7 D. CREATING THE RAW DEVICES ..................................................................................................... 9 E. CONFIGURING THE VSD SYSTEM ON SP NODES, USING PSSP 3.2....................................... 10 F. CONFIGURING HACMP/ES ON A CLUSTER OF PSERIES HOSTS........................................... 12 G. INSTALLING ORACLE ................................................................................................................... 18

Set up the Oracle environment......................................................................................................... 18 Cluster Manager Software................................................................................................................ 19 Unzip the distribution or mount the CDROM .................................................................................... 19

H. INSTALL THE ORACLE SOFTWARE ............................................................................................ 20 I. CREATING THE DATABASE........................................................................................................... 29

Manually ........................................................................................................................................... 29 With the database configuration assistant (DBCA) .......................................................................... 30

J. POST INSTALLATION..................................................................................................................... 31 Startup/shutdown services ............................................................................................................... 31 Configure listener.ora / sqlnet.ora / tnsnames.ora ........................................................................... 31 Configure Oracle Enterprise Manager.............................................................................................. 31 Troubleshooting................................................................................................................................ 32

Appendix A : SSA external disks and adapters microcode management............................................ 33 Appendix B : Logical volumes creation ................................................................................................ 34 Appendix C : VSD management on SP nodes, using PSSP 3.2 ........................................................ 35 Appendix D : Cluster topology & network verification output. .............................................................. 37 Appendix E : Installation Directories. ................................................................................................... 40 Appendix F : RAC Database creation script & init.ora. ........................................................................ 41 Appendix G : Some configuration files… (examples) .......................................................................... 47 Appendix H : Oracle technical notes.................................................................................................... 50


A. THE AIM OF THIS DOCUMENT The aim of this document is to provide help to install Oracle9i Real Application Clusters on IBM eServer pSeries with AIX 4.3.3 through a description of the different steps necessary to run Oracle9i RAC. It is based on our 9i RAC installation experience on AIX and is ever evolving. There are two ways to setup a RAC environment depending on the type of hardware that is used and the software that is installed. Your configuration will therefore direct you towards one or the other of the following two methods : - Virtual Shared Disks (VSD) when available - High Availability Cluster Management Program (HACMP) always possible In this document, both methods are explained. The information contained in this paper resulted from : - Oracle and IBM documentations - Installation runs of Oracle9i RAC - Workshop experiences done in the Oracle/IBM Join Solutions Center - Contributions from Oracle and IBM specialists. Please also refer to Oracle documentation for more information. (http://docs.us.oracle.com). - Oracle9i Quick Installation Procedure Release 9.0.1 for AIX-Based Systems - Oracle9i Installation Guide Release 9.0.1 for UNIX Systems - Oracle9i Administrator's Reference 9.0.1 for UNIX Systems - Oracle9i Release Notes Release 9.0.1 for AIX-Based Systems - Oracle9i Online Generic Documentation CD-ROM Installation and Usage Notes - Oracle9i Real Application Clusters Installation and Configuration - Oracle9i Installation Checklist for AIX-Based Systems - Oracle Enterprise Manager Configuration Guide Your comments are important for us. We want our technical papers to be as helpful as possible. Please send us your comments about this document to the EMEA Oracle/IBM Joint Solutions Center. Use our email address :

[email protected] or our phone number :

+33 (0)4 67 34 67 49


How to install 9i RAC on AIX using this document? Prepare the system.

Chapter B.

Configure the network.

Chapter C.

Create the raw devices.

Chapter D.

Prepare the installation of Oracle.

Chapter G.

pSeries cluster : configure HACMP/ES.

Chapter F.

SP nodes : configure the VSDs.

Chapter E.

Install the Oracle sftware.

Chapter H.

Create the database.

Chapter I.

Post-install tasks.

Chapter J.


B. PREPARING THE SYSTEM HARDWARE REQUIREMENTS • RAM >= 512 MB

Command to check the physical memory : lsattr –El sys0 –a realmem • Internal disk >= 4 GB for the oracle code • Paging space = 2 x RAM, with a minimum of 400 MB

Nota : for big servers with more than 8 GB of memory, the paging space could be less than 2 x RAM. To check the paging space configured : lsps –a

• Temporary Disk Space: The Oracle Universal Installer requires up to 800 MB of free space in the /tmp directory. To check the free temporary space available : df –k /tmpYou can use an other filesystem instead of /tmp. Set the TEMP environment variable (used by Oracle) and the TMPDIR environment variable to the new location. For example : export TEMP=/new_tmp

export TMPDIR=/new_tmpChange the permissions of /new_temp

chown oracle:dba /new_temp• CD-ROM drive. If the server does not have a CD-ROM unit (like a SP node for example), it is still possible

to mount via NFS the drive from an other machine on the network. SOFTWARE REQUIREMENTS • AIX 4.3.3 Maintenance level 9

To determine the current operating system version : oslevel To check the maintenance level applied : instfix -i | grep ML

• PSSP 3.2 ptfset 12 SP machine only • VSD (Virtual Shared Disks) SP machine only • HACMP/ES 4.4 Clusters of pSeries servers only AIX PATCHES REQUIREMENTS

IY01050 (AIXV43 ONLY) SUPPORT FOR NON-ROOT ACCESS TO KERNEL PERF STA IY03478 PTCs rejected because instance number wraps IY04109 PROBLEM WITH HA_GS_INIT() FOR 64-BIT CLIENTS IY04149 PTCs rejected because instance number wraps IY04767 css0 IP driver shouldn't return ENETDOWN during IY07276 (AIXV43 only) AIO_SUSPEND RETURNS WITHOUT I/O COMPLETION IY06749 HAES/ORACLE: add event management variables for IY15138 (AIXV43 only) LIO_LISTIO RETURNS 0 INSTEAD OF -1 FOR EAGAIN E IY20220 RSCT 1.2.1 Maintainence Level PTF. Mandatory to link Oracle binaries IY22458 (AIXV43 only) GETPWUID IN 64BIT APPLICATION FAILS WHILE USI To check if a specific patch is applied : instfix –ik IY15138

You can download new AIX maintenance levels and specified patches on http://techsupport.services.ibm.com/server/nav?fetch=ffa4e

SSA EXTERNAL DISKS MICROCODE

It is particulary important for HACMP that all the SSA disks connected to the cluster are all at the same level of microcode.

http://techsupport.services.ibm.com/server/nav?fetch=ffa4e


Last microcode levels (november 1st, 2001) are : 9911, 0023, 0012 and 0070 (depending on the model of the disk). To list the SSA disks of the cluster : lscfg | grep pdiskTo check the microcode level : lscfg –vl pdisknn (ROS level and ID line) For more information on the procedure of download and installation of new microcode, see appendix A.

SSA ADAPTERS MICROCODE

The latest level of microde is B300 (12/2001) • Check if there is some AIX default limitations (especially on the file size)

File size limitation: ulimit –fAll limitations : ulimit –aSee also the file /etc/security/limits which shows the limits for each user. The default stanza applies to all new user to be created. This file can be modified by root with vi. The default limits should be set to unlimited, except for core (e.g. –1 in the file /etc/security/limits) To turn some user limitation to unlimited, use smit users


C. NETWORK CONFIGURATION

Set up user equivalence for the oracle account, to enable rsh, rcp, rlogin commands. /etc/hosts file.

! When VSD are used :

/etc/hosts for VSD (on both nodes)10.10.10.1 int-rac1 # Internal private net.; fast interconnect10.10.10.2 int-rac2 # Internal private net.; fast interconnect192.128.194.1 rac1 rac1.mop.ibm.com # External, fixed IP address192.128.194.2 rac2 rac2.mop.ibm.com # External, fixed IP address

(This file should contain private network addresses which would use the private link (interconnect) between the two nodes.)

! When HACMP is used :

/etc/hosts for HACMP (on both nodes)9.9.9.1 rac1_boot # HACMP managed addresses9.9.9.11 rac1_service10.10.10.1 rac1_stdby

9.9.9.2 rac2_boot # HACMP managed addresses9.9.9.22 rac2_service10.10.10.2 rac2_stdby

192.128.194.1 rac1_fixed # External, fixed IP address, out of HACMP management192.128.194.2 rac2_fixed # External, fixed IP address, out of HACMP management

/etc/hosts.equiv file. Put the list of machines or nodes into hosts.equiv.

/etc/hosts.equiv

# for VSD configuration :rac1rac2int-rac1int-rac2

# for HACMP configuration :rac1_bootrac1_servicerac1_stdbyrac2_bootrac2_servicerac2_stdbyrac1_fixedrac2_fixed

.rhosts file.

In the root’s home directory, put the list of machines.

$HOME/.rhosts

# for VSD configuration :rac1rac2int-rac1int-rac2


# for HACMP configuration :rac1_bootrac1_servicerac1_stdbyrac2_bootrac2_servicerac2_stdbyrac1_fixedrac2_fixed

Note : It is possible, but not advised because of security reasons, to put a “+” in hosts.equiv and .rhosts files.

Test if the user equivalence is correctly set up (rac2 is the secondary server name) You are logged on rac1 as oracle. $ rlogin rac2 (-> no pwd)$ rcp /tmp/toto rac2:/tmp/toto$ rsh rac2 pwd


D. CREATING THE RAW DEVICES The Oracle code is located on an internal SCSI disk, duplicated on the two machines of the cluster. The data, index, control files and so on are stored on external SSA disks. All the SSA disks are physically linked to the two nodes (open loops). Because theses SSA loops are connected on the two machines, all the disks can be accessed by both of the nodes, in case of a failure of one node. In a normal operation, when the two nodes are up and running, each node has its own disks, its own volume group, its own raw devices. There are shared, but not concurrent. A typical installation requires the following raw devices. Technical file: one raw device to store server configuration information (list of nodes, etc…)

Size Logical Volume name Device name Purpose 100M srvconfig_lv /dev/ rvsd_srvconfig Server configuration file (replacing

db_name.conf in OPS) Database files:

Size Logical Volume name Raw device name Purpose 5M Spfile_lv /dev/rvsd_spfile Server Parameter File (replacing init.ora)

12M Tools_lv /dev/rvsd_tools TOOLS Tablespace 80M Index_lv /dev/rvsd_index INDX Tablespace 90M Drsys_lv /dev/rvsd_drsys DRSYS (intermedia & Ultrasearch) Tablespace

100M Temp_lv /dev/rvsd_temp TEMP Tablespace 100M cmwlite_lv /dev/rvsd_cmwlite CMWLITE (OLAP) Tablespace

10M ctrl1_lv /dev/rvsd_ctrl1 Control File # 1 10M ctrl2_lv /dev/rvsd_ctrl2 Control File # 2

120M Users_lv /dev/rvsd_users USERS Tablespace 120M redolog1_1_lv /dev/rvsd_redolog1_1 Redo Log Thread #1, Group #1 120M redolog1_2_lv /dev/rvsd_ redolog1_2 Redo Log Thread #1, Group #2 120M redolog2_1_lv /dev/rvsd_ redolog2_1 Redo Log Thread #2, Group #1 120M redolog2_1_lv /dev/rvsd_ redolog2_1 Redo Log Thread #2, Group #2 160M example_lv /dev/rvsd_example EXAMPLE Tablespace 512M Undo1_lv /dev/rvsd_undo1 UNDO Tablespace (instance #1) 512M Undo2_lv /dev/rvsd_undo2 UNDO Tablespace (instance #2) 400M System_lv /dev/rvsd_system SYSTEM Tablespace 160M oemrepo_lv /dev/rvsd_oemrepo OEM repository

All these raw devices can be created for our database. But some of them (for exemple drsys, oemrepo, cmwlite…) are not mandatory. Except for the 2 redo logs and undo files, that have to be duplicated on each instance, the others datafiles, for VSD configuration, are splited on the two volumes groups, in order to have roughly the same volume. A script is provided in appendix to create the volumes groups and the logical volume as specified above. The two volumes groups, ora1vg for the primary node, and ora2vg of the secondary are build with a physical partition size of 16MB. The size of the logical volume is expressed in number of PP, not in Kbytes. Once a logical volume new_lv is created on a node, two new entries appears in the /dev directory : /dev/new_lv which is normally used by LVM for file systems. /dev/rnew_lv where the “r” stands for raw device. This is the device name to use with VSD.


E. CONFIGURING THE VSD SYSTEM ON SP NODES, USING PSSP 3.2 The VSD (Virtual Shared Disk) layer is over the AIX Logical Volume Manager layer. When Oracle wants to access a file, it uses the VSD name of the file, no matter if this file have to be accessed via one node, or the other. VSD reside only with raw devices, not with file systems. When we talk about file, or datafile, we have to understand raw device (a logical volume). To read/write a raw device, Oracle knows only the VSD name of a raw device. The VSD layer then send the I/O request to the machine that can physically access the disk. In case of problem on a machine, the VSD configuration files are updated, and all the I/O previously processed by the broken machine are rerouted to the remaining node. It is not a concurrent disk access, but a shared one. Only one node have the access to a particular disk. All the I/O requests to a disk are transmitted to the node who owns the disk. This node handles the physical read/write, and send back the data to the requester node. There are no locks to manage. The VSD are to be used with a SP machine running AIX 4.3.3 and PSSP 3.2. HACMP/ES is used for clusters of pSeries machines with AIX 4.3.3 At the LVM layer, we have two volumes groups for a cluster of two nodes. Each volume group is varyon and used on one node, and varyoff (but defined) on the other node. The volumes groups are NOT opened in concurrent mode. There are shared between all the nodes, but there are varyon on a single node. In case of a hardware problem of a node, the other node of the cluster can varyon the volume group of the broken machine, and access both of the volumes groups. • VSD filesets to be installed on CWS and all nodes : vsd.cmi 3.2.0.0vsd.rvsd.rvsdd 3.2.0.0vsd.rvsd.scripts 3.2.0.0vsd.sysctl 3.2.0.0vsd.vsdd 3.2.0.0

Note : Generally speaking, you have to install first the base software before the patches. A patch will be installed only if the base fileset it fixes is found on the system. • Start the High Availability (HA) using a script. It does start/stop all the processes in the right order. syspar_ctrl –A start all the HA services and processes syspar_ctrl –D stop the HA Check if the High Availability is up and running on the nodes : It is composed of the following services (or sub-systems):

hags (HA Group Services) hagsglsm hats (HA Topology Services) rvsd hc.hc

To list the status of a service : lssrc –g <group of services>To start a service : startsrc –g <group of service>To stop a service : stopsrc –g <group of service> For example : lssrc –a | grep –i ha

hats.cws11 hats 30846 activehags.cws11 hags 25572 activehagsglsm.cws11 hags 15052 active


For example : lssrc –a | grep –i vsd

rvsd.cws11 rvsd 40482 active

The hags daemon creates a socket (size 0) the first time it starts : /var/ha/soc/hagsdsocket.<CWS_name> should belong to the user root and the group hagsuser. The user and the group must both have read and write permissions on this socket. • To enable the VSD of a SP, all the following commands have to be run on the Control WorkStation (CWS)

of the SP. Step 1 : define the nodes of a cluster. The primary node, the secondary node, and the link between them Step 2 : define the VSD volumes groups. For each volume group, indicate the node which normally access the volume group, and the backup node. Step 3 : define the logical volumes (raw devices) that can be used by VSD. A new VSD name is set, which reference the volume group and the logical volume. Step 4 : start the VSD A script is provided in Appendix, to do the 4 steps for all the raw devices involved in the database.


F. CONFIGURING HACMP/ES ON A CLUSTER OF PSERIES HOSTS HACMP/ES is a product which provide a high availability on a cluster of two machines. HACMP layer includes the CLVM, which enable the concurrent logical volume manager. In this 9i RAC installation guide, the purpose is not to set up all the parameter stuff of HACMP. We will focus on the concurrent volumes groups, which is enough to install and run 9i RAC. HACMP/ES is used for clusters of pSeries machines with AIX 4.3.3 The VSD are to be used with a SP machine running AIX 4.3.3 and PSSP 3.2. Filesets to be installed on the two machines of the cluster :

cluster.adt.es.client.demoscluster.adt.es.client.includecluster.adt.es.client.samples.clinfocluster.adt.es.client.samples.clstatcluster.adt.es.client.samples.demoscluster.adt.es.client.samples.libclcluster.adt.es.java.demo.monitorcluster.adt.es.server.demoscluster.adt.es.server.samples.demoscluster.adt.es.server.samples.imagescluster.doc.en_US.es.html 4.4.0.0 C HAES Web-based HTMLcluster.doc.en_US.es.pdf 4.4.0.0 C HAES PDF Documentation - U.S.cluster.doc.en_US.es.ps 4.4.0.0 C HAES Postscript Documentation -cluster.doc.en_US.html 4.4.0.0 C HACMP Web-based HTMLcluster.doc.en_US.pdf 4.4.0.0 C HACMP PDF Documentation - U.S.cluster.doc.en_US.ps 4.4.0.0 C HACMP Postscript Documentation -cluster.es.client.lib 4.4.0.4 C ES Client Librariescluster.es.client.rte 4.4.0.2 C ES Client Runtimecluster.es.client.utils 4.4.0.4 C ES Client Utilitiescluster.es.clvm.rte 4.4.0.0 C ES for AIX Concurrent Accesscluster.es.cspoc.cmds 4.4.0.5 C ES CSPOC Commandscluster.es.cspoc.dsh 4.4.0.0 C ES CSPOC dshcluster.es.cspoc.rte 4.4.0.2 C ES CSPOC Runtime Commandscluster.es.hc.rte 4.4.0.3 C ES HC Daemoncluster.es.server.diag 4.4.0.5 C ES Server Diagscluster.es.server.events 4.4.0.5 C ES Server Eventscluster.es.server.rte 4.4.0.5 C ES Base Server Runtimecluster.es.server.utils 4.4.0.5 C ES Server Utilitiescluster.man.en_US.es.data 4.4.0.0 C ES Man Pages - U.S. Englishcluster.man.en_US.haview.datacluster.msg.en_US.cspoc 4.4.0.0 C HACMP CSPOC Messages - U.S.cluster.msg.en_US.es.clientcluster.msg.en_US.es.servercluster.vsm.es 4.4.0.2 C ES VSM Configuration Utility

The two instances of the same parallel database have a concurrent access on the same external disks. It is a real concurrent access, and not a shared one like in the VSD environment. Because several instances access at the same time the same files and data, locks have to be managed. These locks, at the CLVM layer (including memory cache), are managed by HACMP. After all the filesets have been installed: - add in the PATH environment variable the following directories # /usr/es/sbin/cluster # /usr/es/sbin/cluster/utilities # /usr/es/sbin/cluster/sbin # /usr/es/sbin/cluster/diag

- check the existence of symbolic links from files contained in /usr/sbin/cluster to /usr/es/sbin/cluster. Else, create then with ln –s.


Main steps for implementing an HACMP configuration. 1 – Configure the cluster : “Add a cluster definition” in smitty . In this step you give an identifier and a name to your cluster. 2 – Configure nodes of the cluster: “Add cluster nodes” in smitty. Here, you enter the name of the nodes participating to the cluster. 3 – Configure the network. “Add a network” in section “Configure IP-based networks” in smitty. In the case of the implementation of a concurrent volume group, only a private network is needed. It’s not necessary to define boot, service and standby addresses like in cascading or rotating HACMP configurations. In RAC environments, you will need a high performance private network between the machines participating to the RAC cluster. The network type of this interconnect can be HPS (High Performance Switch) or ethernet (use a gigabit ethernet, for example). Note : It is advised to have also a non-IP network (like RS/232 or Target mode SSA). Consult the HACMP documentation, available on http://xxx.xxx.xxx 4 – Configure the adapters. “Add an adapter” in section “Adapters on IP-based network” in smitty. There must be one adapter by machine. 5 – Synchronize the cluster topology. “Synchronize Cluster Topology” in smitty. 6 – Define a resource group. “Add a resource group” in section “Cluster resources” in smitty. The node relationship must be “concurrent”. All participating nodes must be entered. Then, check the attributes of the resource group in “ Change/Show Resources/Attributes for a Resource Group”. When you will have created a concurrent volume group (see next paragraph), it will be necessary to enter its name in the attributes of your resource group (line “Volume groups” in the “Change/Show Resources/Attributes for a Resource Group” panel of smitty). 7 – Synchronize the cluster resources. “Synchronize Cluster Resources” in smitty. Important note: your cluster must be synchronized after each new modification. How to create and setup a concurrent volume group 1 – Check if the target disks are physically linked to the two machines of the cluster, and seen by both. Run lspv command on both machines. Note : the hdisk number can be different, depending on the others disks connected on each machine. Use the second field of the output (PVid) of lspv to be sure you are dealing with the same physical disk from two hosts.


2 – Create at the AIX level on the first machine a concurrent volume group, myvg smit vg

Add a Volume Group

Type or select values in entry fields.Press Enter AFTER making all desired changes.

[Entry Fields]VOLUME GROUP name [myvg]Physical partition SIZE in megabytes 32 +

* PHYSICAL VOLUME names [] +Activate volume group AUTOMATICALLY no +at system restart?

Volume Group MAJOR NUMBER [64] +#Create VG Concurrent Capable? yes +Auto-varyon in Concurrent Mode? no +

Never choose YES to activate at system restart, neither auto-varyon. These tasks have to be managed by HACMP. The volume group just has to be created with concurrent capability. You must choose the major number to be sure the volume groups have the same major number in all the nodes (attention, before choosing this number, you must be sure it’s free on all the nodes). To check all defined major number, type ls –al /dev/*crw-rw---- 1 root system 57, 0 Aug 02 13:39 /dev/myvg

The major number for myvg volume group is 57. On this volume group, create all the logical volumes and file systems you need for your database. 3 – Import myvg volume group on the second machine On the first machine, type varyoffvg myvg On the second machine, import the definition of the volume group :

Import a Volume Group

Type or select values in entry fields.Press Enter AFTER making all desired changes.

[Entry Fields]VOLUME GROUP name [myvg]

* PHYSICAL VOLUME name [] +Volume Group MAJOR NUMBER [64] +#Make this VG Concurrent Capable? no +Make default varyon of VG Concurrent? no +

The physical volume name (hdisk) could not have the same number on both sides. Check the PVID of the disk, because it’s the only information reliable and common thru the cluster. Be sure to have the same major number. This number has to be undefined on all the nodes. The new volume group is now defined on the all the machines of the cluster, with the concurrent capable feature set on.


4 – Define myvg volume group in an HACMP resource smit hacmp Change/show ressources/attributes for ressource group Service IP label None Concurrent VG myvg Fsck sequential Synchronize the topology and the ressources 5 – Start HACMP smit hacmp services start Is HACMP correctly configured? Checklist. 1 – HAGS socket. The HAGS socket needs to be writable by "oracle" and the "cldomain" executable needs to be executable by "oracle". By configuring the group and permissions for the "grpsvcsdsocket.<domain>" file the instance will be able to communicate with HAGS and the Oracle instance will mount. On all the nodes of the cluster, perform the following tasks : - check hagsuser group exists, else create it - place "oracle" into the "hagsuser" group - change the permissions on the "cldomain" executable :

# chmod a+x /usr/sbin/cluster/utilities/cldomain- change the group to "hagsuser" for the "svcsdsocket.<domain>" socket: # chgrp hagsuser /var/ha/soc/grpsvcsdsocket.`/usr/sbin/cluster/utilities/

cldomain` For more information, see the note # 2064876.102 in Metalink (also presented in appendix H : Oracle Technical notes). 2 – Interconnect. HACMP interconnect network adapter configuration can be checked with : /usr/sbin/cluster/utilities/cllsif


cllsif on a working configuration should look like this: Adapter Type Network Net Type Attribute Node IP Addresspfpdb3 service pfpdb ether private pfpdb3 11.2.18.24pfpdb4 service pfpdb ether private pfpdb4 11.2.18.3

For more information, see the note # 115792.1 in Metalink (also presented in appendix H : Oracle Technical notes). 3 – Concurrent Volume group Check the concurrent volume group is active on all the nodes of the cluster. lsvg myvg

VOLUME GROUP: myvg VG IDENTIFIER: 000915700ab7290dVG STATE: active PP SIZE: 32 megabyte(s)VG PERMISSION: read/write TOTAL PPs: 543 (17376 megabytes)MAX LVs: 256 FREE PPs: 40 (1280 megabytes)LVs: 37 USED PPs: 503 (16096 megabytes)OPEN LVs: 30 QUORUM: 1TOTAL PVs: 1 VG DESCRIPTORS: 2STALE PVs: 0 STALE PPs: 0ACTIVE PVs: 1 AUTO ON: noConcurrent: Capable Auto-Concurrent: DisabledVG Mode: ConcurrentNode ID: 2 Active Nodes: 1MAX PPs per PV: 1016 MAX PVs: 32

4 – Daemons etc… Execute the following command :

lssrc -a | egrep 'svcs|ES'

It should give the following output :

topsvcs topsvcs 45454 activegrpsvcs grpsvcs 59670 activegrpglsm grpsvcs 39028 activeemsvcs emsvcs 59508 activeemaixos emsvcs 45896 activeclstrmgrES cluster 47700 activecllockdES lock 52542 activeclsmuxpdES cluster 37470 activeclinfoES cluster 56110 active

HACMP and PSSP on the same machine. Oracle with HACMP in concurrent mode can be implemented in SP environments. So, PSSP and HACMP will coexist on the same machine. Hereunder, an excerpt taken from a note about the installation of Oracle on AIX-Based Systems". […] When HACMP and PSSP are installed on the same machine, Oracle assumes that customers want to use PSSP functionality. In the default configuration, customers using OPS [or RAC] must place all database files on Virtual Shared Disks (VSDs).


If customers wish to use the Concurrent Logical Volume Manager (CLVM) instead of VSDs, they must enable HACMP functionality by setting the environment variable PGSD_SUBSYS to grpsvcs. Oracle will not allow VSDs and Concurrent Logical Volumes (CLVs) to be used on the same database. If PSSP services are being used, Oracle will report an error if the customer attempts to use CLVs. If HACMP services are used (i.e., PGSD_SUBSYS is set to grpsvcs), Oracle will report an error if the customer attempts to use VSDs. The PGSD_SUBSYS environment variable should be set in all the environments where Oracle is used, including the listener.ora file… […]


G. INSTALLING ORACLE SET UP THE ORACLE ENVIRONMENT • Use smit group or smitty group to create the groups

dba Primary group for the oracle user. hagsuser For high availability (if not already created).

oinstall The ora inventory group. This group is not mandatory. If it exists, it will be the group owner of the oracle code files. This group is a secondary group for oracle user.

• Use smit user to create the users

oracle Owner of the database.

The oracle user must have dba as primary group, oinstall and hagsuser as secondary groups. Also add the secondary group hagsuser to the root account.

Verification : check if the file /etc/group contains lines such as : (the numbers could be different) hagsuser:!:203:oracle, rootdba:!:204:oracleoinstall:!:205:oracle

• Create the file system for Oracle code. This file system of 4 GB, is generally located on an internal SCSI

disk. The external SSA disks will store the datafiles. To list the internal disks : lscfg | grep -i scsi | grep hdiskSuppose we have hdisk1, and internal SCSI free disk of 18.2 GB Create a volume group called oraclevg : mkvg -f -y'oraclevg' -s'16' hdisk1

Create a 4GB file system /oracle in the previous volume group (large file enabled): crfs -v jfs -a bf=true -g'oraclevg' -a size='8388608' -m'/oracle' -A'yes' -p'rw' -t'no' -a nbpi='8192' -a ag='64'

mount /oraclechown oracle:dba /oracle

• Verify that the /etc/oraInst.loc and /etc/oratab files are writable by the oracle account. After installation,

these two files (created by root.sh) will contain information that briefly describes the Oracle software installations and databases on the server. These commands verify that the oracle account has the appropriate permissions : touch /etc/oraInst.loc /etc/oratab /etc/srvConfig.locchown oracle:dba /etc/oraInst.loc /etc/oratab /etc/srvConfig.locchmod 644 /etc/oraInst.loc /etc/oratab /etc/srvConfig.loc

• Edit the /etc/srvConfig.loc file on each node, with a single line : srvconfig_loc=/dev/rvsd_srvconfig

• In Sqlplus, when you “connect / as sysdba” if you get “insufficient privileges” (even if you are the user “oracle” with group “dba”), as root, execute the following command : touch /etc/passwd


• Oracle environment

In the .profile in Oracle’s $HOME : # Oracle specific environmentumask 022export ORACLE_BASE=/oracleexport ORACLE_HOME=$ORACLE_BASE/product/9.0.1export ORA_NLS33=$ORACLE_HOME/ocommon/nls/admin/dataexport NLS_LANG=american_america.WE8ISO8859P1export ORACLE_SID=INSTANCE_NAMEexport PATH=/usr/bin:/etc:/usr/sbin:/usr/ucb:/usr/local/bin:$ORACLE_HOME/bin:/usr/bin/X11:/sbin:.export LD_LIBRARY_PATH=$ORACLE_HOME/lib:/lib:/usr/lib:usr/local/libexport LIBPATH=$ORACLE_HOME/lib:$LIBPATH

export CLASSPATH=$ORACLE_HOME/jre:$ORACLE_HOME/jre/1.1.8/lib:/usr/jdk_base:/usr/jdk_base/lib:$ORACLE_HOME/rdbms/jlib:$ORACLE_HOME/jlib:$ORACLE_HOME/network/jlib

# used by database configuration assistant to map datafiles to raw devicesexport DBCA_RAW_CONFIG=$HOME/dbca_raw_config

# This file contains the raw device for 9i/RAC configexport SRVM_SHARED_CONFIG=/etc/srvConfig.loc

# These variables are not used and incorrectly referred to in some docs.# Use SRVM_SHARED_CONFIG instead#OPSM_SHARED_CONFIG=/etc/srvConfig.loc#SRVM_RAW_CONFIG=/etc/srvConfig.loc

# If tmp has been re-directed (for example in /new_temp) add the 2 following lines:export TEMP=/new_tempexport TEMPDIR=/new_temp

# if you want to use CLVM of HACMP instead of VSD in a SP environmentexport PGSD_SUBSYS=grpsvcs

# set DISPLAY according to the machine to where you want to send the display, and executexhost + as rootexport DISPLAY=<machine_name>:0 export ENV=$HOME/.kshrcexport PS1='{'"$(hostname -s)"'} $PWD> 'set -o vi

CLUSTER MANAGER SOFTWARE If the cluster manager software is correctly set up, the Oracle RAC option should be automatically preselected in Oracle Universal Installer. UNZIP THE DISTRIBUTION OR MOUNT THE CDROM If you have downloaded the distribution, you should have 5 CDs. Note: If you copy the five CDs to the hard disk, use cp –a to maintain directories and links.


H. INSTALL THE ORACLE SOFTWARE Login as oracle and follow the procedure hereunder… $ /<cdrom_mount_point>/runInstaller At the OUI Welcome screen, click Next. The installer will ask the user to run /tmp/orainstRoot.sh in a separate window. This script creates the file /etc/oraInst.loc, which is used by OUI for the list of installed products. OUI asks you to run rootpre.sh as root in another window. Remember that the hagsuser should be secondary group for the user oracle.


A prompt will appear for the Inventory Location (if this is the first time that OUI has been run on this system). This is the base directory into which OUI will install files. Oracle user should have write permissions on this directory. The Oracle Inventory definition can be found in the file /etc/oraInst.loc. Click OK.

Verify the UNIX group name of the user which controls the installation of the Oracle9i software. If the pre-installation steps have not been completed successfully, you are asked to run /tmp/orainstRoot.sh, forcing Oracle Inventory files, and others, to be written to the ORACLE_HOME directory. This screen only appears the first time Oracle9i products are installed on the system. Click Next.


The File Location window will appear. Do not change the Source field. The Destination field defaults to the ORACLE_HOME environment variable. Click Next.

Select the Products to install. In this example, select the Oracle9i Server then click Next.


Select the installation type. Choose the Enterprise Edition option. The selection on this screen refers to the installation operation, not the database configuration. The next screen allows for a customized database configuration to be chosen. Click Next.


Select the configuration type. In this example, “Customized” configuration is selected; so, a customizable database will be created. Click Next.

Select the other nodes on to which the Oracle RDBMS software will be installed. It is not necessary to select the node on which the OUI is currently running. Click Next.


Identify the raw partition in to which the Oracle9i Real Application Clusters (RAC) configuration information will be written. It is recommended that this raw partition has a minimum size of 100MB. Enter the name of the raw device previously created; in our case, “/dev/rvsd_svrconfig”.


An option to Upgrade or Migrate an existing database is presented. Do NOT select the radio button. The Oracle Migration utility is not able to upgrade a RAC database, and will error if selected to do so.

The Summary screen will be presented. Confirm that the RAC database software will be installed and then click Install. The OUI will install the Oracle9i software on to the local node, and then copy this information to the other nodes selected.


During the installation, you will be prompted to give the location of the second disk. And so on until the disk #5. When the installation progress bar will reach 100%, the installer will continue to work because it will copy the files on the other node. The user won’t be informed of this copy.

From the root command prompt, execute /oracle/product/9.0.1/root.sh. This script must be executed on both nodes. The results of this are shown below :


I. CREATING THE DATABASE The database and the two instances can be created by the Oracle Database Assistant. Do not choose to create the database by copying one of the preconfigured database, as the DDL statement used to create the TEMP tablespace does not work against a raw device. On one node, initialize the configuration raw device by running

srvconfig -init Start the Global services Daemon (as oracle) on each node (runs as OPSDaemon)

$ gsd.sh &

$ ps -efl | grep –i 'ops'

gsd (Oracle Global Services Daemon) needs to be running on each node under oracle privileges; start gsd by executing gsd.sh gsd is used by OEM and global tools such as srvctl to execute commands on all the nodes at the same time.

The configuration device should be pointed to by SRVM_SHARED_CONFIG (containing /var/opt/oracle/srvConfig.loc for example). MANUALLY

1) on one node : srvctl add db -p <db_name> -o <oracle_home> 2) for each instance of the database :

srvctl add instance –p <db_name> -i <SID> -n <node_name>

(it is advised to set the SID to db_name plus instance_number)

3) on each node, check the configuration : srvctl config –p <db_name> 4) on each node, create (or update) oratab file (in /etc directory) with the following line :

<db_name>:<$ORACLE_HOME>:N

5) create the udump, cdump and bdump directories.

6) set the SID in .profile of oracle user

7) create the init.ora file for each node. It is possible to prefix parameter which are local to the instance with the instance name. For example, thread, instance_name, rollback_segments… It is possible to create a spfile with “create spfile=’/dev/rawspfile’ from pfile=/…/init.ora”.

8) create the database creation script. You can use $ORACLE_HOME/srvm/clustdb.sql script as sample.

This script is also presented in appendix F. The script must be adapted to your environment. And be careful, there are some errors in this file…

9) you can create a password file. Under $ORACLE_HOME execute orapwd file=orapw password=###.

10) as sysdba, execute the script.


WITH THE DATABASE CONFIGURATION ASSISTANT (DBCA) Create a parameter file $HOME/dbca_raw_config which will be used by DBCA to map the typical tablespaces to raw devices. This parameter file needs to be pointed to by the environment variable DBCA_RAW_CONFIG and makes easier to create the database (see example in Appendix). First use NETCA to create listeners configuration on each node. Start DBCA

Do not use a pre-configured database (as the DDL statement used to create the TEMP tablespace does not work against a raw device); instead have DBCA create a customized database (create database). DBCA can also be used to cleanly remove instances or add new instance to an existing multi-instance database

Optionally back up the spfile

SQL> create pfile='?/dbs/initXXX.ora' from spfile='/dev/rvsd_3'


J. POST INSTALLATION

STARTUP/SHUTDOWN SERVICES

GSD must be running for srvctl to be able to run each command on all the nodes. $ srvctl command -h -> help

To start/stop/check all instances and listeners: su - oracle

srvctl start|stop|status -p <db_name> To start/stop a given instance and listener: srvctl start|stop|status -p <db_name>

-i <instance_name>

To start/stop the listeners only: srvctl start|stop –p <db_name> –s lsnr To start/stop the instances only: srvctl start|stop –p <db_name> –s inst To list the instances: srvctl config –p <db_name> To get environment information : srvctl get env -p <db_name> To set an env. variable globally : srvctl set env -p <db_name> LANG=en To start/stop the Oracle Intelligent Agent: agentctl start To start/stop the Oracle Management Server: oemctl start oms oemctl status oms

oemctl stop oms To start an OEM Console : oemapp console

oemapp dbastudio To start Oracle http server (Apache): OH/Apache/Apache/bin/apachectl start

CONFIGURE LISTENER.ORA / SQLNET.ORA / TNSNAMES.ORA

Use netca and/or netmgr to check the configuration of the listener and configure Oracle Net services (by default the Net service may be equal to the global database name (see instance parameter service_names ).

CONFIGURE ORACLE ENTERPRISE MANAGER

Use the Java assistant emca to configure the Oracle Management Server, then start it : oemctl start oms Enter a username/password with DBA privileges to connect to the instance where the repository is to be set up. The first administrator for the domain will be sysman/oem_temp. For the console to be able to give a single system image of the cluster database, it is necessary to start the Intelligent Agent on each node and to “discover” the nodes.


- Start the Oracle Intelligent Agent on each node Add the following line for Oracle Intelligent Agent to /etc/snmpd.conf (this is specific to AIX). smux 0.0 “” 129.1.11.106 # OEM agent (IP address of current node) The SNMP master agent needs to be restarted: #stopsrc –s snmpd#startsrc –s snmpd

Then start the OEM agent: $agentctl start

- Check /etc/oratab

The file should contain a reference to the database name, not to the instance name. The last field should always be “N” on a RAC environment to avoid 2 instances of the same name to be started.

- Register the database with srvctl (this should not be necessary if the database was created by DBCA)

srvctl add db –p <db_name> –o <ORACLE_HOME path>srvctl add instance –p <db_name> –i <SID1> –n <node1>srvctl add instance –p <db_name> –i <SID2> –n <node1>

TROUBLESHOOTING Useful SQL:

SQL> SELECT * FROM V$ACTIVE_INSTANCES


APPENDIX A : SSA EXTERNAL DISKS AND ADAPTERS MICROCODE MANAGEMENT • SSA Disks The IBM SSA Customer support page resides onhttp://www.storage.ibm.com/hardsoft/products/ssa To list all the SSA disks of the cluster : lscfg | grep pdiskTo check the microcode level : lscfg –vl pdisknn (ROS Level and ID) To determine the disk model : lscfg –vl pdisknn (Machine Type and Model) Depending on the disk model, the latest level of microcode to download and install on all the SSA disks of the cluster is : Release of new disk drive microcode for AIX 4.3.3 (november 1st, 2001): DCHC(CUSM)->9911, DFHC(CUSJ)->9911DGHC(CUSJ)->9911, DGHC(CUSM)->9911DRVC(CUSH)->0023DRHC(CUSS)->0012DMVC(CUSN)->0070 From the site http://www.storage.ibm.com/hardsoft/products/ssa, download the file ssacode433.tar onto your system in your temporary directory How to apply a new disk drive microcode A. 1.Login as Root 2.cd to your temporary directory 3.Type tar -xvf ssacode433.tar 4.Run smit install 5.Select install & update software 6.Select install & update from ALL available software 7.Use the directory that you saved and unpacked the ssacode433.tar file into as the install device 8.Select all filesets in this directory for install 9.Execute the command B. 1. At the prompt, execute the command diag 2. Task Selection(Diagnostics, Advanced Diagnostics, Service Aids, etc.) 3. SSA Service Aids 4.Display/Download Disk Drive Microcode 5.Download Microcode to all SSA Physical Disk Drives 6.Continue with the microcode installation 7.No (because software is in /etc/microcode ) 8.Do you want to continue? Yes.... This will upgrade the microcode on all the disks with a lower level of microcode than the one installed in /etc/microcode during the phase A. • SSA adapter : Advanced SerialRAID Adapter 6225/6230

To install the new adapter microcode, download the file devices.pci.14109100.ucode (4.2.1.6), and install it as decribed above.

http://www.storage.ibm.com/hardsoft/products/ssa

http://www.storage.ibm.com/hardsoft/products/ssa


APPENDIX B : LOGICAL VOLUMES CREATION #!/bin/ksh

export primary_oravg=ora1vgexport secondary_oravg=ora2vg

# To be exectued on the primary node, which owns ssa_disk_11, ssa_disk_12 and ora1vg

# Creation of the volume group, with two disksmkvg -f -y $primary_oravg -s'16' ssa_disk_11 ssa_disk_12

# Creation of the logical volumes (raw devices)# The number is the size of the LV (number of 16MB physical partitions)mklv -y'systemlv' $primary_oravg 25 ssa_disk_11 # 400 MBmklv -y'undo1lv' $primary_oravg 32 ssa_disk_11 # 512 MBmklv -y'templv' $primary_oravg 7 ssa_disk_11 # 112 MBmklv -y'ctrl1lv' $primary_oravg 1 ssa_disk_11 # 16 MBmklv -y'redolog1_1lv' $primary_oravg 8 ssa_disk_11 # 128 MBmklv -y'redolog1_2lv' $primary_oravg 8 ssa_disk_11 # 128 MBmklv -y'userslv' $primary_oravg 8 ssa_disk_11 # 128 MBmklv -y'toolslv' $primary_oravg 1 ssa_disk_11 # 16 MBmklv -y'drsyslv' $primary_oravg 6 ssa_disk_11 # 96 MB

# To be exectued on the secondary node, which owns ssa_disk_21, ssa_disk_22 and ora2vg

# Creation of the volume group, with two disksmkvg -f -y $secondary_oravg -s'16' ssa_disk_21 ssa_disk_22

# Creation of the logical volumes (raw devices)# The number is the size of the LV (number of 16MB physical partitions)mklv -y'ctrl2lv' $secondary_oravg 1 ssa_disk_21 # 16 MBmklv -y'undo2lv' $secondary_oravg 32 ssa_disk_21 # 512 MBmklv -y'redolog2_1lv' $secondary_oravg 8 ssa_disk_21 # 128 MBmklv -y'redolog2_2lv' $secondary_oravg 8 ssa_disk_21 # 128 MBmklv -y'oemrepolv' $secondary_oravg 10 ssa_disk_21 # 160 MBmklv -y'indexlv' $secondary_oravg 5 ssa_disk_21 # 80 MBmklv -y'examplelv' $secondary_oravg 10 ssa_disk_21 # 160 MBmklv -y'spfilelv' $secondary_oravg 1 ssa_disk_21 # 16 MBmklv -y'srvconfiglv' $secondary_oravg 7 ssa_disk_21 # 112 MBmklv -y'cmwlitelv' $secondary_oravg 7 ssa_disk_21 # 112 MB

# To be exectued on the primary node, to recognize ora2vg on the primary nodevaryoffvg $primary_oravgredefinevg -d ssa_disk_21 $secondary_oravgvaryonvg $primary_oravg

chown oracle.dba /dev/*vsd*chmod go+rw /dev/*vsd*

# To be exectued on the secondary node, to recognize ora1vg on the secondary nodevaryoffvg $secondary_oravgredefinevg -d ssa_disk_11$primary_oravgvaryonvg $secondary_oravg

chown oracle.dba /dev/*vsd*chmod go+rw /dev/*vsd*


APPENDIX C : VSD MANAGEMENT ON SP NODES, USING PSSP 3.2 #!/bin/ksh

#____________________________________________________________________# Procedures#____________________________________________________________________

vsdnode_proc (){

/usr/lpp/csd/bin/vsdnode $*}

vsdvg_proc (){

/usr/lpp/csd/bin/vsdvg $*}

defvsd_proc (){

/usr/lpp/csd/bin/defvsd $*}

vsdconfig_proc (){

/usr/lpp/csd/bin/allvsdl $* -c /usr/lpp/csd/bin/cfgvsd}

vsdstart_proc (){

args=$1shift

while [ $# -gt 1 ]do

args="$args $1"shift

done

if [ "$1" = "Primary" ]then

ServerFlag="-p"elif [ "$1" = "Backup" ]

thenServerFlag="-b"

elseargs="$args $1"ServerFlag="-p"

fi

/usr/lpp/csd/bin/allvsdl $args -c /usr/lpp/csd/bin/startvsd $ServerFlag}

#____________________________________________________________________# Main#____________________________________________________________________

export primary_node_name=node1export secondary_node_name=node2

export primary_node_num=1export secondary_node_num=2

export primary_oravg=ora1vgexport secondary_oravg=ora2vg

# VSD node database informationvsdnode_proc "$primary_node_num $secondary_node_num" 'en0' '64' '256' '256' '48' '4096''131072' '4' '61440' "RAC_${primary_node_num}_${secondary_node_num}"


# VSD global Volume Group informationvsdvg_proc -g vsd_$primary_oravg $primary_oravg $primary_node_name $secondary_node_namevsdvg_proc -g vsd_$secondary_oravg $secondary_oravg $secondary_node_name $primary_node_name

# Define a virtual shared disk (logical volume)defvsd_proc systemlv vsd_$primary_oravg vsd_systemdefvsd_proc templv vsd_$primary_oravg vsd_tempdefvsd_proc undolv vsd_$primary_oravg vsd_undodefvsd_proc log1lv vsd_$primary_oravg vsd_log1defvsd_proc ctrl1lv vsd_$primary_oravg vsd_ctrl1defvsd_proc userslv vsd_$primary_oravg vsd_usersdefvsd_proc toolslv vsd_$primary_oravg vsd_tools

defvsd_proc log2lv vsd_$secondary_oravg vsd_log2defvsd_proc ctrl2lv vsd_$secondary_oravg vsd_ctrl2defvsd_proc oemrepolv vsd_$secondary_oravg vsd_oemrepodefvsd_proc indexlv vsd_$secondary_oravg vsd_indexdefvsd_proc examplelv vsd_$secondary_oravg vsd_exampledefvsd_proc spfilelv vsd_$secondary_oravg vsd_spfiledefvsd_proc srvmconfiglv vsd_$secondary_oravg vsd_srvmconfig

# Configure a VSD on 2 nodesvsdconfig_proc -v 'vsd_system' -n $primary_node_num $secondary_node_numvsdconfig_proc -v 'vsd_temp' -n $primary_node_num $secondary_node_numvsdconfig_proc -v 'vsd_undo' -n $primary_node_num $secondary_node_numvsdconfig_proc -v 'vsd_log1' -n $primary_node_num $secondary_node_numvsdconfig_proc -v 'vsd_ctrl1' -n $primary_node_num $secondary_node_numvsdconfig_proc -v 'vsd_users' -n $primary_node_num $secondary_node_numvsdconfig_proc -v 'vsd_tools' -n $primary_node_num $secondary_node_num

vsdconfig_proc -v 'vsd_log2' -n $primary_node_num $secondary_node_numvsdconfig_proc -v 'vsd_ctrl2' -n $primary_node_num $secondary_node_numvsdconfig_proc -v 'vsd_oemrepo' -n $primary_node_num $secondary_node_numvsdconfig_proc -v 'vsd_index' -n $primary_node_num $secondary_node_numvsdconfig_proc -v 'vsd_example' -n $primary_node_num $secondary_node_numvsdconfig_proc -v 'vsd_spfile' -n $primary_node_num $secondary_node_numvsdconfig_proc -v 'vsd_srvmconfig' -n $primary_node_num $secondary_node_num

# Start a single VSD on the cluster#vsdstart_proc -v '<vsd_name>' -n $primary_node_num $secondary_node_num

# Start all VSD on all nodesvsdstart_proc -v 'All_VSDs' -n $primary_node_num $secondary_node_num


APPENDIX D : CLUSTER TOPOLOGY & NETWORK VERIFICATION OUTPUT. All the following outputs come from an HACMP configuration which was including a parallel instance database (HACMP concurrent mode) and a single instance database (HACMP rotating mode). (Information about the concurrent mode configuration is in bold.)

Cluster description : output from a ‘cllscf’ command.

Cluster Description of Cluster opsclusterCluster ID: 68There were 3 networks defined : giga_net, inter_net, rs232_netThere are 2 nodes in this cluster.

NODE gard:This node has 3 service interface(s):

Service Interface gard:IP address: 130.1.1.102Hardware Address:Network: giga_netAttribute: publicAliased Address?: False

Service Interface gard has no boot interfaces.Service Interface gard has no standby interfaces.

Service Interface gard_stby:IP address: 130.1.2.102Hardware Address:Network: inter_netAttribute: privateAliased Address?: False

Service Interface gard_stby has no boot interfaces.Service Interface gard_stby has no standby interfaces.

Service Interface tty_gard:IP address: /dev/tty2Hardware Address:Network: rs232_netAttribute: serialAliased Address?: False

Service Interface tty_gard has no boot interfaces.Service Interface tty_gard has no standby interfaces.

NODE palavas:This node has 3 service interface(s):

Service Interface palavas:IP address: 130.1.1.101Hardware Address:Network: giga_netAttribute: publicAliased Address?: False

Service Interface palavas has no boot interfaces.Service Interface palavas has no standby interfaces.

Service Interface palavas_stby:IP address: 130.1.2.101Hardware Address:Network: inter_netAttribute: privateAliased Address?: False

Service Interface palavas_stby has no boot interfaces.Service Interface palavas_stby has no standby interfaces.


Service Interface tty_palavas:IP address: /dev/tty3Hardware Address:Network: rs232_netAttribute: serialAliased Address?: False

Service Interface tty_palavas has no boot interfaces.Service Interface tty_palavas has no standby interfaces.

Breakdown of network connections:

Connections to network giga_netNode gard is connected to network giga_net by these interfaces:

gard

Node palavas is connected to network giga_net by these interfaces:palavas

Connections to network inter_netNode gard is connected to network inter_net by these interfaces:

gard_stby

Node palavas is connected to network inter_net by these interfaces:palavas_stby

Connections to network rs232_netNode gard is connected to network rs232_net by these interfaces:

tty_gard

Node palavas is connected to network rs232_net by these interfaces:tty_palavas

Cluster network interfaces : output from a ‘cllsif’ command.

{palavas:root}/ ->cllsifAdapter Type Network Net Type Attribute Node IP Address Hardware Address Interface Name GlobalName Netmask

gard service giga_net ether public gard 130.1.1.102gard_stby service inter_net ether private gard 130.1.2.102tty_gard service rs232_net rs232 serial gard /dev/tty2palavas service giga_net ether public palavas 130.1.1.101palavas_stby service inter_net ether private palavas 130.1.2.101tty_palavas service rs232_net rs232 serial palavas /dev/tty3

Cluster resource group : output from a ‘clshowres’ command.

Resource Group Name ops_resNode Relationship concurrentParticipating Node Name(s) palavas gardDynamic Node PriorityService IP LabelFilesystemsFilesystems Consistency Check fsckFilesystems Recovery Method sequentialFilesystems/Directories to be exportedFilesystems to be NFS mountedNetwork For NFS MountVolume GroupsConcurrent Volume Groups opsvgDisksConnections Services


Fast Connect ServicesShared Tape ResourcesApplication ServersHighly Available Communication LinksMiscellaneous DataAutomatically Import Volume Groups falseInactive Takeover falseCascading Without Fallback false9333 Disk Fencing falseSSA Disk Fencing falseFilesystems mounted before IP configured falseRun Time Parameters:

Node Name palavasDebug Level highHost uses NIS or Name Server falseFormat for hacmp.out Standard

Node Name gardDebug Level highHost uses NIS or Name Server falseFormat for hacmp.out Standard


APPENDIX E : INSTALLATION DIRECTORIES. --- /etc oratab --- /usr /local/bin oraenv, coraenv, dbhome --- /etc srvConfig.loc oraInst.loc --- /mnt /oracle --- ($ORACLE_BASE) ex: /u

/admin /<SID> / bdump cdump create pfile udump … /product /9.0.1./ bin dbs rdbms sqlplus ... /oraInventory /jre/1.1.8 /oui


APPENDIX F : RAC DATABASE CREATION SCRIPT & INIT.ORA. clustdb.sql This database creation script is provided with the Oracle 9i software. We have shortened this script in order to provide you only main excerpts. You can find the whole script in your $ORACLE_HOME/svrm/admin directory. RemRem $Header: clustdb.sql 08-may-2001.10:10:37 rajayar Exp $RemRemRem clustdb.sqlRemRem Copyright (c) Oracle Corporation 1999, 2000. All Rights Reserved.RemRem NAMERem clustdb.sql - Example database creation scriptRemRem DESCRIPTIONRem Creates a RAC database on UnixRemRem NOTESREM ******************************************************************REM ** UNIX clustdb.SQL VersionREM **REM ** Please update this file to reflect the correct values forREM ** 1) The init.ora file in startup nomount pfile=REM ** 2) The sysdba account and password if not using connect / as sysdbaREM ** (Note: The connect / as sysdba statement occurs multipleREM ** times in the sql script below)REM ** 3) The location of the sql scripts, to reflect your ORACLE_HOME.REM ** 4) The raw partition names(Symbolic link name) for log and data filesREM ** Note: This script will add two additional log files for the secondREM ** node. If your cluster will contain more nodes(instances) youREM ** must create and enable the additional logfiles for those node.REM ** 5) The name of the database in the "CREATE DATABASE " statement.REM ** 6) The character and national character sets for the databse, seeREM ** "CREATE DATABASE " statement.REM ** 6) The size of tablespaces, if you would like to increase orREM ** decrease the default size.REM **REM ******************************************************************

spool createdb.log

set echo onconnect / as sysdbastartup nomount pfile="%ORACLE_BASE%/admin/clustdb/pfile/init.ora"CREATE DATABASE clustdbCONTROLFILE REUSEMAXLOGMEMBERS 5MAXLOGHISTORY 100MAXDATAFILES 254MAXINSTANCES 32MAXLOGFILES 64DATAFILE '/dev/vx/rdsk/oracle_dg/clustdb_raw_system_400m' SIZE 325M REUSE AUTOEXTENDON NEXT 10240K MAXSIZE UNLIMITEDUNDO TABLESPACE "UNDOTBS1" DATAFILE'/dev/vx/rdsk/oracle_dg/clustdb_raw_undotbs1_290m' SIZE 200M REUSE AUTOEXTEND ONNEXT 5120K MAXSIZE UNLIMITEDLOGFILE GROUP 1 ('/dev/vx/rdsk/oracle_dg/clustdb_raw_log11_120m') REUSE,GROUP 2 ('/dev/vx/rdsk/oracle_dg/clustdb_raw_log12_120m') REUSE


CHARACTER SET US7ASCIINATIONAL CHARACTER SET AL16UTF16;spool off

spool createdb1.logset echo on

REM *** WHEN USING UNDO_MANAGEMENT=MANUAL, DELETE THE UNDO TABLESPACE ...REM LINE FROM THE CREATE DATABASE COMMAND AND UNCOMMENT THE FOLLOWINGREM SQL STATEMENT FOR RBS TABLESPACE.REM CREATE TABLESPACE RBS DATAFILE '/dev/vx/rdsk/oracle_dg/clustdb_rbs1'REM SIZE 520M REUSE MINIMUM EXTENT 512K;

REM ********** TABLESPACE FOR USER **********CREATE TABLESPACE USERS DATAFILE '/dev/vx/rdsk/oracle_dg/clustdb_raw_users_120m'SIZE 25M REUSEAUTOEXTEND ON NEXT 1280K MAXSIZE UNLIMITED MANAGEMENT LOCAL;

REM ********** TABLESPACE FOR TEMPORARY **********CREATE TABLESPACE TEMP DATAFILE '/dev/vx/rdsk/oracle_dg/clustdb_raw_temp_100m' SIZE40M REUSEAUTOEXTEND ON NEXT 640K MAXSIZE UNLIMITED TEMPORARY MANAGEMENT LOCAL;

REM ********** TABLESPACE FOR Tools **********CREATE TABLESPACE TOOLS DATAFILE '/dev/vx/rdsk/oracle_dg/clustdb_raw_temp_12m' SIZE10MREUSEAUTOEXTEND ON NEXT 320K MAXSIZE UNLIMITED MANAGEMENT LOCAL;

REM ********** TABLESPACE FOR INDEX **********CREATE TABLESPACE INDX DATAFILE '/dev/vx/rdsk/oracle_dg/clustdb_raw_indx_70m' SIZE25M REUSEAUTOEXTEND ON NEXT 1280K MAXSIZE UNLIMITED MANAGEMENT LOCAL;

REM ********** UNDO TABLESPACE FOR SECOND INSTANCE **********CREATE TABLESPACE UNDOTBS2 DATAFILE'/dev/vx/rdsk/oracle_dg/clustdb_raw_undotbs2_290m' SIZE 200M REUSE AUTOEXTEND ONNEXT 5120K MAXSIZE UNLIMITED EXTENT MANAGEMENT LOCAL;

REM **** comment out rollback_segments for automatic undo managementREM **** Rollback segments for our 2 nodes ***************REM spool psrbs.log;REM connect / as sysdbaREM create rollback segment rbs1_1 storage(initial 200K next 200K)REM tablespace RBS;REM alter rollback segment rbs1_1 online;REM create rollback segment rbs1_2 storage(initial 200K next 200K)REM tablespace RBS;REM alter rollback segment rbs1_2 online;REM create rollback segment rbs2_1 storage(initial 200K next 200K)REM tablespace RBS;REM alter rollback segment rbs2_1 online;REM create rollback segment rbs2_2 storage(initial 200K next 200K)REM tablespace RBS;REM alter rollback segment rbs2_2 online;REM spool offREM **** End Rollback segments for our 2 nodes ***************

alter user sys temporary tablespace TEMP;

REM **** Various SQL packages ***************@%ORACLE_HOME%/rdbms/admin/catalog.sql;@%ORACLE_HOME%/rdbms/admin/catexp7.sql


@%ORACLE_HOME%/rdbms/admin/catproc.sql@%ORACLE_HOME%/rdbms/admin/caths.sqlconnect system/manager@%ORACLE_HOME%/dbs/pupbld.sqlREM **** End various SQL packages ***************

REM ***** Scott's tables ***************connect / as sysdba@%ORACLE_HOME%/rdbms/admin/scott.sqlREM ***** End Scott table ***************

REM **** Demo support ***************connect / as sysdba@%ORACLE_HOME%/rdbms/admin/demo.sqlconnect / as sysdbaspool offREM **** End Demo ***************

REM **** Redo logfiles for the second instance ***************spool clustlog.log;connect / as sysdbaalter database add logfile thread 2group 3 '/dev/vx/rdsk/oracle_dg/clustdb_raw_log21_120m' reuse,group 4 '/dev/vx/rdsk/oracle_dg/clustdb_raw_log22_120m' reuse;

REM **** Enable the new logfile for thread 2alter database enable public thread 2;spool offREM **** End Logfiles for the second instance ***************

REM **** Cluster Database SQL support ***************spool catclust.log;connect / as sysdba@%ORACLE_HOME%/rdbms/admin/catclust.sqlspool offREM **** End Cluster Database SQL support ***************

connect / as sysdbaalter user system default tablespace TOOLS;alter user system temporary tablespace TEMP;

REM **** Auto extend is turned off *******

REM For undo_management=MANUAL, uncomment next line and comment the 2 linesREM after that.

REM alter database datafile '/dev/vx/rdsk/oracle_dg/clustdb_raw_rbs_580m' autoextendOFF;

alter database datafile '/dev/vx/rdsk/oracle_dg/clustdb_raw_undotbs1_290m'autoextend OFF;alter database datafile '/dev/vx/rdsk/oracle_dg/clustdb_raw_undotbs2_290m'autoextend OFF;

alter database datafile '/dev/vx/rdsk/oracle_dg/clustdb_raw_system_400m' autoextendOFF;

alter database datafile '/dev/vx/rdsk/oracle_dg/clustdb_raw_temp_20m' autoextendOFF;


alter database datafile '/dev/vx/rdsk/oracle_dg/clustdb_raw_users_120m' autoextendOFF;

alter database datafile '/dev/vx/rdsk/oracle_dg/clustdb_raw_indx_70m' autoextendOFF;exit;

init.ora

This init.ora file is provided with the Oracle 9i software (in the same directory as clustdb.sql).

## $Header: init.ora 04-may-2001.17:38:43 rajayar Exp $## Copyright (c) 1991, 2001, Oracle Corporation. All rights reserved. */################################################################################ Example INIT.ORA file## This file is provided by Oracle Corporation to help you customize# your RDBMS installation for your site. Important system parameters# are discussed, and example settings given.## Some parameter settings are generic to any size installation.# For parameters that require different values in different size# installations, three scenarios have been provided: SMALL, MEDIUM# and LARGE. Any parameter that needs to be tuned according to# installation size will have three settings, each one commented# according to installation size.## Use the following table to approximate the SGA size needed for the# three scenarious provided in this file:## -------Installation/Database Size------# SMALL MEDIUM LARGE# Block 2K 4500K 6800K 17000K# Size 4K 5500K 8800K 21000K## To set up a database that multiple instances will be using, use the# same file for all instance. Place all instance-specific parameters# at the end of the file using the <sid>.<parameter_name> = <value> syntax.# This way, when you change a public# parameter, it will automatically change on all instances. This is# necessary, since all instances must run with the same value for many# parameters. For example, if you choose to use private rollback segments,# these must be specified differently for each instance, but since all gc_*# parameters must be the same on all instances, they should be in one file.## INSTRUCTIONS: Edit this file and the other INIT files it calls for# your site, either by using the values provided here or by providing# your own.###############################################################################

# replace "clustdb" with your database namedb_name = clustdbcompatible=9.0.0

db_files = 1024 # INITIAL

control_files = ("/dev/vx/rdsk/oracle_dg/clustdb_raw_1_controlfile_110m","/dev/vx/rdsk/oracle_dg/clustdb_raw_16_controlfile2_110m")

db_file_multiblock_read_count = 8 # INITIAL


db_block_size=8192 # Datawarehouse# db_block_size=4096 # Transaction processingopen_cursors=300

#sort_area_size=524288 #Transaction processingsort_area_size=1048576 #Datawarehouselarge_pool_size=1048576 #Transaction processing, Datawarehousedb_cache_size=50331648 #Datawarehouse, Transaction processingjava_pool_size=67108864 #Datawarehouse, Transaction processing

# db_block_buffers = 200 # INITIAL

#shared_pool_size = 52428800 # INITIALshared_pool_size = 67108864 # datewarehouse, transaction processing

log_checkpoint_interval = 10000

processes = 150 # INITIAL

parallel_max_servers = 5

log_buffer = 8192 # INITIAL

# audit_trail = true # if you want auditing# timed_statistics = true # if you want timed statisticsmax_dump_file_size = 10240 # limit trace file size to 10 K each

# Uncommenting the line below will cause automatic archiving if archiving has# been enabled using ALTER DATABASE ARCHIVELOG.# log_archive_start = true# log_archive_dest = %ORACLE_HOME%/admin/clustdb/arch# log_archive_format = "%%ORACLE_SID%%T%TS%S.ARC"

# If using private rollback segments, place lines of the following# form at the end of this file:# <sid>.rollback_segments = (name1, name2)

# If using public rollback segments, define how many# rollback segments each instance will pick up, using the formula# # of rollback segments = transactions / transactions_per_rollback_segment# In this example each instance will grab 40/10 = 4:# transactions = 40# transactions_per_rollback_segment = 10

# Global Naming -- enforce that a dblink has same name as the db it connects toglobal_names = TRUE

# Edit and uncomment the following line to provide the suffix that will be# appended to the db_name parameter (separated with a dot) and stored as the# global database name when a database is created. If your site uses# Internet Domain names for e-mail, then the part of your e-mail address after# the '@' is a good candidate for this parameter value.

# db_domain = us.acme.com # global database name is db_name.db_domain

# Uncomment the following line if you wish to enable the Oracle Trace product# to trace server activity. This enables scheduling of server collections# from the Oracle Enterprise Manager Console.# Also, if the oracle_trace_collection_name parameter is non-null,# every session will write to the named collection, as well as enabling you# to schedule future collections from the console.

# oracle_trace_enable = TRUE

# define directories to store trace and alert filesbackground_dump_dest=%ORACLE_HOME%/admin/clustdb/bdumpuser_dump_dest=%ORACLE_HOME%/admin/clustdb/udump


db_block_size = 4096

remote_login_passwordfile = exclusive

# text_enable = TRUE

# The following parameters are needed for the Advanced Replication Option

job_queue_processes = 2# job_queue_processes = 4 #datawarehouse

# job_queue_interval = 10# job_queue_keep_connections = false

distributed_transactions = 5

open_links = 4

# The following parameters are instance-specific parameters that are# specified for two instances named clustdb1 and clustdb2

user_dump_dest=%ORACLE_HOME%/admin/clustdb/udumpundo_management=AUTO # For automatic undo management

# = MANUAL For manual/RBS undo managementcluster_database= truecluster_database_instances=2remote_listener=LISTENERS_CLUSTDB

# First instance specific parametersclustdb1.thread=1clustdb1.instance_name=clustdb1clustdb1.instance_number=1clustdb1.local_listener=listener_clustdb1# Comment out clustdb1.undo_tablespace and uncomment clustdb1.rollback_segments# when undo_management=MANUALclustdb1.undo_tablespace = UNDOTBS1# clustdb1.rollback_segments = (rbs1_1,rbs1_2)

# Second instance specific parametersclustdb2.thread=2clustdb2.instance_name = clustdb2clustdb2.instance_number = 2clustdb2.local_listener = listener_clustdb2# Comment out clustdb2.undo_tablespace and uncomment clustdb2.rollback_segments# when undo_management=MANUALclustdb2.undo_tablespace = UNDOTBS2# clustdb2.rollback_segments = (rbs2_1,rbs2_2)


APPENDIX G : SOME CONFIGURATION FILES… (EXAMPLES) This appendix provides examples of the configuration files that are mentioned in the document. Hosts file /etc/hosts

10.10.10.1 int-rac110.10.10.2 int-rac2172.128.194.1 rac1.mop.ibm.com rac1172.128.194.2 rac2.mop.ibm.com rac2

User equivalence (both files are identical; hosts.equiv is not sufficient for root) /etc/hosts.equiv

rac1rac2int-rac1int-rac2 Path to the server configuration raw device /etc/srvConfig.loc

srvconfig_loc=/dev/rvsd_srvconfig

Raw devices parameter file for the Database Configuration Assistant $HOME/dbca_raw_config

system=/dev/rvsd_systemtemp=/dev/rvsd_tempundo1=/dev/rvsd_undo1redo1_1=/dev/rvsd_redolog1_1redo1_2=/dev/rvsd_redolog1_2control1=/dev/rvsd_ctrl1users=/dev/rvsd_userstools=/dev/rvsd_toolsdrsys=/dev/rvsd_drsys

undo2=/dev/rvsd_undo2redo2_1=/dev/rvsd_redolog2_1redo2_2=/dev/rvsd_redolog2_2control2=/dev/rvsd_ctrl2oemrepo=/dev/rvsd_oemrepoindex=/dev/rvsd_indexexample=/dev/rvsd_examplespfile=/dev/rvsd_spfilesrvconfig=/dev/rvsd_srvconfigcmwlite=/dev/rvsd_cmwlite


Example listener.ora listener.ora

# LISTENER.ORA Network Configuration File: /oracle/product/9.0.1/network/admin/listener.ora# Generated by Oracle configuration tools.

LISTENER =(DESCRIPTION_LIST =(DESCRIPTION =(ADDRESS_LIST =(ADDRESS = (PROTOCOL = TCP)(HOST = RAC1)(PORT = 1521))

))

)

SID_LIST_LISTENER =(SID_LIST =(SID_DESC =(SID_NAME = PLSExtProc)(ORACLE_HOME = /oracle/prod/9.0.1)(PROGRAM = extproc)

)(SID_DESC =(ORACLE_HOME = /oracle/product/9.0.1)(SID_NAME = RAC1)

))

Example tnsnames.ora implementing TAF tnsnames.ora# TNSNAMES.ORA Network Configuration File: /oracle/product/9.0.1/network/admin/tnsnames.ora# Generated by Oracle configuration tools.

LISTENERS_TEST =(DESCRIPTION =(ADDRESS_LIST =(ADDRESS = (PROTOCOL = TCP)(HOST = RAC1)(PORT = 1521))(ADDRESS = (PROTOCOL = TCP)(HOST = RAC2)(PORT = 1521))

))

RAC2 =(DESCRIPTION =(ADDRESS_LIST =(ADDRESS = (PROTOCOL = TCP)(HOST = rac2)(PORT = 1521))

)(CONNECT_DATA =(SERVICE_NAME = RAC)(INSTANCE_NAME = RAC2)

))

RAC1 =(DESCRIPTION =(ADDRESS_LIST =(ADDRESS = (PROTOCOL = TCP)(HOST = RAC1)(PORT = 1521))

)(CONNECT_DATA =(SERVICE_NAME = RAC)(INSTANCE_NAME = RAC1)

))

RAC =(DESCRIPTION =(ADDRESS_LIST =(ADDRESS = (PROTOCOL = TCP)(HOST = rac1)(PORT = 1521))(ADDRESS = (PROTOCOL = TCP)(HOST = rac2)(PORT = 1521))

)(CONNECT_DATA =


(SERVICE_NAME = RAC)(FAILOVER_MODE =(TYPE = SELECT)(METHOD = PRECONNECT)(RETRIES = 20)(DELAY = 60)

))

)

INST1_HTTP =(DESCRIPTION =(ADDRESS_LIST =(ADDRESS = (PROTOCOL = TCP)(HOST = RAC1)(PORT = 1521))

)(CONNECT_DATA =(SERVER = SHARED)(SERVICE_NAME = MODOSE)(PRESENTATION = http://HRService)

))

Example listener.ora implementing TAF sqlnet.oraNAMES.DIRECTORY_PATH= (TNSNAMES)

Server configuration file exported by srvconfig srvconfig.txtRAC:/oracle/product/9.0.1:N

RAC.spfile = /dev/rvsd_3RAC.node_list = "int-rac1,int-rac2"RAC.inst_oracle_sid = (RAC1,RAC2)


APPENDIX H : ORACLE TECHNICAL NOTES This appendix provides some useful notes coming from Oracle support. These notes can be found in Metalink. Note #2064876.102 : How to setup High Availability Group Services (HAGS) on IBM AIX/RS6000. Article-ID: <Note:2064876.102>Circulation: PUBLISHED (EXTERNAL)Folder: server.OPS.ParallelserverTopic: - - - IBM RS6000 and SPTitle: How to setup High Availability Group Services (HAGS) on IBM

AIX/RS6000Document-Type: BULLETINImpact: LOWSkill-Level: CASUALServer-Version: 08.01.XX.XX.XXUpdated-Date: 04-FEB-2002 08:21:25References:Shared-Refs:Authors: BLEVE.USAttachments: NONEContent-Type: TEXT/PLAINKeywords: LMON; OPS; PARALLEL; SERVER; SUBCOMP-OPS;Products: 236/RDBMS (08.01.XX.XX.XX);Platforms: 319 (4.3);

Purpose=======

This article gives quick reference instructions on how to configure HighAvailabilityGroup Services (HAGS) on IBM AIX RS6000 for Oracle 8.1.X.

Scope and Application=====================

These instructions are helpful to any customer using Oracle on IBM AIX/RS6000on which HACMP is installed.

How to Configure High Availability Group Services (HAGS)========================================================

In order to configure High Availability Group Services (HAGS), you need to beconnected as root.

Do the following on all nodes that form the cluster:

1. Create the "hagsuser" group and place "oracle" into the "hagsuser" group:

Verify the group does not exists:

# grep hagsuser /etc/group

If this returns nothing do the following:

# smitty groups

Select "Add a Group" and fill in the following:


Group Name ----> hagsuserUSER list ----> oracle

You can take the defaults for the other settings. Also note thatafter the group is created you will have to log out and log back inas "oracle" to be sure "oracle" is part of the "hagsuser" group.

2. Change the permissions on the "cldomain" executable:

# chmod a+x /usr/sbin/cluster/utilities/cldomain

3. Change the group to "hagsuser" for the "svcsdsocket.<domain>" socket:

# chgrp hagsuser /var/ha/soc/grpsvcsdsocket.`/usr/sbin/cluster/utilities/cldomain`

4. Change the group permissions for the "grpsvcsdsocket.<domain>" socket:

# chmod g+w /var/ha/soc/grpsvcsdsocket.`/usr/sbin/cluster/utilities/cldomain`

The HAGS socket needs to be writeable by "oracle" and the "cldomain" executableneeds to be executable by "oracle". By configuring the group and permissionsfor the "grpsvcsdsocket.<domain>" file the instance will be able to communicatewith HAGS and the instance will mount.

References==========Oracle Installation Guide for AIX RS6000, release 8.1.5.

Search Words============OPS HAGS RS6000


Note # 115792.1 , explaining how to setup HACMP cluster interconnect adapter. Article-ID: <Note:115792.1>Circulation: PENDING_DELETE (EXTERNAL)Folder: server.OPS.ParallelserverTopic: - - - IBM RS6000 and SPTitle: AIX: ORA-600 [KCCSBCK_FIRST] starting up second OPS

InstanceDocument-Type: BULLETINImpact: LOWSkill-Level: NOVICEServer-Version: 08.01.06.0X to 08.01.06.0XUpdated-Date: 05-FEB-2002 13:10:49References:Shared-Refs:Authors: RKIRCHHE.DEAttachments: NONEContent-Type: TEXT/PLAINKeywords: HACMP; OPS;Errors: ORA-600; [KCCSBCK_FIRST];Products: 5/RDBMS;Platforms: 319;

*************************************************************This article is being delivered in Draft form and may containerrors. Please use the MetaLink "Feedback" button to adviseOracle of any issues related to this article.*************************************************************

PURPOSE-------

This article helps to resolve problems with Oracle Parallel Serverstartup related to HACMP configuration

SCOPE & APPLICATION-------------------

How to setup HACMP cluster interconnect adapter

-----------------------------

Oracle Parallel Server software is successfully installed.The first OPS instance starts without errors.

Trying to start a second OPS instance on another cluster nodefails with ORA-600 [KCCSBCK_FIRST].

$ORACLE_HOME/bin/lsnodes will list all cluster nodes.

/usr/sbin/cluster/diag/clverify doesn't show any errors.

Check HACMP interconnect network adapter configuration with/usr/sbin/cluster/utilities/cllsif

Adapter Type Network Net Type Attribute Node IPAddresspfpdb3 service pfpdb3 ether private pfpdb311.2.18.24pfpdb4 service pfpdb4 ether private pfpdb411.2.18.3

The network parameter doesn't match. It has to be identical for both adapters.

cllsif on a working configuration should look like this:


Adapter Type Network Net Type Attribute Node IPAddresspfpdb3 service pfpdb ether private pfpdb311.2.18.24pfpdb4 service pfpdb ether private pfpdb411.2.18.3

RELATED DOCUMENTS-----------------

<List related manuals, articles and other documents.>

cookbookaixv1r1

Documents