oracle 11gr2 two node real application cluster installation step by step guide · pdf...

Mohamed Azar | http://mohamedazar.wordpress.com DBA, Blogger, Contributor, Book Reviewer.

Oracle 11gR2 Two Node Real Application Cluster

installation step by step guide on Linux using

Vmware Server 2.0

Mohamed Azar.

Database Administrator, Blogger, Contributor ,Book Reviewer.

http://mohamedazar.wordpress.com

reach me at [email protected]

http://mohamedazar.wordpress.com/

mailto:[email protected]


Step 1: Pre-requisites

Before Start my installation, the below recommanded Memory Configuration needed for you. In this

case onle you can smoothly install RAC .

If you’re plan to Install Two node RAC on your virtual machine, Each node want atleast minimum 2GB

RAM as per Oracle 11gR2 Installation recommand. So you need atelast minimum 6GB RAM for your

laptop. 4 GB for Virtual Machine, remaining for your windows .

Recommanded Browser :

Mozilla Firefox 3.5 or Minimum version. The upgrade version nor support for Vmware Server 2.0.

Internet Explorer 7 or 6, I hope IE 9 not support for VMware Server 2.0.

Step 2: Download Software

Download VMware Server and Install

http://downloads.vmware.com/d/info/datacenter_downloads/vmware_server/2_0?hl=en&rlz=&q=vm

ware%20server%20download&aq=f&aqi=g10&aql=&oq=

Download Oracle Enterprise Linux 32 Bit

https://edelivery.oracle.com/linux

Download Oracle Software

Grid Infrastructure and Database Software

http://www.oracle.com/technetwork/database/enterprise-edition/downloads/112010-linuxsoft-

085393.html

Step 3: Install Oracle Enterprise Linux

We need to Install Oracle Enterprise Linux for Vmware Server infrastructure

environment. Follow procedure for further installation.

Create Virtual Machine

Name of the Virtual Machine racha1

http://downloads.vmware.com/d/info/datacenter_downloads/vmware_server/2_0?hl=en&rlz=&q=vmware%20server%20download&aq=f&aqi=g10&aql=&oq=

http://downloads.vmware.com/d/info/datacenter_downloads/vmware_server/2_0?hl=en&rlz=&q=vmware%20server%20download&aq=f&aqi=g10&aql=&oq=

https://edelivery.oracle.com/linux

http://www.oracle.com/technetwork/database/enterprise-edition/downloads/112010-linuxsoft-085393.html

http://www.oracle.com/technetwork/database/enterprise-edition/downloads/112010-linuxsoft-085393.html


Choose Linux Operating System Red Hat Enterprise Linux 5(32-bit).


Allocate Virtual Machine RAM memory 2048MB (Oracle 11gR2 recommand)

Create New Virtual DISK (Hard Disk) , Size 30 GB


Add a Network Adapter first with bridged Network (RAC Environment We need to configure TWO

adapter, Add another adapter later ).


If you want to use Linux CD – Choose Physical Drive OR

If you plan to use ISO image file, you can use ISO image and after chooses it, you can select your first

disc ISO image using browser option. The software file should be in Virtual Machine Datastore.


Add a USB Controller


Add another one Network Adapter using with bridged network.

Now new virtual machine hardware configuration finished.


Power on Virtual Machine :


Click on console window, the new remote console window will appear. If it’s not appear , it will ask

you to install REMOTE plugin console. Just you install it.

Click Skip option.


Select eth0 put it manual IP , disable IPv6

Your server name with localdomain name


Select eth1 ipaddress


GNOME Desktop Environment

Editors

Graphical Internet

Text-based Internet

Development Libraries

Development Tools

Server Configuration Tools

Administration Tools

Base

System Tools

X Window System


Installation Completed.

Step 4: Install VMware tools , Click right side panel Install VMware Tools


Go to Linux node 1 machine, Install vmware tools like below

[root@racha1 ~]# cd /media/VMware\ Tools/ [root@racha1 ~]# cp VMwareTools-7.7.6-203138.tar.gz /tmp cd /tmp [root@racha1 tmp]# tar -zxvf VMwareTools-7.7.6-203138.tar.gz [root@racha1 tmp]# cd vmware-tools-distrib/ [root@racha1 vmware-tools-distrib]# ls bin doc etc FILES INSTALL installer lib vmware-install.pl [root@racha1 vmware-tools-distrib]# ./vmware-install.pl Creating a new VMware Tools installer database using the tar4 format. Installing VMware Tools. In which directory do you want to install the binary files? [/usr/bin] What is the directory that contains the init directories (rc0.d/ to rc6.d/)? [/etc/rc.d] **Here choose virtual machine screen resolution also and after installation complete, Reboot machine.

vmware-toolbox

Step 5 : Oracle Installation Prerequisites.

Make sure the shared memory file system is big enough for Automatic Memory Manager to work.


[root@racha1 ~]# umount tmpfs

[root@racha1 ~]# mount -t tmpfs shmfs -o size=1500m /dev/shm

[root@racha1 ~]# vi /etc/fstab

tmpfs /dev/shm tmpfs size=1500m 0 0

Install RPM Package :

[root@racha1 rpm]# ls

binutils-2.17.50.0.6-14.el5.i386.rpm

compat-libstdc++-33-3.2.3-63.i386.rpm

elfutils-libelf-devel-0.137-3.el5.i386.rpm

elfutils-libelf-devel-static-0.137-3.el5.i386.rpm

gcc-4.1.2-48.el5.i386.rpm

gcc-c++-4.1.2-48.el5.i386.rpm

glibc-devel-2.5-49.i386.rpm

glibc-headers-2.5-49.i386.rpm

kernel-headers-2.6.18-194.el5.i386.rpm

libacl-devel-2.2.39-6.el5.i386.rpm

libaio-devel-0.3.106-3.2.i386.rpm

libgomp-4.4.0-6.el5.i386.rpm

libstdc++-devel-4.1.2-48.el5.i386.rpm

sysstat-7.0.2-3.el5.i386.rpm

unixODBC-2.2.11-7.1.i386.rpm

unixODBC-devel-2.2.11-7.1.i386.rpm

[root@racha1 rpm]# rpm -Uvh unixODBC-*

warning: unixODBC-2.2.11-7.1.i386.rpm: Header V3 DSA signature: NOKEY, key ID 1e5e0159

Preparing... ########################################### [100%]

1:unixODBC ########################################### [ 50%]

2:unixODBC-devel ########################################### [100%]

[root@racha1 rpm]#

Likewise Install Other rpm packages also.

Add ipaddress in /etc/hosts file

[root@racha1 ~]# vi /etc/hosts

# Do not remove the following line, or various programs

# that require network functionality will fail.

127.0.0.1 localhost.localdomain localhost

# Public

192.168.2.201 racha1.localdomain racha1


192.168.2.202 racha2.localdomain racha2

# Private

10.10.1.1 racha1-priv.localdomain racha1-priv

10.10.1.2 racha2-priv.localdomain racha2-priv

# Virtual

192.168.2.211 racha1-vip.localdomain racha1-vip

192.168.2.212 racha2-vip.localdomain racha2-vip

# SCAN

192.168.2.301 rac-scan.localdomain rac-scan

The SCAN address should not really be defined in the hosts file. Instead is should be defined on

the DNS to round-robin between 3 addresses on the same subnet as the public IPs. For this

installation, we will compromise and use the hosts file.

More Detail about SCAN

http://www.oracleracsig.org/pls/apex/Z?p_url=RAC_SIG.download_my_file?p_file=1004247&p_id=

1004247&p_cat=documents_new&p_user=YURY&p_company=994323795175833

/etc/sysctl.conf

[root@racha1 ~]# vi /etc/sysctl.conf

fs.aio-max-nr = 1048576

fs.file-max = 6815744

kernel.shmall = 2097152

kernel.shmmax = 1054504960

kernel.shmmni = 4096

# semaphores: semmsl, semmns, semopm, semmni

kernel.sem = 250 32000 100 128

net.ipv4.ip_local_port_range = 9000 65500

net.core.rmem_default=262144

net.core.rmem_max=4194304

net.core.wmem_default=262144

net.core.wmem_max=1048586

[root@racha1 ~]# /sbin/sysctl -p

net.ipv4.ip_forward = 0

net.ipv4.conf.default.rp_filter = 1

net.ipv4.conf.default.accept_source_route = 0

kernel.sysrq = 0

kernel.core_uses_pid = 1

net.ipv4.tcp_syncookies = 1

kernel.msgmnb = 65536

http://www.oracleracsig.org/pls/apex/Z?p_url=RAC_SIG.download_my_file?p_file=1004247&p_id=1004247&p_cat=documents_new&p_user=YURY&p_company=994323795175833

http://www.oracleracsig.org/pls/apex/Z?p_url=RAC_SIG.download_my_file?p_file=1004247&p_id=1004247&p_cat=documents_new&p_user=YURY&p_company=994323795175833


kernel.msgmax = 65536

fs.aio-max-nr = 1048576

fs.file-max = 6815744

kernel.shmall = 2097152

kernel.shmmax = 1054504960

kernel.shmmni = 4096

kernel.sem = 250 32000 100 128

net.ipv4.ip_local_port_range = 9000 65500

net.core.rmem_default = 262144

net.core.rmem_max = 4194304

net.core.wmem_default = 262144

net.core.wmem_max = 1048586

[root@racha1 ~]#

[root@racha1 ~]# vi /etc/security/limits.conf

oracle soft nproc 2047

oracle hard nproc 16384

oracle soft nofile 1024

oracle hard nofile 65536

[root@racha1 ~]# vi /etc/pam.d/login

session required pam_limits.so

[root@racha1 ~]# cat /etc/selinux/config

# This file controls the state of SELinux on the system.

# SELINUX= can take one of these three values:

# enforcing - SELinux security policy is enforced.

# permissive - SELinux prints warnings instead of enforcing.

# disabled - SELinux is fully disabled.

SELINUX=disabled

# SELINUXTYPE= type of policy in use. Possible values are:

# targeted - Only targeted network daemons are protected.

# strict - Full SELinux protection.

SELINUXTYPE=targeted

[root@racha1 ~]#


The Oracle Cluster Time Synchronization Service (ctssd) can synchronize the times of the RAC

nodes. In this case we will deconfigure NTP.

[root@racha1 ~]# service ntpd stop

Shutting down ntpd: [FAILED]

[root@racha1 ~]# chkconfig ntpd off

[root@racha1 ~]# mv /etc/ntp.conf /etc/ntp.conf.org

[root@racha1 ~]# rm /var/run/ntpd.pid

rm: cannot lstat `/var/run/ntpd.pid': No such file or directory

User and Group Add

[root@racha1 ~]# groupadd -g 500 oinstall

[root@racha1 ~]# groupadd -g 501 dba

[root@racha1 ~]# groupadd -g 502 oper

[root@racha1 ~]# groupadd -g 503 asmadmin

[root@racha1 ~]# useradd -u 1000 -g oinstall -G dba,oper,asmadmin oracle

[root@racha1 ~]# passwd oracle

Changing password for user oracle.

New UNIX password:

BAD PASSWORD: it is based on a dictionary word

Retype new UNIX password:

passwd: all authentication tokens updated successfully.

[root@racha1 ~]#

Make Directory

[root@racha1 ~]# mkdir -p /u01/app/11.2.0/grid

[root@racha1 ~]# mkdir -p /u01/app/oracle/product/11.2.0/dbhome_1

[root@racha1 ~]# chown -R oracle:oinstall /u01

[root@racha1 ~]# chmod -R 775 /u01

Bash Profile

Log on as Oracle User.

# Oracle Settings

TMP=/tmp; export TMP

TMPDIR=$TMP; export TMPDIR


ORACLE_HOSTNAME=racha1.localdomain; export ORACLE_HOSTNAME

ORACLE_UNQNAME=RAC; export ORACLE_UNQNAME

ORACLE_BASE=/u01/app/oracle; export ORACLE_BASE

ORACLE_HOME=$ORACLE_BASE/product/11.2.0/dbhome_1; export ORACLE_HOME

ORACLE_SID=RAC1; export ORACLE_SID

ORACLE_TERM=xterm; export ORACLE_TERM

BASE_PATH=/usr/sbin:$PATH; export BASE_PATH

PATH=$ORACLE_HOME/bin:$BASE_PATH; export PATH

LD_LIBRARY_PATH=$ORACLE_HOME/lib:/lib:/usr/lib; export LD_LIBRARY_PATH

CLASSPATH=$ORACLE_HOME/JRE:$ORACLE_HOME/jlib:$ORACLE_HOME/rdbms/jlib; export

CLASSPATH

if [ $USER = "oracle" ]; then

if [ $SHELL = "/bin/ksh" ]; then

ulimit -p 16384

ulimit -n 65536

else

ulimit -u 16384 -n 65536

fi

fi

Shutdown NODE 1 machine.

[root@racha1 ~]#shutdown –h now

Create three shared DISK

Making directory for sharing disk in host OS.


C:\Users\Azar>mkdir d:\shareasm

On the VMware Infrastructure web access panel, click Add Hardware and click Add hard disk


10GB allocate for new harddisk


Create other two hard disk as the same above procedure.

After added three disks, Then see your racha1.vmx file , you need to modify like this.


Now Restart the NODE 1 racha1 machine and after restarted , partition your disks like below

[root@racha1 ~]# cd /dev/

[root@racha1 dev]# ls sd*

sda sda1 sda2 sdb sdc sdd

[root@racha1 dev]# fdisk sdb

Device contains neither a valid DOS partition table, nor Sun, SGI or OSF disklabel

Building a new DOS disklabel. Changes will remain in memory only,

until you decide to write them. After that, of course, the previous

content won't be recoverable.

The number of cylinders for this disk is set to 1305.

There is nothing wrong with that, but this is larger than 1024,

and could in certain setups cause problems with:


1) software that runs at boot time (e.g., old versions of LILO)

2) booting and partitioning software from other OSs

(e.g., DOS FDISK, OS/2 FDISK)

Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite)

Command (m for help): Command (m for help): n

Command action

e extended

p primary partition (1-4)

p

Partition number (1-4): 1

First cylinder (1-1305, default 1):

Using default value 1

Last cylinder or +size or +sizeM or +sizeK (1-1305, default 1305):

Using default value 1305

Command (m for help): p

Disk sdb: 10.7 GB, 10737418240 bytes

255 heads, 63 sectors/track, 1305 cylinders

Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot Start End Blocks Id System

sdb1 1 1305 10482381 83 Linux

Command (m for help): w

The partition table has been altered!

Calling ioctl() to re-read partition table.

Syncing disks.

[root@racha1 dev]#

Do It as same again procedure for other remaining Disks.

ASMLIB Install

Download ASMLIB rpm package for your OS requirements

http://www.oracle.com/technetwork/topics/linux/asmlib/index-101839.html

[root@racha1 ~]# cp /media/AZAR/asmlib/*.rpm /install/asmlib

[root@racha1 ~]# cd /install/asmlib

[root@racha1 asmlib]# ls

http://www.oracle.com/technetwork/topics/linux/asmlib/index-101839.html


oracleasm-2.6.18-194.el5-2.0.5-1.el5.i686.rpm

oracleasmlib-2.0.4-1.el5.i386.rpm

oracleasm-support-2.1.3-1.el5.i386.rpm

[root@racha1 asmlib]# rpm -Uvh oracleasm-support-2.1.3-1.el5.i386.rpm

warning: oracleasm-support-2.1.3-1.el5.i386.rpm: Header V3 DSA signature: NOKEY, key ID 1e5e0159

Preparing... ########################################### [100%]

1:oracleasm-support ########################################### [100%]

[root@racha1 asmlib]# rpm -Uvh oracleasm-2.6.18-194.el5-2.0.5-1.el5.i686.rpm

warning: oracleasm-2.6.18-194.el5-2.0.5-1.el5.i686.rpm: Header V3 DSA signature: NOKEY, key ID

1e5e0159

Preparing... ########################################### [100%]

1:oracleasm-2.6.18-194.el########################################### [100%]

[root@racha1 asmlib]# rpm -Uvh oracleasm-2.6.18-194.el5-debuginfo-2.0.5-1.el5.i686.rpm

warning: oracleasm-2.6.18-194.el5-debuginfo-2.0.5-1.el5.i686.rpm: Header V3 DSA signature: NOKEY,

key ID 1e5e0159

Preparing... ########################################### [100%]

1:oracleasm-2.6.18-194.el########################################### [100%]

[root@racha1 ~]# oracleasm configure -i

Configuring the Oracle ASM library driver.

This will configure the on-boot properties of the Oracle ASM library

driver. The following questions will determine whether the driver is

loaded on boot and what permissions it will have. The current values

will be shown in brackets ('[]'). Hitting <ENTER> without typing an

answer will keep that current value. Ctrl-C will abort.

Default user to own the driver interface []: oracle

Default group to own the driver interface []: dba

Start Oracle ASM library driver on boot (y/n) [n]: y

Scan for Oracle ASM disks on boot (y/n) [y]: y

Writing Oracle ASM library driver configuration: done

[root@racha1 ~]# /usr/sbin/oracleasm init

Creating /dev/oracleasm mount point: /dev/oracleasm

Loading module "oracleasm": oracleasm

Mounting ASMlib driver filesystem: /dev/oracleasm


[root@racha1 ~]# /usr/sbin/oracleasm createdisk DISK1 /dev/sdb1

Writing disk header: done

Instantiating disk: done

[root@racha1 ~]# /usr/sbin/oracleasm createdisk DISK2 /dev/sdc1



[root@racha1 ~]# /usr/sbin/oracleasm createdisk DISK3 /dev/sdd1



Unzip Grid Infrastruture Software

[root@racha1 ~]# chown -R oracle:oinstall /install

[root@racha1 ~]# cd /install

[root@racha1 install]# ls

asmlib linux_11gR2_grid.zip rpm

[root@racha1 install]# unzip linux_11gR2_grid.zip

Shutdown Node 1 racha1

[root@racha1 install]# shutdown -h now

Copy racha1 the same location and rename folder racha2

Edit the contents of d:\racha2\racha1.vmx

displayName = "racha2"


In the VMware Infrastructure Web Access Console, select the "Virtual Machine > Add Virtual

Machine to Inventory" and browser for the d:\racha2\racha1.vmx

Start the racha2 machine, just choose i_copied it option and click ok.

Now the another cloned copy of node1 virtual machine running.

Log on to the virtual machine racha2

System > Administration > Network

Remove the devices with the "%.bak" nicknames. To do this, highlight a device, deactivate, then

delete it. This will leave just the regular "eth0" and "eth1" devices. Highlight the "eth0" interface

and click the "Edit" button on the toolbar.


Eth1 Configure Add ip address, subnet mask and Don’t add default gateway ip address here.


If you want restart your network service ,you need to use below commands.

service network restart

Edit Bash_profile of Node 2 Oracle user path correct to ORACLE_SID and HOSTNAME

[root@racha1 ~]# su - oracle

[oracle@racha1 ~]$ vi .bash_profile

ORACLE_HOSTNAME=racha2.localdomain; export ORACLE_HOSTNAME

ORACLE_UNQNAME=RAC; export ORACLE_UNQNAME

ORACLE_BASE=/u01/app/oracle; export ORACLE_BASE

ORACLE_HOME=$ORACLE_BASE/product/11.2.0/dbhome_1; export ORACLE_HOME

ORACLE_SID=RAC2; export ORACLE_SID


PING NETWORK :

Node 1:

[root@racha1 ~]# ping -c 3 racha2

PING racha2.localdomain (192.168.2.202) 56(84) bytes of data.

64 bytes from racha2.localdomain (192.168.2.202): icmp_seq=1 ttl=64 time=62.2 ms



--- racha2.localdomain ping statistics ---

3 packets transmitted, 3 received, 0% packet loss, time 4053ms

rtt min/avg/max/mdev = 0.150/20.874/62.276/29.275 ms

[root@racha1 ~]# ping -c 3 racha2-priv

PING racha2-priv.localdomain (10.10.1.2) 56(84) bytes of data.

64 bytes from racha2-priv.localdomain (10.10.1.2): icmp_seq=1 ttl=64 time=0.923 ms



--- racha2-priv.localdomain ping statistics ---



[root@racha1 ~]#

Node 2:

[root@racha2 ~]# ping -c 3 racha1

PING racha1.localdomain (192.168.2.201) 56(84) bytes of data.





--- racha1.localdomain ping statistics ---



[root@racha2 ~]# ping -c 3 racha1-priv

PING racha1-priv.localdomain (10.10.1.1) 56(84) bytes of data.




SSH Configuration :

Manual SSH configuration not need for Oracle 11gr2, because of During grid infrastructure It will ask

SSH setup. But here I just configured ssh manually.

Node 1 :


[oracle@racha1 ~]$ mkdir ~/.ssh

[oracle@racha1 ~]$ chmod 700 ~/.ssh

[oracle@racha1 ~]$ /usr/bin/ssh-keygen -t rsa

Generating public/private rsa key pair.

Enter file in which to save the key (/home/oracle/.ssh/id_rsa):

Enter passphrase (empty for no passphrase):

Enter same passphrase again:

Your identification has been saved in /home/oracle/.ssh/id_rsa.


Your public key has been saved in /home/oracle/.ssh/id_rsa.pub.

The key fingerprint is:

cf:36:68:1b:be:18:51:7d:7a:67:ee:b7:ee:8e:ca:0b [email protected]

Node 2:


[oracle@racha2 ~]$ mkdir ~/.ssh

[oracle@racha2 ~]$ chmod 700 ~/.ssh

[oracle@racha2 ~]$ /usr/bin/ssh-keygen -t rsa

Generating public/private rsa key pair.

Enter file in which to save the key (/home/oracle/.ssh/id_rsa):

Enter passphrase (empty for no passphrase):

Enter same passphrase again:

Your identification has been saved in /home/oracle/.ssh/id_rsa.

Your public key has been saved in /home/oracle/.ssh/id_rsa.pub.

The key fingerprint is:

7b:69:c1:7d:7e:b1:e8:62:72:1d:3d:ee:9b:d6:26:c6 [email protected]

Node 1:

[oracle@racha1 ~]$ cd ~/.ssh

[oracle@racha1 .ssh]$ cat id_rsa.pub >> authorized_keys

[oracle@racha1 .ssh]$ scp authorized_keys racha2:/home/oracle/.ssh

The authenticity of host 'racha2 (192.168.2.202)' can't be established.

RSA key fingerprint is fd:dd:51:77:0d:46:69:f8:55:fe:2c:53:77:75:90:e0.

Are you sure you want to continue connecting (yes/no)? yes

Warning: Permanently added 'racha2,192.168.2.202' (RSA) to the list of known hosts.

oracle@racha2's password:




authorized_keys 100% 407 0.4KB/s 00:00

Node 2:

[oracle@racha2 ~]$ cd ~/.ssh

[oracle@racha2 .ssh]$ cat id_rsa.pub >> authorized_keys

[oracle@racha2 .ssh]$ scp authorized_keys racha1:/home/oracle/.ssh

The authenticity of host 'racha1 (192.168.2.201)' can't be established.

RSA key fingerprint is fd:dd:51:77:0d:46:69:f8:55:fe:2c:53:77:75:90:e0.

Are you sure you want to continue connecting (yes/no)? yes

Warning: Permanently added 'racha1,192.168.2.201' (RSA) to the list of known hosts.

oracle@racha1's password:

authorized_keys 100% 814 0.8KB/s 00:00

Check below for each node

Node 1:

[oracle@racha1 .ssh]$ ssh date racha1

[oracle@racha1 .ssh]$ ssh racha2 date

[oracle@racha1 .ssh]$ ssh racha2-priv date


Node 2:

[oracle@racha2 .ssh]$ ssh date racha1

[oracle@racha2 .ssh]$ ssh racha2 date



Cluvfy pre –requisites check

11gR2 we would probably use the "runcluvfy.sh" utility in the clusterware root directory to check the

prerequisites have been met.


[oracle@racha1 grid]$ ./runcluvfy.sh stage -pre crsinst -n racha1,racha2 -verbose

Performing pre-checks for cluster services setup

Checking node reachability...

Check: Node reachability from node "racha1"

Destination Node Reachable?

------------------------------------ ------------------------

racha1 yes

racha2 yes

Result: Node reachability check passed from node "racha1"

Checking user equivalence...

Check: User equivalence for user "oracle"

Node Name Comment

------------------------------------ ------------------------

racha1 passed

racha2 passed

Result: User equivalence check passed for user "oracle"

---- -- -- - -- -- -- - - ---- --- ---

------ -- -- - -- - -- - ---- - - - -

Starting Clock synchronization checks using Network Time Protocol(NTP)...

NTP Configuration file check started...

Network Time Protocol(NTP) configuration file not found on any of the nodes. Oracle Cluster Time

Synchronization Service(CTSS) can be used instead of NTP for time synchronization on the

cluster nodes

Result: Clock synchronization check using Network Time Protocol(NTP) passed

Pre-check for cluster services setup was successful.

Oracle Grid Infrastructure installation

[oracle@racha1 grid]$

Install Grid Infrastructure

[root@racha1 ~]# chown -R oracle:oinstall /install/grid

[root@racha1 ~]# xhost +

access control disabled, clients can connect from any host



[oracle@racha1 ~]$ cd /install/grid

[oracle@racha1 grid]$ ls

doc response runcluvfy.sh sshsetup welcome.html

install rpm runInstaller stage

[oracle@racha1 grid]$ ./runInstaller

Starting Oracle Universal Installer...

Checking Temp space: must be greater than 80 MB. Actual 19061 MB Passed

Checking swap space: must be greater than 150 MB. Actual 4031 MB Passed

Checking monitor: must be configured to display at least 256 colors. Actual 16777216 Passed

Preparing to launch Oracle Universal Installer from /tmp/OraInstall2011-07-25_10-18-49AM. Please

wait ...[oracle@racha1 grid]$


Add Second node hostname and virtual ip name


Test SSH Connectivity, Here I was already configured SSH.


Choose ASM storage

Select Disks, If its not show, choose change recovery path and add correct disk path here. These disks

are should be oracle permission.


Run the above screenshot shown root script as a root user in both nodes like below.

Node 1:

[root@racha1 ~]# /u01/app/oraInventory/orainstRoot.sh

Changing permissions of /u01/app/oraInventory.

Adding read,write permissions for group.

Removing read,write,execute permissions for world.

Changing groupname of /u01/app/oraInventory to oinstall.

The execution of the script is complete.

[root@racha1 ~]#

Node 2:

[root@racha2 ~]# /u01/app/oraInventory/orainstRoot.sh

Changing permissions of /u01/app/oraInventory.

Adding read,write permissions for group.

Removing read,write,execute permissions for world.


Changing groupname of /u01/app/oraInventory to oinstall.

The execution of the script is complete.

[root@racha2 ~]#

Node 1:

[root@racha1 ~]# /u01/app/11.2.0/grid/root.sh

Running Oracle 11g root.sh script...

The following environment variables are set as:

ORACLE_OWNER= oracle

ORACLE_HOME= /u01/app/11.2.0/grid

Enter the full pathname of the local bin directory: [/usr/local/bin]:

Copying dbhome to /usr/local/bin ...

Copying oraenv to /usr/local/bin ...

Copying coraenv to /usr/local/bin ...

Creating /etc/oratab file...

Entries will be added to the /etc/oratab file as needed by

Database Configuration Assistant when a database is created

Finished running generic part of root.sh script.

Now product-specific root actions will be performed.

2011-07-25 14:43:57: Parsing the host name

2011-07-25 14:43:57: Checking for super user privileges

2011-07-25 14:43:57: User has super user privileges

Using configuration parameter file: /u01/app/11.2.0/grid/crs/install/crsconfig_params

Creating trace directory

LOCAL ADD MODE

Creating OCR keys for user 'root', privgrp 'root'..

Operation successful.

root wallet

root wallet cert

root cert export

peer wallet

profile reader wallet

pa wallet

peer wallet keys

pa wallet keys

peer cert request

pa cert request

peer cert

pa cert

peer root cert TP


profile reader root cert TP

pa root cert TP

peer pa cert TP

pa peer cert TP

profile reader pa cert TP

profile reader peer cert TP

peer user cert

pa user cert

Adding daemon to inittab

CRS-4123: Oracle High Availability Services has been started.

ohasd is starting

CRS-2672: Attempting to start 'ora.gipcd' on 'racha1'

CRS-2672: Attempting to start 'ora.mdnsd' on 'racha1'

CRS-2676: Start of 'ora.gipcd' on 'racha1' succeeded

CRS-2676: Start of 'ora.mdnsd' on 'racha1' succeeded

CRS-2672: Attempting to start 'ora.gpnpd' on 'racha1'

CRS-2676: Start of 'ora.gpnpd' on 'racha1' succeeded

CRS-2672: Attempting to start 'ora.cssdmonitor' on 'racha1'

CRS-2676: Start of 'ora.cssdmonitor' on 'racha1' succeeded

CRS-2672: Attempting to start 'ora.cssd' on 'racha1'

CRS-2672: Attempting to start 'ora.diskmon' on 'racha1'

CRS-2676: Start of 'ora.diskmon' on 'racha1' succeeded

CRS-2676: Start of 'ora.cssd' on 'racha1' succeeded

CRS-2672: Attempting to start 'ora.ctssd' on 'racha1'

CRS-2676: Start of 'ora.ctssd' on 'racha1' succeeded

ASM created and started successfully.

DiskGroup DATA created successfully.

clscfg: -install mode specified

Successfully accumulated necessary OCR keys.



CRS-2672: Attempting to start 'ora.crsd' on 'racha1'

CRS-2676: Start of 'ora.crsd' on 'racha1' succeeded

CRS-4256: Updating the profile

Successful addition of voting disk 0cf9af8190014f43bf110fdde04e726a.

Successfully replaced voting disk group with +DATA.

CRS-4256: Updating the profile

CRS-4266: Voting file(s) successfully replaced

## STATE File Universal Id File Name Disk group

-- ----- ----------------- --------- ---------

1. ONLINE 0cf9af8190014f43bf110fdde04e726a (/dev/oracleasm/disks/DISK1) [DATA]

Located 1 voting disk(s).

CRS-2673: Attempting to stop 'ora.crsd' on 'racha1'

CRS-2677: Stop of 'ora.crsd' on 'racha1' succeeded

CRS-2673: Attempting to stop 'ora.asm' on 'racha1'


CRS-2677: Stop of 'ora.asm' on 'racha1' succeeded

CRS-2673: Attempting to stop 'ora.ctssd' on 'racha1'

CRS-2677: Stop of 'ora.ctssd' on 'racha1' succeeded

CRS-2673: Attempting to stop 'ora.cssdmonitor' on 'racha1'

CRS-2677: Stop of 'ora.cssdmonitor' on 'racha1' succeeded

CRS-2673: Attempting to stop 'ora.cssd' on 'racha1'

CRS-2677: Stop of 'ora.cssd' on 'racha1' succeeded

CRS-2673: Attempting to stop 'ora.gpnpd' on 'racha1'

CRS-2677: Stop of 'ora.gpnpd' on 'racha1' succeeded

CRS-2673: Attempting to stop 'ora.gipcd' on 'racha1'

CRS-2677: Stop of 'ora.gipcd' on 'racha1' succeeded

CRS-2673: Attempting to stop 'ora.mdnsd' on 'racha1'

CRS-2677: Stop of 'ora.mdnsd' on 'racha1' succeeded















CRS-2672: Attempting to start 'ora.asm' on 'racha1'

CRS-2676: Start of 'ora.asm' on 'racha1' succeeded



CRS-2672: Attempting to start 'ora.evmd' on 'racha1'

CRS-2676: Start of 'ora.evmd' on 'racha1' succeeded



CRS-2672: Attempting to start 'ora.DATA.dg' on 'racha1'

CRS-2676: Start of 'ora.DATA.dg' on 'racha1' succeeded

CRS-2672: Attempting to start 'ora.registry.acfs' on 'racha1'

CRS-2676: Start of 'ora.registry.acfs' on 'racha1' succeeded

racha1 2011/07/25 14:57:56 /u01/app/11.2.0/grid/cdata/racha1/backup_20110725_145756.olr

Preparing packages for installation...

cvuqdisk-1.0.7-1

Configure Oracle Grid Infrastructure for a Cluster ... succeeded

Updating inventory properties for clusterware




The inventory pointer is located at /etc/oraInst.loc

The inventory is located at /u01/app/oraInventory

'UpdateNodeList' was successful.

[root@racha1 ~]#

Node 2:

[root@racha2 ~]# /u01/app/11.2.0/grid/root.sh

]Running Oracle 11g root.sh script...



ORACLE_HOME= /u01/app/11.2.0/grid


Creating ] directory...

Copying dbhome to ] ...

Copying oraenv to ] ...

Copying coraenv to ] ...





2011-07-26 09:35:41: Parsing the host name

2011-07-26 09:35:41: Checking for super user privileges

2011-07-26 09:35:41: User has super user privileges

Using configuration parameter file: /u01/app/11.2.0/grid/crs/install/crsconfig_params

LOCAL ADD MODE




Adding daemon to inittab

CRS-4123: Oracle High Availability Services has been started.

ohasd is starting

CRS-4402: The CSS daemon was started in exclusive mode but found an active CSS daemon on

node racha1, number 1, and is terminating

An active cluster was found during exclusive startup, restarting to join the cluster















CRS-2672: Attempting to start 'ora.drivers.acfs' on 'racha2'

CRS-2676: Start of 'ora.drivers.acfs' on 'racha2' succeeded





CRS-2672: Attempting to start 'ora.evmd' on 'racha2'

CRS-2676: Start of 'ora.evmd' on 'racha2' succeeded


racha2 2011/07/26 09:41:40 /u01/app/11.2.0/grid/cdata/racha2/backup_20110726_094140.olr

Preparing packages for installation...

cvuqdisk-1.0.7-1

Configure Oracle Grid Infrastructure for a Cluster ... succeeded

Updating inventory properties for clusterware



The inventory pointer is located at /etc/oraInst.loc

The inventory is located at /u01/app/oraInventory

'UpdateNodeList' was successful.

[root@racha2 ~]#

We expect the verification phase to fail with an error relating to the SCAN, assuming you are not

using DNS.


Check Cluster status for both nodes

[oracle@racha1 ~]$ export GRID_HOME=/u01/app/11.2.0/grid

[oracle@racha1 ~]$ export PATH=/u01/app/11.2.0/grid/bin

[oracle@racha1 ~]$ crsctl check cluster -all

**************************************************************

racha1:

CRS-4537: Cluster Ready Services is online

CRS-4529: Cluster Synchronization Services is online

CRS-4533: Event Manager is online

**************************************************************

racha2:

CRS-4537: Cluster Ready Services is online

CRS-4529: Cluster Synchronization Services is online

CRS-4533: Event Manager is online

[oracle@racha1 ~]$ olsnodes

racha1

racha2

[oracle@racha1 ~]$ crs_stat -t -v

Name Type R/RA F/FT Target State Host

----------------------------------------------------------------------

ora.DATA.dg ora....up.type 0/5 0/ ONLINE ONLINE racha1

ora....ER.lsnr ora....er.type 0/5 0/ ONLINE ONLINE racha1

ora....N1.lsnr ora....er.type 0/5 0/0 ONLINE ONLINE racha2


ora.asm ora.asm.type 0/5 0/ ONLINE ONLINE racha1

ora.eons ora.eons.type 0/3 0/ ONLINE ONLINE racha1

ora.gsd ora.gsd.type 0/5 0/ OFFLINE OFFLINE

ora....network ora....rk.type 0/5 0/ ONLINE ONLINE racha1

ora.oc4j ora.oc4j.type 0/5 0/0 OFFLINE OFFLINE

ora.ons ora.ons.type 0/3 0/ ONLINE ONLINE racha1

ora.rac.db ora....se.type 0/2 0/1 ONLINE ONLINE racha1

ora....SM1.asm application 0/5 0/0 ONLINE ONLINE racha1

ora....A1.lsnr application 0/5 0/0 ONLINE ONLINE racha1

ora.racha1.gsd application 0/5 0/0 OFFLINE OFFLINE

ora.racha1.ons application 0/3 0/0 ONLINE ONLINE racha1

ora.racha1.vip ora....t1.type 0/0 0/0 ONLINE ONLINE racha1

ora....SM2.asm application 0/5 0/0 ONLINE ONLINE racha2

ora....A2.lsnr application 0/5 0/0 ONLINE ONLINE racha2

ora.racha2.gsd application 0/5 0/0 OFFLINE OFFLINE

ora.racha2.ons application 0/3 0/0 ONLINE ONLINE racha2

ora.racha2.vip ora....t1.type 0/0 0/0 ONLINE ONLINE racha2

ora....ry.acfs ora....fs.type 0/5 0/ ONLINE ONLINE racha1

ora.scan1.vip ora....ip.type 0/0 0/0 ONLINE ONLINE racha2

[oracle@racha1 ~]$

Oracle 11gR2 database Installation

Go to Node 1

Unzip Oracle 11g Software database.

[oracle@racha1 install]$ unzip linux_11gR2_database_1of2.zip


[oracle@racha1 install]$ unzip linux_11gR2_database_2of2.zip

Database Installation

[root@racha1 ~]# xhost +

access control disabled, clients can connect from any host


[oracle@racha1 ~]$ cd /install/database

[oracle@racha1 database]$ ls

doc install response rpm runInstaller sshsetup stage welcome.html

[oracle@racha1 database]$ ./runInstaller


Checking Temp space: must be greater than 80 MB. Actual 9852 MB Passed


Checking monitor: must be configured to display at least 256 colors. Actual 16777216 Passed

Preparing to launch Oracle Universal Installer from /tmp/OraInstall2011-07-25_08-46-27PM. Please

wait ...


During Database creation, suppose to be you may face listener issues like below words

cluster database creation requires default listener configured

In this case, you need to create listener using netca from node 1.


The below errors showing, we need to set Unique name in Node 2 environment and start dbconsole.

However this is not issues, you can start further process.


Node 1:

[root@racha1 ~]# /u01/app/oracle/product/11.2.0/dbhome_1/root.sh




ORACLE_HOME= /u01/app/oracle/product/11.2.0/dbhome_1


The file "dbhome" already exists in /usr/local/bin. Overwrite it? (y/n)

[n]:

The file "oraenv" already exists in /usr/local/bin. Overwrite it? (y/n)

[n]:


The file "coraenv" already exists in /usr/local/bin. Overwrite it? (y/n)

[n]:





Finished product-specific root actions.

[root@racha1 ~]#

Node 2:

[root@racha2 ~]# /u01/app/oracle/product/11.2.0/dbhome_1/root.sh




ORACLE_HOME= /u01/app/oracle/product/11.2.0/dbhome_1


The file "dbhome" already exists in /usr/local/bin. Overwrite it? (y/n)

[n]:

The file "oraenv" already exists in /usr/local/bin. Overwrite it? (y/n)

[n]:

The file "coraenv" already exists in /usr/local/bin. Overwrite it? (y/n)

[n]:






Finished product-specific root actions.

[root@racha2 ~]#

Two Node RAC Installation Successfully completed.

Just Check

Node 1:


[oracle@racha1 ~]$ export ORACLE_SID=RAC1

[oracle@racha1 ~]$ sqlplus


SQL*Plus: Release 11.2.0.1.0 Production on Tue Jul 26 13:00:31 2011

Copyright (c) 1982, 2009, Oracle. All rights reserved.

Enter user-name: / as sysdba

Connected to:

Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 - Production

With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,

Data Mining and Real Application Testing options

SQL> select name from v$database;

NAME

---------

RAC

SQL> select instance_name from v$instance;

INSTANCE_NAME

----------------

RAC1

SQL> create user test identified by test;

User created.


SQL> grant connect,resource to test;

Grant succeeded.

SQL> conn test/test;

Connected.

SQL> create table emp(empno number);

Table created.

SQL> insert into emp values(001);

1 row created.


1 row created.


1 row created.


1 row created.


SQL> commit;

Commit complete.

SQL> select * from emp;

EMPNO

----------

1

2

3

4

Node 2:


[oracle@racha2 ~]$ export ORACLE_SID=RAC

[oracle@racha2 ~]$ export ORACLE_SID=RAC2

[oracle@racha2 ~]$ sqlplus

SQL*Plus: Release 11.2.0.1.0 Production on Tue Jul 26 13:06:31 2011

Copyright (c) 1982, 2009, Oracle. All rights reserved.

Enter user-name: / as sysdba

Connected to:

Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 - Production


With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,

Data Mining and Real Application Testing options

;SQL>select instance_name from v$instance;

INSTANCE_NAME

----------------

RAC2

SQL> conn test/test;

Connected.

SQL> select * from emp;

EMPNO

----------

1

2

3

4


More Details About RAC, my recommended book

http://www.packtpub.com/oracle-11g-r1-r2-real-application-clusters-essentials/book

http://www.packtpub.com/oracle-11g-r1-r2-real-application-clusters-essentials/book

oracle 11gr2 two node real application cluster installation step by step guide · pdf...

Documents