oracle rac with oraclevm installation guide 2
TRANSCRIPT
1
Oracle RAC Virtualization
Installation Guide
Andrei‐Daniel Dumitru, Anton Topurov
[email protected], [email protected]
Supervisors: Anton Topurov, Eric Grancher
Openlab Summer Student Program 2008
CERN, IT‐DES
9th September 2008
2
1. Creating Oracle VM manager node ......................................................................................................... 3
1.1 Oracle enterprise Linux installation. ................................................................................................... 3
1.2 Oracle VM Manager installation. ........................................................................................................ 3
1.3 Oracle VM Manager Migration (only if needed) ................................................................................. 6
2. Creating Oracle VM Server node ............................................................................................................. 6
2.1 BIOS update procedure: ...................................................................................................................... 6
2.2 Installing the OVM Server ................................................................................................................... 6
3. Finding the VM Server and creating VMs on it ....................................................................................... 7
3.1 Creating a Server Pool ......................................................................................................................... 7
3.2 Adding a Server ................................................................................................................................... 8
3.3 Importing an ISO File ......................................................................................................................... 10
3.4 Creating a Virtual Machine from Installation Media ........................................................................ 11
4. Configuring the OS for Oracle use ......................................................................................................... 14
4.1 Installing additional packages, needed by Oracle ............................................................................. 14
4.2 Setting up the NFS storage. ............................................................................................................... 18
4.3 Using Oracle templates: .................................................................................................................... 19
5. Installing CRS .......................................................................................................................................... 21
5.1 Setting up SSH user equivalence ....................................................................................................... 21
5.2 CLUVFY utility .................................................................................................................................... 22
5.3 Installing CRS 10.2.0.1: ...................................................................................................................... 25
5.4 Install 10.2.0.3 Patchset .................................................................................................................... 31
5.5 Install additional patches (optional) ................................................................................................. 31
6. Installing RDBMS .................................................................................................................................... 34
6.1 Installing Oracle Database 10g .......................................................................................................... 34
6.2 Creating the database ....................................................................................................................... 36
7. Swingbech settings and tests ................................................................................................................. 37
8. Pure Xen Setup ....................................................................................................................................... 38
9. Live Migration ......................................................................................................................................... 40
A. Virtual machine installation (steps) order: ............................................................................................. 45
B. Things to know ....................................................................................................................................... 46
Appendix ..................................................................................................................................................... 47
3
1. Creating Oracle VM manager node
1.1 Oracle enterprise Linux installation. Some pictures of the installation: http://www.it‐eye.nl/weblog/2006/10/27/screenshots‐oracle‐enterprise‐linux‐install/ Guides: http://www.oracle.com/technology/pub/articles/smiley‐11gr1‐install.html http://www.oraclebase.com/articles/linux/OracleEnterpriseLinux5Installation.php Appendix:
• 02‐Oracle Enteprise Linux Installation Screenshots.pdf • 03‐Oracle Enteprise Linux Installation – Guide.pdf • 04‐Installing Oracle Database 11g Release 1 on Enterprise Linux 5 (32‐ and 64‐bit).pdf
The installation or Oracle Enterprise Linux was straightforward, following the guide provided by Oracle (the first link or appendix 3). Root password: netapp2008 My user: andrei My password: netapp2008 1.2 Oracle VM Manager installation. OracleVM Manager installs an Oracle 10g Express Edition Database used for keeping all its records about the VM Server(s) and VM(s): Specify the HTTP port that will be used for Oracle Application Express [8080] Specify a port that will be used for the database listener [1521] Installation: sh runInstaller.sh Welcome to Oracle VM Manager Please enter the choice: [1|2|3] 1. Install Oracle VM Manager 2. Uninstall Oracle VM Manager 3. Upgrade Oracle VM Manager 1 Starting Oracle VM Manager installation ... Prepare to install the Oracle XE database ... Checking the supported platforms ... Done Checking the prerequisite packages are installed ... Done Checking the available disk space ... Done Installing the oracle-xe-univ package (rpm) now ...
4
Done Oracle Database 10g Express Edition Configuration ------------------------------------------------- This will configure on-boot properties of Oracle Database 10g Express Edition. The following questions will determine whether the database should be starting upon system boot, the ports it will use, and the passwords that will be used for database accounts. Press <Enter> to accept the defaults. Ctrl-C will abort. Specify the HTTP port that will be used for Oracle Application Express [8080]: Specify a port that will be used for the database listener [1521]:1531 Specify a password to be used for database accounts. Note that the same password will be used for SYS and SYSTEM. Oracle recommends the use of different passwords for each database account. This can be done after initial configuration: Confirm the password: Do you want Oracle Database 10g Express Edition to be started on boot (y/n) [y]:y Starting Oracle Net Listener...Done Configuring Database...Done Starting Oracle Database 10g Express Edition Instance...Done Installation Completed Successfully. To access the Database Home Page go to "http://127.0.0.1:8080/apex" Set default database schema to 'OVS'. Please enter the password for account 'OVS': Confirm the password: Creating the Oracle VM Manager database schema ...Done Installing the ovs-manager package (rpm) ... Done Installing the oc4j package (rpm) ... Done Please enter the password for account 'oc4jadmin': Confirm the password: Starting OC4J ... Done. To access the OC4J Home Page and change the password go to http://127.0.0.1:8888/em
5
Deploying Oracle VM Manager application to OC4J container. Creating connection pool ... Done Creating data source ... Done Deploying application ... Done Deploying application help ... Done Configuring Oracle VM Manager DataCollector ... Done Please enter the password for the default account 'admin': Confirm the password: Configuring SMTP server ... Please enter the outgoing mail server (SMTP) hostname: smtp.cern.ch Mail server checking, may need some time, please wait ... Mail server 'smtp.cern.ch' check failed, want to re-enter it(y|N)?N Setting the SMTP hostname server to smtp.cern.ch ... Done Please enter an e-mail address for account 'admin': [email protected] Confirm the e-mail address:[email protected] Failed to send mail to '[email protected]', want to re-enter the email address(y|N)?N Updating e-mail address for account 'admin' to '[email protected]' ... Done The console feature is not enabled by default. For detailed setup, refer to Oracle VM Manager User's Guide Installation of Oracle VM Manager completed successfully. To access the Oracle VM Manager home page go to: http://127.0.0.1:8888/OVS To access the Oracle VM Manager help page go to: http://127.0.0.1:8888/help/help
The password for the accounts SYS and SYSTEM for Oracle Database 10g Express Edition is oracle After the installation is completed, you can enter in the OracleVM Manager web interface.
6
1.3 Oracle VM Manager Migration (only if needed)
In order to compare Oracle Virtualization with last year’s results, Oracle VM Manager was moved to a different machine, thus we had two identical machines for testing. One was running Oracle VM Server and the other one Oracle Enterprise Linux that comes already patched with Xen.
The migration of Oracle VM Manager to another machine was done following this instructions:
http://download.oracle.com/docs/cd/E11081_01/doc/doc.21/e10901/appendix_b.htm#BGBJGHDG
Appendix: 15‐Backing Up and Restoring Oracle VM Manager.pdf
2. Creating Oracle VM Server node
2.1 BIOS update procedure:
1. Download new BIOS: http://h20000.www2.hp.com/bizsupport/TechSupport/SoftwareDescription.jsp? lang=en&cc=us&prodTypeId=12454&prodSeriesId=3252512&prodNameId=3253418&swEnvOID=1093&swLang=13&mode=2&taskId=135&swItem=ir602351
2. Put the image file 7E3_0234.BIN to a USB disk 3. Insert the USB in the computer and reboot the computer 4. Press F10 to enter the setup menu 5. From the File menu choose Flash System ROM 6. Select the USB and press F10 7. Select the file that you want to use and press F10
(Optional)Pray the power won't shut off. 8. BIOS update will be performed 9. Reboot so the changes will take effect.
After the BIOS update, to enable the hardware virtualization go to the Security menu, choose OS Security and Enable the Virtualization Technology. Now you will have Virtualization hardware support. The computers used are two HP Compaq dc5750. Check the specification files for more details. 2.2 Installing the OVM Server
7
Oracle VM Agent (ovs‐agent) Password: oracle Root password: netapp2008
Guide: http://www.oracle.com/technology/documentation/vm.html Applendix: 01‐08‐Oracle VM Server Installation Guide Release 2.1‐E11081_01.zip Using the guide the installation went smoothly, but when I wanted to login, Oracle gave a warning message: "WARNING: YOUR CPU(S) SUPPORT HARDWARE VIRTUALIZATION BUT YOUR SYSTEM DOES NOT. ENABLE VIRTUALISATION IN YOUR BIOS OR CONTACT YOUR SYSTEM VENDOR." With the help of HP Support I did a BIOS update from the version 786E3 v02.10 to version 786E3 v02.34.
3. Finding the VM Server and creating VMs on it
3.1 Creating a Server Pool
To create a server pool, perform the following: 1. In the Server Pools page, click Create Pool. 2. Enter the Oracle VM Server information and
add the Oracle VM Server to the server pool. You can add multiple Oracle VM Servers to a single server pool. A server pool consists of at least one Server Pool Master, one Utility Server, and one Virtual Machine Server respectively. All can be done in the “all‐in‐one” configuration(see the figure). Enter the following parameters:
o Server Host/IP Enter the host name, or IP address of the Oracle VM Server that functions as the Server Pool Master. For example, enter the following: 192.168.1.1 or hostname1.example.com
o Server Name
8
Name the Server Pool Master. This must be unique. o Agent Password
Enter the password to access Oracle VM Agent installed on the Server Pool Master.
o Server Type Select the Oracle VM Agent options:
Server Pool Master Utility Server Virtual Machine Server
If you select Utility Server you must enter the Oracle VM Agent login credentials. The user must have read/write privileges for the /OVS folder.
o Server Location The location of the Oracle VM Server. For example, Server Room 1.
After you enter the parameters, click Add. If the information is incorrect, or the Oracle VM Server is down, you cannot add it. The Oracle VM Server is displayed in the Servers to be Added to Server Pool table. You can select it, and then edit it or delete it.
3.2 Adding a Server
To add a Utility Server, or a Virtual Machine Server to an existing server pool, perform the following:
1. From the Servers page, click Add Server. 2. Search and select a server pool, and then click Next. 3. Enter the Oracle VM Server parameters. 4. Confirm the information you have entered.
The next pictures show the information of a created server.
9
10
3.3 Importing an ISO File
Depending on where your ISO files are located, on the Virtual Machine Server, or on other computers, you can import ISO files from different resources as follows:
Selecting from the Server Pool
Before importing, copy ISO files to the folder /OVS/iso_pool on the Virtual Machine Server. To select an existing ISO file on the Server Pool, and register it in Oracle VM Manager, perform the following:
1. In the Resources page, click ISO Files, and then click Import. 2. Select Select from Server Pool (Discover and register). 3. Select the following ISO file information:
o Server Pool Name Select the server pool for which the ISO file is to be used.
o ISO Group Select the ISO group. One ISO group may contain multiple ISO files.
o ISO Label Select the ISO file you want to import.
4. Confirm the information. 5. After importing, the status of the ISO file is Pending. You need to approve it to make it
available to use.
Downloading from External Source
To download an ISO file from outside of the Server Pool, such as your own HTTP/FTP server, perform the following:
1. From the Resources page, click ISO Files, and then click Import. 2. Select Download from External Source (HTTP and FTP). 3. Enter the following ISO file information:
o Server Pool Name Select the server pool for which the ISO file is to be used.
o ISO Group Enter the name of the ISO group. One ISO group may contain multiple ISO files.
o ISO Label Enter the ISO label used to identify the imported ISO file. For example, enter CD1.
Note:
The valid ISO label must comply with the following rules:
11
The initial letter of the label must be in uppercase, or lowercase. Valid characters which you can use include, numbers (such as 1, 2, 3, and
so on), letters (from a to z and from A to Z), dash (‐), dot (.) and underscores (_).
o URL Enter the path where the ISO file is located. You can import the ISO file through an FTP service or through an HTTP web service. For example, enter an address similar to the following: http://host/el5_img/disc1.iso ftp://host/el5_img/disc1.iso When using a proxy, select User Proxy and enter the proxy address. For example, enter a proxy address similar to the following: http://host:port
4. Confirm the information. 5. After importing, the status of the ISO file is Pending. You need to approve it to make it
available for creating virtual machines.
Status of ISO Files
The imported ISO file can be one of the following status: • Importing: This status indicates that the ISO file is in the process of being imported. • Pending: This status indicates that the ISO file has been imported successfully, and is
waiting for the approval of the Manager. • Import Error: This status indicates that there were some errors during the importing
process. Reimport the ISO file, or delete it. • Active: This status indicates that the ISO file has been approved, and is available for
creating virtual machines. • Inactive: This status indicates that the ISO file is not available.
3.4 Creating a Virtual Machine from Installation Media
Apart from an existing template, you can create a virtual machine from installation media, and configure the parameters manually. To create a virtual machine from installation media, perform the following:
1. Select Create from installation media. 2. Select the server pool where the virtual machine is to be located, and select a Virtual
Machine Server on which the virtual machine will run. Select Auto to assign a Virtual Machine Server automatically for the virtual machine, or select Manual to specify one or more preferred servers manually for the virtual machine.
3. Select the virtualized method, Fully Virtualized, or Paravirtualized. o Fully Virtualized
Select an ISO file from the list.
12
If you require more than one ISO file to create the virtual machine, select the first ISO file, and after creation, log on to the virtual machine, and click Change CD to continue with the installation. In the fully virtualized method, also known as the hardware virtualized mode (HVM), the unmodified guest operating system runs on the virtual machine. It traps and emulates every I/O and hardware instruction. To apply the fully virtualized mode, you must have either Intel processor with VT (virtualization technology) extensions, or an AMD processor with SVM extensions (also called AMD‐V) available on the host. A complete list of compatible processors is available at: http://wiki.xensource.com/xenwiki/HVM_Compatible_Processors If the CPU does not support HVM, use the paravirtualized method to create the virtual machine.
o Paravirtualized Before you create the virtual machine using the paravirtualized method, mount all ISO files to a single mount point on the Virtual Machine Server using the following command:
# mkdir mount‐point # mount ‐o loop,ro cd1.iso mount‐point
Where, mount‐point refers to the directory to which you want to mount the files. In the Resource Location field, enter the full path of the mount point. HTTP, FTP, and NFS are supported. For example, enter a path similar to the following:
nfs:host:/mnt
In the paravirtualized method, the guest operating system is recompiled before being installed on a virtual machine. Also, the virtual machine does not need to trap privileged instructions. Trapping is a method used to handle unexpected, or unallowable conditions, which is time‐consuming and can impact operating system performance. Without trapping privileged instructions, the paravirtualized operating system runs at near native speed. Currently for the paravirtualized mode, only Oracle Enterprise Linux 4 Update 5, and Oracle Enterprise Linux 5 are supported.
4. Enter the following information for the virtual machine: o Virtual Machine Name
Enter the name of the virtual machine. This must be unique. o Number of Cores
Select the number of cores in the virtual machine. You can allocate a maximum of 32 cores.
o Memory Size (MB)
13
Allocate at least 256 MB of memory. o Virtual Disk Size (MB)
Allocate at least 1024 MB of virtual disk. o Console Password
Set the console password for connecting to the virtual machine using VNC. o Confirm Console Password
Re‐enter the console password. 5. Optionally, click Add Row to add more network interface cards (NIC). By default,
network interface cards are named VIF0, VIF1, VIF2, and so on. You can rename them after creating the virtual machine. The MAC address of a network interface card is generated automatically and randomly. Select a Xen bridge for the network interface card. By default, the number of Xen bridges is the same as that of physical adaptors, and Xen bridges are named after physical adaptors. For example, if the Virtual Machine Server has several physical adaptors, eth0, eth1, and eth2, and so on, Xen bridges are then named xenbr0, xenbr1, and xenbr2 and so on. When creating a virtual machine, you can set a maximum of three network interface cards. After the virtual machine is created, you can set a maximum of eight network interface cards.
6. Confirm the virtual machine information you have entered. 7. The process of creating a virtual machine takes time. Click the Refresh button
periodically until the status of the virtual machine changes from Creating to Running. You can choose to refresh manually, or to refresh every 30 seconds. Now you can log in to the virtual machine and perform the installation.
After a virtual machine has been created you can follow all the steps needed to install CRS and an Oracle Database (see the OEL installation chapter). When the machine is ready for CRS, it can be cloned, thus obtaining another virtual machine identical with the first one. The MAC addresses of the Network Interface(s) will be automatically changed during the clonning process so there will not be a collission between two identical MAC addresses. After that, you will only need to change the hostname of the new machine and the IP address(es) if you have assigned any.
To change the hostname of a machine running Oracle Enterprise Linux just edit the /etc/sysconfig/network file
[oracle@oravm04 ~]$ cat /etc/sysconfig/network NETWORKING=yes NETWORKING_IPV6=no HOSTNAME=oravm04
To change the IP(s) of a network interface card edit the files located in /etc/sysconfig/network‐scripts/
14
[oracle@oravm03 ~]$ ls /etc/sysconfig/network-scripts/ | grep ifcfg-eth ifcfg-eth0 ifcfg-eth1 ifcfg-eth2
4. Configuring the OS for Oracle use
4.1 Installing additional packages, needed by Oracle Just to make sure we have all the packages needed for the installation of CRS and the database, follow the following instructions. Before that, you will need to install from the 1st cd the kernel‐headers‐2.6.18‐92.el5.x86_64.rpm package. The packages are needed for the installation of the Oracle Database 11g: 64-bit (x86_64) Installations Required kernel version: 2.6.18 This kernel, or any of the kernels supplied in updates, works with Oracle Database 11g Release 1 . Check your kernel version by running the following command: uname -rm Ex: # uname -rm 2.6.18-8.el5 x86_64
Required package versions (or later): • binutils-2.17.50.0.6 • compat-libstdc++-33-3.2.3 • compat-libstdc++-33-3.2.3 (32 bit) • elfutils-libelf-0.125 • elfutils-libelf-devel-0.125 • gcc-4.1.1 • gcc-c++-4.1.1 • glibc-2.5-12 • glibc-2.5-12 (32 bit) • glibc-common-2.5 • glibc-devel-2.5 • glibc-devel-2.5-12 (32 bit) • libaio-0.3.106 • libaio-0.3.106 (32 bit) • libaio-devel-0.3.106 • libgcc-4.1.1 • libgcc-4.1.1 (32 bit) • libstdc++-4.1.1 • libstdc++-4.1.1 (32 bit)
15
• libstdc++-devel 4.1.1 • make-3.81 • sysstat-7.0.0
Note that there are a number of packages where both the 64-bit and 32-bit RPMs must be installed. Fortunately, both are provided on the 64-bit Linux installation media. If you've performed a "default RPMs" install as suggested by Oracle, there are still some required packages that must be installed:
• compat-libstdc++-33-3.2.3 • compat-libstdc++-33-3.2.3 (32 bit) • elfutils-libelf-devel-0.125 • gcc-4.1.1 • gcc-c++-4.1.1 • glibc-devel-2.5 • glibc-devel-2.5-12 (32 bit) • libaio-devel-0.3.106 • libstdc++-devel 4.1.1 • sysstat-7.0.0
To install the remaining packages, mount the Oracle Enterprise Linux media and move to the directory containing the RPMs. Some required packages are dependent upon other packages, so the dependant packages must be installed as well. Login as root and run the following commands to install the remaining required packages and their dependent packages:
16
Mount CD 2 mount -t iso9660 -r /dev/cdrom /media cd /media/Server rpm -ivh elfutils-libelf-devel*.x86_64.rpm glibc-devel-2* gcc-4*.x86_64.rpm gcc-c++-4*.x86_64.rpm libstdc++-devel-4*.x86_64.rpm glibc-headers*.x86_64.rpm libgomp*.x86_64.rpm Ex: # rpm -ivh elfutils-libelf-devel*.x86_64.rpm glibc-devel-2* gcc-4*.x86_64.rpm gcc-c++-4*.x86_64.rpm libstdc++-devel-4*.x86_64.rpm glibc-headers*.x86_64.rpm libgomp*.x86_64.rpm warning: elfutils-libelf-devel-0.125-3.el5.x86_64.rpm: Header V3 DSA signature: NOKEY, key ID 1e5e0159 Preparing... ########################################### [100%] 1:glibc-headers ########################################### [ 11%] 2:glibc-devel ########################################### [ 22%] 3:libgomp ########################################### [ 33%] 4:gcc ########################################### [ 44%] 5:libstdc++-devel ########################################### [ 56%] 6:gcc-c++ ########################################### [ 67%] 7:glibc-devel ########################################### [ 78%] 8:elfutils-libelf-devel-s########################################### [ 89%] 9:elfutils-libelf-devel ########################################### [100%] cd / eject Mount CD 3 mount -t iso9660 -r /dev/cdrom /media cd /media/Server rpm -ivh compat-libstdc++-33* libaio-devel*.x86_64.rpm Ex: # rpm -ivh compat-libstdc++-33* libaio-devel*.x86_64.rpm warning: compat-libstdc++-33-3.2.3-61.i386.rpm: Header V3 DSA signature: NOKEY, key ID 1e5e0159 Preparing... ########################################### [100%] 1:libaio-devel ########################################### [ 33%] 2:compat-libstdc++-33 ########################################### [ 67%] 3:compat-libstdc++-33 ########################################### [100%] cd / eject Mount CD 4 mount -t iso9660 -r /dev/cdrom /media cd /media/Server rpm -ivh sysstat* Ex: # rpm -ivh sysstat* warning: sysstat-7.0.0-3.el5.x86_64.rpm: Header V3 DSA signature: NOKEY, key ID 1e5e0159 Preparing... ########################################### [100%] 1:sysstat ########################################### [100%] To verify that the required packages have been installed on your system, run the following command:
17
rpm -q --queryformat "%{NAME}-%{VERSION}-%{RELEASE} (%{ARCH})\n" binutils \ compat-libstdc++-33 elfutils-libelf elfutils-libelf-devel gcc gcc-c++ glibc \ glibc-common glibc-devel-2.5 libaio libaio-devel libgcc libstdc++ libstdc++-devel \ make sysstat Ex: # rpm -q --queryformat "%{NAME}-%{VERSION}-%{RELEASE} (%{ARCH})\n" binutils \ > compat-libstdc++-33 elfutils-libelf elfutils-libelf-devel gcc gcc-c++ glibc \ > glibc-common glibc-devel-2.5 libaio libaio-devel libgcc libstdc++ libstdc++-devel \ > make sysstat binutils-2.17.50.0.6-2.el5 (x86_64) compat-libstdc++-33-3.2.3-61 (i386) compat-libstdc++-33-3.2.3-61 (x86_64) elfutils-libelf-0.125-3.el5 (x86_64) elfutils-libelf-devel-0.125-3.el5 (x86_64) gcc-4.1.1-52.el5 (x86_64) gcc-c++-4.1.1-52.el5 (x86_64) glibc-2.5-12 (i686) glibc-2.5-12 (x86_64) glibc-common-2.5-12 (x86_64) glibc-devel-2.5-12 (x86_64) glibc-devel-2.5-12 (i386) libaio-0.3.106-3.2 (x86_64) libaio-0.3.106-3.2 (i386) libaio-devel-0.3.106-3.2 (x86_64) libgcc-4.1.1-52.el5 (x86_64) libgcc-4.1.1-52.el5 (i386) libstdc++-4.1.1-52.el5 (x86_64) libstdc++-4.1.1-52.el5 (i386) libstdc++-devel-4.1.1-52.el5 (x86_64) make-3.81-1.1 (x86_64) sysstat-7.0.0-3.el5 (x86_64) To install CRS and an Oracle Database some extra configuration steps are necessary. Create user oracle and the groups ci and oinstall: [root@oravm03 ~]# groupadd ci [root@oravm03 ~]# groupadd oinstall [root@oravm03 ~]# useradd -m -g ci –G oinstall oracle [root@xenvmp04 ~]# id oracle uid=500(oracle) gid=500(ci) groups=500(ci),501(oinstall)
18
4.2 Setting up the NFS storage. For the Database actual files, an NFS directory was set up on another machine. Guide: http://www.ubuntugeek.com/nfs‐server‐and‐client‐configuration‐in‐ubuntu.html Appendix: 10‐NFS Server and Client Configuration in Ubuntu.pdf
NFS configuration files: The NFS server: andrei@pcora19:~$ cat /etc/exports /OracleDatabase 137.138.34.1/8(rw,sync,no_root_squash,no_wdelay,anonuid=0,insecure_locks) The NFS client: 137.138.34.28:/OracleDatabase /ORA/dbs03 nfs hard,rw,noac,rsize=32768,wsize=32768,suid,proto=tcp,vers=3 137.138.34.28:/OracleDatabase /ORA/old nfs hard,rw,noac,rsize=32768,wsize=32768,suid,proto=tcp,vers=3,actimeo=0,timeo=600 1 2 137.138.34.28:/OracleXEN /ORA/dbs03 nfs rw,bg,hard,nointr,rsize=32768,wsize=32768,tcp,vers=3,timeo=600,noac !!! /ORA/old is the location of the installation files
Because the interface used for interconnecting the virtual machines is a public one, the routing was manually set so that the interconnect interfaces used for communication between the virtual machines and the NFS server. This way the communication is always done through the desired interface just like a dedicated network card for the storage. Modifyed routing table so the NFS storage will use the appropriate interfaces: [root@oravm03 bin]# route -n Kernel IP routing table Destination Gateway Genmask Flags Metric Ref Use Iface 137.138.34.28 * 255.255.255.255 UH 0 0 0 eth2 137.138.0.0 * 255.255.0.0 U 0 0 0 eth0 137.138.0.0 * 255.255.0.0 U 0 0 0 eth1 169.254.0.0 * 255.255.0.0 U 0 0 0 eth2 10.0.0.0 * 255.0.0.0 U 0 0 0 eth2 default default-route.c 0.0.0.0 UG 0 0 0 eth1
19
So that the routing is done everytime the system boots, the following command was placed in rc.local:
route add ‐host 137.138.34.28 dev eth2
[root@oravm03 bin]# cat /etc/rc.local #!/bin/sh # # This script will be executed *after* all the other init scripts. # You can put your own initialization stuff in here if you don't # want to do the full Sys V style init stuff. touch /var/lock/subsys/local route add -host 137.138.34.28 dev eth2 [root@oravm03 bin]#
4.3 Using Oracle templates: Alternatively to installing and configuring VMs, one can use preformatted Oracle templates, available for download from e‐delivery web site. Guide: http://download.oracle.com/otn_software/virtualization/README.templates Appendix: 17‐README.templates.pdf
I have tried to use the templates, but I cannot get the Oracle Database 11g to mount inside the virtual machine on the Oracle Enterprise Linux 5 system. I tried to use just the Oracle Enterprise Linux template, but that did not work, because I have modified the RAM quantity (from 2 GB to 1 GB) and the Kernel gave an error message. Maybe because it had a fixed size swap partition inside… don’t know for sure.
20
So I have decided to go the classic way, install Oracle Enterprise Linux in the OracleVM using the installation disks. The console feature has some bugs (the mouse in the console is not the mouse in the browser= you have 2 cursors)
21
5. Installing CRS
5.1 Setting up SSH user equivalence
Before the installation of CRS can begin, some additional steps are required.
First, you need to establish user equivalence. This means that on each node where CRS is going to be installed the oracle user can login without a password, using a public key.
We will assume that our cluster has only two nodes: oravm03 and oravm04.
To configure SSH using DSA on that cluster, the following steps must be taken:
1. As the oracle user, create the public and private keys on both nodes:
[oravm03]$ /usr/bin/ssh‐keygen ‐t dsa
[oravm04]$ /usr/bin/ssh‐keygen ‐t dsa
Accept the default location for the key file. When prompted for the pass phrase, just press the Enter key.
2. Concatenate the contents of the id_dsa.pub file from each node into the authorized_keys file on the first node (/home/oracle/.ssh/authorized_keys):
[oravm03]$ ssh oravm03 "cat ~/.ssh/id_dsa.pub" >> \ ~/.ssh/authorized_keys
[oravm04]$ ssh oravm04 "cat ~/.ssh/id_dsa.pub" >> \ ~/.ssh/authorized_keys
3. Copy the authorized_keys file to the other nodes:
oravm04$ scp ~/.ssh/authorized_keys oravm05:/home/oracle/.ssh/
22
/etc/hosts
Each node must know all the hosts in the CRS cluter; the known members of the CRS cluster must be added in the /etc/hosts file on each node. You can see a sample of the /etc/hosts file for a cluster with two nodes: oravm03 and oravm04:
[oracle@oravm03 ~]$ cat /etc/hosts # Do not remove the following line, or various programs # that require network functionality will fail. 127.0.0.1 localhost.localdomain localhost ::1 localhost6.localdomain6 localhost6 134.138.34.16 ORAVM03.CERN.CH ORAVM03 137.138.34.48 ORAVM03-STOR.CERN.CH ORAVM03-STOR 137.138.34.49 ORAVM03-V.CERN.CH ORAVM-V 10.1.1.3 ORAVM03-PRIV ### 137.138.34.50 ORAVM04.CERN.CH ORAVM04 137.138.34.52 ORAVM04-STOR.CERN.CH ORAVM04-STOR 137.138.34.51 ORAVM04-V.CERN.CH ORAVM04-V 10.1.1.4 ORAVM04-PRIV
!!!So no confusion will exist, please use the same case(either small or BIG letters) for all the hostnames.
5.2 CLUVFY utility To check that the system is ready for installing CRS cluvfy can be used:
[oracle@oravm03 bin]$ ./cluvfy stage -post hwos -n oravm03,oravm04 –verbose Performing post-checks for hardware and operating system setup Checking node reachability... Check: Node reachability from node "oravm03" Destination Node Reachable? ------------------------------------ ------------------------ oravm03 yes oravm04 yes Result: Node reachability check passed from node "oravm03".
23
Checking user equivalence... Check: User equivalence for user "oracle" Node Name Comment ------------------------------------ ------------------------ oravm04 passed oravm03 passed Result: User equivalence check passed for user "oracle". Checking node connectivity... Interface information for node "oravm04" Interface Name IP Address Subnet Subnet Gateway Default Gateway Hardware Address ---------------- ------------ ------------ ------------ ------------ ------------ eth0 137.138.34.50 137.138.0.0 0.0.0.0 137.138.1.1 00:16:3E:37:97:6C eth1 10.1.1.4 10.0.0.0 UNKNOWN 137.138.1.1 00:16:3E:2F:B4:5D eth2 137.138.34.52 137.138.0.0 0.0.0.0 137.138.1.1 00:16:3E:1D:BC:D8 Interface information for node "oravm03" Interface Name IP Address Subnet Subnet Gateway Default Gateway Hardware Address ---------------- ------------ ------------ ------------ ------------ ------------ eth0 137.138.34.16 137.138.0.0 0.0.0.0 137.138.1.1 00:16:3E:44:13:BF eth1 137.138.34.48 137.138.0.0 0.0.0.0 137.138.1.1 00:16:3E:46:05:EF eth2 10.1.1.3 10.0.0.0 UNKNOWN 137.138.1.1 00:16:3E:12:49:69 Check: Node connectivity of subnet "137.138.0.0" Source Destination Connected? ------------------------------ ------------------------------ ---------------- oravm04:eth0 oravm04:eth2 yes oravm04:eth0 oravm03:eth0 yes oravm04:eth0 oravm03:eth1 yes oravm04:eth2 oravm03:eth0 yes oravm04:eth2 oravm03:eth1 yes oravm03:eth0 oravm03:eth1 yes
24
Result: Node connectivity check passed for subnet "137.138.0.0" with node(s) oravm04,oravm03. Check: Node connectivity of subnet "10.0.0.0" Source Destination Connected? ------------------------------ ------------------------------ ---------------- oravm04:eth1 oravm03:eth2 yes Result: Node connectivity check passed for subnet "10.0.0.0" with node(s) oravm04,oravm03. Interfaces found on subnet "137.138.0.0" that are likely candidates for VIP: oravm04 eth0:137.138.34.50 oravm03 eth0:137.138.34.16 WARNING: Could not find a suitable set of interfaces for the private interconnect. Result: Node connectivity check passed. Checking shared storage accessibility... NFS Sharing Nodes (2 in count) ------------------------------------ ------------------------ /ora/dbs03 oravm04 oravm03 Shared storage check was successful on nodes "oravm04,oravm03". Post-check for hardware and operating system setup was successful.
The Subnet Gateway for the local interfaces is UNKNOWN because the local interfaces do not have configured a gateway. Although there is a warning about not finding a set of suitable interfaces for the private interconnect during the installation process there was no problem related to that and during the CRS installation all the interfaces were assigned the proper role(public, private, storage).
25
5.3 Installing CRS 10.2.0.1: To install CRS just run the installation script:
[oracle@oravm03 ~]$sh /ORA/dbs03/clusterware/10.2.0.3/installer/runInstaller
You can see bellow some installation screenshots:
26
Specifying the CRS Home Details
27
CRS will now check if all the requirements for the installation are met. You will receive some warning or the installer will not execute the “Checking operating system requirements” and “Checking operating system package requirements”. This is because we have used CRS version 10 with Oracle Enterprise Linux 5. OEL 5 is supported starting from CRS version 11.
28
Here you will have to configure the nodes on which CRS will be installed, using the required hostnames for the public, private and vip interfaces.
29
Here you can specify the role of each network interface. The public interface should be marked as Public, the Private interface as Private and the dedicated storage interface as Do Not Use. Use the Edit button to change the configuration of each interface.
Check the CRS versison:
After the installation you can check the CRS version with the following command:
[oracle@oravm03 bin]$ crsctl query crs activeversion
CRS active version on the cluster is [10.2.0.1.0]
30
Check the CRS is working on all nodes
[oracle@oravm03 bin]$ ./crs_stat -t Name Type Target State Host ------------------------------------------------------------ ora....03.lsnr application ONLINE ONLINE oravm03 ora....m03.gsd application ONLINE ONLINE oravm03 ora....m03.ons application ONLINE ONLINE oravm03 ora....m03.vip application ONLINE ONLINE oravm03 ora....04.lsnr application ONLINE ONLINE oravm04 ora....m04.gsd application ONLINE ONLINE oravm04 ora....m04.ons application ONLINE ONLINE oravm04 ora....m04.vip application ONLINE ONLINE oravm04 ora.racdb.db application ONLINE ONLINE oravm03 ora....b1.inst application ONLINE ONLINE oravm03 ora....b2.inst application ONLINE ONLINE oravm04
Possibe VIPCA problems:
In case you have VIPca problems, check out this:
Links:
• https://metalink.oracle.com/metalink/plsql/f?p=130:14:6948147433366817732::::p14_database_id,p14_docid,p14_show_header,p14_show_help,p14_black_frame,p14_font:NOT,414163.1,1,1,1,helvetica
• https://metalink.oracle.com/metalink/plsql/f?p=130:14:6948147433366817732::::p14_database_id,p14_docid,p14_show_header,p14_show_help,p14_black_frame,p14_font:NOT,276434.1,1,1,1,helvetica
• http://cs.felk.cvut.cz/10gr2/relnotes.102/b15659/toc.htm#CJABAIIF
Appendix:
• 11‐Modifying the VIP or VIP Hostname of a 10g Oracle Clusterware Node.pdf
• 12‐10gR2 RAC Install issues on Oracle EL5 or RHEL5 or SLES10 (VIPCA Failures).pdf
• 13‐SRVCTL and VIPCA Utilities Set the LD_ASSUME_KERNEL Parameter.pdf
A common vipca problem my appear because OEL is not supported default by CRS 10. The first link gives the solution or appendix 12.
31
5.4 Install 10.2.0.3 Patchset
After CRS 10.2.0.1.0 the upgrade to 10.2.0.3.0 was applied (located in /clusterware/10.2.0.3/patchset)
[oracle@oravm03 patchset]$ sh runInstaller
[oracle@oravm03 bin]$ ./crsctl query crs activeversion CRS active version on the cluster is [10.2.0.3.0]
5.5 Install additional patches (optional) The patches located in the ../clusterware/10.2.0.3/patches directory are specific patches made by CERN.
More information about them(1st link) and installation instructions(2nd link) can be found here:
• https://twiki.cern.ch/twiki/bin/view/DESgroup/OracleCERNPatchLevels
• https://twiki.cern.ch/twiki/bin/view/DESgroup/CreatingAnRpmOfOracleCrsandRACRdbms
CERN Level 1 patches Running the default command it says that it does not know the JDK location and specifying the JDK location returns another error. See below for details: [oracle@oravm03 OPatch]$ ./opatch lsinventory -detail -oh $RDBMS_HOME ls: /ORA/dbs01/oracle/product/rdbms/jre: No such file or directory Java could not be located. OPatch cannot proceed! OPatch returns with error code = 1 [oracle@oravm03 OPatch]$ ./opatch lsinventory -detail -oh $RDBMS_HOME -jdk /ORA/dbs01/oracle/product/crs/jdk/ java.lang.NoClassDefFoundError: oracle/ops/mgmt/cluster/ClusterException at oracle.opatch.OPatchSession.<clinit>(OPatchSession.java:168) at oracle.opatch.OPatch.<clinit>(OPatch.java:124) OPatch failed with error code 255 [oracle@oravm03 OPatch]$
Solution Link:https://metalink2.oracle.com/metalink/plsql/f?p=130:14:1864059879221541198::::p14_database_id,p14_docid,p14_show_header,p14_show_help,p14_black_frame,p14_font:NOT,427728.1,1,1,1,helvetica
32
Appendix: 18‐OPatch Java Error.pdf
CERN Level 2 patch
[oracle@oravm03 mega-6741817]$ /ORA/dbs01/oracle/product/crs/OPatch/opatch apply -local -oh $CRS_HOME Invoking OPatch 10.2.0.3.0 Oracle interim Patch Installer version 10.2.0.3.0 Copyright (c) 2005, Oracle Corporation. All rights reserved.. Oracle Home : /ORA/dbs01/oracle/product/crs Central Inventory : /ORA/dbs01/oracle/product/rdbms/oraInventory from : /etc/oraInst.loc OPatch version : 10.2.0.3.0 OUI version : 10.2.0.3.0 OUI location : /ORA/dbs01/oracle/product/crs/oui Log file location : /ORA/dbs01/oracle/product/crs/cfgtoollogs/opatch/opatch2008-07-30_18-28-50PM.log ApplySession applying interim patch '6741817' to OH '/ORA/dbs01/oracle/product/crs' You selected -local option, hence OPatch will patch the local system only. Please shutdown Oracle instances running out of this ORACLE_HOME on the local system. (Oracle Home = '/ORA/dbs01/oracle/product/crs') Is the local system ready for patching? Do you want to proceed? [y|n] y User Responded with: Y Final… Verifying the update... Inventory check OK: Patch ID 6741817 is registered in Oracle Home inventory with proper meta-data. Files check OK: Files from Patch ID 6741817 are present in Oracle Home. The local system has been patched and can be restarted. OPatch succeeded.
33
After patching(CERN patchlevels 1 and 2) the CRS will not start:
CRS-0184: Cannot communicate with the CRS daemon.
Also, there aren’t any logs in /ORA/dbs01/oracle/product/crs/log/
I removed the CRS using: Link:https://metalink.oracle.com/metalink/plsql/f?p=130:14:7095958152507896948::::p14_database_id,p14_docid,p14_show_header,p14_show_help,p14_black_frame,p14_font:NOT,239998.1,1,1,1,helvetica
Appendix: 14‐10g RAC How to Clean Up After a Failed CRS Install.pdf After this, a clean CRS installation was done and patched to CRS 10.2.0.3 version without applying any CERN patches.
Misspelling Error
When installing CRS, this Error Message apperead:
The location <loc> entered for Oracle Cluster Registry (OCR) is not shared across all the nodes in the cluster.
SOLUTION: CHECK /etc/hosts to see if it is correct on all nodes !!!!!!!!!!
[in my case, I had a wrong dot (.) somewhere in the file]
34
6. Installing RDBMS
6.1 Installing Oracle Database 10g Directory structure: /ORA/dbs00 /ORA/dbs01/oracle/product/rdbms (instance files) /ORA/dbs03 (database files – mounted NFS directory from another computer) /ORA/old (installation binaries – mounted NFS directory from another computer) ORABASE: /ORA/dbs01/oracle (ORACLE_BASE) ORAHOME: /ORA/dbs01/oracle/product/rdbms (ORACLE_HOME)
Environment variables
To make sure that Oracle software has all the necessary environment variables set up when the system starts a script is executed when the system shell is started by the oracle user:
[oracle@oravm03 ~]$ cat .bashrc # .bashrc # Source global definitions # User specific aliases and functions source set_racdb1_database;
The contents of the file:
[root@OraVM01 oracle]#source set_racdb1_database
[root@OraVM01 oracle]# cat set_oravm_database ORACLE_BASE=/ORA/dbs01/oracle; export ORACLE_BASE ORACLE_PRODUCT=$ORACLE_BASE/product; export ORACLE_PRODUCT ORACLE_HOME=$ORACLE_PRODUCT/rdbms; export ORACLE_HOME ORACLE_SID=racdb1; export ORACLE_SID TNS_ADMIN=$ORACLE_HOME/network/admin export TNS_ADMIN LD_LIBRARY_PATH=$ORACLE_HOME/lib; export LD_LIBRARY_PATH PATH=$PATH:$ORACLE_HOME/bin:$ORACLE_HOME; export PATH
You can find the set_ovm_database file in the appendix directory.
35
Installation After the installation of Oracle Enterprise Linux 5 Release 2, I tried to install Oracle 10g, but Oracle Database 10g does not work directly with Oracle Enterprise Linux 5, because OEL was released after the Database 10g. Starting Oracle Universal Installer... Checking installer requirements... Checking operating system version: must be redhat-3, SuSE-9, redhat-4, UnitedLinux-1.0, asianux-1, asianux-2 or redhat-5 Failed
See metalink Note:456634.1 (Applendix: 06‐Metalink Note 456634.1.pdf) for details: https://metalink.oracle.com/metalink/plsql/f?p=130:14:6948147433366817732::::p14_database_id,p14_docid,p14_show_header,p14_show_help,p14_black_frame,p14_font:NOT,456634.1,1,1,1,helvetica After that, came this:
36
See metalink.oracle.com Note:443617.1 (07‐Metalink Note 443617.1.pdf) for details:
https://metalink.oracle.com/metalink/plsql/f?p=130:14:6948147433366817732::::p14_database_id,p14_docid,p14_show_header,p14_show_help,p14_black_frame,p14_font:NOT,443617.1,1,1,1,helvetica
Basically, there was a need for another package, libXp‐1.0.0‐8.1.el5.i386.rpm.
Yes, it is a package for a 32 bit machine!!!
After that, the installation went normally and the update as well. The RDBMS was updated from 10.2.0.1.0 to 10.2.0.3.0 using the patch located in (../database/10.2.0.3/patchset). The CERN patches (../database/10.2.0.3/patches) were not applied.
Checking database version
$ sqlplus / as sysdba SQL*Plus: Release 10.2.0.3.0 - Production on Wed Sep 24 23:05:57 2008 Copyright (c) 1982, 2006, Oracle. All Rights Reserved. Connected to: Oracle Database 10g Enterprise Edition Release 10.2.0.3.0 - 64bit Production With the Partitioning, Real Application Clusters, OLAP and Data Mining options
6.2 Creating the database
After the software was installed the database was created using dbca.
[oracle@OraVM01 ~]$ source set_oravm_database [in case you did not configured the environment yet] [oracle@OraVM03 ~]$ dbca
Important
When creating the database use the following values (to have an accurate comparison with the previous years results): SGA: 512 MB PGA: 272 MB Redo Logs: 1 GB
37
7. Swingbech settings and tests
Swingbench was the only tool used for testing the performance of the database.
Two tests were used for Order Entry (OE) and Stress Test (ST). Both tests have been done in Load Balancing (LB) and No Load Balancing (NLB) mode.
For the LB mode, in the tnsnames.ora file was specified LOAD_BALANCE = yes.
For the NLB mode, 2 Swingbench instances were used, each instance testing a different node.
You cand find in the Appendix directory all the configuration and test files.
38
8. Pure Xen Setup
After successful moving the VM Manager, OEL was installed on the machine that will host OEL XEN virtualization. The installation was normal. For creating the virtual machines virt‐manager (GUI provided by RHEL) was used. To finish the installation, the installer had to reboot... the XEN will not start. The problem is that, if you followed the instruction, you have removed the cdrom (iso iamge file) and the machine is setup to boot from cdrom. Edit the config file /etc/xen/xenvm01 (/etc/xen/<your_vm_name>) and specify boot="c" (to boot from the hard drive). Check this for more details on configuration options(section: Configuration file for unmodified HVM guests): http://www.cl.cam.ac.uk/research/srg/netos/xen/readmes/user/user.html#SECTION04320000000000000000 Appendix: 16‐Xen Manual.pdf After the machine has rebooted, the installation worked fine until I tryed to use xm console xenvm01. This command hangs and does not work. I never managed to make this work. After the machine obtained an IP address I could ssh to it and user virt‐manager. At first, virt‐manager would not work. After creating and setting up the virtual machine, I was able to use virt‐manager(picture above) utility just enough to add 2 extra network cards to the virtual machine. After that, the virt‐manager will not do anything upon connecting to the localhost. The log (/root/.virt‐manager/virt‐manager.log) output: [Wed, 27 Aug 2008 16:59:12 virt-manager 32586] ERROR (virt-manager:132) Traceback (most recent call last): File "/usr/share/virt-manager/virtManager/engine.py", line 185, in _do_show_console self.show_console(uri, uuid)
39
File "/usr/share/virt-manager/virtManager/engine.py", line 239, in show_console con.get_vm(uuid)) File "/usr/share/virt-manager/virtManager/connection.py", line 278, in get_vm return self.vms[uuid] KeyError: '2a77f13e-a5ee-d73f-b612-69e8ab35e73c' None
So, I used virt‐viewer <machine‐id/name> and I could see the VNC interface. Maybe it was because of the X server implementation I used from the Windows Vista machine I was working on. There is a chance that there will not be any problems if an authentic linux is used with X forwarding. After performing all the necessary steps in preparing the system for Oracle installation (adding more network interfaces, creating the oracle user and the group ci, setting the kernel parameter, installing the required packages) I cloned the machine using virt‐clone. If no parameters are supplied, it will start in interactive mode. [root@pcitdes013 ~]# virt-clone What would you like to use as the cloned disk (file path)? /var/lib/xen/images/xenvm02.img What is the name or uuid of the original virtual machine? xenvm01 What is the name for the cloned virtual machine? xenvm02 Cloning from /var/lib/xen/images/xenvm01.img to /var/lib/xen/images/xenvm02.img
The cloning process was successful . The first created machines were done using hardware virtualization and after all the testing was done we moved to paravirtualization. Because we ran out of IPs we started using local IPs. At some point, CRS complained about a gateway – that did not exist because we were using local ips – so I used “masquerading” and set the machine I was working on as a gateway. Sharing Internet Connection in Ubuntu: http://www.ubuntugeek.com/sharing‐internet‐connection‐in‐ubuntu.html Appendix: 19‐Sharing Internet Connection.pdf
40
9. Live Migration
Setup In order to test live migration, I used Oracle VM Server running on multiple machines (3 servers with Oracle VM Server). For Live Migration to work, the servers need to have a common storage. The pssibilities are an NFS Cluster or iSCSI. For my test I have used an iSCSI cluster. For iSCSI to work you need the iSCSI Initiator Utils [from Wikipedia: An initiator functions as an iSCSI client. An initiator typically serves the same purpose to a computer as a SCSI bus adapter would, except that instead of physically cabling SCSI devices (like hard drives and tape changers), an iSCSI initiator sends SCSI commands over an IP network. ] The iSCSI initiator is provided by the iscsi‐initiator‐utils‐6.2.0.868‐0.7.el5.x86_64.rpm package from the Oracle Enterprise Linux cds. Also, for the actual storage an iSCSI target [from Wikipedia: iSCSI refers to a storage resource located on an iSCSI server (more generally, one of potentially many instances of iSCSI running on that server) as a "target". An iSCSI target usually represents hard disk storage. As with initiators, software to provide an iSCSI target is available for most mainstream operating systems.] is needed. In my cause I used Openfiler (http://www.openfiler.com/) to create an iSCSI target. After I created the target I used the instructions from Oracle VM Server User's manual to connect the servers to the storage. Since this is a shared storage , a clustered file system [http://en.wikipedia.org/wiki/Clustered_file_system ] was needed. Oracle provides OCFS2 [From http://oss.oracle.com/projects/ocfs2/ : OCFS2 is a POSIX‐compliant shared‐disk cluster file system for Linux capable of providing both high performance and high availability. As it provides local file system semantics, it can be used with any application. Cluster‐aware applications will be able to make use of parallel I/O for higher performance. Other applications, if not able to benefit from parallel I/O, will be able to make use of the file system to provide a fail‐over setup to increase the availability of the application. ] Following instructions provided by Oracle VM Server's Guide i was able to make all the server have a common view of the storage:
41
Creating a Shared Virtual Disk for Live Migration If you want to perform live migration of domains to other, identical, computers, you must create a shared virtual disk to be used during the live migration. You can set up a shared virtual disk in the following configurations: * OCFS2 (Oracle Cluster File System) using the iSCSI (Internet SCSI) network protocol * OCFS2 using SAN (Storage Area Network) * NFS (Network File System) This section discusses creating a shared virtual disk in these configurations. 6.1.1 Creating a Shared Virtual Disk Using OCFS2 on iSCSI To create a shared virtual disk using OCFS2 on iSCSI: 1. Install the iscsi‐initiator‐utils RPM. The iscsi‐initiator‐utils RPM is available on the Oracle VM Server CDROM or ISO file. rpm ‐Uvh iscsi‐initiator‐utils‐6.2.0.742‐0.5.el5.i386.rpm 2. Start the iSCSI service: service iscsi start 3. Run discovery on the iSCSI target. In this example, the target is 10.1.1.1: iscsiadm ‐m discovery ‐t sendtargets ‐p 10.1.1.1 This command returns output similar to: 10.1.1.1:3260,5 iqn.1992‐04.com.emc:cx.apm00070202838.a2 10.1.1.1:3260,6 iqn.1992‐04.com.emc:cx.apm00070202838.a3 10.2.1.250:3260,4 iqn.1992‐04.com.emc:cx.apm00070202838.b1 10.1.0.249:3260,1 iqn.1992‐04.com.emc:cx.apm00070202838.a0
42
10.1.1.249:3260,2 iqn.1992‐04.com.emc:cx.apm00070202838.a1 10.2.0.250:3260,3 iqn.1992‐04.com.emc:cx.apm00070202838.b0 4. Delete entries that you do not want to use, for example: iscsiadm ‐m node ‐p 10.2.0.250:3260,3 ‐T iqn.1992‐04.com.emc:cx.apm00070202838.b0 ‐o delete iscsiadm ‐m node ‐p 10.1.0.249:3260,1 ‐T iqn.1992‐04.com.emc:cx.apm00070202838.a0 ‐o delete iscsiadm ‐m node ‐p 10.2.1.250:3260,4 ‐T iqn.1992‐04.com.emc:cx.apm00070202838.b1 ‐o delete iscsiadm ‐m node ‐p 10.1.1.249:3260,2 ‐T iqn.1992‐04.com.emc:cx.apm00070202838.a1 ‐o delete iscsiadm ‐m node ‐p 10.0.1.249:3260,5 ‐T iqn.1992‐04.com.emc:cx.apm00070202838.a2 ‐o delete 5. Verify that only the necessary entries are visible: iscsiadm ‐m node 6. Review the partitions by checking /proc/partitions: cat /proc/partitions major minor #blocks name 8 0 71687372 sda 8 1 104391 sda1 8 2 71577607 sda2 253 0 70516736 dm‐0 253 1 1048576 dm‐1 7. Restart the iSCSI service: service iscsi restart 8. Review the partitions by checking /proc/partitions. A new device is listed. cat /proc/partitions major minor #blocks name 8 0 71687372 sda 8 1 104391 sda1 8 2 71577607 sda2 253 0 70516736 dm‐0 253 1 1048576 dm‐1 8 16 1048576 sdb 9. The new device can now be used.
43
fdisk ‐l /dev/sdb 10. Create a new directory named /etc/ocfs2 directory: mkdir /etc/ocfs2 11. Create the OCSF2 configuration file as /etc/ocfs2/cluster.conf. The following is a sample cluster.conf file: node: ip_port = 7777 ip_address = 10.1.1.1 number = 0 name = example1.com cluster = ocfs2 node: ip_port = 7777 ip_address = 10.1.1.2 number = 1 name = example2.com cluster = ocfs2 cluster: node_count = 2 name = ocfs2 12. Review the status of the OCFS2 cluster service: service o2cb status 13.Load the OCFS2 module: service o2cb load 14. Set the OCFS2 service to be online: service o2cb online 15. Configure the OCFS2 service to start automatically when the computer boots: service o2cb configure 16. Start up the OCFS2 service.
44
service o2cb start 17.[ONLY DO THIS STEP FOR ONE NODE] Format the shared virtual disk from ANY of the servers in the cluster: mkfs.ocfs2 /dev/sdb1 18. Mount the shared virtual disk from ALL the servers in the cluster on /OVS/remote: mount /dev/sdb1 /OVS/remote/ ‐t ocfs2 [I MOUNTED /OVS DIRECTLY, NOT /OVS/remote] 19. Change the /etc/fstab file to include the shared virtual disk mounted at boot: /dev/sdb1 /OVS/remote ocfs2 defaults 1 0 After the Oracle VM Servers have been set up I created multiple virtual machines and used the Live Migration feature from the Oracle VM Manager. The virtual machine(s) I choose to migrate was moved from the server it was running on, to another server. During the Live Migration I keept PINGing the virtual machine and it responded without loosing any packages. Next step was to test a virtual machine running a single database. I installed a single instance Oracle Database using the instructions provided in this installation guide. After that I started running an Order Entry test using Swingbench. During the test, the machine performed a Live Migration. This did not affect the Order Entry test and Swinbench performed as usual.
45
A. Virtual machine installation (steps) order:
VM part: 1. create virtual machine 2. create ORACLE user 3. create directory structure for the database as oracle user (/ORA/…) 4. mount NFS 5. install required packages and set required parameters (follow the guide) Oracle part: 1. Install CRS (+ patch it) 2. Install RDBMS (+patch it) 3. Create the database When cloning VMs:
• change hostname of the new VM
• edit /etc/hosts with the IPs and hostnames
• make oracle user equivalency on all nodes (ssh with keys)
46
B. Things to know Command Description ./emctl start dbconsole Start Oracle Enterprise
Manager ./emctl stop dbconsole
Sqlplus>select * from GV$instanc
Sqlplus>select * from V$instance
Sqlplus>startup
Start‐up the database
Sqlplus>shutdown
Shutdown the database
./crs_stat –t
Get CRS status
./srvctl config database -d racdb
./lsnrctl status Listener status crsctl query crs softwareversion crsctl query crs activeversion
Get CRS version
select * from v$version where banner like ‘Oracle%’ Database Software version Sqlplus / as sysdba Another way to see the
database software version ./servctl stop/start database <name> Start/Stop database ./runcluvfy.sh comp ssa ‐n node_list Check for storage
47
Appendix Filename Description 01‐08‐Oracle VM Server Installation Guide Release 2.1‐E11081_01.zip Oracle VM Installation Guide
(for OracleVM Server and Manager)
02‐Oracle Enteprise Linux Installation Screenshots.pdf Some Oracle Enterprise Linux Installation Screenshoots
03‐Oracle Enteprise Linux Installation – Guide.pdf An Oracle Enterprise Linux Installation Guide
04‐Installing Oracle Database 11g Release 1 on Enterprise Linux 5 (32‐ and 64‐bit).pdf
Official Install Guide For Oracle Database 11g R1 and Oracle Enterprise Linux 5
sp39197.exe Original file downloaded from the HP website. It contains the SP39197 directory.
SP39197 (directory) All the files necessary to update the BIOS
SP39197/Bios Flash.htm BIOS Update Procedures 05‐README.templates This document contains
information on installing the Oracle VM templates.
06‐Metalink Note 456634.1.pdf Solution to the operating system check fail.
07‐Metalink Note 443617.1.pdf Solution to the java error received when running the installer – the missing package is libXp
08‐Installation of Oracle 10g.pdf An installation of Oracle Database 10g on CentOS – useful for the common mistakes (@end of the guide) ☺
09‐set_oravm_database Sets the environment variables for the Oracle Database
10‐NFS Server and Client Configuration in Ubuntu.pdf Configuring the NFS server on Ubuntu Linux
11‐Modifying the VIP or VIP Hostname of a 10g Oracle Clusterware Node.pdf
Modifying the VIP or VIP Hostname of a 10g Oracle Clusterware Node
12‐10gR2 RAC Install issues on Oracle EL5 or RHEL5 or SLES10 (VIPCA Failures).pdf
When installing 10gR2 RAC on Oracle Enterprise Linux 5 or RHEL5 or SLES10 there are
48
three issues that users must be aware of
13‐SRVCTL and VIPCA Utilities Set the LD_ASSUME_KERNEL Parameter.pdf
Some more VIPCA and SRVCTL debug information
14‐10g RAC How to Clean Up After a Failed CRS Install.pdf The purpose of this document is to help DBA's and support analysts understand how to clean up a failed CRS (Cluster Ready Services) install for 10g RAC.
15‐Backing Up and Restoring Oracle VM Manager.pdf How to back up and restore Oracle VM Manager (good for moving it to another machine)
16‐Xen Manual.pdf Xen Manual 17‐README.templates.pdf Oracle VM templates
information 18‐OPatch Java Error.pdf OPatch Java Error Solution from
Metalink 19‐Sharing Internet Connection.pdf Sharing Internet Connection in
Ubuntu