Oracle RAC on MacBook Pro with VMWare Fusion

IntroductionVirtual machines make our IT life so much easier than days prior to their existence. Even a decade ago, if you wanted to have a RAC testbed, you had to cobble together a few old workstations repurposed for the exercise. You also had to come up with some shared disk and a network switch as well. Getting all of the pieces together proved to be very difficult for most peoples. With today’s VM technology, you can have Oracle RAC up and running on a laptop with very little effort compared to what you had to do in the past. No additional hardware is needed. Just make sure your laptop or workstation has enough disk and memory to support the VM’s you will be creating.

It seems like I keep coming up across another article on how to set up Oracle RAC on a laptop or workstation using virtual machines. I even saw one whitepaper on how to set it up on a Mac. However, I haven’t seen any such papers using VMWare Fusion as the hypervisor. I could certainly download Oracle’s Virtual Box, but I already have Fusion installed on my laptop to support Windows guest operating systems. So why not have a little fun and see if I can get this thing up and running on my laptop?

Over a year ago, I wrote a blog entry about the importance of having testbeds at your disposal. Setting up Oracle RAC on my laptop provides another testbed for me to use and enjoy.

The hardest part of setting up RAC on your laptop is to get the network and shared disks configured correctly. Otherwise, the rest is no different than installing RAC on traditional systems. This paper will discuss how to set up the virtual machines on VMWare Fusion on the MacBook Pro and get the network and storage up and running. After that, getting Oracle RAC up and running is no different than any other system.

Guest OSSince this is a VM environment, I have a choice of guest operating systems that I could use. To make my life simpler, I downloaded the Oracle Linux validated RPM so that it would have all of the required packages and be ready to go.

Shared Storage HostOracle RAC relies heavily on shared disk storage. There are a number of different ways to accomplish this. The easiest way for me was to set up a VM that serves as an iSCSI host. The shared storage will be iSCSI mounted devices which we will run ASM

on. iSCSI will simulate a Storage Area Network for our RAC cluster. We will need a VM to act as the iSCSI SAN. The disk local to that VM will be attached to our hosts.

In VMWare Fusion, I will choose File New to create a new VM. I will install from ithe disk image I downloaded (.iso file) of Oracle Linux 6. I navigate to the ISO file and work my way through the wizard to the Finish screen.

Next I press the Customize Settings button. This lets me change the name and perform a few other configurations before creating the VM. I will name this VM “iscsi”. I should now be at the Settings screen.

We do not need much memory at all for this VM. Click on Processors & Memory and set the RAM to 1024MB.

Many people shy away from the iSCSI option for setting up their shared storage on their laptop because of the overhead involved. However, the memory and CPU usage is negligible compared to the RAC nodes we will set up later. And this is a very easy way to get the shared storage up and running. If desired, you can pare down the memory usage of this VM to 512MB.

Next, I will set the virtual hard disk device to be 40GB. That should give us plenty of space for our testbed we are setting up here. But if not, we can increase the space at a later date.

For the Network Adapter, we will make this Private to My Mac.

At this point, I power on the VM. I choose the option to Install the system. After a minute or two, you will be asked to perform the media test. I skip the test. What follows next is a standard Linux installation. I will answer the default questions unless noted in the next paragraph.

For the hostname, I put in “iscsi.localdomain”. On the same screen where the hostname is defined, press the Configure Network button. Select the eth0 device and press Edit. Make sure Connect Automatically is checked. I make sure the “System Clock uses UTC” box is not checked. On the screen to choose the Basic Server Install, make sure the radio button next to Customize Now is selected. On the next screen, make sure to choose Servers Network Storage Server. Without this option, you will not have the appropriate package to create the iSCSI devices.

Those are the only non-default choices I made. I eventually got to the screen where Linux is being installed on the VM. After the installation is complete, the VM is rebooted and I log in as root.

First, we will set up the network for the SAN VM. Let’s find out the IP address that was dynamically assigned to the VM. We’ll use that IP at a static IP instead. I use the /sbin/ifconfig command to see my IP address.

We can see that the IP address assigned to this VM is 172.16.150.138. Let’s make that static.

vi /etc/sysconfig/network-scripts/ifcfg-eth0

I will change BOOTPROTO to “static”. I will set IPADDR=172.16.150.138 and I set the NETMASK to the usual value. I also make sure to set the GATEWAY to 172.16.150.0. The config file now looks like:

I quickly reboot the VM and verify that I can ssh to it from my local MAC Terminal application.

brians-mbp:~ brianpeasland$ ssh root@172.16.150.138The authenticity of host '172.16.150.138 (172.16.150.138)' can't be established.RSA key fingerprint is e0:40:da:a6:b3:10:1a:63:6b:20:7b:db:1b:93:7a:19.Are you sure you want to continue connecting (yes/no)? yesWarning: Permanently added '172.16.150.138' (RSA) to the list of known hosts.root@172.16.150.138's password: Last login: Tue Jun 17 11:03:50 2014[root@iscsi ~]# hostnameiscsi.localdomain

Awesome. Works as planned!

Now let’s create some disk devices for our SAN. First, issue this command:

chkconfig –level 345 tgtd on

Next, we’ll make a directory on this VM to store files that will be mounted as iSCSI disks on our RAC nodes.

mkdir /disks

In that directory, we will create empty files that will be used as “disk devices”. These files are called “backing stores” in the iSCSI world.

cd /disksdd if=/dev/zero of=/disks/dev01.dat bs=1M count=4096

The above will create a 4GB file. Our disk devices will be 4GB. Repeat until you have 5 files created.

In the future, if you need to add more space to your RAC shared storage, you may need to create files similar to this on the iSCSI SAN host.

To get things started up at boot time, do:

vi /etc/rc.d/rc.local

Enter these lines into the file:

#stop firewall from killing connections to iSCSIservice iptables stoptgtadm --lld iscsi --op new --mode target --tid 1 -T iscsi.localdomain:vols-santgtadm --lld iscsi --op new --mode logicalunit --tid 1 --lun 1 -b /disks/disk01.dattgtadm --lld iscsi --op new --mode logicalunit --tid 1 --lun 2 -b /disks/disk02.dattgtadm --lld iscsi --op new --mode logicalunit --tid 1 --lun 3 -b /disks/disk03.dattgtadm --lld iscsi --op new --mode logicalunit --tid 1 --lun 4 -b /disks/disk04.dattgtadm --lld iscsi --op new --mode logicalunit --tid 1 --lun 5 -b /disks/disk05.dattgtadm --lld iscsi --op bind --mode target --tid 1 -I ALL

The first tgtadm line gets up the iSCSI target. The last line lets any and all use the target. The middle lines define the mappings between the files we created and the LUNs we are stetting up.

I had some problems with SELINUX being enabled. So I modified the contents of /etc/selinux/config to have “SELINUX=disabled”.

Now that this is set up, reboot the VM. Verify the targets are available.

tgtadm --lld iscsi --op show --mode targets

You should see the LUN’s present. Here is one of them:

At this point, we are done configuring our iSCSI SAN system for our RAC shared storage.

Create HostsFor my RAC cluster, I will be creating two Linux hosts. I will name them HOST01 and HOST02. In the VMWare Fusion main panel, Virtual Machine Library, I simply click on the Add button at the top left and choose New.

On the next screen, I choose to Install from Image and press Continue.

On the next screen, I navigate to my ISO file that I downloaded. Fusion recognizes this as OEL6 64-bit. I then press Continue.

On the next screen, Fusion is ready to create the VM for me. But I want to do things like give this VM a different name and change its default disk size. So I press the Customize Settings button. I fill in “host01” for the name and press Save. In the Processors and Memory section, I changed my host memory size to 2048MB. In the HardDisk section, I changed made sure the disk size is 40GB.

Since this is for Oracle RAC, we will need three network adapters, for the public, private, and shared storage connections. Make sure Private to my Mac is turned on for each of the interfaces. This keeps the RAC cluster accessible only to the laptop and not seen by devices on my LAN, which means I won’t potentially have conflicts with anything else on the network. First, we will generate a MAC address for the first adapter. Press the Generate button.

Make sure you click on the Generate button to create a MAC address for this adapter. Now click the Add Device button to create a second and third NIC, generating the MAC address for them as well.

Now that I have handled the initial configuration, I go to Virtual Machine Startup.

What follows next is the installation of Oracle Linux. For step-by-step instructions, see this from Oracle Corp. I will show the different options I used but I otherwise followed much of the documentation.

I am going to install a new system here. The installation routine asked me to Test the media, which skipped.

Next, I just went through the install wizard. When prompted for the host name, I entered “host01.localdomain”.

At the bottom of this screen is a button named Configure Network. Press that button. You will see the eth0 device. Make sure it is selected and press the Edit button.

Make sure the Connect Automatically option is checked to have this device start on bootup. Press the Apply button. Do this for each NIC device.

Note: At this point, we’ll use DHCP to generate an IP address. Later on, we’ll make the IP’s static.

On the next screen, choose your timezone. Make sure the “System Clock uses UTC” box is not checked.

The next big question was what type of installation. I made sure to select this will be for a Database Server. Also make sure the Customize Now button is selected.

The next screen will let you pick the packages to install. I want a X-windows interface. So under the Desktops section, I added the following:

Desktop Desktop Platform Fonts General Purpose Desktop Graphical Administration Tools X Windows System

Also make sure to add Base System iSCSI Storage Client. Without this one, we won’t be able to connect to our storage system.

After the packages were installed, a reboot finished the installation. There were a few more screens to walk through which I just accepted the defaults.

After the OS was up, I started a terminal window and verified my three network adapters were up and running:

/sbin/ifconfigEnsure that eth0, eth1 and eth2 are present and that they an IP address. Make note of the IP address. In a Terminal session on the laptop, verify that you can ping each IP address successfully. Also verify you can SSH from the laptop to the VM.

The same process is repeated to set up host02. I now have two VMs running, host01 and host02.

Network Setup

Now that I have two hosts running, they need to be able to talk to each other. And in the end, I’m going to create a connection from SQL Developer running locally on my laptop to the RAC database. So I need to do a little bit of setup on the network side. I need to configure the following:

Give each host a unique IP address within my laptop’s network. I won’t be access this testbed outside of my laptop so it can stay private.

Provide a VIP for each host. Provide SCAN VIPs for the Scan Listeners.

On each host, if I use the /sbin/ifconfig command I can see the IP addresses of each interface on that host. The IP addresses are assigned via DHCP, which can change after we restart each host. To fix that, we will first start both hosts so both have obtained IP addresses for their interfaces. If I start both hosts, I can see the following IP assignments:

host01 eth0 172.16.150.129host01 eth1 172.16.150.130host01 eth2 172.16.150.131host02 eth0 172.16.150.137host02 eth1 172.16.150.135host02 eth2 172.16.150.136

Now let’s make these IP addresses static. Sign on to the Gnome interface with root and go to System Preference Network Connections. Select the eth0 device and press the Edit button. Click on the IPv4 tab. In the Method pulldown menu, choose Manual. Press the Add button. Enter in 172.16.150.129 for the IP address, 172.16.150 and 255.255.255.0 for the Netmask and 172.16.150.0 for the Gateway.

Note, you may have different IP addresses that will lead to a different gateway as well. Repeat for the eth1 adapater and repeat for the second host. Once all IP addresses have been statically defined, reboot the virtual machines. Finally, ping and ssh from the laptop’s Terminal application to the IP addresses.

We will be using eth0 for the public network, eth1 for the private network, and eth2 for the shared storage. So that we don’t have to register the IP addresses in DNS, we will just add them to the hosts file. I also have to add entries for the VIPs and SCAN VIP. Since I am using a host file, I can only have one IP for the SCAN VIP. Finally, I will need to add the DNS entry for my iSCSI storage that was created earlier. So I add the following to my hosts file:

172.16.150.129 host01172.16.150.130 host01-priv172.16.150.137 host02172.16.150.135 host02-priv172.16.150.140 host01-vip172.16.150.141 host02-vip172.16.150.142 mac-cluster-scan172.16.150.138 iscsi

The host file needs to be updated on the Mac laptop and on each VM host. You can see the public and private names. The next three entries are my traditional VIPs and the SCAN VIP. The last entry is for the iSCSI host for our shared storage.

Shared Storage

The network setup is now complete. So we need to turn out attention to setting up the shared storage.

On the host, issue:

service iscsi startchkconfig --level 345 iscsi on

Next, let’s probe our iSCSI VM to see if it responds for us:

[root@host01 ~]# iscsiadm -m discovery -t sendtargets -p iscsi172.16.150.138:3260,1 iscsi.localdomain:vols-san

Perfect! We want to see port 3260 and the name “iscsi.localdomain:vols-san” that we defined earlier.

Before we can map the luns, we need to set up the ‘oracle’ user and its groups so that we can change ownership.

[root@host01 ~]# /usr/sbin/groupadd dba[root@host01 ~]# /usr/sbin/groupadd oinstall[root@host01 ~]# /usr/sbin/groupadd asmdba[root@host01 ~]# /usr/sbin/groupadd asmadmin

[root@host01 ~]# /usr/sbin/useradd -g dba -G oinstall,asmdba oracle[root@host01 ~]# passwd oracleChanging password for user oracle.New password: Retype new password: passwd: all authentication tokens updated successfully.

I have worked with ‘dba’ as the primary group for years so I deviate from the documentation which says to have ‘oinstall’ as the primary group. The documentation also says to have a user named ‘grid’, but since this will just be for me, everything will be done with the ‘oracle’ user.

Now let’s get some persistent mounts:

vi /etc/rc.d/rc.local

Add this line:

chown oracle:dba /dev/sd[b-f]

Now reboot the host. After it comes back up, do:

[root@host01 ~]# cd /dev[root@host01 dev]# ls -l sd*brw-rw----. 1 root disk 8, 0 Jun 19 09:34 sdabrw-rw----. 1 root disk 8, 1 Jun 19 14:34 sda1brw-rw----. 1 root disk 8, 2 Jun 19 09:34 sda2brw-rw----. 1 oracle dba 8, 16 Jun 19 14:35 sdbbrw-rw----. 1 oracle dba 8, 32 Jun 19 14:35 sdcbrw-rw----. 1 oracle dba 8, 48 Jun 19 14:35 sddbrw-rw----. 1 oracle dba 8, 64 Jun 19 14:35 sdebrw-rw----. 1 oracle dba 8, 80 Jun 19 14:35 sdf

Awesome! We can see devices sdb through sdf. These are the five iSCSI disks mounted to our host. All owned by oracle:dba exactly as we desire.

At this point, everything should be repeated for host02. As a DBA in real life, I like backups. I like to see my hard worked backed up so that I don’t have to repeat it. At this point, I take a snapshot of all three VM’s that I have set up. I have a good config right now and I’d like to be able to revert to this known, good state should I need to.

Conclusion

The work above is the hardest part of setting up RAC on this laptop configuration. The first half focused on setting up the VMs, configuring the network, and getting our shared storage up and running. What remains is pretty much a standard GI and RDBMS install for RAC. As such, I won’t be discussing the remainder of the work in this paper. There are plenty of resources, including the Oracle documentation, that can be used to install Grid Infrastructure and get your Oracle RAC database up and running.

I hsould also mention that you can take down any of your VM’s, but if you take down the ‘iscsi’ VM, your RAC nodes (host01 and host02) will lose the shared storage. Those RAC VM’s won’t go down, but Oracle running on them sure will. If you restart your laptop or take down these VM’s, start the iSCSI VM first.

This paper was written specifically for running Oracle RAC on a MacBook Pro with VMWare Fusion for the hypervisor. The basic steps of getting the network and shared storage can be ported to other platforms as well. You just may have to work out the kinks in the process if you use another hypervisor or another physical host.